This action might not be possible to undo. Are you sure you want to continue?
Basic Concepts of Software Testing
Basic Concepts and Industry awareness
Page 1 of 60
Basic Concepts of Software Testing
Table of Contents 1. Introduction.................................................................................................4 2. Types of Testing...........................................................................................4 2.1. White Box Testing...................................................................................4 2.2. Black Box Testing....................................................................................4 2.3. Unit Testing............................................................................................5 2.3.1. Benefits ...........................................................................................5 2.3.2. Encourages change ...........................................................................5 2.3.3. Simplifies Integration ........................................................................5 2.3.4. Documents the code .........................................................................5 2.3.5. Separation of Interface from Implementation .......................................6 2.3.6. Limitations .......................................................................................6 2.4. Integration testing .................................................................................6 2.4.1. Purpose ...........................................................................................6 2.5. Performance Testing................................................................................7 2.5.1. Technology ......................................................................................7 2.5.2. Performance specifications .................................................................7 2.5.3. Tasks to undertake ...........................................................................8 2.6. Stress Testing ........................................................................................8 2.7. Security Testing......................................................................................9 2.7.1. Security Testing Techniques ...............................................................9 2.8. Usability Testing.....................................................................................9 2.9. Stability Testing....................................................................................10 2.10. Acceptance Testing..............................................................................10 2.11. Installation Testing..............................................................................10 2.12. Alfa Testing........................................................................................11 2.13. Beta Testing.......................................................................................11 2.14. Product Testing...................................................................................11 2.15. System Testing...................................................................................12 2.16. Regression Testing...............................................................................12 2.17. Compatibility Testing...........................................................................13 2.18. Test Cases, Suits, Scripts and Scenario..................................................13 2.19. Defect Tracking...................................................................................14 2.20. Formal Verification...............................................................................14 2.20.1. Validation and Verification ..............................................................14 2.21. Fuzz Testing.......................................................................................15 2.21.1. Uses ............................................................................................15 2.21.2. Fuzz testing methods .....................................................................15 2.21.3. Event-driven fuzz ..........................................................................16 2.21.4. Character-driven fuzz ....................................................................16 2.21.5. Database fuzz ...............................................................................16 3. Manual Testing...........................................................................................17 3.1. Facts...................................................................................................17 3.2. Software Crisis .....................................................................................17 3.3. Software Myths ....................................................................................17 3.3.1. Management Myths .........................................................................17 3.3.2. Developers Myths ...........................................................................17 3.3.3. Customer’s Myth .............................................................................18 3.3.4. What do we do? ..............................................................................18 3.4. Software Quality Assurance: ..................................................................18 3.4.1. Verification: ...................................................................................18 3.4.2. Validation: .....................................................................................18 3.5. Software Life Cycle Models: ...................................................................18
Page 2 of 60
Basic Concepts of Software Testing
3.6. What makes a good Software QA engineer? .............................................18 3.7. Testing: ..............................................................................................19 3.7.1. Why Testing? .................................................................................19 3.8. Test Life Cycle .....................................................................................19 3.9. Testing Techniques ...............................................................................19 3.10. Test Plan: ..........................................................................................19 3.10.1. Test Specification: .........................................................................20 4. Testing Procedure.......................................................................................20 4.1. Bug Tracking .......................................................................................20 5. Testing Tools and Software..........................................................................23 5.1. Load and Performance Test Tools ...........................................................23 5.2. Java test Tools......................................................................................23 5.3. Link Checking Tools...............................................................................27 5.4. Perl Testing Tools..................................................................................28 5.5. Web Functional and Regression Testing Tools............................................28 5.6. Web Site Security Test Tools...................................................................33 5.7. Web Site Management Tools...................................................................37 5.8. Other Web Testing Tools........................................................................45 6. Testing FAQ ..............................................................................................50
Page 3 of 60
clear box and open box testing. White box testing does not account for errors caused by omission. Although most of the intellectual processes of testing are nearly identical to that of review or inspection. but effective testing of complex products is essentially a process of investigation. glass-box and clear-box. portability. and it focuses specifically on using internal knowledge of the software to guide the selection of test data. 2. Synonyms for white-box include: structural. Black-box and white-box are test design methods. In other words Testing is nothing but CRITICISM or COMPARISION. Refer to the ISO standard ISO 9126 for a more complete list of attributes and criteria. so it doesn't explicitly use knowledge of the internal structure. and closed-box. 2. Page 4 of 60 . Types of Testing 2. This is a software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data. Here comparison in the sense comparing the actual value with expected one.2.Aditi Technologies Basic Concepts of Software Testing 1. functional. maintainability and usability. White Box Testing White box testing is also known as glass box. The quality of the application can and normally does vary widely from system to system but some of the common quality attributes include reliability. He or she can then see if the program diverges from its intended goal. and the product answers with its behavior in reaction to the probing of the tester. White-box test design allows one to peek inside the "box". Black Box Testing Testing of a function without knowing internal structure of the program. Black-box test design treats the system as a "black-box". structural. There are many approaches to software testing. Unlike black box testing. The test is accurate only if the tester knows what the program is supposed to do.1. Black-box test design is usually described as focusing on testing functional requirements. stability. completeness and quality of developed computer software. white box testing uses specific knowledge of programming code to examine outputs. Introduction Testing is a process used to help identify the correctness. and all visible code must also be readable. testing can never completely establish the correctness of computer software. where the "questions" are things the tester tries to do with the product. With that in mind. not merely a matter of creating and following rote procedure. One definition of testing is "the process of questioning a product in order to evaluate it". the word testing is connoted to mean the dynamic analysis of the product—putting the product through its paces. Synonyms for black-box include: behavioral. opaque-box.
Behavioral test design is slightly different from black-box test design because the use of internal knowledge isn't strictly forbidden. This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly.2. Unit testing is usually associated with structural test design. 2.Simplifies Integration Unit testing helps eliminate uncertainty in the pieces themselves and can be used in a bottom-up testing style approach.3.Aditi Technologies Basic Concepts of Software Testing While black-box and white-box are terms that are still in popular use. 2. and make sure the module still works correctly (regression testing). a unit test is a method of testing the correctness of a particular module of source code. Clients looking to learn how to use the class can look at the unit tests to determine how to use the class to fit their needs. It provides a written contract that the piece must satisfy. By testing the parts of a program first and then testing the sum of its parts will make integration testing easier. Note that any level of testing (unit testing. and their influence is hard to see in the tests once they're implemented.3. In practice. but it's still discouraged.Encourages change Unit testing allows the programmer to re-factor code at a later date. but others wish we'd stop talking about boxes altogether. but this is because testers usually don't have well-defined requirements at the unit level to validate.3.) can use any test design methods. etc.1.3. 2. it hasn't proven useful to use a single test design method.4. It is important to understand that these methods are used during the test design phase. This isolated testing provides four main benefits: 2.3.Documents the code Unit testing provides a sort of "living document" for the class being tested.Benefits The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. This type of testing is mostly done by the developers. Unit Testing In computer programming. One has to use a mixture of different methods so that they aren't hindered by the limitations of a particular one. system testing. many people prefer the terms "behavioral" and "structural". The idea is to write test cases for every non-trivial function or method in the module so that each test case is separate from the others if possible. Page 5 of 60 . 2. Some call this "gray-box" or "translucent-box" test design.3.
e. 2. because a unit test should never go outside of its own class boundary. thus minimizing dependencies in the system. This is a mistake.Separation of Interface from Implementation Because some classes may have references to other classes. the tester finds herself writing code that interacts with the database. These "design items". 2. testing a class can frequently spill over into testing another class. Integration testing Integration Testing is the phase of software testing in which individual software modules are combined and tested as a group.4. This results in loosely coupled code.4. groups them in larger aggregates. A common example of this is classes that depend on a database. for example.Purpose The purpose of Integration testing is to verify functional.3. takes as its input modules that have been checked out by unit testing. In addition. and delivers as its output the integrated system ready for system testing. performance problems and any other system-wide issues.3. performance and reliability requirements placed on major design items. across procedure calls or process activations. the software developer abstracts an interface around the database connection. By definition. Page 6 of 60 . individual subsystems are exercised through their input interface. it will not catch integration errors. i. It follows unit testing and precedes system testing.1. As a result. Therefore. it may not be trivial to anticipate all special cases of input the program unit under study may receive in reality. Simulated usage of shared data areas and inter-process communication is tested.6. it only tests the functionality of the units themselves. in order to test the class. assemblages (or groups of units). The overall idea is the "building block" approach in which verified assemblages are added to a verified base which is then used to support the Integration testing of further assemblages.Aditi Technologies Basic Concepts of Software Testing 2. 2.Limitations It is important to realize that unit-testing will not catch every error in the program. All test cases are constructed to test that all components within assemblages interact correctly. applies tests defined in an Integration test plan to those aggregates.5. and then implements that interface with their own Mock Object. are exercised through their interfaces using Black box testing. Unit testing is only effective if it is used in conjunction with other software testing activities. success and error cases being simulated via appropriate parameter and data inputs.
Performance specifications Performance testing is frequently not performed against a specification.5. e. Performance Testing In software engineering. The test result shows how the performance varies with the load. It can compare two systems to find which performs better. software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time. in order to see what happens when an acceptable load is exceeded –does the system crash? How long does it take to recover if a large load is reduced? Does it fail in a way that causes collateral damage? 2. coordinating and gathering metrics from each of the injectors and collating performance data for reporting purposes. etc. no one will have expressed what the maximum acceptable response time for a given population of users is. performance testing is frequently used as part of the process of performance profile tuning. It is sometimes a difficult task to identify which part of the system represents this critical path. Performance testing can be combined with stress testing. performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. Tools in this category usually execute a suite of tests which will emulate real users against the system.1. However.5. It can demonstrate that the system meets performance criteria. will result in the overall system running faster. there are outliers of a few key transactions that take considerably longer to complete – something that might be caused by inefficient database queries. given as number of users vs. response time. Usually.2. that while the average response time might be acceptable. The usual sequence is to ramp up the load – starting with a small number of virtual users and increasing the number over a period to some maximum. Or it can measure what parts of the system or workload cause the system to perform badly. or as a series of scripts to emulate different types of user interaction) with the host whose performance is being tested. are available to perform such tests. The idea is to identify the “weakest link” – there is inevitably a part of the system which.Technology Performance testing technology employs one or more PCs to act as injectors – each emulating the presence or numbers of users and each running an automated sequence of interactions (recorded as a script. Sometimes the results can reveal oddities.5.Aditi Technologies Basic Concepts of Software Testing 2. i. Various tools. 2.e. if it is made to respond faster. it is often crucial (and often difficult to arrange) for the test conditions to be similar to the expected actual use.g. Performance testing can serve different purposes.. including Compuware Corporation's QACenter Performance Edition. In performance testing. and some test tools come provided with (or can have add- Page 7 of 60 . a separate PC acts as a test conductor. In the diagnostic case.
deployment of server instrumentation. They ended up rewriting the system’s ‘idle loop’. which can be analyzed together with the raw performance statistics. database access times. For example. It can also be done in-house. router configuration. 2. often to a breaking point.Tasks to undertake Tasks to perform such a test would include: • • • • • • Analysis of the types of interaction that should be emulated and the production of scripts to do those emulations Decision whether to use internal or external resources to perform the tests. or investigation of critical path and recommendation of corrective action. since it is known that the response times of the internet itself vary regionally.3. Running the tests – probably repeatedly in order to see whether any unaccounted for factor might affect the results. then an injector configuration could be used to test whether the proposed system met that specification. It involves testing beyond normal operational capacity. a web server may be stress tested Page 8 of 60 . network overhead. but even having the most efficient idle loop in the world obviously didn’t improve overall performance one iota! Performance testing almost invariably identifies that it is parts of the software (rather than hardware) that contribute most to delays in processing users’ requests. where they had found the system spent most of its time. Performance testing can be performed across the web. etc. Set up of a configuration of injectors/controller Set up of the test configuration (ideally identical hardware to the production platform).6. Without such instrumentation one might have to have someone crouched over Windows Task Manager at the server to see how much CPU load the performance tests are generating. quiet network (we don’t want results upset by other users). It is always helpful to have a statement of the likely peak numbers of users that might be expected to use the system at peak times. in order to observe the results. 2. If there can also be a statement of what constitutes the maximum allowable 95 percentile response time. although routers would then need to be configured to introduce the lag what would typically occur on public networks.5. and even done in different parts of the country. either pass/fail. Stress Testing Stress Testing is a form of testing that is used to determine the stability of a given system or entity.Aditi Technologies Basic Concepts of Software Testing ons that provide) instrumentation that runs on the server and reports transaction times. Analyzing the results. There is an apocryphal story of a company that spent a large amount optimizing their software without having performed a proper analysis of the problem.
An external security vulnerability review by Third Eye Testing will give you the best possible confidence that your application is as secure as possible. and various denial of service tools to observe the performance of a web site during peak loads. a document. During usability testing. or interpreting feedback.Security Testing Techniques • • • • • • • • Vulnerability Scanning Network Scanning Password Cracking Log Views Virus Detect Penetration Testing File Integrity Checkers War Dialing 2. or a device) for its intended purpose. then developers should improve the design and test it again. Stress testing a subset of load testing. A designers' primary function should be more than appearance. bots.8. Downtime. i. 2. whereas general human-computer interaction studies attempt to formulate universal principles.7.1. including making things work with people. to discover errors and areas of improvement.7. software testing. manipulating parts. If usability testing uncovers difficulties. 2. Security within an application or web service is crucial to avoid such vulnerabilities and new threats. performance testing.e." Page 9 of 60 . Data theft. While automated tools can help to eliminate many generic security issues.Aditi Technologies Basic Concepts of Software Testing using scripts. "Caution: simply gathering opinions is not usability testing -. Data corruption and application Defacement.you must arrange an experiment that measures a subject's ability to use your document. such as people having difficulty understanding instructions. Usability Testing Usability testing is a means for measuring how well people can use some humanmade object (such as a web page. usability testing measures the usability of the object. Also see testing. This is often caused by pressure from the people in charge. compromising usability and functionality. the detection of application vulnerabilities requires independent evaluation of your specific application's features and functions by experts. forcing designers to develop systems based on management expectations instead of people's needs. Security Testing Application vulnerabilities leave your system open to attacks. Usability testing focuses on a particular object or a small set of objects. the aim is to observe people using the product in as realistic a situation as possible. Designers commonly focus excessively on creating designs that look "cool". a computer interface.
and what people like. Rather. Page 10 of 60 . ideally. 2.). to test the attachment function of an e-mail program. the test subjects should be given the instructions and a box of parts. Instruction phrasing. and ask him or her to undertake this task. 2. The technique popularly used to gather data during a usability test is called a think aloud protocol. when testing instructions for assembling a toy. or realistic situation. Several other test instruments such as scripted instructions. paper prototypes. cosmetic problems) and show stoppers (major problems like the software crashing. 2. to which the system should conform.11. so that developers can see problem areas.10. Users of the system will perform these tests which. Stability Testing In software testing. software will not run etc. the focus is on a final verification of the required business function and flow of the system. illustration quality. The focus in this type of testing is less on simple problems (spelling mistakes. Test designers will draw up a formal test plan and devise a range of severity levels. it refers to a period of time during which a multi-dose product retains its quality after the container is opened. usability testing involves watching people trying to use something for its intended purpose. wherein the person performs a list of tasks using the product being tested while observers watch and take notes. For example. The test scripts will emulate real-world usage of the system.9. developers have derived from the User Requirements Specification. For example. The aim is to observe how people function in a realistic manner. Developers should have worked out these issues during unit testing and integration testing.and post-test questionnaires are also used to gather feedback on the product being tested. and pre.Aditi Technologies Basic Concepts of Software Testing Rather than showing users a rough draft and asking. The idea is that if the software works as intended and without issues during a simulation of normal use. a scenario would describe a situation where a person needs to send an e-mail attachment. Acceptance Testing User acceptance testing (UAT) is one of the final stages of a software project and will often occur before the customer accepts a new system. Setting up a usability test involves carefully creating a scenario. stability testing is an attempt to determine if an application will crash. it will work just the same in production. In the pharmaceutical field. and the toy's design all affect the assembly process. Results of these tests will allow both the customers and the developers to be confident that the system will work as intended. "Do you understand this?". Installation Testing Installation testing (in software engineering) can simply be defined as any testing that occurs outside of the development environment.
Only suitably organized and executed test process can contribute to the success of a software product.Aditi Technologies Basic Concepts of Software Testing Such testing will frequently occur on the computer system the software product will eventually be installed on. testing is usually required before release to the general public. however no serious or critical bugs would exist. This process helps in determining whether the final software meets its intended purpose and whether the end users would accept the same. the software is released to a limited audience who would finally form the end users.12. A beta release is very close to the final release. Whilst the ideal installation might simply appear to be to run a setup program. The product handed out as a Beta Release is not bug free. Product Testing Software Product development companies face unique challenges in testing. Product testing experts design the test process to take advantage of the economies of scope and scale that are present in a software product. Ideally. This technique is known as black box testing. 2.13. In-house developers often test the software in what is known as 'ALPHA' testing which is often performed under a debugger or with hardware-assisted debugging to catch bugs quickly. the generation of that setup program itself and its efficacy in a variety of machine and operating system environments can require extensive testing before it can be used with confidence. Page 11 of 60 . particularly where software is to be released into an already live target environment (such as an operational web site) installation (or deployment as it is sometimes called) can involve database schema changes as well as the installation of new software.14. Deployment plans in such circumstances may include back-out procedures whose use is intended to roll the target environment back in the event that the deployment is unsuccessful. A factor that can increase the organizational requirements of such an exercise is the need to synchronize the data in the test deployment environment with that in the live environment with minimum disruption to live operation. the deployment plan itself should be tested in an environment that is a replica of the live environment. 2. Beta Testing Many a time. 2. to use it / test it and come back with feedback or bugs. Alfa Testing In software development. It can then be handed over to testing staff for additional inspection in an environment similar to how it was intended to be used. In distributed systems. This is often known as the second stage of alpha testing.
integrated system to evaluate the system's compliance with its specified requirements. all bug previously identified and fixed is tested along with it's impacted areas to confirm the fix and it's impact if any. Regression testing falls within the scope of Black box testing.). According to the IEEE Standard Computer Dictionary. 2. 2. where it seeks to detect both defects within the "inter-assemblages" and also the system as a whole. As a rule. Regression testing is testing conducted on a complete. As a rule. System testing takes. Regression Testing Regression Testing is typically carried out at the end of the development cycle. where it seeks to detect both defects within the "inter-assemblages" and also the system as a whole. During this testing. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. Regression testing is more of a limiting type of testing.15. NY. all of the "integrated" software components that have successfully passed Integration testing and also the software Regression itself integrated with any applicable hardware Regression(s). Alpha testing and Beta testing are sub-categories of Regression testing. and as such. 1990. New York. New York. Alpha testing and Beta testing are sub-categories of System testing. should require no knowledge of the inner design of the code or logic (IEEE. The purpose of Integration testing is to detect any inconsistencies between the software units that are integrated together called assemblages or between any of the assemblages and hardware. Page 12 of 60 . integrated system to evaluate the system's compliance with its specified requirements.). should require no knowledge of the inner design of the code or logic (IEEE. System testing is more of a limiting type of testing. and as such. The purpose of Integration testing is to detect any inconsistencies between the software units that are integrated together called assemblages or between any of the assemblages and hardware. as its input.Aditi Technologies Basic Concepts of Software Testing These activities are sequenced and scheduled so that a test activity occurs immediately following the construction activity whose output the test is intended to validate. Regression testing takes. System testing falls within the scope of Black box testing. System testing is testing conducted on a complete. all of the "integrated" software components that have successfully passed Integration testing and also the software system itself integrated with any applicable hardware system(s). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. NY.16. as its input. System Testing According to the IEEE Standard Computer Dictionary. 1990.
2. Test scripts should be written for modules with the highest risk of failure and the highest impact if the risk becomes an issue. In a database system. Compatibility testing service aims at locating application problems by running them in real environments. It definitely contains a section where the tester identifies the system configuration used during testing. The ideal scenario test has five key characteristics. A scenario test is a test based on a hypothetical story used to help a person think through a complex problem or system.Aditi Technologies Basic Concepts of Software Testing 2. system. along with various additional pieces of information. These steps can be stored in a word processor document. Larger test cases may also contain prerequisite states or steps. author. and descriptions of the following tests. Test suites and scenarios can be used in concert for complete system tests. The test suite often also contains more detailed instructions or goals for each collection of test cases. Page 13 of 60 . It is (a) a story that is (b) motivating. and (e) easy to evaluate. It can occasionally be a series of steps but with one expected result or expected outcome. (d) complex. A test case is usually a single step. test step or order of execution number. A group of test cases may also contain prerequisite states or steps. spreadsheet. and regression testing. database or other common repository.17. operating system and browser versions. Test Cases. The most common term for a collection of test cases is a test suite. thus ensuring you that the application is compatible with various hardware. Scripts and Scenario Black box testers usually write test cases for the majority of their testing activities. They can be as simple as a diagram for a testing environment or they could be a description written in prose. (c) credible. Most white box tester write and use test scripts in unit. A test case should also contain a place for the actual result. related requirement(s). depth.18. test category. The optional fields are a test case ID. Suits. and its expected result. Compatibility Testing One of the challenges of software development is ensuring that the application works properly on the different platforms and operating systems on the market and also with the applications and devices in its environment. you may also be able to see past test results and who generated the results and the system configuration used to generate those results. They may also be called a test script. or even a test scenario. They are usually different from test cases in that test cases are single steps and scenarios cover a number of steps. Collections of test cases are sometimes incorrectly termed a test plan. Most companies that use automated testing will call the code that is used their test scripts. and check boxes for whether the test is automatable and has been automated. and descriptions. These past results would usually be stored in a separate table.
.Aditi Technologies Basic Concepts of Software Testing Scenario testing is similar to. Validation: "Are we building the right product?” i. Usually formal verification is carried out algorithmically. using formal methods. (by inspection. and abstractions of general software components.20. Defect Tracking In engineering. Defect tracking is important in software engineering as complex software systems typically have tens or hundreds of thousands of defects: managing.e. Often one refers to the overall checking process as V & V. defect tracking is the process of finding defects in a product. Validation and Verification Verification is one aspect of testing a product's fitness for purpose. The properties to be verified are often described in temporal logics. but not the same as session-based testing. timed automata and hybrid automata.19. abstract interpretation. The verification process consists of static and dynamic parts. 2.. 2. does the product conform to the specifications. Defect tracking systems are computer database systems that store defects and help people to manage them. Page 14 of 60 . 2.20. but the two concepts can be used in conjunction. The main approaches to implementing formal verification include state space enumeration.e. digital circuits with internal memory. Verification: "Are we building the product right?” i. cryptographic protocols. testing. combinatorial circuits. formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property. Validation is the complementary aspect. symbolic state space enumeration. labeled transition systems (LTS) and their compositions. and tracking them to closure. E. processalgebraic methods. or recording feedback from customers). System types that are considered in the literature for formal verification include finite state machines (FSM). Petri nets.. for a software product one can inspect the source code (static) and run against specific test cases (dynamic). and reasoning with the aid of automatic theorem provers such as HOL or Isabelle.g.1. does the product do what the user really requires. evaluating and prioritizing these defects is a difficult task. which is more closely related to exploratory testing. such as linear temporal logic (LTL) or computational tree logic (CTL). Formal Verification In the context of hardware and software systems. abstraction refinement.
The great advantage of fuzz testing is that the test design is extremely simple. For example. usually before applying it to the software. Fuzz testing is thought to enhance software security and software safety because it often finds odd oversights and defects which human testers would fail to find. If the program fails (for example. rather than a direct measure. then there are defects to correct. i. or possibly from a mechanism in an embedded system. by crashing. • Database inputs are from tabular data. fuzz testing can only be regarded as a proxy for program correctness. rather than behaving correctly. with fuzz test failures actually being more useful as a bug-finding tool than fuzz test passes as an assurance of quality. so that if the computer fails dramatically.. 2. Fuzz testing is also used as a gross measurement of a large software system's quality. almost all fuzz testing makes a record of the data it manufactures. Fuzz testing methods As a practical matter. the test data is preserved. developers need to reproduce errors in order to fix them. Modern software has several different types of inputs: • Event driven inputs are usually from a graphical user interface. and even careful human test designers would fail to create tests for.2. and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing.21. 2. For this reason. or by failing in-built code assertions). fuzz testing is not a substitute for exhaustive testing or formal methods: it can only provide a random sample of the system's behavior. Uses Fuzz testing is often used in large software development projects that perform black box testing. and fuzz testing is one of the techniques which offer a high benefit to cost ratio. Fuzz Testing Fuzz testing is a software testing technique. However.e.21. The advantage here is that the cost of generating the tests is relatively low. such as relational databases. The basic idea is to attach the inputs of a program to a source of random data. the product is tested by putting it through typical usages and atypical usages ("Can we break it?"). • Character driven inputs are from files or data streams.21.Aditi Technologies Basic Concepts of Software Testing Validation usually can only be done dynamically.1. 2. and free of preconceptions about system behavior. third party testers have used fuzz testing to evaluate the relative merits of different operating systems and application programs. There are at least two different forms of fuzz testing: Page 15 of 60 . These usually have a budget to develop test tools. Thus.
21. Simple fuzz usually uses a pseudo random number generator to provide input. simply providing error status can cause resource problems or a crash. To succeed in a fuzz-tested environment. Another is that decode tables or logic may be incomplete. 2. often the database is more tolerant of odd data than its client software. because input and comparison constraints reduce the invalid data in a database. However. not handling every possible binary value. One common problem with a character driven program is a buffer overrun. 2. Character-driven fuzz Normally this is provided as a stream of random data. The classic source in UNIX is the random data generator.21.3. Some IT shops use software tools to migrate and manipulate such databases. or conforms to actual production data. 2.4. when the character data exceeds the available buffer space. Database fuzz The standard database scheme is usually filled with fuzz that is random data of random sizes. Database fuzz is controversial. Robust error detection systems will report only the most significant or most recent error over a period of time. The most common problem with an event-driven program is that it will often simply use the data in the queue. fuzz-generated randomness can test the un-designed behavior surrounding a wider range of designed system states. without even crude validation.5. software must validate all fields of every queue entry. Fuzz testing may use tools to simulate all of these domains. Event-driven fuzz Normally this is provided as a queue of data-structures. and a general-purpose interface is available to users.Aditi Technologies Basic Concepts of Software Testing • • • Valid fuzz attempts to assure that the random input is reasonable. This problem tends to recur in every instance in which a string or number is parsed from the data stream and placed in a limited-size area. The queue is filled with data structures that have random values. decode every possible binary value. One of the more interesting issues with real-time event handling is that if error reporting is too verbose. By using all of these techniques in combination. Since major customer and enterprise management Page 16 of 60 . Often the same schema descriptions can be used to automatically generate fuzz databases. and then ignore impossible requests. A combined approach uses valid test data with some proportion of totally random input injected.21.
Basic Concepts of Software Testing
software is starting to be open-source, database-based security attacks are becoming more credible. A common problem with fuzz databases is buffer overrun. A common data dictionary, with some form of automated enforcement is quite helpful and entirely possible. To enforce this, normally all the database clients need to be recompiled and retested at the same time. Another common problem is that database clients may not understand the binary possibilities of the database field type, or, legacy software might have been ported to a new database system with different possible binary values. A normal, inexpensive solution is to have each program validate database inputs in the same fashion as user inputs. The normal way to achieve this is to periodically "clean" production databases with automated verifiers.
3. Manual Testing
• • •
In India itself, Software industry growth has been phenomenal. IT field has enormously grown in the past 50 years. IT industry in India is expected to touch 10,000 crores of which software share is dramatically increasing.
• • •
Software cost/schedules are grossly inaccurate. Cost overruns of several times, schedule slippage’s by months, or even years are common. Productivity of people has not kept pace with demand. Added to it is the shortage of skilled people. Productivity of people has not kept pace with demand Added to it is the shortage of skilled people.
• • • • • •
Software Management is different. Why change or approach to development? We have provided the state-of-the-art hardware. Problems are technical If project is late, add more engineers. We need better people.
• • • • We must start with firm requirements Why bother about Software Engineering techniques, I will go to terminal and code it. Once coding is complete, my job is done. How can you measure the quality...it is so intangible.
Page 17 of 60
Basic Concepts of Software Testing
• • A general statement of objective is good enough to produce software. Anyway software is “Flex-ware”, it can accommodate my changing needs.
3.3.4.What do we do?
• • • • Use Software Engineering techniques/processes. Institutionalize them and make them as part of your development culture. Adopt Quality Assurance Frameworks : ISO, CMM Choose the one that meets your requirements and adopt where necessary.
Software Quality Assurance:
The purpose of Software Quality Assurance is to provide management with appropriate visibility into the process being used by the software project and of the products being built. • Software Quality Assurance involves reviewing and auditing the software products and activities to verify that they comply with the applicable procedures and standards and providing the software project and other appropriate managers with the results of these reviews and audits.
• • Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications. The determination of consistency, correctness & completeness of a program at each stage.
• • Validation typically involves actual testing and takes place after verifications are completed The determination of correctness of a final program with respect to its requirements.
• • • •
Software Life Cycle Models:
Prototyping Model Waterfall Model – Sequential Spiral Model V Model - Sequential
What makes a good Software QA engineer?
The same qualities a good tester has are useful for a QA engineer. Additionally, they must be able to understand the entire software development process and how it can fit into the business approach and goals of the organization.
Page 18 of 60
Basic Concepts of Software Testing
Communication skills and the ability to understand various sides of issues are important. In organizations in the early stages of implementing QA processes, patience and diplomacy are especially needed. An ability to find problems as well as to see 'what's missing' is important for inspections and reviews.
• • • • • •
An examination of the behavior of a program by executing on sample data sets. Testing comprises of set of activities to detect defects in a produced material. To unearth & correct defects. To detect defects early & to reduce cost of defect fixing. To avoid user detecting problems. To ensure that product works as users expected it to.
• • • • To To To To unearth and correct defects. detect defects early and to reduce cost of defect fixing. ensure that product works as user expected it to. avoid user detecting problems.
• • • • • • • •
Test Life Cycle
Identify Test Candidates Test Plan Design Test Cases Execute Tests Evaluate Results Document Test Results Casual Analysis/ Preparation of Validation Reports Regression Testing / Follow up on reported bugs.
• • •
Black Box Testing White Box Testing Regression Testing
These principles & techniques can be applied to any type of testing.
A Test Plan is a detailed project plan for testing, covering the scope of testing, the methodology to be used, the tasks to be performed, resources, schedules, risks, and dependencies. A Test Plan is developed prior to the implementation of a project to provide a well defined and understood project roadmap.
Page 19 of 60
functional. required standards and processes (such as release processes. comprehensive definition of a testing campaign. test tracking. usability tests. system. Bug Tracking What's a 'test case'? Page 20 of 60 . It provides a repeatable. set up test tracking processes. 4. etc. software. Determine project-related personnel and their responsibilities. test cases. milestones Determine input equivalence classes. communications. and labor requirements Set schedule estimates. integration. those responsible for tasks.unit. timelines. and internal design specifications and other necessary documents. Testing Procedure The following are some of the steps to consider: • • Obtain requirements. and determine scope and limitations of tests. Test Specification: A Test Specification defines exactly what tests will be performed and what their scope and objectives will be. test environment. Obtain budget and schedule requirements. Determine test environment requirements (hardware. functional design.1.Aditi Technologies Basic Concepts of Software Testing 3. coverage analyzers. prior to the onset of manual testing and/or automated test suite development. A Test Specification is produced as the first step in implementing a Test Plan. obtain needed user manuals/reference documents/configuration guides/installation guides.) Determine test-ware requirements (record/playback tools. change processes.10.) Identify application's higher-risk aspects. load. set up logging and archiving processes. set up or obtain test input data Obtain and install software releases Perform tests Evaluate and report results Track problems/bugs and fixes Retest as needed Maintain and update test plans. and test ware through life cycle • • • • • • • • • • • • • • • • • • 4.) Determine test input data requirements Identify tasks. boundary value analyses. set priorities. reporting requirements. error classes Prepare test plan document and have needed reviews/approvals Write test cases Have needed reviews/inspections/approvals of test cases Prepare test environment and test-ware. Determine test approaches and methods . etc. problem/bug tracking.1. etc. etc.
) The application name or identifier and version The function. A variety of commercial problem-tracking/management software tools are available (see the 'Tools' section for web resources with listings of such tools). For this reason. etc. action. objective. fixes should be re-tested. system. Bug identifier (number. it should encapsulate these processes. feature. where the bug occurred Environment specifics. screen. and determinations made regarding requirements for regression testing to check that fixes didn't create problems elsewhere. to determine if a feature of an application is working correctly. If a problem-tracking system is in place. input data requirements. relevant hardware specifics Test case name/number/identifier One-line bug description Full bug description Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case/test script/test tool Names and/or descriptions of file/data/messages/etc. A test case should contain particulars such as test case identifier. 'Released for Retest'. After the problem is resolved.g. or event and an expected response. used in test File excerpts/error messages/log file excerpts/screen shots/test tool logs that would be helpful in finding the cause of the problem Severity estimate (a 5-level range such as 1-5 or 'critical'-to-'low' is common Was the bug reproducible? Tester name Test date Bug reporting date Name of developer/group/organization the problem is assigned to Description of problem cause Description of fix Code section/file/module/class/method that was fixed Date of fix Application version that contains the fix Tester responsible for retest Retest date Retest results • • • • • • • • • • • • • • • • • • • • • • • • • • Page 21 of 60 . What should be done after a bug is found? • The bug needs to be communicated and assigned to developers that can fix it. etc. test case name. and reproduce it if necessary. it's useful to prepare test cases early in the development cycle if possible. ID. steps. get an idea of it's severity. The following are items to consider in the tracking process: Complete information such that developers can understand the bug. and expected results. etc. Note that the process of developing test cases can help find problems in the requirements or design of an application.) Current bug status (e.Aditi Technologies Basic Concepts of Software Testing • • A test case is a document that describes an input. object. since it requires completely thinking through the operation of the application.. module. platform. 'New'. test conditions/setup.
In this case. etc. testers need to know when retesting is needed. rescheduling of engineers. hardware requirements that may be affected.redesign. and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control. Programming errors . Egos . In many • • • • • • • Page 22 of 60 . Why does software have bugs? • • Miscommunication or no communication . like anyone else. and reporting/summary capabilities are needed for managers. and sheer size of applications have all contributed to the exponential growth in software/system complexity. we'll wing it' o 'I can't estimate how long it will take. effects on other projects. For instance. Poorly documented code . And the use of objectoriented techniques can complicate instead of simplify a project unless it is well engineered. developers need to know when bugs are found and how to get the needed information. Software complexity . work already completed that may have to be redone or thrown out. mistakes will be made.the customer may not understand the effects of changes. client-server and distributed applications. Changing requirements .Aditi Technologies Basic Concepts of Software Testing • • • • Regression testing requirements Tester responsible for regression tests Regression testing results A reporting or tracking process should enable notification of appropriate personnel at various stages. until I take a close look at it' o 'we can't figure out what that old spaghetti code did in the first place' If there are too many unrealistic 'no problems'. enormous relational databases.the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. continuously modified requirements may be a fact of life.programmers.it's tough to maintain and modify code that is badly written or poorly documented. Windows-type interfaces. When deadlines loom and the crunch comes.people prefer to say things like: o 'no problem' o 'piece of cake' o 'I can whip that out in a few hours' o 'it should be easy to update that old code' Instead of: o 'that adds a lot of complexity and we could end up o making a lot of mistakes' o 'we have no idea if we can do that. can make mistakes. known and unknown dependencies among parts of the project are likely to interact and cause problems. the result is bugs. or may understand and request them anyway .scheduling of software projects is difficult at best. Time pressures . often requiring a lot of guesswork.as to specifics of what an application should or shouldn't do (the application's requirements). Enthusiasm of engineering staff may be affected. management must understand the resulting risks. the result is bugs. If there are many minor changes or any major changes. In some fast-changing business environments. data communications. and the complexity of keeping track of changes may result in errors.
presents data in easy-to-use customizable dashboards which enable deep. and other server technologies as well. GUI interface emphasizing ease of use. EJBs. In fact. Software development tools . etc. and a component recognition algorithm that takes into account a variety of attributes. Included 'LeakHunter'identifies potential memory leaks. GJTester Java unit. AppPerfect DevSuite Suite of testing. Filters.visual tools.Aditi Technologies Basic Concepts of Software Testing • organizations management provides no incentive for programmers to document their code or write clear. profiler. Testing Tools and Software 5. Useful for testing CORBA. Cactus A simple open-source test framework for unit testing server-side java code (Servlets. Enables test case and test script development without programming. regression testing for JAVA VM upgrades.1. Includes a natural user interface. Java/J2EE Profiler and other modules. 5. Java test Tools Java Development Tools Java coverage. it's usually the opposite: they get points mostly for quickly turning out code. 'Transaction Tracer' can provide detailed tracing of execution paths and component response times for individual transactions in production systems. user actions and associated data are automatically integrated into an editable tree view reflecting the hierarchical structure of the application's GUI. Recorded GUI elements. Introscope Performance monitoring tool from Wily Technology. and monitoring products for java development from AppPerfect Corp. and clone detection tools from Semantic Designs. without implementing test clients. and server application's modules. etc. it should be hard to read'). Code Analyzer. compilers.). Monitors applications as soon as installed no coding is needed. Load and Performance Test Tools AppPerfect DevSuite 5. Includes: Unit Tester. class libraries. regression.2. intuitive views of interrelation between system components and application infrastructure. Extensive documentation. Intent is to allow fine-grained continuous testing of all files Page 23 of 60 . and contract (black box) test tool from TreborSoft. resulting in added bugs. understandable code. Test private and protected functions. Tag Libs. often introduce their own bugs or are poorly documented. scripting tools. tuning. RMI. qftestJUI Record/playback test tool from Quality First Software for creation. metrics. execution and management of automated Java/Swing application tests. and there's job security if nobody else can understand it ('if it was hard to write. scripting capabilities.
Suitable for use both by developers for unit tests and QA for functional testing. for early detection of software defects and automatic assessment of code quality.Framework to write repeatable java unit tests A regression testing framework written by Erich Gamma and Kent Beck. Abbot Java GUI Test Framework Testing framework by Timothy Wall provides automated event generation and validation of Java GUI components. Free Open Source Software released under the IBM Public License and hosted on SourceForge. Intended for use where there are performance/scalability requirements that need re-checking while refactoring code. capability of working directly on Java method binaries (no recompilation). Free Open Source Software from SourceForge site. The framework may be invoked directly from Java code or accessed without programming through the use of scripts via 'Costello'. which can then be run automatically and independent of other JUnit tests. CSV. Koalog Code Coverage Code coverage analyzer for Java applications from Koalog SARL. for which JUnit may be sufficient). TEXT) or custom report generation. or from the command line. It uses JUnit and extends it.available under the GNU Lesser General Public License. a script editor/recorder. predefined (XML.Robot class. Couples advanced static analysis capabilities to ISO 9126 quality standard framework. LaTex.Aditi Technologies Basic Concepts of Software Testing making up an application: source code but also meta-data files (such as deployment descriptors. From Apache Software Foundation. Free . By Mike Clark/Clarkware Consulting. etc) through an in-container approach. Jemmy A Java library that is used to create automated tests for Java GUI applications. improving upon the very basic functions provided by the java. For use by developers implementing unit tests in Java. Integrates with leading Java Development Environments and platforms.awt. Typically use within your IDE. Also available as plugins for JBuilder and Eclipse. JUnitPerf Allows performance testing to be dynamically added to existing JUnit tests. (Abbot = "A Better 'Bot'). licensed under the BSD License. Contains Page 24 of 60 . Provides recording and playback capabilities. and session merging to allow compilation of overall results for distinct executions. QStudio for Java Java code inspection tool from QA Systems allows automation of a major portion of code inspection process. using Ant. Enables quick composition of a performance test suite. Supports customizing existing rules and defining custom rules. jfcUnit Framework for developing automated testing of Java Swing-based applications at the UI layer (as opposed to testing at lower layers. Site includes a large collection of extensions and documentation. JUnit . Integrates with Ant and JUnit. HTML. Includes: in-process or remote coverage computation.
Fifty different audits and metrics. button pushing. Coverage API provided. Gathers coverage measures of branches. etc. and dependencies for Java. TrueJ Source code audit and metrics tool from BlueBay systems. Inc.. JMSAssert. JSynTest. highly scalable. and other IDE's. JEvolve. Works with source or compiled files. JBrowser class browser. Halstead Measures. Fully integrated plugin for NetBeans. JemmyTest is a program written in Java which uses the Jemmy API to test applications. C. a syntax testing tool that automatically builds a Java-based test data generator. JBench Freeware Java benchmarking framework to compare algorithms. statements.e. source distribution. Shows all dependencies. components. reView Java source code visualization tool from Headway Software. application. J_DocGen for Java code static analysis. and J_Playback for GUI operation capture and automatic playback. integrates with build tools for quality gate and reporting. JVerify Java class/API testing tool uses an invasive testing model allowing access to internals of Java objects from within a test script and utilizes a proprietary OO scripting language... package. user-friendly performance diagnosis tool from Quest Software for Page 25 of 60 . J_Diagrammer for Java code logic analysis. Available as binary distribution (including documentation). etc.Aditi Technologies Basic Concepts of Software Testing methods to reproduce all user actions which can be performed on Swing/AWT components (i. PerformaSure Low-overhead. Cyclomatic Number. etc. can 'reason' about selective need for regression testing Java classes. Panorama for Java Visual environment containing six integrated java tools from ISA. classes. compiler-style output. configurable. an intelligent Java code evolution analyzer that automatically analyzes multiple versions of a Java program and shows how various classes have evolved across versions. Java Tool Suite from Man Machine Systems Includes JStyle. J_SQA for ObjectOriented software quality measurement. method. Reverse engineer and automatically lay out and view code. View coverage data in XML. JCover Java code test coverage analysis tool from Codework Limited. Clover Code coverage tool for Java from Cenqua. it can be used separately as well as together with the NetBeans IDE. JBuilder. a Java source analyzer to generate code comments and metrics such as inheritance depth. tree node expanding. class. J_Structure for Java code structure analysis and diagramming. a tool and technique for writing reliable software. HTML. or jar file.). Seamless integration with projects using Apache ANT. Coverage difference comparison between runs. virtual machines. J_Test for test coverage analysis and test case minimization. JPretty reformats Java code according to specified options. at all levels and between all levels. integrates with a variety of editors/IDE's. JCover test coverage analyzer. text typing. or via a Swing GUI. . control flow analysis and diagramming. and C++ applications. package and produces reports in multiple formats. for speed. methods. file. PDF.
Aditi Technologies Basic Concepts of Software Testing distributed J2EE applications. includes Java support. servlets. Includes Cyclomatic Complexity. to diagnose and resolve performance bottlenecks. (Note: some other tools in these listings also handle testing. Hundreds of easily-confugured run-time. Has online advisor for quality improvement. VTune Intel's performance tuning tool for applications running on Intel processors. TCAT for Java Part of Software Research's TestWorks suite of test tools. testability. and applications.) Page 26 of 60 . and size metrics related to reusability. code coverage analyzer and code analysis for Java. complexity. maintainability. procedural. coverage analysis. It automatically generates and executes JUnit tests and checks whether code follows 400 coding standards and can automatically correct for many. or are planning to add such capabilities. stalls. and clarity. or load testing of java applets. includes JProbe Profiler and JProbe Memory Debugger for finding performance bottlenecks and memory leaks. LOC metrics and MOOD metrics. automatic unit testing and standards compliance tool for Java. 2000 Sun discontinued accepting orders for these products. Halstead Software Science metrics. Traces and reconstructs execution path of end-user transactions across all components of a clustered multi-tieer J2EE system. Sun's Java Test Tools As of February 4. thread debugger. Integrates with several Java IDE's. Enhanced Cyclomatic Complexity. and JProbe Threadalyzer for finding deadlocks. LProbe Coverage code coverage tool. and code coverage tool suite from Borland (formerly from VMGear). OS. Krakatau Metrics for Java Software metrics tool from Power Software includes more than 70 OO. written in Java. JProfiler freeware version available. JProbe Developer Suite Collection of Java debugging tools from Quest Software. management. DevPartner Java Edition Compuware's (formerly NuMega) debugging/productivity tool to detect and diagnose Java bugs and memory and performance problems. and network metrics. thread and event analysis. Jtest ParaSoft's Jtest is an integrated. and race conditions. OptimizeIt Profiler. Check listed web sites for current information. Includes suggestions for optimization techniques.
free trial period available.Aditi Technologies Basic Concepts of Software Testing 5. domain names. ChangeAgent Link checking and repair tool from Expandable Language. free version or low-cost pro version. and FTP protocols. InfoLink Link checker program from BiggByte Software. Handles HTTP. replaces links but does not reformat or restructure HTML code. For Windows platforms. can also produce a graphical site map of entire web site. Automatically-scheduled reporting by e-mail. Linkalarm Low cost on-the-web link checker from Link Alarm Inc. DNS servers and SSL certificates. Excel. employs a simple. Xenu's Link Sleuth Freeware link checker by Tilman Hausherr. fixes broken links with an easy. Ideal for dynamic sites requiring frequent link checking. Runs as often as every hour. CSV. HTML. Export to text. such as internal and external links. updates links to moved and renamed files. For Windows.. Web Link Validator Link checker from REL Software checks links for accuracy and availability. Page 27 of 60 . provides multiple-level undo/redo for all operations. For Windows.3. detects and reports redirected URL. several report formats available. multiple page list and site list capabilities. Link Checker Pro Link check tool from KyoSoft. customizable reports. supports SSL websites. RTF. Link Checking Tools HiSoftware Link Validation Utility Link validation tool. evaluation version available. includes FTP link checking. 3-click process. previews files when fixing broken links and before orphan removal. Freeware 'REL Link Checker Lite' version available for small sites. Identifies orphan files and broken links when browsing files. SiteAnalysis Hosted service from Webmetrics. Site Map. HTTPS. Site Audit Low-cost on-the-web link-checking service from Blossom Software. For Windows. or as infrequent as once a week. finds broken links or paths and links with syntactic errors. can be automatically scheduled. used to test and validate critical website components. Alert Linkrunner Link check tool from Viable Software Alternatives. familiar interface for managing files. partial testing of ftp and gopher sites. for Windows.
relocations. Windows. ability to check password-protected areas. Web Functional and Regression Testing Tools IeUnit IeUnit is an open-source simple framework to test logical behaviors of web pages. Available as source code or binaries. by Francois Pottier. allowing for customizable queries and reports.5. reports of orphan files and files with mismatching case. timeouts. PERL script for a web spider for web site maintenance. For automatic fixing of HTML errors. unauthorized-access. results can be exported to database. and finding problem HTML areas. randomized order checking. LinkScan Electronic Software Publishing Co. doc-not-found. via the WWW Consortium (the folks who set web standards). Mac. MOMspider Multi-Owner Maintenance Spider. link checker. For Windows. page size analysis. for UNIX and PERL. can check for bad links due to specified problems such as server-not-found. For Windows.'s link checker/site mapping tool. support of proxy servers for remote URL checking. Big Brother Freeware command-line link checker for Unix. This site contains an interesting discussion on the use of META tags. Discontinued. For all UNIX flavors.0 or 3. Handles one URL at a time. HTMLchek for awk or perl Old but still useful HTML 2. Results stored in database. Perl Testing Tools W3C Link Checker Link checker PERL source code.0 validator programs for AWK or PERL by H. Windows. Distributed under Gnu General Public License. 5. Not updated in recent years. ID/Password entries. LinkLint Open source Perl program checks local/remote HTML links. HTML TIDY Free utility available from SourceForget. 5. It helps users to create. binary avaialable for Linux. 'what's new' reporting. Utilizes the HTTP 'HEAD' request instead of the 'GET' request so that it does not require retreival of the entire html page. Not updated in recent years. capabilities include specified URL exclusions. site has much documentation and related info. but old versions still available as freeware.Aditi Technologies Basic Concepts of Software Testing changed-link checking. formatting disorganized editing. configurable. reports URLs changed since last checked.4. CyberSpyder Link Test Shareware link checker by Aman Software. Has not been updated in recent years. capabilities include automated retesting of problem links. Validates hyperlinks for all major protocols. Churchyard. released under IBM's Common Public License. support for all standard server-side image maps. originally by Dave Raggett. Includes cross referenced and hyperlinked output reports. HTML syntax error checking.net. Includes capabilities for central management of large multiple intranet/internet sites. organize and Page 28 of 60 . Available as source code. test resumption at interruption point.
Can group tests into a testsuite that again can be part of a bigger testsuite. Includes capability to expand scope of macros by editing and adding loops. Fuzzy matching capabilities. XMLbased test script code is editable with user's preferred XML editor. Reports information needed to troubleshoot problems. Handles Applets. ensure robustness of scripts with Synchronization commands. Free including source code. Includes an Intelligent. cookie testing and form submission. such as bitmap or text matching. etc. Key features besides capture/playback include automatically detecting and capturing standard and custom content errors. WinTask Macro recorder from TaskWare. and Runner executes the test case document. animated bitmaps. etc. Includes all of the typical functions that would be expected from JUnit and the PHPUnit ports. 90-day evaluation copy available. Flash. test case documents are always synchronized with the application. C++ or C++/MFC. HTML/DOMAware and Object Mode Recording Engine. The simple navigation methods and ready-to-use assertions allow for more rapid test creation than using only JUnit and HttpUnit. STC Master and STC Master Service. Enables 'Persistent Acceptance Testing' that activates tests each time a web application is used. playback window/size can be different than in capture. Editable scripts can be recorded in SmithSript language or in Java. TestAgent Capture/playback tool for user acceptance testing from Strenuus. Test results are reported in either plain text or XML format for later presentation via XSLT. can be inserted during a recording. and a Data-Driven. with its HTML objects recognition. Includes a WinTask Scheduler. jWebUnit Open source Java framework that facilitates creation of acceptance tests for web applications. automates repetitive tasks for Web site testing (and standard Windows applications). scripts have to be developed manually. Utilizes a distributed testing model and consists of three parts: STC Administrator. validation of table contents. TestSmith Functional/Regression test tool from Quality Forge. but also adds mock objects. Page 30 of 60 . SimpleTest Open source unit testing framework which aims to be a complete PHP developer test solution. form entry and submission. Utilizes HttpUnit behind the scenes. Adaptable and MultiThreaded Playback Engine. Maker creates test case documents. but all recorded items are validated and logged 'on the fly'. Has an XMLbased scripting capability to enable modifying captured scripts or creating new scripts. LLC. until recording capabilities are added. Controls are recorded as individual objects independent of screen positions or resolution. Provides a high-level API for navigating a web application combined with a set of assertions to verify the application's correctness including navigation via links. This includes web page navigation. has some JWebUnit functionality as well. TestCaseMaker/Runner Test case document driven functional test tool for web applications from Agile Web Development.Aditi Technologies Basic Concepts of Software Testing Site Test Center Functional and performance test tool from Alliance Software Engineering. Standard reporting XSLT stylesheets included. Active-X controls. branching statements. and can be adapted to any reporting style or requirements. (300+ commands). and other typical business web application features. Canoo WebTest Free Java Open Source tool for automatic functional testing of web applications. Special validation points.
can audit UNIX. SANS/ISTS certified. Provides detailed reports and vulnerability fix recommendations. Includes instructions to help immediately remedy security problems. and more for security vulnerabilities which may lead to interrupted service. Automatically downloads latest IDS attack signatures. Win servers STAT Scanner Page 34 of 60 . Inc. plug-in facility for third party apps. CVE standards support.a set of scripts that scan a Unix system for security problems. websites. real-time IDS event analysis. Perimeter Check SecurityMetrics 'Perimeter Check' service analyzes external network devices like servers. Qualys Free Security Scans Several free security scan services from Qualys. data theft or system destruction. and browser checkup tool. SARA 'Security Auditor's Research Assistant' Unix-based security analysis tool from Advanced Research Corp. remediation management of vulnerabilities. network topology discovery and organizational executive reporting for hundreds of administrators via easy-to-use web interface. Lightning Console Security management tool from Tenable Network Security for multiple security and network administrators across multiple organizations. including SANS/FBI Top 20 Vulnerabilities Scan. inside and outside the firewall. Qualys Guard Online service that does remote network security assessments. an upgrade to the TAMU 'tiger' program .Aditi Technologies Basic Concepts of Software Testing Integrated software and hardware device includes Intrusion Detection and Prevention Systems and Vulnerability Assessment. Scheduled vulnerability scanning. provides proactive 'Managed Vulnerability Assessment'. routers. Runs on Macintosh. network security scan. firewalls. MacAnalysis database is updated regularly with a "Vulnerability Update" function to update database of more than 100 new holes per month. MacAnalysis Low cost Network Security audit tool from Lagoon Software. asset management. Freeware. Also available is 'Tiger Analytical Research Assistant' (TARA). updated bi-monthly. Can automatically schedule vulnerability assessment of designated IP addresses during low traffic times. Mac. Operates as a Layer 2 Bridge no network configuration needed. Supports the FBI/SANS Top 20 Consensus. vulnerability assessment scripts and program enhancements nightly. based on the SATAN model. remote self scan and API facilities.
eliminate vulnerabilities using recommended solutions with links to related websites and knowledge-base articles. track vulnerability trends via analyses comparing current and previous assessments. Works with a variety of operating systems. the server (nessusd) manages the 'attacks'. web server requests-per-second benchmark. Port scans. based on 'never trust the version number' and 'never trust that a given service is listening on the good port'. for security analysis of Windows/UNIX/Linux and other resources. and SAINTbox scanner appliance. Utilizes comprehensive updatabale vulnerability databases to automatically detect vulnerabilities. enabling identification of exactly where to correct vulnerabilities to enforce corporate security policies. by Renaud Deraison. Scans systems and services on the network and safely simulates common intrusion or attack scenarios. correct vulnerabilities across the network with 'AutoFix' function. Customizable reporting. NMap Network Mapper Page 35 of 60 . Secure-Me Automated security test scanning service from Broadbandreports. Single machine or full network scans. Limited free or full licensed versions available. CVE compatible. Nessus is made up of two parts : a server and a client. analysis reports of vulnerabilities with detailed information relating to the name/description/risk level of each vulnerability. Utilizes a root-cause and path-analysis engine to illustrate exact sequence of steps taken to uncover vulnerabilities. customizable security reports for management and technical personnel. Updated regularly. Runs on many UNIX flavors. denial-of-service checks. 45 common web server vulnerability checks. NetRecon Network vulnerability assessment scanner from Symantec. Includes DoS testing. whereas the client is a frontend designed to collect the results. Nessus Security Scanner Free.Aditi Technologies Basic Concepts of Software Testing Tool from Harris Corp. product runs on Windows. Capabilities include: scan and analyze an entire network domain and/or a single machine. open-source remote network security auditing tool. Learns about network as it scans.Security testing tool from SAINT Corporation. retest corrected vulnerabilities immediately. Also available is 'WebSAINT' self-guided scanning service. adapting penetration strategy based on previous results. select or ignore specific vulnerabilities via configuration files. Includes more than 1000 tests in 23 vulnerability categories. An updated and enhanced version of the SATAN network security testing tool. Can scan a variety of operating systems. and Nessus Attack Scripting Language. and a wide variety of other tests. reports specify severity levels of problems. SAINT Security Administrator's Integrated Network Tool .com for individual machines.
Page 36 of 60 . bv-Control for Internet Security Secutiry scanning tool from BindView. CERIAS Security Archive Purdue University's 'Center for Education and Research in Information Assurance and Security' site. NetIQ Security Analyzer Multi-platform vulnerability scanning and assessment product. Runs on most flavors of UNIX as well as Windows. intrusion detection tools. Remotely evaluates security of network perimeter. to uncover and report systems vulnerabilities that might be open to attack. and many other characteristics. etc. what services (ports) they are offering. what type of packet filters/firewalls are in use. provides automated vulnerability assessment for measuring online security risks. Systems are analyzed on demand or at scheduled intervals. Performs scheduled and selective probes of network services. 'hotlist' section includes extensive collection of links. electronic law. etc. servers. Also includes an FTP site with a large collection of (mostly older) security-related utilities. includes capabilities for checking and remediation of security problems of servers. what operating system (and OS version) they are running. or managed service. hubs. designed to rapidly scan large networks or single hosts. routers.Aditi Technologies Basic Concepts of Software Testing Free open source utility for network exploration or security auditing. remediation information. DMZ and externally visible assets. intrusion detection resources. routers. Available as a hardware appliance. workstations. For Windows/Solaris/Linux. desktops. etc. operating systems. and any other devices with an IP address on a network. Automatic update service allows updating with latest security tests. publications. prepares remediation advice. CyberCop ASaP Managed vulnerability assessment service from McAfee/Network Associates. Uses raw IP packets in novel ways to determine what hosts are available on the network. Includes a Software Developer's Kit to allow custom security test additions. InternetScanner Tool from Internet Security Systems. Foundstone Vulnerability management software tools from McAfee/Network Associates can provide comprehensive enterprise vulnerability assessments. scanners. Scans any device on network regardless of platform. to hundreds of security information resources and tools. firewalls. software product. etc. organized by subject. Provides flexible risk management reports. printers.
combining site content. to enable detailed analysis of what is going on behind the scenes of Web operations. and trend analyses using intuitive browser-based reports. not just for testing.. for monitoring compliance standards related to Privacy. Web Site Management Tools (This section includes products that contain: site version control tools. server management and optimization tools. and OS's. service types. structure and visitor traffic data into one searchable data base. promoting. combined utilities/tools. Desktop product for multiple platforms. Integrates performance Page 37 of 60 . WebCEO Tool from Radiocom Ltd. webSensor Integrated site measurement and analysis tool from Moniforce BV. Passive monitoring records real user sessions and reliably detects and logs all error situations encountered by real-users. WYSIWYG editor. actively monitors and evaluates the health of a site's live applications utilizing synthetic business transactions.7. Limitedcapability free version also available. site scores. Accessibility.) 5.. analyzing web sites. and Web Governance. Recorded transactions can then be analyzed when production errors occur and fed back into SilkPerformer for real-world performance testing scenarios. Information Assurance. and online service available.) COAST WebCentral Site management tool from Coast Software. and authoring/publishing/deployment tools that include significant site management or testing capabilities. Inc. (Includes various types of security tools. complete inventory of Web sites and Comparison Inventory Reports to compare different versions of a Web site. works with a variety of web servers. and site monitoring capabilities. Capabilities include: high-level summary reports.org. Maxamine Knowledge Platform Integrated site managment tool from Maxamine Inc. SilkCentral Performance Monitor Tool from Segue Software. Top 75 Security Tools Listing of 'top 75' network security tools from survey by Insecure. Includes link checker. traffic analysis. ManageEngine Applications Manager Site management tool from AdventNet.Aditi Technologies Basic Concepts of Software Testing trend analyses and comprehensive data sets to support policy enforcement.roll-up of results from different business units or global Web sites for overall views of site compliance. Free and professional versions available. database servers. for maintaining. Similar managed service also available. Inc. FPT/publishing.
including server errors. Site Manager to create and manage your web site. PROGNOSIS Comprehensive tool from Integrated Research Ltd. network management. click path analysis. net send. For Windows. real-time executive dash board. for performance and availability monitoring. pagers. capabilities include notifications via email. Other capabilities include search engine optimization. and 'Real Scripts' automatically generated from recorded user sessions for use in specified other load testing tools. providing context and correlation data for application failure analysis. and popup. Freeware version of GFI Network Server Monitor is also available. visitor segmentation. etc. sends email notification if site is slow or down. and diagnostics. bandwidth usage. run scripts. Legacy-to-Web Tools can "bulk-load" online catalog from legacy Page 38 of 60 . Captures and monitors real user sessions. Web Site Monitoring Performance Monitoring . customizable views of success/failure rates for key online business processes and other critical metrics. page sizes. GFI Network Server Monitor Server management tool from GFI Software Ltd. restart services. RealiTea Web application management solution that provides detailed visibility into availability and functionality issues to enable efficient problem identification. isolation. from AllScoop. eliminates need for programmers and special production staff to maintain catalogs. and more. includes modules to check HTTP and ICMP/ping for checking availability of HTTP and HTTPS sites. Cuevision Network Monitor Monitoring tool from Cuevision for monitoring website. Alerts via email. Web Compliance Manager to manage integrity and accessibility. page aborts. RedDot CMS Web content managment system from RedDot Solutions includes modules such as SmartEdit. ContentStudio E-catalog management tool from TechniCon Systems with Win Explorer-type interface with drag and drop functionality. and repair. suited to large systems.Free open-source website performance monitoring and uptime notification application in PERL. services. response errors. Add-on capabilities include a 'Dashboard' to provide real-time. applications.Aditi Technologies Basic Concepts of Software Testing and availability data with site traffic data. and network. network errors. automatically reboot servers. checks network and servers for failures and fixes them automatically. etc. SMS. server. restart apps and services. Asset Manager to securely centralize images. response times.
FTP Assistant for Webmasters FTP client from Byteway. link verification. spatial relationships between products. WebLight HTML validator and link checking tool from Illumit LLC. PHP-Code Insertion. All can be integrated without writing code. Works on multiple platforms. comprehensive Communications Gateway for inbound and outbound data. SpinPike Flexible and scalable content management system from SavvyBox Systems. drag-and-drop. Message Board. based on database-driven. Statistical Reports.Aditi Technologies Basic Concepts of Software Testing product data. TestWEB Monitor Web site monitoring tool from Original Software checks 5 or more URL's up to every 30 seconds. and more. high-level functions save template coding time. Installer easily installs system on your server. CrownPeak CMS Content management service from CrownPeak Technology. additional features include manual or customizable automated control.com for web site updating. Constructioner Light Edition available as Freeware. Free for use on small sites. Database Backup. Multilingualism. Design/administrate database connected PHP web applications in combination with individual webdesign. Search. can manage all components across complex Web-enabled applications by measuring and monitoring performance. reporting and fixing problems before users see them. etc. verifying content and functional integrity. Content and Table Management (WYSIWYG-Editor). Trellian InternetStudio Page 39 of 60 . low cost for large sites. and a robust API. Receive instant notification by email whenever one of the sites is unavailable. load testing. site cleanup. detecting. such as option compatibilities on a single product. User Administration. Dynamic Menu. scheduler. Provides complete software developers environment. Includes: Ready-to-use Backoffice. Tonic Suite of web site management/monitoring/test tools from Tonic Software. upload resuming. template-based dynamically-created content. Constructioner Website development software with integrated content management system from Artware Multimedia GmbH. which hosts the management system application and the client's administrative interfaces and pushes the final assembled pages to client Web servers. HTML and GIF optimization. WYSIWYG editor. Capabilities include defining intra-configuration rules.
CGI scripts return errors or the SQL server is not reachable. allows a distributed team of non-technical content experts to manage every aspect of site content. securing and sharing digital and paper-based documents and reports. 'Stellent Content Server' uses a web-based repository. Rhythmyx Content Manager Web content management product from Percussion Software. and archiving. Page 41 of 60 . integrates with other OpenView products to provide a variety of management and monitoring services and capabilities. For Windows. It can constantly monitor any number of servers and ports. based on native XML and XSL technologies. replication and administration. editing. Manages Web content. Mac. Enables end-user emulation of major business-critical applications as well as a single integrated view of the complete Internet infrastructure. services for capturing. Eliminates problems encountered when moving files between Windows. IP Check Server monitoring tool from Paessler GmbH. management of workflow. Enables scheduling of content refreshes. Other modules provide additional services such as: services for creating. reuse and access. publishing. digital assets. and UNIX systems and publishing to CD-ROM. workflow. documents. and customizable workflow. managing and publishing Web content and supporting from one to thousands of Web sites. Content Management Server Windows based content mgmt tool from Microsoft (formerly 'nResolution' from nCompass Labs). Enables services such as library services. index/search. personalization. including creation. diagnose and troubleshoot problem occurrences. portals and scanned images. HP OpenView Internet Services Internet services monitoring/management tool from HP. version control. and maintaining highly dynamic web sites. Enterprise web content management system that enables quickly and efficiently building. Stellent Universal Content Management System Content management tool from Stellent. Alerts webmasters if a webserver is not working correctly because the web service is down. Broadvision Suite of content and publishing management tools from Broadvision Inc.Aditi Technologies Basic Concepts of Software Testing correcting the links to those files automatically.. where all content and content types are stored for management. security. production. conversion services. isolate. and manage and report on service level agreements. tracking of revisions. Designed to help IT staff efficiently predict. For Mac or Windows. content development. anticipate capacity shortfalls. Inc. deploying. staging. and indexing content by means of a browser window or via MS Word. and services for collaborative environments and for digital asset and records management.
server management. Evaluation copy available. version control. PowerMapper From Electrum Multimedia. UNIX. HTML validation. Includes a 'Database Import Wizard'. network management. logging.Aditi Technologies Basic Concepts of Software Testing Web-content workflow management system with browser-based interface includes configurable workflow management. number of pending requests. WebTrends Analysis Suite Web site management tool from NetIQ includes log analysis. etc. scalable web content management solution that enables collaboration. and can automatically convert output to CALS or HTML table models or to XML. XMetal XML development tool from BlastRadius for XML-based web site authoring and validation. metrics. Page 43 of 60 . requires Windows and MSIE. E-mail/pager notifications. Supports multiple platforms and browsers. e-mail submission of web content. For Windows. alerting. logging capabilities. Site/C 'Set-and-forget' utility from Robomagic Software. for periodic server monitoring for web server connection problems. link analysis and quality control. monitoring. For Windows. proxy server traffic analysis and reporting. access control. for customizable automated site mapping. and e-mail notifications. Macromedia Web Publishing System Macromedia's complete. allows defining and applying existing workflow and approval rules to web content management process. monitoring and recovery. link checking. real-time server monitoring of server response time. Interwoven Team Site Web development. Unicenter TNG w/Web Management Option Site management application from Computer Associates includes access and security control. management. For MS and Netscape web servers. works with many servers. server failure detection. content management and site visualization. Web based interface for load balancing administration. standards-based environment. link problems. and platforms. For Windows. OS's. For Windows. and publishing control tool. and publishing to websites within a centrally controlled. Equalizer Load balancing server appliance and site management tool from Coyote Point Systems.
etc. NetObjects Fusion Site authoring/management tool. links. For Windows. scheduled and on-demand reporting. XML repositories. HTML-to-Text converter. databases. Date stamper. Content Management Server Vignette Corporation's product for web site collaborative content. administration. HTML PowerTools HTML validator. link checker. thumbnail viewer. etc. variable lock controls. Support for managing content stored in databases. and maintenance. For Windows or Unix. For Windows. API's. file uploading. publishing. error info is linked to error in html. page in screen is validated with results displayed below. TeamSite Interwoven's collaborative web site production control. For Windows. Includes published API for creating custom monitors. provides notifications of problems. Meta manager. management. and management product for enterprise-wide internet and intranet projects. Microsoft FrontPage Microsoft's web site authoring and site management tool. For Windows and Unix. global search-and-replace. includes site management capabilities. staging/publishing control. and encryption capabilities. logs. Validates against various HTML versions. spell checker. HomeSite Macromeida's web site project management/authoring/validator tool. browser extensions. link checking. security. browser interface. and servers. Server side compatible with many available web servers. Monitors mimic users' end-to-end actions.. From Talicom. customizable reports. Page 44 of 60 .Aditi Technologies Basic Concepts of Software Testing SiteScope Mercury's product for agentless site monitoring and maintenance. comparison capabilitie. graphics management. Link checker. Visual site structure editor. Includes version control. Runs on servers and monitors server performance. connections. and static files. Supports a wide variety of web attributes. Includes automated deployment. image tag checker. has updateable rulebase. OpenDeploy Interwoven's configurable control system for deploying from development to production environments. layout editor. file edit merging.
The Manager component includes integrated editing functionality for assembly of important video clips to share with stakeholders. For anyone interested in css.8. examine server's HTTP headers. analyzing and sharing of usability data. from how it breaks down the HTML.reports on how a search engine may view a webpage. Other Web Testing Tools Morae Usability test tool for web sites and software. more. can specify the screen width(s). for automated recording. and MS Excel for charting and modeling. Free. web standards and elastic design. system operations in 7 different categories. Any application which accepts an HTTP Proxy can be configured to run through Page 45 of 60 . A Recorder records and synchronizes video and data. and share with others. where network traffic is captured over a LAN. font magnification(s) and page position(s) for the proofs. Remote Viewer observers can set markers and add text notes. it displays test user's computer screen along with a picture-in-picture window displaying the test participant's face and audio. TestGen Free open-source web test data generation program that allows developers to quickly generate test data for their web-services before publicly or internally releasing the web service for production. A Remote Viewer enables geographically dispersed observers to watch usability tests from any location. from TechSmith Corp. URL2image. how it interprets page's robot exclusion rules and more. system resource usage. Fiddler An HTTP Debugging tool by Eric Lawrence. For Windows.Aditi Technologies Basic Concepts of Software Testing 5. Extreme Web Analyzer Web site performance analysis tool by Dale Feiste for use in a development or controlled test environment. SiteTimer service . Consists of 3 components. requires a system capable of running Ethereal and/or Sniffer.com Service from HREF Tools to check web page appearance in different Browser/OS combinations. Allows user to save and review relevant information for bug reports. Acts as an HTTP Proxy running on port 8888 of local PC. Enter URL and receive back report with screenshots taken in real time on real hardware. EngineViewer and SiteTimer Free basic services: EngineViewer . Repro Manual testing 'helper' tool that records desktop video. upload to a bug tracking system. to which links it extracts. and compress the result into a very small file to replay. Instruments in memory the target application at runtime so no changes are required to application under test. Perl. The predictable network performance of this environment enables reliable modeling of trace data.Find out how long it takes various connection types to get a page. check all the graphical links to ensure they're correct. creating a digital record of system activity and user interaction. and system configuration information.
etc. and delay. FunnelIT Free utility from ByStorm Software helps streamline test reporting and analysis by gathering Windows platform environment/configuration information (running tasks. and other versions available in multiple languages. IP Traffic Test and Measure Network traffic simulation and test tool from Omnicor Corp. system file listings. Designed to be much simpler than using NetMon or Achilles. processes. and much more. loss. SetSim Pro Northern Webs' free on-the-web meta tag checker. XML. html. Available as an executable intended to leave no footprint. and "fiddle" with incoming or outgoing data. Professional. Free. processor information. Services information. tune-up wizard. set breakpoints. Also allows comparisons among results from multiple systems. ports. OS version. users can watch their web traffic as if they're watching a movie. physical memory information. etc. etc. modules. loaded modules. Logs all HTTP traffic between between computer and the Internet. drives. Drive Free Space/Size/Used info.) in small text files.Aditi Technologies Basic Concepts of Software Testing Fiddler. Installed Software and Installed XP Updates. manage and monitor throughput. file system and memory bandwidth benchmarking. VisitorVille Site traffic monitoring tool from World Market Watch Inc. more. component details. and allows inspection of the HTTP data. Provides performance enhancing tips. data creation or capture and replay. Full-featured version (not free) has configurable reporting. for Windows.NET event-based scripting subsystem. Network Adaptor Information. Path Application Manager Page 46 of 60 . Includes CPU. FREEping Free ping software utility from Tools4ever which will ping all your Windows-based servers (or any other IP address) in freely-definable intervals. and includes a simple but powerful JScript. memory details.). device drivers. Provides large variety of information about a Windows system's hardware and software. virtual memory settings. environment settings. can generate TCP/UDP connections using different IP addresses. mainboard. that depicts website visitors as animated characters in a virtual village. Sandra 'System ANalyser. and also allows gathering TCP/IP Address/Network Information. includes additional search-enginerelated capabilities. Reporting via save/print/fax/email in text. ODBC sources. Will send a popup when one of the servers stops responding. Free. Diagnostic and Reporting Assistant' utility from SiSoftware. services.
For Windows.E. captures and displays packet data between local client and specified web server. in most versions of every browser ever released. originally designed for testing networking protocols. SpySmith Simple but powerful diagnostic tool from Quality Forge. Other info: runs on FreeBSD packages. HTTP Interceptor A real-time HTTP protocol analysis and troubleshooting tool from AllHTTP. It allows the user to peek inside I.Aditi Technologies Basic Concepts of Software Testing Application Monitoring and management tool from Winmoore. RAMP Section 508 and W3C Accessibility Guidelines tool from Deque Systems that automates analysis and remediation of non-compliant web functionality. available as on-line service. Linux. Uses pattern recognition technology to peer deep inside customized or COTS applications. Page 47 of 60 . Windows (on-demand). packet losses. caching proxies. and multipath effects. SpySmith can also spy on Windows objects. and XML switches. delays. Browser-based Documents (including those without a 'view source' command) to extract precise information about the DOM elements in an HTML source. security. Dummynet Flexible tool developed by Luigi Rizzo. any version. robustness. Browser Cam Service for web developers and testers that creates screen captures of web pages loaded in any browser. Macintosh. Inc. especially useful when testing web sites and web-based applications. Split-screen display and dual logs for request and response data. Interceptor also allows changing of select request headers on-the-fly. CoAdvisor can test for protocol compatibility. such as "Referrer" and "User Agent". Co-Advisor Tool from The Measurement Factory for testing quality of protocol implementations. and troubleshooting with code coverage capabilities. binaries. Free 90-day trial. with GUI.com. Enables enhancement of QA. Can be used on user's workstations. filters. PocketSOAP Packet-capture tool by Simon Fell. analogous to running an MRI scan. freeware. Allows viewing of web page appearance on Windows. View all headers and data that travel between your browser and the server. can be used in testing to simulate queue and bandwidth limitations. Also available is PocketXML-RPC and PocketHTTP. binaries and source available. Linux RPMs. Co-Advisor's current focus is on HTTP intermediaries such as firewalls. any operating system. testing. For Windows. or source code. compliance. Can log captures to disk. and other quality factors. or on FreeBSD machines acting as routers or bridges.
Allows developers to test down to the unit level by simulating the internal software interfaces (message queues. window advertisements. link validation.GUI XML Conformance Test Suite XML conformance test suites from W3C and NIST. tcptrace Tool written by Shawn Ostermann for analysis of TCP dumpfiles. Custom Checks and Test Suites to meet organization's standards. For managing and developing test cases and scripts. etc. etc). mailboxes. The test report contains background information on conformance testing for XML as well as test descriptions for each of the test files.) Tool can learn what request/response scenarios are being tested for future tests and can work with any protocol. Freeware. and various graphs. throughput.comm Tool from Omsphere LLC for simulating virtually any software interface (internal or external). or WinDump. contains over 2000 test files and an associated test report (also in XML). can also programmatically fix most common errors found. retransmissions. Allows testing without pitfalls associated with live connections to other systems (TCP/IP. HP Net Metrix.Aditi Technologies Basic Concepts of Software Testing TcpTrace Tool by Simon Fell acts as a relay between client and server for monitoring packet data. For windows. for Windows. and segments sent and received. and generating test scripts. This is a set of metrics for determining conformance to the listed W3C XML Recommendation. and as source code. Certify Test automation management tool from WorkSoft. Privacy and Intellectual Property policy verification. Overall Site Quality. ProxyTrace Tool by Simon Fell acts as a proxy server to allow tracing of HTTP data. Runs on Windows platforms. Available for various UNIX flavors. FTP. and mainframe applications. Also available: MITS. HiSoftware Web Site Monitor Tool allows user to monitor your server and send alerts. Works with all text-based IP protocols. HiSoftware AccVerify Tool for testing site Accessibility & Usability. any message definitions. Can crawl a site and report errors. Searchability. Page 48 of 60 . snoop. and any network. client/server. such as those produced by tcpdump. can be used by setting browser to use it as a proxy server and then can monitor all traffic to and from browser. freeware. etherpeek. freeware. allows monitoring web sites for changes or misuse of your intellectual property in metadata or in the presented document. bytes. MITs. Inc. Can produce various types of output with info on each connection seen such as elapsed time. round trip times. Runs on Windows. Ethernet. For automated testing of Web.
and ensuring that problems are found and dealt with. What is 'good code'? 14. How can new Software QA processes be introduced in an existing organization? 7. What is 'Software Quality Assurance'? 2. What are 5 common problems in the software development process? 12. What is 'Software Quality Assurance'? Software QA involves the entire software development Process . then D should happen'). Testing FAQ 1. What is a 'walkthrough'? 9. Also common are project teams that include a mix of testers and developers who work closely together. Why is it often hard for management to get serious about quality assurance? 5. What is 'Software Testing'? Testing involves operation of a system or application under controlled conditions and evaluating the results (eg. It is oriented to 'prevention'. Organizations vary considerably in how they assign responsibility for QA and testing. (See the Books section for a list of useful books on Software Quality Assurance. 'if the user is in interface A of the application while using hardware B. What is verification and Validation? 8.Aditi Technologies Basic Concepts of Software Testing 6. What are some recent major computer system failures caused software bugs? by Page 50 of 60 .) 2. and does C. with overall QA processes monitored by project managers. What are some recent major computer system failures caused by software bugs? 4. It will depend on what best fits an organization's size and business structure.monitoring and improving the process. Will automated testing tools make testing easier? 1. What is SEI? CMM? CMMI? ISO? Will it help? 16. What is 'good design'?Top of the Document 15. What's an 'inspection'? 10. It is oriented to 'detection'. The controlled conditions should include both normal and abnormal conditions. making sure that any agreed-upon standards and procedures are followed. What kinds of testing should be considered? 11. Why does software have bugs? 6. What is 'Software Testing'? 3. 3. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. What is software 'quality'? 13. What is the 'software life cycle'? 17. Sometimes they're the combined responsibility of one group or individual.
forced shutdown of 100 power plants. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. Articles about the incident stated that it took two weeks to fix all the resulting errors. a software bug was determined to be a major contributor to the 2003 Northeast blackout.S. Reportedly the original contract allowed for only 6 weeks of acceptance testing and the system was never tested for its ability to handle a rate increase. A bug in site management software utilized by companies with a significant percentage of worldwide web traffic was reported in May of 2004. The company found and reported the bug itself and initiated the recall procedure in which a software upgrade fixed the problems. In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects of the Y2K bug. The error was found and corrected after examining millions of lines of code. the vendor had reportedly delivered Page 51 of 60 . that additional problems resulted when the incident drew a large number of e-mail phishing attacks against the bank's customers. According to news reports in April of 2004. which was unable to correctly handle and report on an unusual confluence of initially localized events. Studies were under way to determine which. Millions of bank accounts were impacted by errors due to installation of inadequately tested software code in the transaction processing system of a major North American bank. The bug resulted in performance problems for many of the sites simultaneously and required disabling of the software until the bug was fixed. and economic losses estimated at $6 billion. news reports revealed the intentional use of a software bug as a counter-espionage tool. in the early 1980's one nation surreptitiously allowed a hostile nation's espionage service to steal a version of sophisticated industrial software that had intentionally-added flaws. The failure involved loss of electrical power to 50 million customers. This eventually resulted in major industrial disruption in the country that used the stolen flawed software.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • • • Media reports in January of 2005 detailed severe problems with a $170 million high-profile U.S. if any. according to mid-2004 news reports. government IT systems project. News reports in September of 2000 told of a software vendor settling a lawsuit with a large mortgage lender. According to the report. and that the total cost of the incident could exceed $100 million. News stories in the fall of 2003 stated that a manufacturing company recalled all their transportation products in order to fix a software problem causing instability in certain circumstances. In July 2004 newspapers reported that a new government welfare management system in Canada costing several hundred million dollars was unable to handle a simple benefits rate increase after being put into live operation. The bug was reportedly in one utility company's vendor-supplied power monitoring and management system. the worst power system failure in North American history. the trains were started by altering the control system's date settings. In early 2004. The company found that many of their newer trains would not run due to their inability to recognize the date '31/12/2000'. retailer was reportedly hit with a large government fine in October of 2003 due to web site errors that enabled customers to view one anothers' online orders. portions of the project could be salvaged. A major U.
one of whom was known throughout the land and employed as a physician to a great lord. rescheduling of engineers. January 1998 news reports told of software problems at a major U. The school district decided to reinstate it's original 25-year old system for at least a year until the bugs were worked out of the new system by the software vendors.as to specifics of what an application should or shouldn't do (the application's requirements). In October of 1999 the $125 million NASA Mars Climate Orbiter spacecraft was believed to be lost in space due to a simple data conversion error. Changing requirements (whether documented or undocumented) . The problem went undetected until customers called up with questions about their bills. was delivered late. data communications. enormous relational databases. Why does software have bugs? • • Miscommunication or no communication .S.S.000 business customers over a period of 8 days in August of 1999. Programming errors .the enduser may not understand the effects of changes. Among those affected was the electronic trading system of the largest U. like anyone else. can make mistakes. hardware requirements that may be affected. Why is it often hard for management to get serious about quality assurance? • Solving problems is a high-visibility process. and sheer size of applications have all contributed to the exponential growth in software/system complexity. work already completed that may have to be redone or thrown out. Several investigating panels were convened to determine the process failures that allowed the error to go undetected. effects on other projects. 4. preventing problems is lowvisibility. client-server and distributed applications. Multi-tiered applications. problems included 10. Bugs in software supporting a large commercial high-speed data network affected 70. major problems were reported with a new computer system in a large suburban U.000 erroneous report cards and students left stranded by failed class registration systems. public school district with 100. the orbiter was to serve as a communications relay for the Mars Polar Lander mission.000+ students. It was determined that spacecraft software used certain data in English units that should have been in metric units. In early 2000. futures exchange. known and unknown dependencies among • • Page 52 of 60 . or may understand and request them anyway . which was shut down for most of a week as a result of the outages. and didn't work.S. If there are many minor changes or any major changes. etc. the district's CIO was fired.Aditi Technologies Basic Concepts of Software Testing • • • • an online mortgage processing system that did not meet specifications. This is illustrated by an old parable: In ancient China there was a family of healers.redesign.the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development.000 customers. which failed for unknown reasons in December 1999. Among other tasks.programmers. Software complexity . telecommunications company that resulted in no charges for long distance calls for a month for 400. 5.
and the complexity of coordinating changes may result in errors. it should be hard to read'). complete. and there's job security if nobody else can understand it ('if it was hard to write. often requiring a lot of guesswork. resulting in added bugs. with a goal of clear.scheduling of software projects is difficult at best. Also see information about 'agile' approaches such as XP. A lot will depend on team leads or managers. the result is bugs. depending on the type of customers and projects. testable requirement • • • Page 53 of 60 .it's tough to maintain and modify code that is badly written or poorly documented. Software development tools . management must understand the resulting risks. often introduce their own bugs or are poorly documented. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand. etc. The most value for effort will often be in (a) requirements management processes. until I take a close look at it' o 'we can't figure out what that old spaghetti code did in the first place' If there are too many unrealistic 'no problem's'. How can new Software QA processes be introduced in an existing organization? • A lot depends on the size of the organization and the risks involved. mistakes will be made.visual tools.people prefer to say things like: o 'no problem' o 'piece of cake' o 'I can whip that out in a few hours' o 'it should be easy to update that old code' instead of: o 'that adds a lot of complexity and we could end up making a lot of mistakes' o 'we have no idea if we can do that. In fact. Poorly documented code .Aditi Technologies Basic Concepts of Software Testing • • • • • • parts of the project are likely to interact and cause problems. maintainable code. Enthusiasm of engineering staff may be affected. also in Part 2 of the FAQ. class libraries. managers. step-at-a-time process. Time pressures . it's usually the opposite: they get points mostly for quickly turning out code. In this case. we'll wing it' o 'I can't estimate how long it will take. continuously modified requirements may be a fact of life.see 'What can be done if requirements are changing continuously?' in Part 2 of the FAQ. and ensuring adequate communications among customers. In many organizations management provides no incentive for programmers to document their code or write clear. feedback to developers. When deadlines loom and the crunch comes. the result is bugs. understandable. egos . Where the risk is lower. 6. For small groups or projects. and testers. scripting tools. serious management buy-in is required and a formalized QA process is necessary. management and organizational buy-in and QA implementation may be a slower. developers. In some fast-changing business environments. and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control . a more ad-hoc process may be appropriate. compilers. For large organizations with high-risk (in terms of lives or property) projects.
• • Page 54 of 60 .the most 'micro' scale of testing. Unit testing . not to fix anything. 8. Tests are based on coverage of code statements. Not always easily done unless the application has a well-designed architecture with tight code. 10. requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed. plans. requirements. (b) design inspections and code inspections. The result of the inspection meeting should be a written report. and inspection meetings. What is verification and validation? • Verification typically involves reviews and meetings to evaluate documents. typically with 3-8 people including a moderator. Validation typically involves actual testing and takes place after verifications are completed. or that test drivers be developed as needed. Tests are based on requirements and functionality. The subject of the inspection is typically a document such as a requirements spec or a test plan. individual applications. and (c) postmortems/retrospectives.testing of combined parts of an application to determine if they function together correctly. walkthroughs. may require developing test driver modules or test harnesses. code. 7. What is a 'walkthrough'? • A 'walkthrough' is an informal meeting for evaluation or informational purposes. Typically done by the programmer and not by testers.based on knowledge of the internal logic of an application's code. What kinds of testing should be considered? • • • Black box testing . as it requires detailed knowledge of the internal program design and code. most problems will be found during this preparation. branches. 9. The term 'IV & V' refers to Independent Verification and Validation. Integration testing . done by programmers or by testers. issues lists. and a recorder to take notes.not based on any knowledge of internal design or code. and the purpose is to find problems and see what's missing. and specifications. This can be done with checklists.Aditi Technologies Basic Concepts of Software Testing specifications embodied in requirements or design documentation. The 'parts' can be code modules. Attendees should prepare for this type of meeting by reading thru the document. Incremental integration testing . What's an 'inspection'? • An inspection is more formalized than a 'walkthrough'. to test particular functions or code modules. paths. or in 'agile'-type environments extensive continuous coordination with end-users. White box testing .continuous testing of an application as new functionality is added. reader. Little or no preparation is usually required. client and server applications on a network. conditions.
Load testing . It can be difficult to determine how much re-testing is needed.similar to system testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.testing of full. input of large numerical values. For example.term often used interchangeably with 'load' and 'performance' testing. Compatability testing . surveys.testing an application under heavy loads. hardware failures.testing how well software performs in a particular hardware/software/operating system/network/etc. Recovery testing . Functional testing .testing how well the system protects against unauthorized internal or external access. Performance testing . or corrupting databases. if the new software is crashing systems every 5 minutes. covers all combined parts of a system.re-testing after fixes or modifications of the software or its environment. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing. large complex queries to a database system. such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails. this type of testing should be done by testers. End-to-end testing . Also used to describe such tests as system functional testing while under unusually heavy loads.typically used interchangeably with 'recovery testing' Security testing . may require sophisticated testing techniques. Page 55 of 60 .testing how well a system recovers from crashes. Acceptance testing . or based on use by end-users/customers over some limited period of time.testing for 'user-friendliness'. Clearly this is subjective.) System testing . or interacting with other hardware. Automated testing tools can be especially useful for this type of testing. Regression testing . video recording of user sessions. Stress testing . Sanity testing or smoke testing . environment. This type of testing is especially relevant to client/server and distributed systems.term often used interchangeably with 'stress' and 'load' testing. etc. partial. using network communications.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • • • • • • • • etc. and other techniques can be used. etc.black-box type testing geared to functional requirements of an application. Usability testing . such as interacting with a database. especially near the end of the development cycle.black-box type testing that is based on overall requirements specifications. the 'macro' end of the test scale. User interviews. or systems if appropriate. Failover testing . willful damage. the software may not be in a 'sane' enough condition to warrant further testing in its current state. Programmers and testers are usually not appropriate as usability testers. heavy repetition of certain actions or inputs. or upgrade install/uninstall processes. applications. and will depend on the targeted end-user or customer.typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. Install/uninstall testing . or other catastrophic problems. involves testing of a complete application environment in a situation that mimics real-world use. bogging down systems to a crawl.final testing based on specifications of the end-user or customer.
plan for adequate time for testing and bug-fixing. detailed. Communication . minor design changes may still be made as a result of such testing. Context-driven testing .start testing early on.e-mail. 'Early' testing ideally includes unit testing by developers and built-in testing and diagnostic capabilities. insure that information/documentation is available and upto-date . informal software test that is not based on formal test plans or test cases. work closely with customers/end-users to manage expectations.. For example. changes.testing of an application when development is nearing completion. re-testing.similar to exploratory testing.often taken to mean a creative. re-test after fixes or changes. testers may be learning the software as they test it. design.preferably electronic. bug fixing. groupware. not paper.testing when development and testing are essentially completed and final bugs and problems need to be found before final release.require walkthroughs and inspections when appropriate. Beta testing . Alpha testing . If possible. and be prepared to explain consequences. Realistic schedules . make extensive use of group communication tools . culture. not by programmers or testers. and intended use of software. but often taken to mean that the testers have significant understanding of the software before testing it. testing. Comparison testing . intranet capabilities. What are 5 common problems in the software development process? • Solid requirements . If changes are necessary. Proper implementation requires large computational resources. Adequate testing . they should be adequately reflected in related schedule changes. and documentation. User acceptance testing . This will provide them a higher comfort level with their requirements decisions and minimize excessive changes later on.clear. continuous coordination with customers/end-users is necessary. testable requirements that are agreed to by all players.comparing software weaknesses and strengths to competing products.determining if software is satisfactory to an enduser or customer. Stick to initial requirements as much as possible . • • • • Page 56 of 60 . Typically done by end-users or others. attainable. use protoypes if possible to clarify customers' expectations. by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected.allow adequate time for planning.a method for determining if a set of test data or test cases is useful. promote teamwork and cooperation.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • Exploratory testing . networked bug-tracking tools and change management tools.testing driven by an understanding of the environment. Mutation testing . 11.be prepared to defend against excessive changes and additions once development has begun. complete. not by programmers or testers. etc. Typically done by end-users or others. Ad-hoc testing . In 'agile'-type environments. cohesive. Use prototypes to help nail down requirements. the testing approach for life-critical medical equipment software would be completely different than that for a low-cost computer game. personnel should be able to complete the project without burning out.
etc. Some organizations have coding 'standards' that all developers are supposed to adhere to. be consistent in naming conventions. Each type of 'customer' will have their own slant on 'quality' .use both upper and lower case. but everyone has different ideas about what's best.Aditi Technologies Basic Concepts of Software Testing 12. Use descriptive function and method names . One code statement per line. A wideangle view of the 'customers' of a software development project might include end-users. What is software 'quality'? • Quality software is reasonably bug-free. use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line). is bug free. What is 'good code'? • 'Good code' is code that works. Function descriptions should be clearly spelled out in comments preceding a function's code. the development organization's. customer contract officers. etc. can be used to check for problems and enforce standards. customer management. use of brackets. less than 100 lines of code is good. delivered on time and within budget. However.vertically and horizontally. customer acceptance testers. such as McCabe Complexity metrics. and is readable and maintainable. magazine columnists. indentations. For C and C++ coding. 'buddy checks' code analysis tools. avoid abbreviations.the accounting department might define quality in terms of profits while an end-user might define quality as user-friendly and bug-free. stockholders. less than 50 lines is preferable. 'Peer reviews'. Management/accountants/testers/salespeople. Use whitespace generously .use both upper and lower case. avoid abbreviations. here are some typical ideas to consider in setting rules/standards. and is maintainable. Use descriptive variable names . No matter how small. There are also various theories and metrics. err on the side of too many rather than too few comments. It will depend on who the 'customer' is and their overall influence in the scheme of things. • 13. meets requirements and/or expectations. etc. use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line).) In adding comments. Coding style should be consistent throught a program (eg. be consistent in naming conventions. Function and method sizes should be minimized. a common rule of thumb is that there should be at least as many lines of comments (including header blocks) as lines of code. naming conventions. or what is too many or too few rules. quality is obviously a subjective term. these may or may not apply to a particular situation: Minimize or eliminate use of global variables. an application should include documentaion of the overall program function and flow (even a few paragraphs is better than • • • • • • • • • • • • Page 57 of 60 . future software maintenance engineers. It should be kept in mind that excessive use of standards and rules can stifle productivity and creativity. Each line of code should contain 70 characters max. Organize code for readability.
S. Defense Department to help improve software development processes. make liberal use of exception handlers. It's a model of 5 levels of process 'maturity' that determine effectiveness in delivering quality software. now called the CMMI ('Capability Maturity Model Integration'). and maintainable. some common rules-of-thumb include: The program should act in a way that least surprises the user It should always be evident to the user what can be done next and how to exit The program shouldn't let the users do something stupid without warning them. Good functional design is indicated by an application whose functionality can be traced back to customer and end-user requirements. periodic panics. For C++.characterized by chaos. Level 1 . What is SEI? CMM? CMMI? ISO? IEEE? ANSI? Will it help? • • SEI = 'Software Engineering Institute' at Carnegie-Mellon University. Defense Department contractors. less than 50 lines of code per method is preferable. it's often a good idea to assume that the end user will have little computer knowledge and may not read a user manual or even the on-line help. but often refers to 'functional design' or 'internal design'. and minimize use of operator overloading (note that the Java programming language eliminates multiple inheritance and operator overloading. For C++. or if possible a separate flow chart and detailed program documentation. keep class methods small. avoid too many levels of inheritance in class heirarchies (relative to the size and complexity of the application). Level 3 . CMM = 'Capability Maturity Model'. a Software Engineering Process • • • Page 58 of 60 . and if reasonably applied can be helpful. and works correctly when implemented. Good internal design is indicated by software code whose overall structure is clear. initiated by the U. It is geared to large organizations such as large U.) For C++. successes may not be repeatable.S. understandable. What is 'good design'? • 'Design' could refer to many things. and configuration management processes are in place. easily modifiable. and heroic efforts required by individuals to successfully complete projects. Level 2 . successful practices can be repeated.For programs that have a user interface.Aditi Technologies Basic Concepts of Software Testing • • • • nothing).standard software development and maintenance processes are integrated throughout an organization. Few if any processes in place. is robust with sufficient error-handling and status logging capability. developed by the SEI. many of the QA processes involved are appropriate to any organization. realistic planning. to minimize complexity and increase maintainability. Organizations can receive CMMI ratings by undergoing assessments by qualified auditors. However. Minimize use of multiple inheritance. requirements management.software project tracking. 14. • • • 15. Make extensive use of error handling procedures and status and error logging.
the focus is on continuous process improvement. 23% at 3.it indicates only that documented processes are followed. Bootstrap. installation. the primary industrial standards body in the U. Also see http://www. test planning.S. publishes some software-related standards in conjunction with the IEEE and ASQ (American Society for Quality). the standards can be purchased via the ASQ web site at http://e-standards. It includes aspects such as initial concept. Of those. What is the 'software life cycle'? • The life cycle begins when an application is first conceived and ends when it is no longer in use. (For ratings during the period 1992-96. coding. (b)Q90002000 . It covers documentation. For those rated at Level 1. and others. and certification is typically good for about 3 years. the most problematical key process area was in Software Quality Assurance. 32% of organizations were U. and training programs are used to ensure understanding and compliance. production. 6% at 4.) The median size of organizations was 100 software engineering/maintenance personnel. document preparation. 39% at 2.iso. not just software. Project performance is predictable. 13% at 3.S. and other processes. Note that ISO certification does not necessarily indicate quality products .among other things. and quality is consistently high. In the U. development. Perspective on CMM ratings: During 1997-2001.Quality Management Systems: Fundamentals and Vocabulary. internal design. testing. 27% were rated at Level 1.. and 0. maintenance. design. Level 4 . after which a complete reassessment is required. documentation planning. Level 5 .S. and it applies to many kinds of production and manufacturing organizations. 1018 organizations were assessed. 16. 62% were at Level 1. integration. updates. Will automated testing tools make testing easier? Page 59 of 60 . 23% at 2. a third-party auditor assesses an organization. Trillium. ITIL. (c)Q9004-2000 . ANSI = 'American National Standards Institute'. 'IEEE Standard for Software Quality Assurance Plans' (IEEE/ANSI Standard 730). requirements analysis. processes. functional design. To be ISO 9001 certified.asq. and 5% at 5. phase-out. Other software development/IT management process assessment methods besides CMMI and ISO 9000 include SPICE.The ISO 9001:2000 standard (which replaces the previous standard of 1994) concerns quality systems that are assessed by outside auditors. TickIT. servicing. retesting. 2% at 4.4% at 5. MOF. and CobiT. ISO = 'International Organization for Standardization' .ch/ for the latest information. 17. and products. testing. The full set of standards consists of: (a)Q9001-2000 . creates standards such as 'IEEE Standard for Software Test Documentation' (IEEE/ANSI Standard 829). 'IEEE Standard of Software Unit Testing (IEEE/ANSI Standard 1008). The impact of new processes and technologies can be predicted and effectively implemented when required.org/ IEEE = 'Institute of Electrical and Electronics Engineers' .Quality Management Systems: Requirements. and other aspects. federal contractors or agencies.Quality Management Systems: Guidelines for Performance Improvements.Aditi Technologies Basic Concepts of Software Testing • • • • • • • Group is is in place to oversee software processes.metrics are used to track productivity.
Coverage analyzers .such as via a spreadsheet . data.such as bounds-checkers and leak detectors. and comparing the logging results to check effects of the changes. or some underlying code in the application is changed. development. the time needed to learn and implement them may not be worth it. The data and actions can be more easily maintained . This approach can enable more efficient control. dialog box choices. If new buttons are added. in an application GUI and have them 'recorded' and the results logged by a tool. or on-going long-term projects they can be valuable. etc. bug reporting.vyomworld. a tester could click through all combinations of menu choices. in which the test drivers are separated from the data and/or actions utilized in testing (an 'action' would be something like 'enter a value in a text box'). The test drivers 'read' the data/action information to perform specified tests.com/testing/. buttons. Load/performance test tools . condition coverage. etc. etc. The same has then been reviewed and restructured for better reading and understanding. the primary being: http://www. Additionally.for test case management. a web site's interactions are secure. The problem with such tools is that if there are continual changes to the system being tested. and configuration management. client-side and server-side programs work. adherence to standards.monitor code complexity.to check that links are valid. A common type of automated tool is the 'record/playback' type. Note that there are record/playback tools for text-based interfaces also.) can be a difficult task. Page 60 of 60 .since they are separate from the test drivers. Another common type of approach for automation of functional testing is 'data-driven' or 'keyword-driven' automated testing. and may be oriented to code statement coverage. All contents of this document have been collated from various resources. etc. For example. path coverage.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • • • Possibly For small projects.these tools check which parts of the code have been exercised by a test. The 'recording' is typically in the form of text based on a scripting language that is interpretable by the testing tool. Memory analyzers . Other automated tools can include: Code analyzers . Web test tools . Test drivers can be in the form of automated test tools or custom-written testing software. logs. Other tools . the 'recordings' may have to be changed so much that it becomes very time-consuming to continuously update the scripts. and maintenance of automated tests/test cases.for testing client/server and web applications under various load levels. This document is for internal purposes only. documentation. interpretation and analysis of results (screens. HTML code usage is correct. etc. and for all types of platforms. documentation management. For larger projects. the application might then be retested by just 'playing back' the 'recorded' actions.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.