You are on page 1of 12

TESTING USER MANUALS

INTRODUCTION
Testing is the process of testing software for defects. It requires a tester to play the role of an end user, and use most of all features of the application to ensure correct behavior. To ensure completeness of testing, the tester often follows a written test plan that leads them through a set of important test cases.

OVERVIEW
A key step in the process of software engineering is testing the software for correct behavior prior to release to end users. For small scale engineering efforts (including prototypes), exploratory testing may be sufficient. With this informal approach, the tester does not follow any rigorous testing procedure, but rather explores the user interface of the application using as many of its features as possible, using information gained in prior tests to intuitively derive additional tests. The success of exploratory manual testing relies heavily on the domain expertise of the tester, because a lack of knowledge will lead to incompleteness in testing. One of the key advantages of an informal approach is to gain an intuitive insight to how it feels to use the application. Large scale engineering projects that rely on manual software testing follow a more rigorous methodology in order to maximize the number of defects that can be found. A systematic approach focuses on predetermined test cases and generally involves the following steps 1. Choose a high level test plan where a general methodology is chosen, and resources such as people, computers, and software licenses are identified and acquired. 2. Write detailed test cases, identifying clear and concise steps to be taken by the tester, with expected outcomes. 3. Assign the test cases to testers, who manually follow the steps and record the results. 4. Author a test report, detailing the findings of the testers. The report is used by managers to determine whether the software can be released, and if not, it is used by engineers to identify and correct the problems. A rigorous test case based approach is often traditional for large software engineering projects that follow a Waterfall model. However, at least one recent study did not show a dramatic difference in defect detection efficiency between exploratory testing and test case based testing.

1|Page

STAGES
There are several stages. They are: Unit Testing This initial stage in testing normally carried out by the developer who wrote the code and sometimes by a peer using the white box testing technique. Integration Testing This stage is carried out in two modes, as a complete package or as a increment to the earlier package. Most of the time black box testing technique is used. However, sometimes a combination of Black and White box testing is also used in this stage. System Testing In this stage the software is tested from all possible dimensions for all intended purposes and platforms. In this stage Black box testing technique is normally used. User Acceptance Testing This testing stage carried out in order to get customer sign-off of finished product. A 'pass' in this stage also ensures that the customer has accepted the software and is ready for their use.

DOCUMENTATION EFFECTIVENESS
Documentation is fully effective with following benefits: Enhances salability: Documentation quality is often a selling feature. It helps retail salespeople explain and demonstrate the product. It also plays a major role in many software reviews. Improves installability: Customers have to install software products on their computers. They may have to copy the software onto their systems, then customize it or load a special database that reflects their needs and equipment. Installation software is written last. Developers take it less seriously than other parts of the product because customers will install the product only once or twice. These routines get the least amount of testing and development support. However, your customer's first experience with the product is installation. Difficulties will lead to refunds and expensive technical support demands. Clear, correct installation instructions are among the most important parts of the documentation. Installation of some types of products (such as telephone systems) is so complex that customers hire an installer. As a rule, installers work with many different products. Don't expect them to be expert users or installers of your product. Expect them to have to look things up in the manual. The harder it is for them to find information, the more they have to charge the customer. Some vendors refuse to carry products that cost too much to install. Installation software also needs re-installation instructions. The manual must explain how to change options and how to upgrade from a previous version. Reduces liability: Your Company makes a false claim about the program's capabilities when the manual says that the program does something that it doesn't. Your company misleads the reader 2|Page

when the manual says to do something in a way that doesn't work. Incorrect instructions cost the reader unnecessary time, effort, and mental anguish. Improves usability: A well-documented product is easier to use. Schneider man (1987) reviews evidence that better documentation leads to faster learning, fewer errors, and better throughput (i.e., people get more work done). Lowers customer support costs: When one of your customers can't figure out how to use the product, she'll call for help (or for a refund). Telephone-based technical support is an expensive service. A good manual prevents many unnecessary support calls. Improves reliability: Unclear and inaccurate documentation makes the product less reliable because the people who use it make errors. Excellent documentation helps reduce the number of user errors, even (or especially) when the program's design is awkward. Increases maintainability: Much time and money is spent tracking down problems that turn out to be user errors. Many product changes merely repackage the same capabilities. They're made in the hope that people will finally understand how to use these capabilities and quit complaining that they aren't there or don't work. Better manuals reduce these problems substantially; poor documentation contributes to them

OBJECTIVES OF THE DOCUMENTATION TESTER


Reviewers of documentation are concerned with improving its accuracy, completeness, clarity, ease of use, and the degree to which it captures the spirit of the design. The documentation will have problems in all of these areas.Plan to test the manual, online help, and any other documentation many times. As a tester working with the manual (or help), you are responsible for checking the technical accuracy of every word. There is no substitute for a thorough, thoughtful comparison between the claims and implications in the manual and the actual behavior of the running program. This is what you do. (As a side benefit, you find lots of bugs this way too). Be on the lookout for confusions in the writing. Many of them stem from confusions and complexities inherent in the product's design. The writers must describe the product as it is. You can help them substantially by pressing for changes that make the program more documentable (and more usable). Look out, too, for missing features. Writers work from specifications, notes, and rumors. Developers try to keep writers up to date, but sometimes they forget to mention recently added features. You often discover changes much sooner than the writermake sure these get into the manual. And don't assume that they got into the help text just because they got into the manual. These are probably being written by different people; updating information can easily get lost. Realize that you are a reviewer, not the writer. Most testers who think they know more about writing than the writer are wrong. You have no more right to demand changes in the manual than in the program. It is your responsibility to find and identify problems, but after giving them a fair hearing, writer and programmer alike can choose not to follow your suggestions. 3|Page

TESTING DOCUMENTATION CONTRIBUTES TO SOFTWARE RELIABILITY


You 'II find more bugs than you expect Surprisingly many bugs show up when a competent tester thoroughly checks the manual (or help) against the program. The writer looks at the program from a different angle than the programmer and the tester, so the manual will reveal different problems than the ones programmers and testers look for. We've seen this happen on so many projects that we now take it for granted that tests of the manual will reveal many serious errors that other testing has not yet turned up. Documentation testing doesn't always reveal significant new problems. Testers who don't do a thorough job don't find many problems. A full test of the manual takes about an hour per three to five pages. Testers who speed through the material more quickly find much less in the book and in the program. We make a point of monitoring for this problem and retraining or reassigning staff to deal with it. It's an important source of real world combination test cases. You can't hope to test all the combinations of features or other options in the product; there are just too many. But you can test every combination that the manual describes as interesting or useful. Any time the manual even hints that two aspects of the program work well together, test them together. Bug reports arising out of documentation testing are particularly credible. The manual is your company's instruction to the customer on how to use the product. It's hard to dismiss a bug as "esoteric" when you report that the program failed while you were following the manual's instructions or suggestions, or were checking one of its statements of fact. These are mainstream tests. These are things many people will do. These are errors that magazine reviewers can publish without fear of correction. These bugs are hard to defereither the manual will change or the program will change. We've often seen previously deferred bugs reconsidered and fixed when they showed up again during testing of the manual. In your main test of the manual, you should sit with it at the computer and: Use the program exactly as the manual says. Enter every keystroke in every example. Customers make mistakes when they try to follow instructions. Feel free to make mistakes too. How does the computer respond? Bad error handling in the program will look worse when you show that it happens in response to an obvious, common mistake that several people will make when they try to follow the manual's instructions. Try every suggestion, even if the suggestions aren't fully spelled out, step by step. Do what a reasonable customer would do who was trying to follow the suggestion. Check every statement of fact and every obvious inference from the stated facts, instructions, and suggestions. The manual is the product's final specification, and the customer's first place to check whether the program is working correctly.

4|Page

It also pays to retest the documentation when you add a tester to the project. This keeps the manual current while the software is changing, and it educates new testers about the program. Consider assigning every new tester to work through the most recent draft of the manual as his first task.

BECOME THE TECHNICAL EDITOR


If possible, the Testing Group should assign one person to the manual as technical editor. He might play other roles too, but in this role he is the primary technical reviewer, even if many other people also review the manual. It is very common when two or more people check a product for none of them to take ownership of the task. Rather than improving the thoroughness of the review by adding another tester, thoroughness declines because everyone expects the other person to do the job. The technical editor should be encouraged to feel ownership of the technical accuracy of the book.

WORKING MANUALLY THROUGH ITS DEVELOPMENT STAGES


The manual is developed in stages. The four major stages are: Conceptualization and initial design: The writer makes decisions about the scope, target audience, degree of coverage, and general organization of the manual. Preparation: The manual is written, reviewed, rewritten, etc. The manual is in the preparation stage until its content is in final form. Production: The manual is laid out for publication. This might involve typeset ting or making revisions to make the manual look as good as possible when printed by a laser printer or a daisy wheel. Typefaces are chosen, page style (margins, etc.) is designed, final artwork is prepared, and so forth. Publication: The manual is printed or copied and bound, ready for circulation.

Testing is concentrated in the preparation stage, with some spillover into production. You review the manual's content, not the layout, unless you have expertise in layout. You probably won't be involved in the initial design of the manual. You will rarely be involved in publication: the writer checks that the printed manual contains all the pages, none upside down, and so forth. Some comments are more welcome at some times than others. Sometimes the writer wants to work on accuracy, sometimes on style, sometimes on organization. The following sections look at the value and appropriateness of different types of comments across various preparation and production stages. These notes describe the needs of the "typical" writer; there are many individual differences. Talk with the person you work with about her needs and schedule. 5|Page

THE FIRST DRAFT


You will rarely see the first draft of the manual. Excellent writers often write a horrid first draft. It might be badly written and full of spelling mistakes and factual errors. It might be badly and inconsistently organizedwriters often experiment in this draft. Think of it as a set of personal notes. If you are freshly assigned to a project, you don't know how the software should work and can't find any specifications, or if you are desperate, you might beg the writer for any documentation she has, even if it is first draft material. If she gives it to you, xe_alize that it is for your use only. It is not for review, circulation, or criticism. It is not ready. Breach this trust and you will embarrass the writer and ensure that you never get first draft material again. The writer will find some comments helpful. Correct factual errors. If you think the writer doesn't understand something, volunteer to share your understanding. Treat this as a shared learning experience, not as a set of comments and criticisms. Finally, make no comments about the manuscript's style, structure, or organization unless you are explicitly asked for them. Even then, make them cautiously.

THE SECOND DRAFT


It goes to the programmers, managers, and you. It is not ready for review by the user community, except for users who have been explicitly assigned to the development team. Do the following:
Make your structural comments early: If you don't like the order of chapters, or think that material should be combined into one chapter or split into two, say so early. You can wait a little longer (not much) before saying that the order of topics within a chapter is wrong. The longer you wait, the harder it is to change the book's structure. Some documentation groups review a document plan before they write the first line of text. A good document plan names each chapter and each section within each chapter. It gives an estimated page count for each section, and breaks sections longer than 10 pages into subsections. You might be invited to a meeting to review this document plan. This is the best time to make structural comments. If you think that some aspect of the program's design should be difficult to explain, but the document plan doesn't assign many pages to it, ask why not. The writer probably doesn't realize what she's in for. Explain the complexity in a factual, non-judgmental, non-sarcastic way. On hearing your explanation, the project manager may revise the design to eliminate excess options or to make it more consistent with the rest of the program. Do a general review. Read the manual with an eye to improving its accuracy, clarity, usability, and completeness. Don't be afraid to make comments like "I had to read this three times to understand it." Even if you can't say why it was difficult, the writer wants to know that a careful reader found a section, paragraph, or sentence difficult.

6|Page

Look for areas that need discussion. Some features have not yet been described in the manual. The writer may not know that a new feature has finally made it into the program and is ready to be examined, and described, carefully. Look for violations of the spirit of the design. The writer might miss a simple conceptual relationship between features, and describe each independently. Carefully devised large-scale consistencies are lost. The writer might imply disapproval of a program's restrictions. While some restrictions are arbitrary, often to simplify the user interface. The writer might approve if she understood the reasoning. The manual might suggest inefficient strategies for some tasks -they work, but a user who understood the program well would do things differently. In cases like these, the writer has missed something fundamental. Once she gains understanding, she may make significant changes, redoing not only this material but also discussions of related topics. These might he big revisions. The writer must understand the need for them as soon as possible. Look for things that mislead. Some examples and feature descriptions aren't incorrect, but a reasonable reader might generalize incorrectly from them. She might expect the program to be more capable than it is. She might believe she can do something under more general circumstances than she can. Or she might imagine restrictions that don't exist on the use of a feature. It is particularly important to flag misleading material early because the writer might believe those incorrect generalizations. She might make significant changes if she understood the product differently.

Check the error message: The writer will probably include a list of error messages in an appendix, with notes on how the reader probably got to this message and what to do about it. If you've been keeping a list of every way you've found to get each message, this is invaluable for the writer. The writer will base her explanations on your list and on information from the project manager and the technical support staff (who rely on these explanations to help keep customers from flooding the company with phone calls). It pays to test every message as it's explained in the book you'll find more bugs. After you've tested a message, give the writer your additional notes about other message meanings, ways to get the message, or things the customer must do or avoid doing as soon as he gets the message. Look for confusions that reflect on the program. If the writer can't describe some aspect of the program in a clear and consistent way, evaluate the program before condemning the manual. If the program has many inconsistent options, it's a mess and so will be the documentation. Suggest improvements to the writer (if you can). Don't spend hours rewriting sections of the manual, but if you can do it quickly, do provide a description that you think is clear and acceptable. Also, write Problem Reports if you think confusion in the program is the cause of confusion in the manual. It's easy and common to condemn writers for documentation that accurately describes an incompetent design. We find it more profitable to start with the assumption that the writer is competent and that bad text is telling us something about the program.

7|Page

THE BETA TEST DRAFT


This draft or a revision addressing comments to this draft, will be the last one you'll see before production. (In companies that don't rely heavily on beta tests, the final circulating draft is the user interface freeze draft, circulated after the software's design has frozen.) Beta testers don't work for your company. They use the product in the same ways they would have had they bought it in finished form. They report their difficulties with the product, their suggestions for improvement, and any bugs they find. You should review their reports about the software and about the documentation. Up to this point the marketers, programmers, writers, and you have been making assumptions about how people will react to the product and about what they'll understand. Some of those assumptions are wrong. Some seemingly obvious aspects of the program may be incomprehensible to beta testers. Many changes to the manual come from user testing. Users often complain if the documentation is not task-oriented. A task-oriented manual anticipates what users want to do with the product and explains how to do each task. It describes features in the context of using them to get specific tasks done. In contrast, a feature-oriented manual describes features individually, maybe in alphabetical order. Each section includes everything you ever wanted to know about one feature. Brockmann (1990) notes that task-oriented manuals are much longer, but review some further evidence that they are more effective. If the product is so widely useful that people could do thousands of different types of tasks with it, the writer could never finish a task-oriented manual. As a compromise, writers often write a task-oriented tutorial that covers the most popular tasks. Beta test comments may convince the writer to improve the task orientation with more examples, more illustrations, a different index, or a different organization. Customers will raise many other issues about the documentation. As always, ask yourself whether their confusion is really due to poor documentation. We repeat this point because it's so often missed. The manual is often blamed for the faults of a poorly designed program, but no accurate description of a fundamentally confused program can be clear. Complaints about documentation should often lead you to file Problem Reports about the program's user interface.

8|Page

PRODUCTION
Your main concern during production is that the document stays accurate. Someone in the documentation group, the writer or an editor or editorial assistant, will do the main proofreading of the laid out or typeset manuscript. You too can note spelling mistakes, misaligned headers, etc., and these notes will be welcome, but if that's all you can provide at this point, you're wasting your time. If the company wants to release the product as soon as the software is complete and tested, documentation production must start 8 to 14 weeks before the program is finished The program will change over those many weeks. Some parts of the manual will no longer be correct. Further, some bugs that everyone expected to be fixed will not be. Sections of the manual that assumed that a given bug would be fixed, in good faith and on good authority, have to be revised. Not all desirable changes can be made during production. The writer will (should) change as little as she can get away with. You can get her to make more changes, and help keep the cost of the changes down, by designing the changes to match production constraints. As soon as a manual enters production, it stops being an organized collection of words. It is now a bunch of pages. There happen to be words and pictures on the pages, but each page is separate from all others. Each was carefully laid out; each will be photographed on its own. The writer will not make changes that affect more than one page unless they are essential. At the other extreme, it is easy to make a change that affects only one line, without moving words down to the next line or needing words from the line above. If you can keep a change within a line, a paragraph, or a page, you have some hope of convincing the writer to make it. It is your responsibility to convince her that the change will stay within those limits. Be prepared to provide a suggested wording, and to show how the words fit on the lines. The wording must be stylistically acceptable to the writer. This degree of editing is beyond the formal scope of your job. You can be asked to stop doing it. You don't have to do it and you shouldn't try unless you can do it well without taking too long. If you don't do it, you should send the writer a memo describing your problems with the manual. She will save it and incorporate your comments in revisions made for the next printing of the manual, or in the manual for the next version of the product. Also, if she thinks that one of the problems you raise is critical, she will work on the wording and get it td fit. Another area to test during production is the index. The earlier you can get your hands on a draft index the better. You can improve the index's completeness by working with the draft manual and constantly trying to look things up in the index as you use the manual. Many words you expect in the index won't be there. Report them to the writer. You (or an editorial staff member) must also check the index when everything is finished, just before the book goes to the printer. The "final" index may miss entries from one chapter, or it may still be based on a previous version of one chapter. At a minimum, check at least two index entries in each five pages of the book.

9|Page

POST-PRODUCTION
Some companies don't print the manual until after the software is finished. In these cases, there are no post-production tasks. (The writer still has some tasks, such as checking bluelines, a first print run from the printer, but you probably won't have to check bluelines.) If your company does send the manual to print before the software goes to the duplicator, the writers probably have to write two further documents. One is a printed supplement that includes corrections, troubleshooting notes, and discussion of additional features. The typical supplement goes to print a few days before the disks go to the duplicator. Later-breaking information must go into a README file on the disk. Apart from checking the accuracy of material in the supplement and README, your most valuable contribution during this period is identifying troubleshooting tips and explaining them to the writer. Every deferred bug is a potential idea for a troubleshooting tip. If you (or the writer) can describe the bug in a positive tone, and tell the customer something usefiil, it's a good candidate for the troubleshooting section.

ONLINE HELP
Most of what we've said about the manual is equally true for help. Here are a few additional notes. Accuracy: You must check the accuracy of help at least as closely as the manual. Help text is generally not well done, not well tested, and not well respected by customers. A customer will probably quit using the help immediately if she finds factual errors. Good reading: The best book we've read (or seen) on online help is Horton (1990). Help is a combination of writing and programming: You have to check the accuracy of the text and the reliability of the code. Tf the programmers implement help using special system-provided tools (common in GUI environments), it will pay to read the system's instructions to the writer and to the programmer. You will find bugs that stem from the writer and programmer not understanding each other's job well enough to cooperate perfectly. Test hypertext links: If the program includes hypertext links (cross-references that will take you directly to other topics), you have to check each link. Suppose the writer crossreferences to "Keyboard layout" in two different places. In most systems, she could have the program jump to one help message if you select the first "Keyboard layout" and to a different message if you select the second "Keyboard layout." What you see doesn't necessary identify correctly where you will go. You have to check it in each case. Test the index: If the program includes an index or a list of topics, and lets you jump from the index to the topics, you must check each one.

10 | P a g e

More on the index: If the program includes an index, or hypertext links, you should note whether the index entries or the list of linked topics (per topic) are sensible. Some help topics never appear in the index or appear only under odd names. Customers will back away from the system if they can't quickly find information they need. Watch the style: Few customers take a leisurely read through help. They come to_help with a specific question, or specific task they have to do, or error state they have to attend to. Help readers are often nervous or distracted, and they are often impatient. Expect the help text to be much more concise than the manual. Its style should also be much simpler (grade 5 reading level is sometimes recommended). Good help text is also very task- or action-oriented. It must say something useful, which the customer can do right away. If you find anything confusing or drawn out in help, report it as a problem.

11 | P a g e

REFERENCE
I hereby declare that all the contents of this report is taken from the book Testing computer software2nd edition written by Jack L. Falk, Hung Quoc Nguyen. And the link en.wikipedia.org/wiki/ & http://naveengoogle.blogspot.in/.

12 | P a g e