Professional Documents
Culture Documents
Profession Manual Testing PDF
Profession Manual Testing PDF
Managing
manual
testing
Robot, I
by Bogdan Bereza-Jarociński
Jarociński envisages Others think there must always be a place Some of these limitations of automation
for manual test execution – for exactly the may diminish in the future: but few doubt
an approach which same reasons: that in the present at least some manual
reverses current execution is essential.
sometimes inconsistent execution,
common practice inadvertent or not, increases coverage So the variation in test execution and
and therefore potential to detect checking introduced by people is
defects which automated execution sometimes desirable, sometimes not.
would miss When it is not, how can we eliminate it? I
suggest that the answer is by defining the
sometimes people notice anomalies tests more explicitly. Much of the
which a tool has been configured, weakness of manual execution comes
wrongly, not to look for or to ignore from its association with manual test
because it was not foreseen preparation. Even when standards and
templates are used, test specifications
automated execution can validate have some room for interpretation. If that
software, but a person can evaluate it: could be eliminated, manual testers could
he/she adds business knowledge, still use them as a basis for useful
understanding, intuition, imagination variations in both the actions taken and
and empathy with users a tool cannot what they look for, but could be trusted far
emulate more to execute them correctly and not
miss any significant discrepancy when
the act of executing a test can lead a needed. All the advantages and almost
person to create additional valuable none of the disadvantages (the exceptions
tests being speed and use of human resources)
of manual testing would be realised.
NE
W
TE
ST
Tester’s S
iven the definition of such a language or
expectation
and other description system – which because
understanding
of its purpose would need to be both
Exploration
simple and small – it should be quite easy
Figure 1: routes from test descriptions to executable tests – and back to write a “translator” program that
generates it from the test specification
Could this be achieved by using a tool to Test description telling a person how language.
create consistent, unambiguous tests for to e ecute the test
people to execute? Manual test preparation usually involves to Ideally, it would be possible to define new
some degree the use of natural language. tests directly using the “execution
Test specification defining what the This creates a lot of problems: different test language” or system too. It would be
test is analysts have different description styles extremely useful in incident reporting and
To automate the creation of detailed test different organizations use different retesting when an execution-time variation
procedures, the test cases (pre-conditions, description guidelines and different testers of the procedure, or a new test created in
input, expected output and post-conditions) may understand a description differently. an ad-hoc or exploratory way, detects an
must be described unambiguously. Most anomaly, or when it is desired to add such
formal languages used to define test cases Using a formal language at this stage too a test, passed or not, to a regression suite.
are developed and used locally within an should eliminate the first two problems but Depending on the form the language or
organization or even for a specific project. would only change the third. The formal system takes, an extra interface or
Some tool vendors provide basic languages used to specify tests are “development kit” might be necessary to
frameworks for such languages, for designed for machine, not human, achieve this, and/or syntax checking tools
example HP's Business rocess Testing, readability. Executing tests expressed in could be used to verify and debug
which enables the creation of test cases as them manually would be difficult, “handcoded” tests.
blocks of words which can then be painstaking, error-prone work. The
manipulated graphically. They are not very keyword-driven approach, designed to Finally, the execution procedure could be
much like programming languages, but make it easier to create and maintain used also as input for automated
more like business modelling languages, automated tests, may be a partial solution, generation of tests for automated
so that business rather than technical but the person executing would need to execution, on the same principle as
people can learn to write test cases and either know, or make constant reference to, keyword-driven automation methods. Thus
test scenarios using them. the definitions of the keywords. Again this the same tests could be run manually or
would probably be excessively demanding automatically as most appropriate for the
Tailored languages can be made 100 work in most circumstances. current objectives. Doing both and
suitable for the purposes of an comparing the results, such as resulting
organization and project. On the other So a second language is needed: more change to back-end data etc, could be
hand, building, teaching and learning such abstract, easily readable but still formal interesting too: it may help to reveal some
languages is expensive, and they tend to enough to define detailed procedures with subtle and dangerous defects such as
hamper collaboration, so a standard no ambiguity. It may resemble natural timing issues that either manual or
language for this purpose would be language, express the actions and inputs automated execution alone cannot
desirable. Perhaps one could be based on
a meta-language such as BPML or UML, Bogdan Bere a arocinski is a testing consultant, speaker and trainer and a long time
or adapted from a language used to contributor to rofessional Tester. e is the proprietor of ict http victo.eu
describe test cases such as TTCN or
LabVIEW?
www.nl.capgemini.com
Managing manual testing
party, libraries can be used to encode and causing the script always to amend more
decode data during testing without fields.
needing access to proprietary or otherwise
unavailable source code. Now, suppose the form is designed to
change depending on the data: that is,
Object-oriented scripting it can have more, fewer or different
The class structure of the scripts enables fields depending on the
the tester to override any of the base customer type and/or history.
methods, introducing his or her own logic, With many tools, it is
conditions and validation. I used this necessary to determine every
recently when the application under test possible variant of the form –
required client-side timestamping and pre- which itself can be difficult
validation of data before every HTTP or impossible – and create
POST request. Coding this once, in the specific scripts to handle
custom virtual user, means it is each. With Forecast,
automatically implemented in all requests provided the fields they
sent by all scripts, including ones yet to be amend are still present,
created. These concepts are of course scripts always execute.
nothing new to OO programmers, but Values of all other fields,
many other performance testing tools try no matter what they are,
to hide the real code behind user are captured when the
interfaces or simplified procedural form is served and sent
languages that serve only to restrict what automatically when it is
you can do. submitted
Global editing
Heavily UI-based tools can be very
cumbersome, requiring user input for
every data item to be correlated or
modified, making for a great deal of error-
prone editing. Instead, Forecast has a
wizard to define script generation rules.
When a pattern, eg a header type, URL or
specific document content is detected, the
rule inserts code for correlation, checking,
extraction, header creation and so on.
Rather than editing the scripts, one edits
and extends the rules: the scripts are then
regenerated according to the new rules.
In their strivings for operational efficiency, quality and to satisfy growing government regulation, the number
of companies that test software professionally is growing.
The book provides a clearer and more effective manual for a well-oiled testing approach. This knowledge
allows you to arrange custom software testing and integrate it in any business environment.
The book ties in with the knowledge needed to gain the ISTQB Advanced Test Management Certificate in
Software Testing. ps_testware is an accredited ISTQB Foundation and Advanced training provider.
We can remember it
for you wholesale
by Marek Kucharski
Marek Kucharski is CEO Europe of Parasoft. For more information about Concerto see
http://parasoft.com/concerto
The things that delay testing that cause delay to testing and what might
be done to eliminate it.
and how to avoid them
I recently carried out questionnaire-based
research with testers in multiple industries,
Software testing is usually on the critical aiming to discover the causes of the
path. Most testers feel the burden of the wasted waiting time. This article discusses
anxiety and impatience of managers and the three cited most often, and suggests
stakeholders, who expect testing activity approaches to arguing for project change
to reflect it, and often express surprise to that might help to mitigate them. That
find this is not the case: when the storm should keep testers busy for more and
everywhere else is reaching its height, the unduly pressurized for less, of the time,
testers… wait. Why, when even small helping to bring about what everyone
delays can have severe impact on the wants – quality products delivered faster.
project timeline? Because while test
execution is given high priority, providing Unavailability of test environment
the things needed to do it is not. Not Test execution time is typically greatly
having those things at the right times (i) increased because the test environment is
delays the start of test execution; (ii) unavailable, unstable or unusable.
makes test execution take longer; and (iii)
makes testing less effective and This is a familiar but still common situation
dependable by requiring more for testers. I recently worked on a large
Why are we waiting? assumptions to be made. project with a one-week release cycle.
Derk-Jan de Grood Unfortunately “release” meant only
Good test managers try to emphasize the delivery of code. The deployment and
finds out advantages of early involvement and configuration required to enable
working in parallel with development, so meaningful test execution and results
that test execution can begin immediately checking took several days, reducing
and be done efficiently whenever work testing time to two or three days a week.
products are released. Unfortunately those
concepts are still not well understood – or Unstable and slow environments are also
are not taken seriously – by other leaders, commonly experienced. Both cause very
whose concern with project timescales significant delay and risk. In the first case,
tends to make them concentrate on the environment crashes before the test is
removing any potential cause of delay to complete so it must be repeated
development, forcing more of the testing unnecessarily. Casual testers – ie
work to take place later, actually causing stakeholders and developers – who
worse delays. “explore” software often do not understand
that “carrying on where you left off” is
To help managers to reduce time-to- usually impossible and always dangerous
market, we as testers need to get the in systematic testing. As well as wasting
measures required for better, more timely their time waiting for responses, sluggish
testing higher on their agenda. Trying to do interfaces affect testers' concentration and
this by making them understand testing lead to mistakes.
better has failed. We might achieve more
by focusing instead on what they do Discussions with project managers
understand, highlighting the project issues regarding test environments tend to be rare
because they are seen as a side issue, not Delay in fixing show-stopping bugs To create and justify a strategy that will not
contributing directly to the delivered Testing aims to find the most important fail because of waiting caused by unsafe
product. It needs to be made clear that if bugs first, but those bugs are often assumptions, ask management:
release and testing of code are on the showstoppers which cause testing to be
critical path then so is making the release suspended until they are fixed. But, how expressed quantitatively (eg on a scale
testable. The concept of a timeline event long will that take? It's hard to say. Bug of 1 to 10, where 1 is a blank sheet of
called “release to testing” – which occurs fixing is seldom a well-managed process. paper and 10 is the complete, perfect
only after (i) the developers have released It's important to ensure that management design), how detailed can we expect
not a build but a full installation and (ii) the realises and takes into account the fact documented requirements to be (i)
testers have verified it might help. Failing that while incidents are being reported, before the testing project begins; (ii) at
that, asking the following questions will help reproduced, discussed and resolved, management-defined milestones in the
to anticipate problems and taking action to testers will often be waiting. critical path of the development
make more of the answers positive will When discussing this problem with project?
reduce execution time. management, ask the following questions:
how accurate can we expect
are test environment considerations is the incident management documented requirements to be, as a
being included in project risk mechanism sufficient and are all who proportion of the final features of the
management? should using it correctly? accepted product, at those critical
milestones?
has the number of environments, and if an incident occurs and a tester is
instances of each, been established or unsure whether to raise it, what should is the information needed by the
estimated? he or she do? various project participants, including
testers, identified and agreed before
does the test plan include identification does the project plan allocate sufficient system definitions are documented?
and the project plan creation and time for incidents to be resolved
maintenance of test data? (investigated and, if necessary, fixed)? where detailed documentation of that
Do its estimates take into account information is not to be available, are
have the specifications of environments quality, complexity and commenting of information sharing activities to replace
yet to be created been documented and code and availability of the people it planned?
agreed? needed to resolve incidents?
Other reasons testing time is lost
have the stability and performance of Lack of sufficient information about the The research identified four other common
environments already created been system issues that force testers to wait. The full
assessed? Every tester understands the importance results and many more suggestions are
of getting system definitions included in a comprehensive checklist,
will there be any requirement to share (specifications, requirements, use cases, intended to offer a fresh approach to
environments with other activities or stories etc) as early as possible. If they are opening and maintaining productive
projects? late, test preparation is made difficult, dialogue. Its questions align with the
causing subsequent time-consuming concerns and areas of expertise of project
does the project plan include allocation change. If they must be chased, the time management, helping to eliminate time
of environments to testing and does the for test preparation is reduced, with the wasting from any testing effort. The
schedule show the associated same result. If they are inadequate, the checklist is available free at
dependencies? time taken to discuss incidents is http://www.smartest.nl/toolstemplates/
increased, impacting bug-fix time. procesverbetering
have technical support resources
required to provide and configure user Derk-Jan de Grood is a test manager at Valori (http://valori.nl) and author of
accounts and then to assist users of the TestGoal: Result-Driven Testing (Springer, ISBN 9783540788287). His new book in
environments? Dutch, Grip op IT: De Held Die Voor Mijn Nachtrust Zorgt (Academic Service, ISBN
9789012582599) will be published later this year. He speaks frequently at international
has configuration management to enable testing conferences, including about his passion for aligning IT and business
environments to be reproduced exactly,
and to track change and difference
between environments, been
implemented?
Like CMMI, TMMi defines maturity levels, Rather than being simply necessary to
process areas, improvement goals and detect defects, testing at this level is
practices. An organization that has not evaluation: everything that is done to
implemented TMMi is assumed to be at check the quality of all work products,
maturity level 1. Being at level 2, called throughout the software lifecycle. That
“Managed”, requires the practices most quality is understood quantitatively,
testers would consider basic and essential supporting the achievement of specified
to any test project: decision on approach, quality needs, attributes and metrics. Work
production of plans and application of products are evaluated against these
techniques. I call it “the project-oriented quantitative criteria and management is
level”. informed and driven by that evaluation
throughout the lifecycle. All of these
The goals and practices required by level practices are covered in the Product
3, “Defined”, invoke a test organization, Quality Evaluation process area.
professional testers (that is people whose
main role is testing and who are trained to The Advanced Peer Reviews process area
perform it) earlier and more strategic test is introduced and builds on the review
planning, non-functional testing and practices from level 3. Peer review
specific, difficult fields including US Social Figure 1: DFT in test and test data generation
Security number, UK National Insurance
number and credit card details. The data Ashwin Palaparthi (ashwin.p@applabs.com) is VP, innovation at AppLabs, which he
can also be very rich. DFT includes facilities rejoined recently when it acquired ValueMinds, the company he left to found three years
to calculate and insert values that: ago and which has created many innovative test tools including testersdesk.com
Equality Unconquered
by George Wilson
Revelation space
very short with many in tables, and element
Advanced Test Management types and subheadings are plentiful and
by Patrick Hendrickx and Chris Van Bael
diverse. It works brilliantly for study and
ps_testware, ISBN 9789090257273
reference, but less well for personal
Available from http://amazon.fr, soon from http://amazon.com
learning and understanding, because the
fragmentation makes linear reading heavy
In his foreword Alain Bultink says that syllabus requires, much is drawn from going: there is almost no narrative. So, this
ps_testware and ISTQB share the same other sources, but the whole adheres is a fine study aid, positively essential for
philosophy of testing. That is the great tightly to ISTQB’s prescriptions – exactly anyone taking ISTQB-AL-TM. It’s also a
strength of this book: the authors have what someone aiming to pass Advanced good textbook, although it’s better to dip
embraced completely the often mysterious Level Test Manager needs. Indeed, the into than read through, and would be even
ISTQB Advanced Level Syllabus and tripartite nature of the syllabus means that better if it contained fewer defects: typos
explained their interpretations of it more portions of the book can be used also by etc fall to the eye too readily, and those
clearly than anyone else has yet managed. those studying for Test Analyst and publishing books for testers should do
Even better, the explanation is practical: Technical Test Analyst. A subset of the more to convince readers that they are
the titles of many sections begin with “How content would also be good, in my opinion eating their own dog food. Whether it’s a
to...” and they really do demonstrate better than BCS’s official book, for sourcebook for a test manager depends on
actually doing the things needed to choose Foundation Level candidates. The large to what extent he or she agrees with
the right exam answer. Diagrams (not page count (685) is due to the use of ISTQB, but it contains much valuable,
including the silly photos at the start of “structured writing" which is granular, accessible information and is undeniably a
each chapter) are well executed. As the organized and formulaic. Paragraphs are worthy addition to the genre.
The Little TMMi: Objective-Driven Test a less formal way, with examples, may
Process Improvement have achieved the objective better. The last
by Erik van Veenendaal and Jan Jaap Cannegieter 20 pages are original: they describe
UTN, ISBN 9789490986032 Available from http://www.utn.nl assessments, then implementation using
IDEAL. These sections are much easier to
As contributors to and enthusiasts for model, but with the low-level detail read and are worthwhile, but again are
TMMi, the authors want it to be adopted by removed: over 56,000 words cut down to prescriptive rather than instructive. This
more test organizations. Their book aims to about 13,000. That obviously makes it book is a convenient way to learn what
promote that and is deliberately compact in easier to digest, but the very stiff and TMMi says we should do, but a practical,
order to make the model more accessible. general style remains. Explaining what it self-contained guide to tell us how is
In fact 50 or so pages are the text of the means (which is often far from obvious) in needed too.
Test library
Not only does software testing save time and money; it also protects your good reputation.
Ensuring high quality in your IT projects is challenging and requires continuous training.
Specialist training can solve your real-life testing and QA problems. Give your testing team
the winning edge with best practices drawn from successful real-life projects!
Learning how to best use industry-leading tools from vendors including HP and Microsoft
will secure you maximise your investment and increase your team’s efficiency.
For practitioners, our ISTQB® Certifed Tester training series drives confidence and guarantees
a secure platform to build success on.
Our TrainingFLEX scheme will help you squeeze more value from your training budget
and ensure success.
* Courses must be booked and taken by 30 April 2011, quote code PT1102 to qualify