European Software Testing

Benchmark Report
2016
AUTOMATION
C2

Automation TEST Magazine publishes

the European Software
Testing Benchmark Report,
which provides an impartial
and insightful view of the
current software testing
market, on an annual basis.

This first part of the survey

covers automation.

T
he results from the first European
Software Testing Benchmark Report
2016 survey on test automation are
in, highlighting that automation is not yet as
commonplace as firms want; there are still
numerous challenges hindering some; and
sourcing test automation specialists needs to
start in‑house.
KEY FINDINGS FROM THE
AUTOMATION REPORT INCLUDE:
• Only 5% of survey respondents
said they currently carry out 0:100
manual:automation testing. The majority
(66%) are either at 75:25 or 50:50, and 9%
said they are only doing manual testing.
• When asked what they'd like to achieve
in the next five years, the majority (73%)
said they'd like to see 50:50 to 25:75
manual:automation testing. 14% said
they'd like to have no manual testing at all.
• The majority of respondents reported
seeing a return on their test automation
investment immediately (24%) or within
the first 6 months (24%). The remainder
saw a result within 6‑12 months (28%) or
after one year (15%). Only 9% reported
never getting an ROI.
• The majority of respondents (94%)
said they use test execution tools and
automation to support testing efforts.
Other popular answers include generation
of test data (57%) and deployment of
environments (49%).
• The survey respondents reported that the
primary users of their testing tools are test
automation specialists (55%) and testers
(27%). Developers make up just 6%. Other
responses included offshore contractors
software engineers, and product owners.
• When asked what automation currently
exists in organisations, there were a wide
variety of responses. UI (82%) is the
most popular; APIs (62%), cross browser/
platform (55%), test data/environment
(55%), performance (53%) and integration
(47%) are all very even. Interestingly,
automation in the ecosystem scored the
lowest at just 6%.
THE BIGGEST CHALLENGES FACING
AUTOMATED TESTING IN THE
FUTURE
The survey also asked what the biggest
challenges facing automated testing in the
future are. Common responses included
maintenance overhead and continuity of
ownership; automation in sprint; deciding
what to automate; skilled resources;
tool selection; budget/funding; updating
automation scripts; mind set; buy‑in from
1
CECILIA REHN
EDITOR OF TEST MAGAZINE
stakeholders; legacy projects; and lack of
experience.
The sheer number of different responses
underscores the complexity of implementing
test automation and how solutions are never EUROPEAN SOFTWARE TESTING
BENCHMARK REPORT
'one size fit all' due to differentiating sets of
circumstances and resources at hand.
SOURCING AUTOMATION TESTERS © 2016 31 Media Limited. All rights reserved.
The survey showed that test automation TEST Magazine is edited, designed, and
is overwhelmingly carried out in‑house, published by 31 Media Limited.
primarily by in‑house resources (62%) but No part of TEST Magazine may be reproduced,
transmitted, stored electronically, distributed,
also by contractors working in‑house (12%).
or copied, in whole or part without the prior
A few (9%) responded 'other', and specified
written consent of the publisher. A reprint
a combination of in‑house and outsourced service is available. Opinions expressed in this
efforts. journal do not necessarily reflect those of
A majority (51%) said that they prefer the editor of TEST Magazine or its publisher,
to retrain existing staff to meet automation 31 Media Limited.
testing resource requirements. Another 38%
said they prefer to hire skilled permanent ISSN 2040‑01‑60
staff. Only 8% said they'd hire skilled contract
staff and even fewer said they would engage GENERAL MANAGER AND EDITOR
Cecilia Rehn
consultancy services (3%).
cecilia.rehn@31media.co.uk
When asked about what attributes/
+44 (0)203 056 4599
characteristics are sought after in team
members who are to be retrained in test ADVERTISING ENQUIRIES
automation, it was clear from the volume of Anna Chubb
responses that a combination of technical anna.chubb@31media.co.uk
(often coding) skills and soft (primarily +44 (0)203 668 6945
communication and motivation) skills are
attractive. PRODUCTION & DESIGN
JJ Jordan
jj@31media.co.uk
CONCLUSION
For more in depth insight and to see all the
survey results please read on in this report. 31 Media Ltd,
TEST Magazine will be running more 41‑42 Daisy Business Park
surveys throughout 2016, if you are 19‑35 Sylvan Grove
interested in participating please visit: London, SE15 1PD
www.softwaretestingnews.co.uk/survey/ +44 (0)870 863 6930
info@31media.co.uk
www.testingmagazine.com
softwaretestingnews
@testmagazine
TEST Magazine Group
cecilia.rehn@31media.co.uk
T E S T M a g a z i n e | E u r o p e a n S o f t w a r e Te s t i n g B e nc h m a r k R e p o r t 2 01 6
2

SURVEY RESULTS IN DETAIL

100%

80%

60%

40%

20% 11%
30% 36% 23%
0%
Less than 10% 20%-30% 50%-75% More than 75%

WHAT % OF YOUR TEST CASES ARE AUTOMATED?

Interestingly, only 11% or respondents said that they automate 75% or more of their
test cases. And a third (30%) said it is 10% or less; suggesting that test automation is
not as commonplace as many might think.

100%

80%

60%

40%

20% 47% 13% 9% 40%

21% 21%
0%
ROI Difficult No proof Unreliable Reports Other
not achieved /expensive script hard to analyse
to maintain execution

WHAT ARE THE MAIN ISSUES WITH THE QUALITY

OF YOUR AUTOMATED TESTING?
Respondents highlighted various issues with the quality of their automated testing.
Difficulty and cost of maintenance came out on top (47%), but many highlighted other
concerns including:

Infrastructure not set up to support; skilled staff; legacy systems; unstable

environments; adapting existing test cases to automation; automatic deployment;
co‑ordination with other teams; and costly test automation training.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
 3

100%

80%

60%

40%

20%
11% 60% 13% 16%

0%
Execution time
Defect reduction Cost reduction Other
detection ratio & speed to market

WHAT IS YOUR PRIMARY MEASURE OF THE

SUCCESS OF YOUR AUTOMATED TESTING?
Execution time reduction and speed to market are considered the top measures
of automated testing success (60%). Defect detection ratio and cost reduction
were highlighted as important, but respondents also reported using regression test
coverage and a quick feedback loop.

100%

80%

60%

40%

20%
14% 13% 33% 16%
23%
0%
Manual Error seeding Specific test project Verify Other
code review by manual test

HOW DO YOU TEST CHANGES TO YOUR

AUTOMATED TEST SUITE?
A third of respondents (33%) said manual testing stills plays an important role when
testing changes to automated test suites. Other responses included manual code
review (23%), error seeding (14%), specific test project (13%) and utilising a range of
the above.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
4

WHAT ARE THE BIGGEST CHALLENGES FACING

AUTOMATED TESTING IN THE FUTURE FOR YOUR
ORGANISATION?
The survey also asked what the biggest challenges facing automated testing in the
future are. Common responses included maintenance overhead and continuity
of ownership; automation in sprint; deciding what to automate; skilled resources;
tool selection; budget/funding; updating automation scripts; mind set; buy‑in from
stakeholders; legacy projects; and lack of experience.

The sheer number of different responses underscores the complexity of implementing

test automation and how solutions are never 'one size fit all' due to differentiating sets
of circumstances and resources at hand.

100%

80%

60%

40%

20%
44%
3% 21% 24% 9%
0%
Evaluation Design Implementation Maintenance Other
& tool selection

WHERE IS MOST OF YOUR AUTOMATED TESTING

TIME SPENT?
Survey respondents clearly stated that implementation (44%) and maintenance (24%)
are the two areas where test automation time is spent. Design (21%) came a close
third. Other answers include evaluation and tool selection, writing specifications and
awaiting sign‑off.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
 5

100%

80%

60%

40%
57%
20%
9% 25% 6% 3%
0%
Reduce Speed up Reduce risk A combination Other
cost of testing cycle time and failure of all
of software testing of the above

WHAT IS THE MAIN REASON FOR TEST

AUTOMATION INVESTMENT IN YOUR
ORGANISATION?
A combination of reducing the cost and the risk of testing, and speeding up cycle time
is the main reason for investing in test automation in the respondents' organisations.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
6

100%

80%

60%

40%

20% 15%
24% 24% 28% 9%
0%
Immediately 0-6 months 6-12 months 1+ years Never

HOW LONG BEFORE YOU SEE ANY ROI FOR YOUR

TEST AUTOMATION?
The majority of respondents reported seeing a return on their test automation
investment immediately (24%) or within the first 6 months (24%). The remainder saw
a result within 6‑12 months (28%) or after one year (15%). Only 9% reported never
getting an ROI.

WHAT ARE THE KEY POINTS TO PUT INTO A

BUSINESS CASE BEING PRESENTED TO UPPER
MANAGEMENT THAT WOULD ALLOW YOU TO
GET THE NECESSARY TIME AND MONEY TO
INVEST INTO TEST AUTOMATION?
Putting together a business case in order to secure test automation investment is
critical. Suggestions for key points to include in such a report included:
Cost savings: immediate and long‑term ROI, and cost against coverage.
Improved quality: freeing up of manual test resources to add extra capacity to
previously untested functionality; extra test coverage with each release; test
automation results in better quality software.
Speed to market: increase pace of delivery, reduce cycle time, manual effort in time/
money; supporting larger development teams and software complexity; flexibility and
reusability.
Risk report: highlight scare cases, risk scenarios and the cost of production failures.
Relate to business objectives: it is advised to present tangible business benefits,
including improvement in end client confidence and fast delivery in a competitive
landscape.

One respondent gave this confident answer: "You show the basics of how many test
cases a manual tester can execute in an eight hour period then juxtaposition the
results against a single threaded automation system. Typically shows it."

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
 7

100%

80%

60% 94%

40%
49% 57%
20%
39% 39% 24% 7%
0%
Deployment Generation Generation Test Analysis Recording Other
of environments of test data of test cases execution of test results of exploratory
testing

IN WHAT WAYS DO YOU USE TOOLS AND

AUTOMATION TO SUPPORT TESTING?
The majority of respondents (94%) said they use test execution tools and automation
to support testing efforts. Other popular answers include generation of test data
(57%) and deployment of environments (49%).

100%

80%

60%

40%

20%
6% 55% 12%
27%
0%
Developers Test automation Testers Other
specialists

WHO ARE YOUR PRIMARY USERS OF TESTING

TOOLS?
The survey respondents reported that the primary users of their testing tools are test
automation specialists (55%) and testers (27%). Developers make up just 6%. Other
responses included offshore contractors software engineers and product owners.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
8

21% 32% 6% 41%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Proprietary (i.e. vendor) Open source Home grown A combination

ARE THE MAJORITY OF YOUR TOOLS

OPEN SOURCE, PROPRIETY, HOME GROWN
OR A COMBINATION?
Highlighting the growing trend for open source tools, the survey found that 32% use
open source test automation tools; while only 21% said they use propriety tools. But
the majority (41%) did report using a combination of both.

In addition, 56% of respondents said that the majority of their test automation tools
are general, whereas 44% said theirs were specialised.

100%

80%

60%

40%

20% 9% 35% 33% 14% 9%

0%
Better than Speed Increased Reduced Other
manual testing of execution coverage costs

WHICH OF THE FOLLOWING WOULD YOU

CONSIDER TO BE THE PRIMARY DRIVER FOR THE
ADOPTION OF TEST AUTOMATION?
Speed of execution (35%) and increased coverage (33%) were both cited as the top
reasons for adopting test automation. Reducing costs was the primary driver for 14%
of respondents, and only 9% said it was because it is better than manual testing.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
 9

100%

80%

60%

40% 76%
20%
29% 27% 6% 17% 45% 29%
0%
Selenium HP functional In-house Watir Ranorex SOAPui Other
testing developed

WHAT UI TEST AUTOMATION TOOLS DO YOU

USE?
Selenium is by far the most popular tool for UI test automation, with 76% of
respondents using it regularly. The survey respondents also highlighted tools such as
SoapUI, HP, Ranorex and in‑house solutions. Some other responses include: Capybara;
Telerik; Autoit; Sahi; CA Technologies; JMeter; Neoload; Protractor; TestComplete;
QA Wizard; Behat; Calabash; Appium; Eggplant and Odin Axe.

100%

80%

60%
79%
40%

20% 19% 63% 35%

5% 6%
0%
Virtual Dockers Playbooks Continuous Continuous Other
machines integration delivery

WHAT VIRTUALISATION AND ENVIRONMENT

AUTOMATION SOLUTIONS HAVE YOU
IMPLEMENTED?

When asked what virtualisation and environment automation solutions have been
implemented, a majority said virtual machines (79%) as well as continuous integration
(63%). Continuous delivery (35%) came in at third place, and Dockers (19%) fourth.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
10

100%

80%

60%

40%

20%
6% 48% 17% 18% 11%

0%
Developers Specific Manual testers Everybody Other
automation
developers

WHO WRITES THE MAJORITY OF YOUR TEST

AUTOMATION SCRIPTS?
The survey highlighted that a specific automation developers write the majority of test
automation scripts (48%). Manual testers make up 17%, and others specified the role
of automation testers and general developers in this capacity.

100%

80%

60%

40%

20%
82% 62% 55% 53% 18% 39% 55% 47%
6% 5%
0%
UI APIs Cross Performance Security Deployment Test Integration Ecosystem Other
browser data
/platform /environment

WHAT AUTOMATION HAVE YOU GOT?

When asked what automation currently exists in organisations, there were a wide
variety of responses. UI (82%) is the most popular; APIs (62%), cross browser/platform
(55%), test data/environment (55%), performance (53%) and integration (47%) are all
very even. Interestingly, automation in the ecosystem scored the lowest at just 6%.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
 11

100%

80%

60%

40%
62%

20% 11% 12% 3% 3% 9%

0%
In-house Outsourced Contracted Local Off-site Other
resources but in-house

WHO CARRIES OUT YOUR TEST AUTOMATION?

The survey shows that test automation is overwhelmingly carried out in‑house,
primarily by in‑house resources (62%) but also by contractors working in‑house (12%).
In the other category (9%), many mentioned using a combination of in‑house and
outsourced efforts.

100%

80%

60%

40%

20% 35% 49% 15%

0%
Coding skills Test automation Tools
principles

WHEN SELECTING TRAINING COURSES TO GET

TEAM MEMBERS INTO AUTOMATION WHAT DO
YOU FOCUS ON?
Test automation principles (49%) ranked higher than both coding skills (35%) and tools
(15%) when selecting automation training courses.

We also asked what type of course they prefer, and respondents answered preferring
in‑house coaching /mentoring (43%) to other options such as classroom based (23%),
online course (18%) or self study (12%).

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
12

100%

80%

60%

40%

20%
9% 34% 22% 31% 5%
0%
100 : 0 75 : 25 50 : 50 25 : 75 0 : 100

WHAT RATIO OF MANUAL VERSUS AUTOMATION

TESTING DO YOU CURRENTLY DO?
(MANUAL : AUTOMATION)
Only 5% of survey respondents said they currently carry out 0:100 manual:automation
testing. The majority (66%) are either at 75:25 or 50:50, and 9% said they are only
doing manual testing.

100%

80%

60%

40%

20% 2% 11% 29% 44% 14%

0%
100 : 0 75 : 25 50 : 50 25 : 75 0 : 100

WHAT RATIO OF MANUAL VERSUS AUTOMATION

TESTING DO YOU WANT TO ACHIEVE IN THE
NEXT 5 YEARS? (MANUAL : AUTOMATION)
When asked what they'd like to achieve in the next five years, the majority (73%) said
they'd like to see 50:50 to 25:75 manual:automation testing. 14% said they'd like to
have no manual testing at all.

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
 13

100%

80%

60%

40%

20% 51% 38%

8% 3%
0%
Retrain Hire skilled Engage Hire skilled
existing staff contract staff consultancy services permanent staff

WHAT IS YOUR PREFERENCE FOR SOURCING

AUTOMATION TESTERS?
A majority 51% said that they prefer to retrain existing staff to meet automation
testing resource requirements. Another 38% said they prefer to hire skilled permanent
staff. Only 8% said they'd hire skilled contract staff and even fewer said they would
engage consultancy services (3%).

WHAT ATTRIBUTES/CHARACTERISTICS DO YOU

LOOK FOR IN SELECTING TEAM MEMBERS TO
RETRAIN IN TEST AUTOMATION?
Attributes/characteristics sought in test automation team members include:

Technical skills: coding & scripting skills, good understanding of testing and test
automation concepts and principles; test scenario identification for best test
coverage; an ability to transition to a more programming‑oriented environment;
and a mathematical understanding of extended finite state machines and notions of
completeness.

A development/analytical/scrum background was highlighted as preferable.

Soft skills: inquisitive nature; motivated and a desire to learn; focus with an eye for
detail; being a team player; communication skills; well‑organised; sociable; as well as
being passionate and willing to change the world.

One respondent gave this pragmatic answer: "I look for aptitude not pre‑existing skills.
Finding people is like looking for unicorns. Hire good people, up skill them and give
them a career opportunity. It will pay off."

T E S T M a g a z i ne | E ur o p e a n S of t wa r e Te s t i n g B e nc h ma r k R e p o r t 201 6
14
Special
thanks
The European ROD ARMSTRONG
PROGRAMME QUALITY MANAGER
Software Testing EASYJET
Benchmark Report is Rod Armstrong has over 20 years’
put together experience in the QA and testing space,
holding senior positions in a range of
with help from the organisations spanning application
TEST Magazine development, insurance, finance & risk,
telco and ecommerce. Currently, Rod is
Editorial Board. working with easyJet, assisting them in
driving a number of change/transition
programmes across the enterprise.
IAIN MCCOWATT ASIA SHAHZAD
HEAD OF TESTING FOR THE QA MANAGER
TREASURY FUNCTION HOTELS.COM
HIGH STREET BANK
Asia is currently working as a QA Manager
Iain is the author of software testing blog at Hotels.com (part of the Expedia Inc
Exploring Uncertainty. He has almost brand), where she is responsible for the
two decades of experience as a tester, QA strategy for several teams across
automator, test manager, and software multiple locations in Europe. Prior to this
testing consultant. role Asia was a freelance tester working
on some challenging high profile projects
across ITV, BBC, Talk Talk, AOL and Cable
& Wireless. Asia’s QA career spans over
10 years.
T E S T M a g a z i n e | E u r o p e a n S o f t w a r e Te s t i n g B e nc h m a r k R e p o r t 2 01 6
RIEL CAROL
HEAD OF TEST
YOUVIEW TV LIMITED
Riel has 20 years' experience in quality
assurance and testing applied to telecoms,
TV and internet industries. From an original
background in software development,
he has become a highly technical senior
manager that has worked in successful
start-ups as well as very well established
companies. Riel has extensive experience in
creating teams and quality processes that
focus in automation.
JIM WOODS
DIRECTOR OF DEVELOPMENT SERVICES
SEGA WEST
Jim has worked in the gaming sector
since 1989. At SEGA since 2008, his
responsibilities cover product development,
quality assurance and localisation, and
since his return, Jim has overseen the
consolidation of SEGA West’s QA in
London, and the relocation of the entire
QA team to new premises alongside the
publishing business in Brentford. At its peak,
Jim is responsible for a team in excess of
350 who work on SEGA’s latest games on
console, PC and mobile platforms.
European Software Testing
Benchmark Report
2016
AUTOMATION