You are on page 1of 80



2 REPORT 2016-17

World Quality Report 2016-17
Eighth Edition

Regional and country reports are available
online from or from
Capgemini, Sogeti and HPE local offices covering:
North America, United Kingdom, France, Germany
Benelux, Southern Europe (covering Italy, Spain and
Portugal) Eastern Europe (covering Poland, Hungary
and The Czech Republic), China, Australia and New
Zealand, Middle East and the Nordic Region

Contents 3

04 Introduction 74 About the Study
06 Executive Summary 78 About the Sponsors

14 Current trends in quality 56 Sector Analysis
assurance & testing 58 Automotive
16 Digital Transformation – Disrupting business 60 Consumer Products, Retail and Distribution
models for a better customer experience
62 Energy and Utilities
25 Internet of Things and Security Testing –
Stretching the need for quality and security 64 Financial Services

32 Agile and DevOps – More intelligence-led 66 Healthcare and Life Sciences
testing will meet speed, quality and 68 High Tech
cost imperatives
70 Public Sector
38 Industrialization and Testing Centers
of Excellence – Reflecting the shift to 72 Telecom, Media and Entertainment
an intelligence-led test approach
45 Test Environments and Test Data
Management – Key-enablers for
quality in agile and DevOps
51 QA and Testing Budgets – Getting
a grip on test spending

Previous editions
World Quality Report

2009 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16
First Second Third Fourth Fifth Sixth Seventh
Edition Edition Edition Edition Edition Edition Edition

4 REPORT 2016-17


the QA and DevOps principles to ensure organizations can achieve the Testing function is rapidly re-inventing itself. Accelerating software trends that dominate the world of QA and Testing today. In this maelstrom of change. creating matter experts. enabling continuous would encourage you to get in touch with your Capgemini testing and high fidelity feedback. Many businesses must decreasing time-to-market. we thank you for your time and contribution. . I delivery though agile and DevOps. performance and security to move towards hybridization of TCOEs as well as a change validate the value of IT from the point of view of the customer. are more important than ever. digital transformation continues to accelerate into more industries. reason that we see trends such as the increased adoption of agile and DevOps. performance evolve and transform to adopt this digital mindset. of doing business happen. security. there is an increase in the adoption of agile and In order to deliver towards such objectives. Users expect an exceptional well as QA and Testing) towards business goals such as experience and responsive services. in the skills required from quality professionals. for their insights and analysis as well as the what is defined as the idea economy. to give you the insight you need to address the challenges that are most critical to If you are one of the 1600 executives across 32 countries your business success. innovating with predictive analytics to help software teams tap into historical data to learn and anticipate the future. We share the results. We help you will provide tangible benefits and food for thought. We would also like to thank our subject Software drives our interactions and relationships. The report also contains an analysis of quality practices HPE partners with and helps IT leaders like you to increase across different sectors as well as countries and as you velocity and deliver amazing apps so you can confidently go through the report you will notice certain overarching meet the needs of your consumers. Testing Global Service Line SVP and GM. Capgemini Group HPE Software Welcome to the eighth edition of the World Quality Report by I am very happy to sponsor and share the results from the Capgemini. I hope that the information contained in this report further streamlining their software delivery. more and more of an organization’s services are exposed to direct customer To compete requires businesses embrace digital and interaction. increasing I want to acknowledge everyone – especially the team at Capgemini and Sogeti – who contributed in the research and creation of this eighth edition of the World Quality Report.a comprehensive and balanced 2016/17 World Quality Report. As the and customer satisfaction. it is clear that for our and outlines the directions in which the QA and Testing respondents. the validation of functionality. quality and scale needed to succeed. As Digital Transformation sweeps the world. and it is this idea team that worked on bringing out this edition of the World economy . our solutions are designed or Sogeti account manager to discuss how this research to help delivery teams accelerate and succeed. scale and are secure. It is for this speed. Each year we explore the overview of the key trends driving Quality Assurance (QA) major trends and issues that are the forefront of IT and QA and Testing. This is one of the biggest factors driving IT (as software in a new paradigm. new ideas and creative ways Quality Report. customer experience and quality function is evolving. To a certain Testing should validate the optimal user experience. Happy accelerate app delivery and to ensure that the apps you reading! build can perform. Sogeti and HPE . If you would like to learn more go to: hpe. ADM. Introduction 5 Hans van Waayenburg Raffi Margaliot Leader. Finally. these objectives are also driving the adoption of cloud technologies. the use of predictive analytics and We see the focus of testing changing as a pure technical increasing automation of QA and Testing practices. who participated in the research. leader’s minds.where innovation. extent. We’re might shape your future plans. Software is the canvas and paint that enables the idea economy. This report tracks all of these changes.

reflects a desire to stay close to the essential objective of Instead it is about transparent business risk profiling and the testing. along with upgrading production. skills and testing these trends on the Quality Assurance (QA) and Testing organizations co-existing. WORLD QUALITY 6 REPORT 2016-17 Mark Buenen Govind Muthukrishnan Vice President Senior Vice President Global Leader. Our study findings thus reveal function. Transformation programs. such as customer IT and testing leaders from 32 countries across the globe. which is to prevent serious defects from reaching shifting left of quality responsibilities. such as test data. which is increasingly required to transform into various multiple approaches and trends. but always related to the higher objectives of critical QA enablement components. as well as technologies. QA and Testing Practice Testing Leader. revenue growth. What clearly emerges in this year’s survey is that organizations continue to evolve their QA and Testing function in their . technologies. including the Internet of Things. Together. It’s a change that service to market quickly and with as few defects as possible. There has been a shift towards a focus on how QA and This year’s research was conducted among 1. environments and test network. these will drive effective decision making and deliver business results. This eighth This QA evolution in these parallel. seek to disrupt with digital at an ever faster pace. but different. Financial Services Sogeti Capgemini Executive Summary The past year has seen an unrelenting uptake of new traditional Core Enterprise IT (systems of record). and uninterrupted business It reveals a shifting set of primary objectives for the QA and operations. experience. but at other times running in tandem. worlds often edition of the World Quality Report illustrates the impact of sees a duality of processes.600 CIOs and Testing can improve business outcomes. as organizations in their nimbler digital Business IT (systems of interaction). sometimes partially a business enabler and secure client value from Digital shared. test customer value and business outcomes. It is no longer simply about getting a product or Testing function compared with last year.

An agile projects are increasing. This will be research clearly shows that despite the continued adoption the next big wave of change after the introduction of risk- of agile methodologies. and of past projects and releases as well as day-to-day data the realization of benefits from automation. newer challenges and contrasting expectations are of equally high importance. QA and Testing results. Organizations must adopt a fit-for-purpose test strategy to We believe that the most important solution to overcome manage these differing challenges in their bi-modal IT. the centralized Testing and Testing approach. So while there is clarity on the part of this a very different abstraction level: to contribute to business year’s study participants in terms of their understanding of growth and business objectives. EXECUTIVE SUMMARY 7 When combined with the relentless push for speed When asked for their opinions concerning the objectives of through rapid release cycles enabled by agile and QA and Testing. challenges with quality and testing in based test strategies and test automation technologies. which currently relies too heavily Center of Excellence (TCOE) has played an enormous part on largely subjective manual decision making. preparation in bringing efficiency and higher maturity of testing. On the other hand. On the one hand. there are expectations at and technologies. Figure 1 Executive Management Objective with QA & Testing 2016 65% 44% 42% 42% 42% Enhance Security Higher Quality of Enhance Customer Cost optimization Higher responsiveness Software Solutions Experience of IT to business demands 38% 37% 37% 36% 35% Implement DevOps Faster Time to Market Increased output Implemented third Better alignment party SaaS to Business . Our introduction of machine-based intelligence. technologies and interconnected systems. our senior management survey participants DevOps. but to comply priorities seem to emerge as they seek to understand what with them will require a very different level of reporting on we refer to as Business QA. digital is resulting in an escalating volume of new revealed very distinct. as well as to ensure end- the challenges in the Core QA (a term we coined to support user satisfaction. In the and execution. The QA team must be aware that these the Core Enterprise IT). It will radically change the QA traditional Core Enterprise IT systems. in increasing QA and Testing challenges will be the emerging both their legacy and agile development approaches. The most notable difficulties are intelligence-led QA approach enables QA activities to be in identifying the right areas on which to test. developments. which is to prevent defects from approach to quality validation and testing for these systems production. In the world of points from production. the availability automatically defined and adjusted to reflect the realities of flexible and reliable test environments and test data. de-centralized and agile world this is apparently much more difficult to achieve. they have a basic and essential We see organizations increasingly struggling to find the right expectation of testing. expectations. yet equally high scoring.


and the Internet of Things). The challenges include the need to manage escalating test volumes at increased speeds. it is evident that the issues around Figure 2 Top 7 challenges in Application Development 50 2016 2015 45% 43% 42% 42% 41% 40 39% 39% 37% 32% 30 23% 24% 22% 22% 20 18% 10 0 Lack of effective Too much Difficulty Lack of access Reliance on Manual Lack of effective build/integration design complexity integrating to dev and manual testing environment release candidate automation new applications test management quality information with the existing environments and deployment application ecosystem . While both agile and DevOps offer significant opportunity to drive faster release cycles. Keeping tight control of budgets is also a challenge. Agile and DevOps continue to grow in adoption. with QA in multiple areas are still relatively immature or scarce. that the agile and enabling QA activities in agile and DevOps settings. This trend looks set to continue for new developments now consumed by digital technology and grow in the future. tools in the mobile and digital testing space to measure. For the QA and Testing in the strategic imperative to embed customer experience function tasked with assuring customer value and business at the heart of Digital Transformation programs. with previously immature sectors catching up. while specific skillsets 2. year. This technologies to accelerate change and connect in gives QA and Testing professionals a bigger role to play new ways with their customers. The ability to bring new products and services to market driven QA and Testing strategies. Organizations in all sectors continue to use new digital quantify and better understand the user experience. albeit still large and growing percentage of the QA and Testing budget as an emerging practice. EXECUTIVE SUMMARY 9 World Quality Report Findings 1. solutions (mobile. It is no wonder. We also performance. with a and now DevOps principles are growing in use. cloud. Digital Transformation continues to drive IT strategy and There is also a growing opportunity to use the host of new make itself felt in the QA and Testing function. test data management. most notable lack of skill is in the areas of: intelligence. newer test automation ahead of the competition is a crucial differentiator for today’s skills. or even overtaking. last year’s digital leaders. optimizing test environments. business intelligence and business analytics. methodology is firmly embedded in development lifecycles. The making a corresponding move. Digital Transformation brings both challenges observe a levelling in the digital maturity across sectors this and opportunities. therefore. digital enterprise.

After an initial focus on delivery pipeline automation and environment management. rather than purely on execution. be the enabling glue that keeps teams focused on customer increased challenges with test environments are cited as value and business relevance. it is clear that there’s a whole new environments and data. A striking development in the environments. and data. For example. development. so in addition to the key skills such as achieving and maintaining good test coverage. cited by 46% of respondents of more cloud-based test environments. TEM is slowly maturing. The emergence of Internet of Things functionality is a Despite the growing challenges. • Lack of the right tools for testing has jumped again. will become ever more automated and virtualized. 44% of this year’s survey Assuring the quality of Internet of Things functionality will participants using agile claim involving the testing team in become an increasingly bigger part of the QA and Testing the inception and sprint planning phases is a challenge. Organizations are struggling Internet of Things products or solutions play a role do not to find the answer to the question of how to achieve speed currently have a test strategy for this specific aspect of (or velocity) with the right level of quality. a form of artificial intelligence and Organizations report a lack of specific QA and Testing skills machine learning is viewed as valuable functionality for to be a challenge. while agile implies a shift left in QA and Testing to earlier in the lifecycle. quality and operations requires efficiency. with 46% of respondents citing it as a challenge. 3. although it will be of failure. it is clear that DevOps implementation challenges go far beyond just issues with While there is an increasing trend for investing in test QA. claimed by 48% of survey respondents The study findings pertaining to DevOps show an increasing maturity with a better appreciation of the true purpose • Lack of facilities to book and manage their own and application of DevOps. specialists mentioned earlier. year are to establish distributed agile and test approaches. we see an emerging need for data scientists to implement a data-led predictive QA 4. revealed this year are: rather than on tools. quality validation and validations performed by the device manufacturers. However. • Having to maintain multiple versions of test environments. Indeed. both reported by 42% of this year’s study respondents. study saying Internet of Things products are part of their using analytics and self-managing capability for testing business operations. go. WORLD QUALITY 10 REPORT 2016-17 managing and implementing the quality and test measures It is very worrying that 68% of the organizations in which are an increasing challenge. test environment specialists and test data a high level of product quality. disrupting force with the potential to increase the impact Previously scarce skillsets are increasing. The challenges around managing and driving down the cost approach. As the needs of agile and DevOps world of connectivity and functionality that must be validated cycles escalate. among other generic and potentially more pressing DevOps implementation challenges. the key themes for testing in agile this of test environment management are growing. This will see the This year we have also identified a need for agile testing emergence of new test tools and approaches designed expertise to become more strategic. although there is still a long way to an Internet of Things test strategy in the near future. Investment in test environments is aimed at a culture change. . Apparently many of them are relying on the quality boundaries between the disciplines. leading to test strategists. The key challenges in test environment management (TEM) and to focus on business outcomes and customer value. function’s remit as more and more things are connected and able to share data across networks. Despite the growing maturity. Core systems With 85% of participants in the World Quality Report 2016 built to support testing are becoming increasingly smart. the rising cost of test tool implementation. along with the increased adoption demand. There is a risk that the implementation of removing the testing bottlenecks in the development cycle Quality Assurance in the development cycle will be buried that are a barrier to rapid releases. For example. with a focus to deal with the exponential increase in test requirements on transformation. This testing activities are being more fully integrated in the agile situation must change and it appears that 30% are planning development lifecycle. By removing their IT. it is evident that breaking down the silos environments is turning the spotlight on how to drive TEM of business. a while before TEM is fully professionalized. cited by 46% of respondents approach to QA is the uptake of continuous monitoring • Lack of visibility to test environment availability and with predictive analysis. having the third biggest impact on QA and Testing budgets. test environments and test data management and verified. this would be a serious The relative lack of TEM maturity gives rise to inefficiency mistake because a QA guideline and toolkit for DevOps can and this has an ensuing impact on the budget. Overall.

There to validation and verification activities. the application in a continuous mode. centralizing to use automatic and self-learning analysis and decision test functions and increasing automation levels. This will be required in addition to these traditional ones to ensure that extended across the development and operations phase of the containment of QA costs can improve. digital technologies and the Internet of Things is creating an This year’s share of the IT budget devoted to QA and Testing increasingly heavy workload for the QA and Testing function. that the growing adoption of digital innovation will lead to an increase of test cycles and require further investment in test This year’s study findings reveal that 40% of participants platforms and test tools.95 4.07 5. has dropped to 31% after a significant and worrying increase it also offers the potential to take an intelligence-led approach from 18% to 35% during the preceding four years. however. EXECUTIVE SUMMARY 11 5. output DevOps time to alignment third party Experience software of IT ness to market to business SaaS solutions business services demands . So other and new measures are test execution and production quality decisions. Traditional measures for techniques in the coming year.97 4. The continued requirement to find efficiencies at every level the need for a next level of change in the QA and Testing in QA and Testing remains evident despite this year’s success operations of all organizations. next disruptive change. making technologies on all QA aspects: from strategy agile and DevOps adoption limits the possibility of low cost decisions (what should be tested and to what depth).85 5 Mean summary 4 3 2 1 0 Enhance Enhance Higher Cost Higher Increased Implement Faster Better Implement Security Customer quality of optimization responsive. This need is brought into sharp focus by the knowledge including production and user data. new increase to 40% in 2019. This far exceeds reasonable levels approaches and tools are available. While the advent of new in containing costs. test environment preparation. with more emerging. However. All the previous findings highlight Figure 3 Top 10 most important aspects of your IT strategy 7 2016 6 5. there is an overall prediction that spending will of what to test and improving environment availability.67 5.94 4. to test offshoring. with minimal human- led decisions in true zero-touch testing or test environment Transformation to intelligence-led QA and Testing will be the and data provisioning.08 5. Despite this year’s understanding of the end user.93 4. to addressing challenges reduction. This will potentially drive up costs still foresee using predictive analysis as one of their automation further if not sufficiently controlled. of up to 25% and points to a continued urgent need to find efficiencies. From gaining a better is no room for complacency. These can inform and drive more efficient test cycles with predictive analytics and the ability to leverage Big Data.07 4.90 4. and automation is still only delivering a partial set generation and adjustment. return on investment (ROI). This will enable organizations driving efficiency have been: low-cost offshoring.

data-led testing. Whereas the challenges in Business QA are more likely left quality. BDD. and to extend the agile and Lean data provisioning more as a service that can remain as a principles from software development to the software shared service to serve the needs of a highly efficient and deployment phase. done traditionally. testing critical to address this: activities. be successful if the Quality Assurance aspect is integrated and maximum automation is achieved in each step of the The increasing challenges with TEM and TDM are not only DevOps lifecycle. test data management. environment virtualization capabilities. analyze coverage and efficiency of test sets. We believe that a few steps are increasing throughput and velocity. and plan the The TEM and TDM challenges differ from a Core QA and a priority of test cases. behavior driven development (BDD). We recommend beginning your agile and DevOps • Invest in virtualized and cloud-based environments for journey by first shifting testing left to involve the test team the digital Business IT right from inception. therefore putting your organization’s reputation at risk. predict risk areas and risk levels of projects. test data issues before they occur in production. as well as the continuous invest in and experiment with tools that enable the user to: improvement in the traditional development projects. and test execution. The challenges in Core QA are one of seeking a test-focused ecosystem. continuous monitoring. and • Establish service-based solutions for test environment predictive analytics capabilities for DevOps. to be the simple and basic need of what to test and how to The core idea behind DevOps is to remove the boundaries test. To begin to realize the benefits promised resulting in cost inefficiencies in traditional Core IT systems. management. Business QA perspective. by the DevOps philosophy. service virtualization. A way to resolve both efficient. Invest in as-a-service solutions for test environment environments. Then shift testing right over time once • Invest in intelligent test data generation and TDM solutions you achieve scale. Adopt a QA approach for DevOps. will become the constraining • Assign the responsibility of optimizing Core IT test factor. for example data or 2. processing and interpreting the huge automation standpoint. Intelligent. which ensures the right areas of quality decisions is one of the major emerging degree of automation coverage and tools defined for the opportunities. volumes of data that it is not humanly possible to analyze. and functional and non-functional automation. analyze utilization of resources and 3. IT development. From an artificial intelligence. In other words. Given the different needs. Invest in intelligent self-learning QA and Testing platforms The right skillsets for the agile test organization should for all areas of the application landscape. automation. and build continuous testing. we recommend organizations between the traditional silos of business. cloud-based environments. self-learning technology will drive analytics and as opposed to pure play automation specialists. include test strategists and environment and data specialists. In this way the QMO acts as an enabler for driving velocity The key on the journey toward intelligence-driven QA is to in agile and DevOps initiatives. strategic identification of what to test and predict quality environment virtualization. such as TDD. . WORLD QUALITY 12 REPORT 2016-17 Key recommendations in order of importance 1. predict test estimation based on requirements. This intelligent data analysis will facilitate the various levers. make use of mature solutions that handle environment and QA and IT operations. Or the alternative is that testing will not be done environments to a TEM team adequately. we believe automation of QA but are also interfering with the velocity targets of digital activities is not only required but is the core enabler of Business IT applications. Leveraging predictive analytics strategy and competency challenges is to use a centralized from various sources to bring speed efficiencies in all Quality Management Office (QMO). test driven development (TDD). agile and traditional environment provisioning with a view to unlocking resource – powered by enablement teams that help to truly shift idling. white box testing skills It will become far easier to identify what to test and where and services virtualization should take on greater importance more testing is required to bring down costs with more than functional automation skills. analyze the root cause of defects. It is our view that this process can only offshored Core QA and an increasingly agile Business QA. provision and test data provisioning for projects.

And as we increasingly rely on connected. But as the Internet of Things is playing a more platform and application level. and validation of the potentially multiple applications by which humans can control and interact with the device. together with a few managerial important role. Use of different automated tools is key to success of a DevOps program. EXECUTIVE SUMMARY 13 4. the validation of the have passed exit criteria. The minimum recommended tested safely. test of a test environment set up in which the device can be productivity. Develop Internet of Things-specific test strategies 5. Clearly this validation requires higher levels of quality indicators per application are: status of production engineering skills because it means dealing with: validation incidents. operational reliability. chains of enterprises by expanding the machine-to-machine With the transformation to more agile and DevOps-based and machine-to-human communication and data exchange. One way to establish the balance and take the right decisions is to keep a real-time dashboard that tracks the enterprise- The test approach and test depth will differ depending on quality balanced scorecard with metrics that are critical to whether you are a manufacturer of an Internet of Things both velocity and quality. each and every enterprise must have a performance indicators. This dynamic dashboard should device. ease of to this challenge is to balance the conflicting needs within use. and total number of features connection of the device in its environment. processing of data. release defect status. On top of this there is a series of the context of resource constraint. interoperability. CIO. Retail. and performance. which is a good thing. Sweden . and test velocity. or an end-user of the represent a handy set of critical quality indicators on a device. and resource utilization. The solution be considered are: security. to more flexibility and speed. the storage and delivered. organizations are relying more on self- New devices. installability. self-steering and the flip-side of this is that organizations now lack insight into self-learning devices. Manage quality with simple balanced scorecards per Line Internet of Things developments are changing the value of Business and per application or process. as well as availability managerial performance indicators are: release quality. the key quality aspects that should the quality at an overarching enterprise level. development. However. percentage features that of the operation of the device itself. developments and content exchange between empowered and self-steering teams. The minimum recommended specific risk-based test strategy for it. technical level quality attributes: compatibility. an integrator of the device. This obviously leads these devices are happening at a faster and faster pace. Automation plays a major role in transition to DevOps.


quality and cost imperatives 38 Industrialization and Testing Centers of Excellence – Reflecting the shift to an intelligence-led test approach 45 Test Environments and Test Data Management – Key- enablers for quality in agile and DevOps 51 QA and Testing Budgets – Getting a grip on test spending . Current Trends in Quality Assurance & Testing 15 Current Trends in Quality Assurance & Testing 16 Digital Transformation – Disrupting business models for a better customer experience 25 Internet of Things and Security Testing – Stretching the need for quality and security 32 Agile and DevOps – More intelligence-led testing will meet speed.

it is safe Social media. solutions have equipped the digital enterprise with the tools Instead. and reacting to emerging external market years. WORLD QUALITY 16 REPORT 2016-17 Digital Transformation Disrupting business models for a better customer experience John Jeremiah. with profound implications for the Quality every device is becoming a smart device that’s connected to Assurance and Testing functions. empower employees. whereby than ever before. it is an everyday means by which organizations it needs to get closer to customers. business analytics and cloud to say that digital is no longer viewed as something new. mobility. Figure 4 Percentage of Organization with a dedicated Chief Digital Officer 100 100 No we don’t need a Chief Digital Officer 90 16% as digital transformation strategy resides 90 19% with our Business Leadership (CE) No we don’t have a Chief Digital 80 80 Officer and NO plans are in place 17% No we don’t need a Chief Digital 9% 70 Officer as digital transformation strategy 70 resides with our IT Leadership (CIO) No we don’t have a Chief Digital 60 16% 60 23% Officer but plans are in place No we don’t need a Chief Digital (for 2017+) 50 Officer as digital transformation is 50 18% NOT part of our strategy 40 40 No we don’t have a Chief Digital No we don’t have a Chief Digital Officer but plans are in place 30 30 Officer but plans are in place (for 2015-16) 49% 20 (for coming years) 20 33% 10 Yes we have a Chief 10 Yes we already have a Digital Officer Chief Digital Officer 0 0 . rethinking their business models and transform internal business processes for the past five and operations. continuously sharing and providing customer- relevant data. are accelerating change. Capgemini While Digital Transformation continues at pace. Digital Research Team Leader. the internet. But technology changes rapidly and there is a marked forces. The end customer is a lot closer to the business growth in the need for Internet of Things integration. Capgemini Sathish Natarajan. Service Delivery Director. HPE Renu Rajani. Vice President.

Transportation too has moved from its laggard in light of our new categorization is in the 3% drop in those position in 2015 to join CPRD and Financial Services on organizations with no plan for this role. . of the QA and Testing budget for new developments. customer cloud. This is a big increase from experience and corporate image remain among the last year’s 53%. 18% of those currently (57%) and Energy & Utilities (54%). that 66% now either have a dedicated CDO or have assigned the role of CDO to the newly defined categories of the IT We can clearly see a jump in maturity by the CPRD sector leadership (17%) or the business leadership (16%) as part of (68%) from the bottom of the maturity curve to a mid-way their Digital Transformation. little change. which stands at a solid 13%. in a rush to meet the Digital Transformation trend recorded by the World Quality objective of improved time to market. either with the appointment of a dedicated Chief Public Sector survey participants have gone down the route Digital Officer (CDO). and Internet of digital focus does not automatically lead to an increase in Things continue to consume a major share of the QA and QA and Testing spend. DevOps and continuous integration methods. we also specifically measured the percentage of the QA and Testing budget spent on Internet Drilling down into the figures. which do affect the QA process. However. with the latter showing without a CDO. A number of last year’s digital laggards have caught up with the digital leaders. Products. Additionally. High Tech 2015 to 16% in 2016. and Manufacturing. Nonetheless. followed by the Public Sector (73%) and Telecoms to which organizations are making digital a senior-level & Media (69%) at the top of the leader board. down from 17% last year. where only 25% have a dedicated CDO. cloud and BI/BA. This year’s findings reveal across the world. For Testing objectives. organizations adopt Report over the past three years: 30% to 53% to 65%. so potentially is the management of digital. or with existing executive positions of appointing a dedicated CDO far more enthusiastically assuming responsibility for digital. Retail and Distribution (CPRD). customer-facing applications and digital innovation shown in Figure 5. This is an ongoing of QA and Testing costs. Thus digital is not a primary driver Testing budgets for new developments. there is also renewed little to 14%. there is a slight correlation of Things solutions. between the absence of a digital test strategy and a relatively higher proportion of IT budget spent on QA and Testing. that the key strategic IT drivers to securing the quality of 60% is consumed by new digital technology solutions (mobile. BI/BA. while ERP and legacy focus on efficiency and effectiveness as important QA and back office solutions is 4% down on last year at 27%. This demonstrates that a higher level of solutions around mobile. Internet of Things). as Clearly. One measure of digital progress position. have plans in place for the coming years. This year’s priority. Current Trends in Quality Assurance & Testing 17 Digital maturity measures reveal steady growth As part of our survey into the impact of Digital Transformation sector state they have either a dedicated CDO or assigned on QA and Testing. Higher digital maturity in the Public Sector aligns with the increased Last year. our research showed that 48% of the organizations move to a digitally inclusive model for the provision of citizen interviewed had a CDO and 23% were planning to establish services and stretched efficiency targets set by governments the role over the coming months. 76% of respondents in the Automotive Renewed focus on efficiency and effectiveness The World Quality Report 2016 survey findings show Overall. we investigated in more detail the extent CDO role. Digital Transformation are still relevant: security. This year’s laggards are Healthcare (61%). This acknowledges that than most other sectors with 51% compared to Consumer as digital is increasingly embedded in day-to-day activities. Spending on front office solutions dipped a important objectives. down from 19% in 68%. the first time this year.

When combined. we have also recorded the level of on-premise Public Sector (29%) and Healthcare and Life Sciences (28%) cloud applications for the first time. Public Sector (45%) solutions are a significant component of the overall private and Healthcare (43%) are at the lower end of the spectrum. with a combined total of 49%. private of on-premise cloud applications. Sector/ Retail industry hardware Life Media Government Sciences Level of cloud-based applications predicted to increase over the coming three years There has been a significant drop in the level of public and Public Sector and Healthcare organizations. tation Utilities Services turing including & nications. up just 1% to 45%. When combined with the volume it with the private cloud metric. Clearly on-premise cloud and Financial Services at 47%. year) and hybrid cloud applications (down to 18% from 28%) is due to concerns about both security and control of data. This year. WORLD QUALITY 18 REPORT 2016-17 Figure 5 Percentage of organizations with a Chief Digital Officer and the IT budget allocated to the Testing and Quality Function 2016 responses Yes 25% 32% 27% 31% 25% 31% 28% 32% 51% 43% 34 33% 33% 32% 32% 32 31% 31% 30% 30% 30% 30 29% 28 26 Consumer Transpor. cloud options. Regionally. Again. Energy. while Central Europe records a 20% lower . The Middle East is the most where the highest percentage users of private cloud were the prolific user at 52%. It’s likely that the high figure for private cloud although at 39% it is the Transportation sector that uses the when compared to public cloud (down to 23% from 29% last least private and on-premise cloud applications. only measure. CPRD is still using the cloud (30%) and on-premise cloud (15%) applications as most. honor goes to CPRD (36%) and Manufacturing (33%). there are significant differences between the biggest users of private cloud-based and on-premise cloud There is a shift at a sector level compared with last year applications as a combined total. Financial Manufac. as opposed to bundling among the lowest users. that hybrid cloud-based applications over the past twelve months. Public Automotive Products. Automotive. with This year. followed by the Energy a percentage are little changed from last year’s private. and Manufacturing sectors. & Utilities. HI Tech Healthcare Telecommu. all at 48%.

and Asia Pacific stands at 46%. from 18% to 21%. cloud. followed by public maintenance. cloud applications. Public Automotive Products. Based on a Energy & Utilities. private cloud proves that the concerns around data and privacy extend beyond the traditionally data sensitive sectors This year’s respondents to the question of cloud-based like Financial Services to other sectors. tation Utilities Services turing including & nications. HI Tech Healthcare Telecommu. Energy. The strongest growth in records a 49% usage. Hybrid shows a moderate growth Figure 7 Percentage split of cloud based application 100 90 80 13% 16% 15% 15% 14% 15% 17% 16% 16% 70 14% 17% 60 18% 18% 15% 17% 17% 19% 19% 16% 18% 50 40 23% 23% 25% 23% 24% 24% 24% 24% 23% 21% 30 20 36% 25% 32% 32% 33% 28% 28% 26% 29% 32% 10 0 Consumer Transpor. and scalability perspectives. from 45% to 55%. Sector/ Retail industry hardware Life Media Government Sciences On-premise Cloud Hybrid Cloud Public Cloud Private Cloud . such as CPRD and applications usage believe that it will grow. In between these polarities. North America expectation. Current Trends in Quality Assurance & Testing 19 Figure 6 Percentage split of cloud based application 2016 2015 50 44% 40 30% 29% 28% 30 23% 18% 20 15% 10 0 Private Cloud Public Cloud Hybrid Cloud On Premise Cloud usage at 32%. The moderate growth in public cloud- three-year horizon. from 23% to 28%. they predict strongest growth in a based applications is a signal that organizations are also combination of private cloud applications and on-premise expecting open cloud solutions to be beneficial from cost. Financial Manufac.

more mature cloud environment. we see a significant increase from last year Our study findings also reveal that many organizations in the percentage of study respondents saying they pay are currently ignoring the need to validate their move to special attention to peak load testing. Understandably and customer engagement. Both of these are up just 1% to 53% a sign that organizations have not yet realized the need to this year. If they in a number of striking changes to the specialist approach were. This is up from 38% cloud. This is and performance. for which scalability and this has a direct impact on the testing approach used. security is increasing. up from 24% in 2015 to 49% this year. cloud-based and/or third party software-as-a-service (SaaS) This finding also suggests our study participants may not solutions having a specific risk profile. This is demonstrated yet be using true cloud native applications services. The percentage of respondents that do not follow a to 54% and brings peak load testing in line with the other specific test approach for testing cloud-based applications two areas covered in this aspect of QA and Testing. be attributed to the fact that organizations are using more Figure 8 Approach towards testing of cloud based and third party SAAS (Software as a Services) 2016 2015 53% 52% 54% 53% We pay special 38% attention to performance 52% requirements and risks 49% We pay special We pay special attention to peak load attention to security requirements and risks requirements and risks 24% We do not use a specific approach to test cloud based or third party SAAS services . with responsiveness to peak loads are increasingly important. peak loading would not require special attention in a of testing these applications. The increase in focus on peak load testing can specifically test for cloud migration. In particular. WORLD QUALITY 20 REPORT 2016-17 Higher focus on performance testing while functional validation on cloud is ignored More and more organizations are transforming ways of cloud-based services for day-to-day business operations working to become cloud-first enterprises.

This leapfrogs the challenge above last year’s top three challenges: lack of an in-house test environment. and traditional) value use cases. and test and deliver based on this respondents reporting challenges with testing mobile and usage and adoption to avoid wasting time and energy on low multi-channel (mobile. The first is to understand user-base With growing integration. that given the increasing complexity of the mobile landscape. The next activity is to prioritize and create a applications this year. wearables. up from 36% to management and operations required for mobile testing. and a lack of mobile test experts. not enough time to test. up from 29% to 46%.Wearables. 44%. delivery of features increases. testing becomes more complex usage patterns and device demographics. Social. devices. what is Figure 9 Challenges in testing mobile and Multi-channel (Mobile. A lack of the right testing process or method has should be accelerated though continuous integration and seen the biggest jump at 20% more than 2015. all the issues in this area covered by of devices. Current Trends in Quality Assurance & Testing 21 Challenges increase for mobile and Multi-channel applications testing There has been a surprising increase in the number of features being developed. For example. cited by 48% of respondents. with feedback leveraged to greatest challenge this year. address this challenge. social. and methodologies. It is also the continuous delivery (DevOps). Finally. Operating System. up this year Clearly. and Traditional) application 2016 2015 2014 48% Don't have the right testing process / method 28% 36% 46% No mobile testing experts 29% 33% 46% Don't have in-house testing environment 38% 30% 45% Don't have the right tools to test 26% 34% 44% Not enough time to test 36% 40% 44% Don't have the devices readily available 26% 35% 1% We don't do mobile testing 8% 13% 0% 10% 20% 30% 40% 50% 60% . organizations are struggling to cope There is a core high level sequence of activity that can help to with the variation in tools. while minimizing the World Quality Report 2016 survey have seen marked variation in the testing pool. and carriers. Then prioritize the and new areas must be validated. shape each iteration. there is an overall sharp deterioration in the from 38% to 46%. Having reduced in volume across sufficiently representative development and test ecosystem the board in 2015. It is possible.

which offer use of mobile solutions. The top three remain security (46%). . This can be explained by the fact that. this While the challenges with testing mobile applications have is the least tested area of mobile applications and is down increased and changed in priority over the last year. the focus from 48% last year. not least with regards security. with limited complexity of functionality the number one IT strategy priority by a long measure. Additionally. WORLD QUALITY 22 REPORT 2016-17 Figure 10 Focus area for testing mobile applications 2016 2015 2014 46% Security 55% 54% 44% Efficiency / Performance 54% 51% 44% User interface / Ease of Use 47% 45% 43% Compatibility/Regression Testing 45% 42% 43% Portability 34% 40% 39% Certification of Application 33% 35% 34% Functionality 48% 54% 0% 10% 20% 30% 40% 50% 60% the overall integrated user experience? How secure is data possible reason for the drop in percentage effort afforded exchange and storage – is the analytic engine correct? In to security is the strong desire to move away from public addition. testing is becoming more This is surprising. although many more organizations are making cloud to private and on-premise environments. with performance (44%) and user interface/ease of use (44%). are down in terms of the focus effort. mobile testing is still viewed as a greater control over security. functionality validation can be 2015. areas for testing mobile solutions are fairly consistent with for many mobile solutions. however. One to test. efficiency and moved to field testing by real (subset) users. relatively new skill in the development lifecycle. a reduction in large releases of complex upgrades in favor All three. which is granular and minimal. With just 34% of respondents focusing on functionality. of smaller DevOps style releases.

it has become QA and Testing challenge of creating scenarios to genuinely easier to measure. This could be reduced still of study participants. Current Trends in Quality Assurance & Testing 23 Customer experience testing challenges reduce This year has seen a reduction in all challenges with testing The challenge of identifying end user expectations and the customer experience of multi-channel applications. Overall. It is an indication that organizations are better able to manage can be a real challenge to understand how customers use their customer experience testing and have improved their an application and. This will characterize QA and Testing in the coming years. both ranked by 36% with last year’s study findings. it is clear that the tools in the mobile and digital testing space. while these two challenges are being understanding of the user experience. the reduction in challenges tools for customer experience testing (down from 43%). requirements has dropped from 30% to 35% compared The two most challenging aspects. experience. and better understand the user reflect the flow of what users do remains. quantify. Figure 11 Challenges for Testing Customer Experience for Multi-channel applications 2016 2015 60 43% 41% 40% 41% 41% 40% 38% 40 36% 36% 35% 35% 35% 34% 33% 20 0 Designing the Implementation/ Establishing the Identifying the Establishing the Getting the right Identifying the test cases usage of test test data for end user environments for coverage of end systems and apps tools for customer customer expectations customer user expectations to be covered experience experience and experience and requirements in test testing testing requirements testing in the test set . With a host of new experienced to a lesser extent this year. are designing the test case (down further through the wider adoption of behavior-driven- from 41% in 2015) and implementation and use of test development techniques.

it is our belief that only by truly knowing cost to quality. private and public cloud) (Mobile. Instead. Today we see IT driving business The use of intelligence-led testing and predictive analysis as digital enables new ways of working. One is causing specific test challenges. this the key will be to test the right things. it is impossible to test 100% of all releases. At 29% of respondents. Historically. business speed. are more lakes. Figure 12 Most challenging aspects of testing digital implementation 40% 35% 2016 2015 29% 30% 28% 30% 24% 20% 18% 17% 20% 10% 0% Integration services (including Multi Channel Interface Data and Service End to end workflows local. integration of different IT architecture components (front with the result that they must have developed more robust office systems vs back office systems). although there is clearly still a long way to users. and turning data into strategic insights. mainstream IT strategy as digital comes of age. The achieved by some of the respondent organizations in their speed of ongoing Digital Transformation is seeing the rapid understanding of how to test the omni-channel experience. situation has reversed. with the customer can QA and Testing activities be effectively many organizations struggling to make the shift to adopt prioritized to meet the needs of the digital enterprise and cloud native applications. and new off-the-shelf services with back office and pronounced this year. engaging with end will enable this. possible explanation for this could be the level of maturity the challenge with integration services is the most cited. dropping from 35% to 28%. such those associated with data and legacy systems. WORLD QUALITY 24 REPORT 2016-17 Digital continues to disrupt and the World Quality Report 2016 Testing the multi-channel interface appears to have become study findings reveal that testing new digital implementations less of a challenge this year. demand drove IT. it is essential that this speed doesn’t come at a Nonetheless. This speed is causing a major challenge and service orchestration. go to achieve it: 35% of respondents say that establishing the data for testing customer experience for multi-channel While digital has also created an expectation of speed-to. QA and Testing has to continue its customers. however. direct access to data testing strategies. Digital Transformation is also fundamentally changing the With the proliferation of application releases and demand for way in which business and IT interact. and end-to-end workflows. Traditional) orchestration A digital future cannot come at the cost of quality The consumer appetite to engage via digital channels is its transformation and work in new ways adapted to this matched only by intolerance for any defects that might have digital world. an impact on their user experience and business outcomes. . Digital Transformation is still evolving. Social. at the right time. applications is an important or very important challenge. This is the focus of QA and Testing should be targeted at securing possibly due to the adoption of digital implementation into the quality across all system layers. market. Other challenges. but with greater digital capability.

HI Tech Healthcare Telecommu. Cybersecurity Services. Internet of Things test strategy is needed. This is demonstrated by the 85% of participants of Things device themselves. Sector/ Retail industry hardware Life Media Government Sciences Our products have IoT functionality. Vice President. High Tech. even though it Figure 13 Organization with specific test strategy for testing products in an Internet-of-Things (IoT) environment 120 100 7% 16% 22% 13% 11% 15% 17% 16% 20% 24% 15% 18% 80 18% 16% 17% 14% 16% 18% 21% 19% 60 36% 41% 37% 39% 29% 46% 40% 33% 42% 34% 40 20 30% 27% 25% 30% 28% 28% 28% 32% 33% 29% 0 Consumer Transpor. Senior Test Consultant. but plan to include near future We have a fairly mature IoT test strategy We do not have or work with IoT products/solutions . tation Utilities Services turing including & nications. test strategy currently. 68% of respondents with Internet of Things and how we live. Capgemini The Internet of Things is revolutionizing the way we work However. Financial Manufac. relying instead on the device in the World Quality Report 2016 study that say Internet of manufacturer to plan and execute tests. The amount of devices and level of data functionality do not have a specific test strategy for it. But we believe an Things products are part of their business operations. Public Automotive Products. Director Technology – V&V. but currently Our IoT products and solutions do not have any specific do not have any specific test strategy. Sogeti Tom van de Ven. Energy. This exchanged in our connected landscape is growing to epic is perhaps because most do not manufacture the Internet proportions. Current Trends in Quality Assurance & Testing 25 Internet of Things and Security Testing Stretching the need for quality and security Yves Le Floch. Sogeti Vivek Jaykrishnan.

66). and testing non-functional aspects (4. Worryingly. data metrics to ascertain what the user feels about the product. new ways of working must be number of reasons.70 experience. rated by virtualization of Internet of Things services and APIs (51%). It’s possible that these around whether security is adequately guaranteed and if the companies are confident their existing testing is adequate. Internet of Things development process. including a lack of expertise. management experts all talking and collaborating. although this has security implications test strategy and no plans for one. another factor layers receives an average 4. and the aim is for the product or application deployed no surprise due to the difficulty in testing between each to simply just work. and Intelligence. with improved diagnostics. rush to introduced. is the opportunity for Operational tools for stress testing middleware and gateways (50%). products is largely due to the need to switch from a physical Opportunities for further testing identified This year’s study participants were also asked about the recorders and players providing real-time monitoring and possible opportunities for solution development for Internet analytics of Internet of Things devices. it is currently rarely applied in the products (4. including performance. they may not realize the end user expectation of being able to switch seamlessly huge potential for risk if they don’t have a comprehensive between device operating systems and different application Internet of Things test strategy. in fact 38% concede that they have There are a number of techniques coming to the fore as plans for one in the near future. although the secure by design approach those of testing the integration with third party solutions/ is the right one to adopt. Internet testing and creating a test environment with virtualized end of Things functionality is typically centered around the end products.57). test craftsmanship and project market. they might simply be content to rely on the technique that we are seeing more and more of to meet the device manufacturer. there is little chance it can really be The challenge with creating test environments for virtualized taken into account in the testing process. 54% of respondents. and business intelligence. The challenge of testing this interaction between Looking more closely at the security challenge. our World Quality Report aspects for the first time. WORLD QUALITY 26 REPORT 2016-17 will require the introduction of new skills and extension of tests in virtualized environments. This is followed by: of Things testing. It is a concern that 30% building blocks for Internet of Things testing. protocols. One of these is of respondents with functionality claim to have no specific the use of crowdtesting.69. follows closely behind (on a scale of 1-7 with 7 being most challenging) to security these two leading challenges with a score of 4. While 68% without a test strategy for their Internet of Things functionality is high. However. and the thing itself. This is due to a To overcome this challenge. Testing the end-to-end user 2016 study participants give an equally high score of 4. putting it above to consider is that. . or test data. Other building blocks include structural IT system Security presents a challenge in the Internet of Things When asked about the key challenges with testing products in environment to a virtual one and how best to connect all this Internet of Things environment. communications bridge. versions. and quality monitoring scope than traditional software testing. Organizations must in production. exploratory testing. and low security maturity compared to traditional IT. This is because testing these smart. device data protocol and device simulators (47%). IP is sufficiently protected.67 score. At the top of the leader board. with domain. if security is not taken into account in the design planning and process. validation of communication the test environment. Interoperability testing is another Alternatively. made up of the application. scenario- connected solutions demands a broader and more complex based testing. The security challenge is user. also pay attention to defining a strategy for system integration testing and/or real-life testing. devices. integration testing with test devices. The challenge lies in defining the right Internet of Things layer.

It is also interesting to note the relatively low score given and the ability to allow for early real-life validation checks to protocol and device simulators (47%) in terms of the by launching beta-versions. in the development cycle. These two high scoring sectors place a big There is an even 42% average across the remaining three value on their customer satisfaction scores. thus a low solution and APIs lies in the potential for moving testing much earlier opportunity score. a shift from testing with domain knowledge to testing the different speeds at which a product (the thing) and its with test craftsmanship also receives an average of 42% associated software are developed. be integrated. One of the key challenge here lies in Thirdly. both Energy & Utilities machine learning to test to the best extent”. which is possibly statements regarding their ideas on Internet of Things why they believe testing the experience should be a priority. Internet of Things developments. and this The high ranking given to the opportunity to develop testing lack of standardization means there is insufficient breadth solutions for the virtualization of Internet of Things services and depth for customization purposes. so there is a need to bring the two together machine learning will increase for testing and validation of at testing stage. device Virtualization of IoT data recorders and players) Tools for stress services and APIs testing IoT middleware and gateways Protocol and Device simulators The popularity of Operational Intelligence aligns with the or the testing of the end-product in which the device will start-up mentality in Internet of Things product development. requiring a form of artificial intelligence and characterizes testing in this area. This is borne out by the 46% of our survey participants agreeing with the idea that Secondly. the difficulty with Internet of Things testing lies in the things part. testing. and Financial Services come in above average. at 53% and 52% respectively. Firstly. Intelligence in testing: the use of real-time data collection to analyze and predict potential failures before they occur. it is possible to more easily test scenarios in There are two potential testing opportunities for Operational early stage development. although there is a marked difference in . There is currently no generic Internet of Things protocol as one standard. in response to the idea that less functional “Internet of Things solutions have an unlimited amount of testing and more Internet of Things experience testing situations to test. Software development in this topic area. Current Trends in Quality Assurance & Testing 27 Figure 14 Opportunities for solutions for IoT testing 2016 54% 51% 50% 47% Operational Intelligence (improved diagnostics. By virtualizing device test-beds Artificial intelligence will be part of future Internet of Things testing It is evident that the adoption of artificial intelligence and is much faster. opportunities for testing solutions.

rate security as important or very important. thus this is not for testing Internet of Things functionality. 65% of respondents vulnerable than ever. towards companies relying on their product domain experts. This is companies in this sector tend to be more conservative and likely to be because domain expertise is no longer sufficient work from domain knowledge as a base. For the rest. which requires a an unexpected result. performed security testing activities over the past twelve Industry reports suggest that 84% of breaches occur in months. Retail Transportation Energy. with the escalating adoption of security overall is the most important aspect of IT strategy cloud services. Utilities Financial Services industry Manufacturing HI Tech including hardware Healthcare & Life Sciences Media Automotive Public Sector/ Government Protocol and Device simulators Virtualization of IoT services and APIs Tools for stress testing IOT middleware and gateways Operational Intelligence (improved diagnostics. mobile applications and Internet of Things by a long measure. Figure 15 Opportunities for IoT testing across industries 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% Total across industries Consumer Products. WORLD QUALITY 28 REPORT 2016-17 the Energy & Utilities industry’s score (19%).67 on a scale of 1 to 7 (with capabilities. with the next closest priority of higher quality software solutions coming There have been a number of changes in the most commonly in at more than 20% lower (44%). It scores 5. Historically. Telecommunications. device data recorders and players) Security stands out more clearly as the number one IT priority This year’s World Quality Report study reveals that enhancing the applications layer and. there is clearly a move towards test craftsmanship. This comes as no surprise. . Dynamic application security testing (DAST) testing. Indeed. while there is still a bias more theoretical and broader knowledge of testing. today’s digital enterprise is potentially more 7 being the most important).

respondents (up from 34% in 2015) is the use of an external . Just 1% of participants claim to carry out no to 51% this year. Both DAST and static application security kept back for this. result of the increasing uptake of agile. from 52% from 57% to 49%. Similarly. internal security testing teams application security testing. while penetration testing is up from 39% in 2015 to 49% in 2016. production. We view this as likely to be the to 43%. compared with 17% last year. These promote rapid The logical conclusion for the drop in DAST testing is that development. although occasionally afterwards. which potentially leaves less time for manual some companies are buying in more expertise rather than code review. has dropped Manual code review is also slightly down. DevOps. yet hackers are human and report we suggested that penetration testing still had room look for increasingly smart ways to attack applications and to grow and it is clear that this growth is underway. using pay-per-use tools has increased as an approach from which demonstrates that security testing is no longer viewed 32% last year to 51% this year. preferably before the application is in testing (SAST). at 50% of as an option. Thus expert reviews and penetration testing in combination with automated SAST and DAST affords a greater level of security. and continuous integration methodologies. but is part of strategic intent. In last year’s are very much automated. it seems likely that time is being this requirement. penetration testing meets of the development cycle. which is performed by 50% of respondents. Increase in contributors to security testing The type of contributors to security testing has also seen The level at which managed application security testing big changes this year. with significant increases in all but services are performed has increased from just 35% in 2015 one area. gain access to enterprise assets. As penetration testing comes towards the end relying too heavily on automation. Close behind. Current Trends in Quality Assurance & Testing 29 Figure 16 Commonly performed security testing activities 2016 2015 60 57% 52% 52% 50% 49% 49% 43% 39% 40 20 0 Static application Dynamic application Manual code review Penetration Testing security testing security testing where code running on the server is checked.

prefer to rely on specialized external teams. has nonetheless significantly grown volumes. More respondents actively that security is increasingly being viewed as an important participate in it on average during the QA phase (29%) than aspect of an application. The use of an internal security difficult to hire and keep properly qualified resources to testing team with its own tools is the one area that remains provide application security testing. It is positive to see the percentage of security assurance activities carried out at the earlier requirements definition . delays and poor security. It is clear from these findings that the increasing workload as a result of spiraling application development has made external Figure 17 Contributors performing security testing 2016 2015 51% Managed application security testing services 35% 51% Internal security testing team with pay-per-use tools 32% External security testing team on a 50% project-by-project basis 34% 50% Internal security testing team with own tools 51% 46% Staff augmentation (time and material) 30% 1% No application security testing 17% 0% 10% 20% 30% 40% 50% 60% Late stage testing continues to dominate While security assurance activities are undertaken at all stage because this moves it up in priority from the least stages in the application lifecycle. once again there is a used phase (alongside design) in the 2015 study. This takes QA above the we know that security by design is the only way to ensure development and coding phase (25%). not just an afterthought. at any other time in the lifecycle. In addition. while security requirements definition phase (25%). more and more of them almost static this year at 50%. WORLD QUALITY 30 REPORT 2016-17 security testing team on a project-by-project basis. many organizations find it extremely in use from 30% in 2015. and above the that protection is robust against most threats. Even support in one form or another a growing requirement as staff augmentation (time and material). This shows preference for late-stage testing. after design is likely to result in additional costs. although not the most internal security teams become overwhelmed with testing favored approach (46%). Indeed.

for example in the requirement phase. As this exciting technology the approaches used for validation and security checks are development grows exponentially. This will move security testing from the end of the development lifecycle to security engineering in all stages. But the impact of the Internet of Things on testing is the development lifecycle. few. and compatibility. Different types of test focus areas are needed for the end-to-end validation of an Internet The proliferation of interconnected devices and the collection. we also believe that security awareness and the responsibility for assuring security will now be assumed by all team members. There them. With the growing adoption of agile and DevOps. Current Trends in Quality Assurance & Testing 31 Figure 18 Application security assurance activities 2016 35 30 29% 25 25% 25% 20 15 10 11% 5 10% 0 Quality Assurance Development/Coding Requirements Design Deployment Staging definition Production Bringing mainstream security and Internet of Things together in future QA and Testing activity Overall. an increasing call to security possible. to name just a driven security to the top of the CXO agenda. far broader than security alone. interoperability. Nonetheless. must be taken into account. of Things solution. For example. analysis. even if end- testing specialists to support and augment in-house testing user experience or time-to-market are bigger drivers for the capabilities. are clear trends: a steady increase in systematic testing security by design assumes that security is tested as early as of all applications/releases. not yet mature and require closer scrutiny. and sharing of personal data and images have efficiency. in all sectors and all geographies. performance. and more security in the requirements stage of projects. testing must keep up. In the quest to improve Internet of Things security. this year’s study findings are strong confirmation Connected products and the Internet of Things impact the that enhancing security is by far the most important aspect physical world. as would a security breach associated with of IT strategies. usability. .

Other agile development methods in use are: agile methods: Scrum and Scaled Agile framework (SAFe) XP. and to focus on single projects. program adopt agile development methods. Looking ahead. and from It is clear from our survey findings that organizations are a focus on test execution tools to strategy capabilities. but in business professionals must adopt integrated quality engagement understanding. Financial Services SBU. budgeting and multiple preferences in terms of the percentage of projects adopting practices. used in 17% of projects. The most used of these is to work in a business outcomes and customer value. With this change comes the need for agile testing expertise to move away from execution to transformation. development and new tool technologies models. Lean Software Development are both used in 24% of projects. Quality Assurance and Testing All of this requires skills. Principal. QA and Testing organizations becoming more informed about various experts should be fully integrated with the agile teams. There are three clear management. and Kanban (15%). To address this. WORLD QUALITY 32 REPORT 2016-17 Agile and DevOps More intelligence-led testing will meet speed. They DevOps levers. We increasingly struggling to match the right level of quality are seeing a similar maturing of DevOps practices with with the demand for speed. such testing in agile projects. not just in testing. adopted by 42% of those using tools. no longer enough. and make use of intelligent tools as vital enablers. With a total in excess of 100%. quality and acceptance criteria that can be automated. data analysis to form test strategies and create automated optimized validation checks and test sets. once again. up significantly from last year’s 24%. Capgemini This year’s World Quality Report marks a new impetus for right test focus areas and creating nimble processes. must take a business-centric approach to choosing the Multiple hybrid agile frameworks shape the agile testing future Just 1% of this year’s study participants say they do not addresses large-scale portfolio management. and the use of intelligent as well. quality and cost imperatives Deepika Mamnani. The increased demands for what as such as behavior driven development (BDD) to define appear to be opposing objectives of speed. rather than on distributed agile model. release management. They should also cost require organizations to rethink their quality validation leverage techniques such as mind maps and model-based and testing activities. SAFe has these figures demonstrate a mix of multiple methods within gained popularity in all sectors. while Dynamic Systems (16%). of testing approaches and. due to the fact that it different projects. Standard automation of test cases is tests. Instead. Those study participants using agile adopt a range agile. it is clear that a distributed agile model will be critical to successful agile . a business outcome focus. Development Method (DSDM) is used in 22%. Distributed agile model increases in use The key themes for testing in agile this year are to establish multiple guidelines and frameworks are combined within distributed agile and test approaches.

budgeting. and demand management will be required to balance core In agile development. both development method is cited by 43% of respondents this cited by 44% of the 2016 study respondents. up from 31% last year. The late involvement of testing teams has as a challenge is a lack of professional expertise in agile ramifications. with less time for the in-depth analysis on which teams. The agile TCOE. The top two Lack of a good testing aproach that fits with the agile challenges in applying testing to agile development. Figure 19 Challenges faced in applying testing to agile development 2016 2015 Early involvement of testing team in inception 44% phase or sprint planning Difficulty in identifying the right areas 44% on which test should focus 33% 43% Lack of appropriate test environment and data 31% Lack of a good testing approach that fits 43% with the agile development method 29% 43% Lack of professional test expertise in agile teams 31% 41% Inability to apply test automation at appropriate levels 23% Difficulty to re-use and repeat tests 40% across sprints/iterations 29% 1% No real difficulties with testing in agile 18% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% . This reflects a shift in behavior the right focus areas for testing. offshore. close and immediate collaboration of and flexible teams and drive economies of scale. disciplines is needed and this can conflict with the objective to optimize testing costs. This creates difficulties with an increase in the challenges encountered with testing in with allocating the appropriate level of test types and test agile. Familiar challenges in higher proportion The increasing adoption of agile development has been met test strategy decisions are made. 18% of respondents to this question said they depth. these challenges are heightened. to offer a more optimum mix of onshore. We technologies. As a consequence. and set up of the can see that as agile becomes enterprise wide. Last year. portfolio management. needs In setting up an agile Testing Center of Excellence (TCOE). Also cited by 43% of participants planning phases. Current Trends in Quality Assurance & Testing 33 development projects due to the ability to include low cost. agile agile cite this as a challenge due to its potential to deny program. insufficient throughput of new functionality. the selection of automation had no real difficulties. challenge. up from 29% in 2015. are: identifying year. and poor containment of overall IT costs. and data challenges for testing exponentially increase. With quality demands now taking on a more strategic character. up from 33% in 2015. To this end. therefore. near shore options. and where testing is being viewed as the biggest bottleneck in the early involvement of the testing team in inception and sprint delivery of agile projects. 46% of respondents whose organizations use same time zone. contrasting with just 1% this year. governance cost savings through globally distributed delivery models. leading to quality issues in software. preparation of test data. and near the need to co-locate teams is viewed as the biggest shore teams to balance both cost and flexibility. environment test environment do not receive the consideration needed.

WORLD QUALITY 34 REPORT 2016-17 Skills shortage must be addressed to avoid bottlenecks The shortage of testing skills in all stages of agile projects levers for agile initiatives. and virtualization skills to keep up with delivering shippable increments at an ever faster pace. test data. software development. In last year’s study some DevOps. virtualization skills (21%). understanding of business process (20%). with an average project to predict the quality impacts of new functionality plans. agile containing at least 15-20 Scrum teams at a minimum backlogs. QA transformation and strategy consulting capabilities are also needed to define the appropriate speed-to-test DevOps matures with a new appreciation of its purpose The use of DevOps principles has continued to grow. by a lack of: non-functional skills (21%). and to an earlier stage in the development process. production quality monitoring skills. test data setup skills (20%). 26% of respondents said they used DevOps principles in a Figure 20 Projects using DevOps principles 40 2016 2015 32% 32% 30 26% 21% 18% 17% 20 16% 12% 11% 7% 8% 10 0 Not applicable as Less than 20% of 20 . This is followed coding skills (both 19%). with 40% of respondents saying they use them in 12% of CIO respondents this year say they are not using more than 50% of their projects. Test Driven Development (TDD) or Behavior Driven One other challenge should be noted as testing shifts left Development (BDD) (20%). An average 22% of respondents state that they development engineer testing (SDET) and development and need to build test strategy and design skills. as follows: TDD or BDD (20%). This requires test environment.50% of 50-70% of 70-90% of 90-95% of we do not use our projects use our projects use our projects use our projects use our projects use DevOps principles DevOps principles DevOps principles DevOps principles DevOps principles DevOps principles . on data facts. Just projects. on average. Our study participants also noted has to be addressed to avoid testing being a bottleneck in other skills gaps. compared with 18% in 2015. Bottlenecks predictive analytics skills (20%). open defects. test environment and functional test automation expertise (both 18%). and production incidents based becoming the new norm. This latter skillset will be around both environment and data arise as a result of the needed in a future intelligence-led QA and Testing landscape adoption of agile in the enterprise. being applied in 38% of although it is still very much in an emerging stage. DevOps principles are.

used by 51% of respondents. That number has dropped in organizations were eagerly jumping on the hype of DevOps 2016. which is zero-touch testing in a true integration of build and integration of code with unit testing. popularity. and more cloud-based test environments. the delivery pipeline (83%). Last year they use DevOps in 20-70% of their projects. TDD. . That the use Shifting testing left and right in combination is the way ahead It is apparent that a combination of test approaches is used Testing is also moving right towards later stages. This has dropped to 38% and been overtaken by several other practices. Continuous testing possible. with in a DevOps environment. This Use of loosely coupled architecture and microservices has demonstrates recognition that DevOps is about more than increased this year to 41%. which early on in the lifecycle. Current Trends in Quality Assurance & Testing 35 high 70-90% of their projects. This is made possible by increasing stands at 86%. of more further mandates the ability to spin up multiple environments relevance is the combined current and planned use. continuous testing. These shifting patterns are a sign of more to use both continuous monitoring with predictive analytics strategic thinking and approaches to DevOps. with 64% of respondents claiming and a better understanding of what DevOps is. As a result. compared with 11% of Today there is better recognition that adopting DevOps projects last year. and continuous integration techniques. BDD that we believe will characterize the future testing landscape. combined with the figure for those planning to use it. There is a shift in the various DevOps showing that those already using and those planning to practices used this year. of work. Leveraging predictive analytics to bring speed has gone down this year from 50% in 2015 to 41%. functional and non-functional tests. and treating infrastructure as This is confirmed by the fact that 42% of respondents claim a code (81%). Among the success factors are the ability the need for big up-front design and enables the release to decide on the right things to test in the shortest time of the software early and continuously. this leaps to 88%. Last year’s most used practice was use cloud-based test environments rise to 86% and 84% to break down large development efforts into smaller batches respectively when combined as datapoints. This This is a promising sign of testing maturing as it shifts left in is again indicative of the growth in intelligence-led testing the DevOps process with service virtualization. Multiple approaches are used to achieve quality in DevOps Almost all organizations embarking on DevOps initiatives use of these levers will grow is evident from the study finding some flavor of agile. but efficiencies in all areas of quality decisions is one of the we believe this is due to a better understanding of true major emerging opportunities. The use of predictive analytics production monitoring and the provision of predictive to determine or optimize test coverage emerges alongside analytics feedback into development and testing gaining TDD and/or BDD as the two most used approaches (45%). down to just 8%. There is no usage in the extremely but without fully understanding its real impact and application. Again. implies a complete change of culture with the breakdown of barriers between disciplines. the averages are We interpret this significant drop to be proof of realism now shifting to the center. up from 31% in 2015. This is not surprising in light of the fact that Speed-to-market is particularly critical for the development adopting flexible and service-based architecture reduces of digital platforms. high 90-95% category this year. When just cutting up large releases in smaller batches. This is also high for the practice of automating the use of cloud-based test environments with virtualization. The drive to increase speed of testing in agile and DevOps BDD. along with automated test data generation.

but overall. but asking the right questions to guide the low average score for these two skillsets is questionable. Understandably. Thus we would argue needed from QA and Testing professionals. WORLD QUALITY 36 REPORT 2016-17 Figure 21 Approaches to testing in agile in your organization 2016 2015 We use predictive analytics to determine 45% or optimize test coverage We use Test Driven Development (TDD) and or 45% Behaviour Driven Development (BDD) 48% We use continuous quality monitoring 44% on IT systems in production 43% We use automated test data generation 43% We use service virtualization of test environments 51% 42% We use automated provisioning of test environments 43% We use test automation integrated with 42% continuous integration (CI) techniques 47% 0% 20% 40% 60% The biggest change in QA and Testing in the transition to DevOps is the human mind-set change. although with the inclusion of more predictive analytics to help there is a marked difference in this across our respondent turn data-led insights into a better customer experience base. there is a fairly consistent level of satisfaction (42%-50%) where skills Focusing on enhancing the mix of expertise in agile-DevOps are deemed to need no change.and historic than any other skillset (average 37% of respondents). Both test data setup expertise and test environment and improved business outcomes. commented on the very real skills gap. however. right decisions will become more important than ever. as proven . We have already that these two areas should be in higher demand. QA/Test Manager. Consumer Products. However. it is clear that teams will be a critical success factor in future. this and virtualization expertise are seen as less relevant today intelligence-led environment will be heavily tool. Germany Changing skillsets catering to a new world This year’s study participants were asked to comment on by the fact that the environment and data issue scores among the extent to which agile and DevOps changed the skills they the top challenges with testing in agile. The data-enabled. Agile and DevOps are highly demanding. for example some are considered less relevant than others.

The way forward is to find a better flexible teams to drive economies of scale. continuous testing and continuous quality monitoring. domain experts. The smart ecosystems capable of using predictive analytics and solution here is to build multi-skilled agile testing expertise automated decision making to achieve the two objectives comprising transformation experts. we infer that for an ideal agile and DevOps transformation and enable all testing phases by providing engagement. One option is to create a centralized Quality Management Office to drive Finally. strategy of speed and quality. and. Given the complexity of all of this working together in an agile The basic skills mismatch discussed earlier also stands in the environment. specialized test experts and data scientists. These on speed in agile and DevOps is causing increasing skills should be balanced in a combination of core and challenges with quality. balance between this speed and quality. we recommend that organizations invest in way of achieving a balance between velocity and quality. tools. SDETs. test data .No change needed Skills are lacking and required more Steps to take agile and DevOps deeper into the enterprise The survey clearly demonstrates that the primary focus experts. environment virtualization. test flexible (loosely coupled) and cloud-based. consultants. security) Predictive analysis skills TDD (Test Driven Development) or BDD (Behaviour Driven Development) Skills are less relevant Skills are OK. of course. Current Trends in Quality Assurance & Testing 37 Figure 22 Skills needed from QA and Test professional for agile and DevOps 100 20% 21% 22% 20% 18% 19% 18% 19% 21% 20% 20% 75 43% 42% 42% 44% 47% 46% 48% 47% 46% 47% 50% 50 25 37% 37% 36% 36% 35% 35% 34% 34% 33% 33% 30% 0 Test data set up expertise Test environment and virtualization expertise Test strategy and test design skills Understanding of Business Processes Production quality monitoring skills Software development engineer testing skills (SDET) Functional test automation expertise Development and coding skills Non-functional testing skills (performance. organizations need test environments that are the right set of nimble processes. which support competencies for all projects.

In While centralized QA and Testing operations still remain a fact. This typically comes from large Lines of Business (LoBs) in which were originally set up to industrialize all test operations enterprises and from the adoption of agile/digital practices. there is a reduction in traditional back-end system Setting up test operations leans towards flexible solutions The key conclusion to infer from Figure 23 is that the primary accessible TCOE models. With increased digital and DevOps survey data does not give any evidence for this. which was pushed further with the QA needs between the worlds of core IT and the business growing adoption of agile and DevOps. we predict that at least 90% of enterprises the traditional engagement model of staged disciplines will adopt bi-modal development by 2018. This decentralization IT landscape of flexible customer-facing applications. that work sequentially (business analysis. Now. agile and DevOps is acknowledge the emerging needs of a decentralized model. organizations of operational test factory models and captive TCOEs is still seek managed service models from expert QA vendors. amid this shift. decentralized TCOE. WORLD QUALITY 38 REPORT 2016-17 Industrialization and Testing Centers of Excellence Reflecting the shift to an intelligence-led test approach Prabakaran Karuppiah. with its incumbent overhead is trailing behind other delivery options. does not negate the need for a core central function. bi-modal. focus on achieving maximum cost efficiency in the form It is interesting to note that. the hybrid TCOE has a thin central enablement layer predominant theme. Vice President. nimble and easily . or at least an increase in of core IT and business IT on the modern TCOE. the TCOE has to adapt to facilitate the changing Evidence shows that preference for a full centralized TCOE QA demands. Here. Capgemini The adoption of digital transformation. with cost efficiency as the primary focus. forcing a change in Testing Centers of Excellence (TCOEs). design. correlation analysis on the rather than just testers. This is The response to this year’s World Quality Report survey consistent with last year’s three-year trend away from central shows increased maturity and balance in fulfilling the differing to decentralized delivery. Indeed. production) co-exist in the TCOE with the Interestingly we assumed that the cost of QA and Testing agile development model that sees these divisions merging. giving way to the demand for more flexible. to an independent third party. although the difference is not significant. build. more than ever. either in-house or through outsourcing providing QA standardization and governance across units. there is enough data to Traditional and agile work alongside each other in bi-modal development We observe an increase in the impact of the bi-modal world development and testing methods. such as crowdsourcing options. QA and Testing. would be lower in a centralized TCOE compared to a with the testing activity carried out by multiple disciplines. adoption. However. Financial Services SBU.

of course. This is especially so where a Line of Business testers which. South Korea Changing skills and new challenges for agile TCOE setup The make-up of an agile TCOE in terms of skillsets shows a This changing make-up is in line with greater use of the hybrid number of changes in the roles shared across multiple Scrum model and a decentralized TCOE where a wider range of teams and those that are localized within individual Scrum skillsets are localized in Scrum teams as and when they’re teams. a hybrid TCOE factory model (45%) and a TCOE in the use of a crowdsourcing solution model for quick an offshore location (42%) Figure 23 Important elements in setting up your test operations 2016 Use a managed service model with 49% expert QA vendor 2015 Use crowd-sourcing solution model 47% for quick test capacity Decentralize TCOE for improved 47% agility and efficiency 65% Captive TCOE factory in a 45% build-operate-transfer model 60% TCOE factory in a hybrid (near shore/offshore model) 45% with optimum cost savings and high speed to market 63% TCOE factory in an offshore location 42% with high cost savings 56% 0% 20% 40% 60% 80% The most important opportunity for using intelligent data analysis in testing is to decide on our strategy based on the data analysis. This moves the specialist tester above last year’s is limited overhead in getting tasks executed. rather than a single large TCOE that doesn’t have to 40% in the 2016 survey responses. with tools that help to visualize the data. software for flexibility is driving the transition to clusters of smaller development engineer tester (SDET). Current Trends in Quality Assurance & Testing 39 In the setup of test operations. from being the least shared role across needs to get critical features to market quickly. Skills are multiple Scrum teams (38%) in 2015. is now the most shared directly accessible and available within the teams and there (41%). which drops from 43% TCOEs. alongside reveal a significant interest in crowdsourcing. The reverse of this. Financial Services. The key change concerns specialist technology needed. While a a decentralized TCOE for improved agility and efficiency. The need most shared role across multiple Scrum teams. this year’s survey findings test capacity is used by 47% of respondents. managed service model provided by an expert QA vendor This puts crowdsourcing ahead of a captive TCOE factory ranks the highest in importance (49% of respondents). IT Director. (45%). is that with . this flexibility.

Ireland Challenges in setting up an agile TCOE Setting up an agile TCOE brings a number of challenges. This year’s survey data leads us to conclude that there is a whole new set of special skills expected from the test The number one challenge is finding the skilled people. and for enabling development platforms. Pharmaceuticals. Figure 24 Challenges in setting up the Agile TCOE 2016 2015 Necessity to co-locate teams. but also demands retaining a central corporate TCOE layer that is responsible hands-on expertise with various modern and DevOps for standards. It is not only about deep test redundancy and inconsistency of methods and tools when automation skills and specialized non-functional test skills. with the exception of together. test guidelines and test platforms. some organizations are or domain and technical tester skills. denying cost savings 46% through globally distributed delivery models 41% Independent Testing Vendors lack of willingness toward 45% 'pair programming' with development teams 36% Agile being used as an excuse to not develop 44% documentation resulting in low quality applications 45% Emergence of new testing tools which 43% needs development skill sets Traditional test design methods used that 43% are unsuitable for agile delivery models 42% 42% Testing teams not being as agile as development teams 42% 0% 20% 40% 60% . WORLD QUALITY 40 REPORT 2016-17 the breaking up of the central TCOE there is a risk of skills experts of today and tomorrow. yet brings with it the challenge when setting up an agile TCOE is the necessity to issues of sub-optimal costs and the potential challenge that co-locate teams. As such. analytics leveraging best practices. and cloud. This has grown as a challenge When compared with the 2015 survey findings. the top two challenges. including digital. IT Director. distributed delivery models. The most encountered (or expected) Co-location is a major enabler of this. the World from 41% in 2015 to 46% this year. simultaneously in a highly collaborative mode. largely because agile and Quality Report 2016 results for the type of challenge DevOps approaches demand multi-skilled teams working encountered are relatively unchanged. we look across LoBs. which denies cost savings through globally skills are not always locally available.

tight time constraints. Significant drop in test automation after last year’s increase Test automation is a vital contributor to testing efficiency. one to write and one to true essence of agile at all times. quality can be managed with greater visibility through the use of transparent QA and value dashboarding. low automation ROI. Figure 25 Test Automation Challenges 2016 We don't have the right automation tools 45% Challenge with the Test Data and Environment Availability 40% Delivery methodology doesn’t support test automation 35% We have difficulties to integrate the different 37% automation tools together Application functionality changes too frequently 36% We don't have the right automation testing process/method 35% Lack of stability of test script 34% Lack of skilled and experience test automation resources 34% We have difficulties to automate because of use 32% of multiple development lifecycles 0% 20% 40% 60% . It is all too easy to place quality at can create difficulties with co-locating the validation and the bottom of the priorities pile and our recommendation is to verification team with development teams. This is especially embrace a clear standard methodology enterprise wide. Another cause is a short-term focus maintenance. To enable this. that the case in heavily low-cost driven contracts where there is puts quality alongside the pursuit of speed and low costs. automation projects fail or don’t result in the anticipated Return on Investment. rather than long-term strategic planning activities are automated. Current Trends in Quality Assurance & Testing 41 The second most perceived challenge. which is the fragile balance review code. this means that from 36% last year to 45%. is up even more. no clear agreed co-location demand upfront. a high percentage of and maintenance. There are a number of reasons for this. that of the lack of The third most important challenge in setting up an agile willingness by independent testing vendors to participate in TCOE is that of agile being used as an excuse to avoid pair programming with development teams. developing documentation. Pair programming typically brings organizations are not necessarily taking into account the together two developers in one place. quality and cost. We also see other challenges standing in the way of beginning with a lack of central coordination and guidelines increased automation. However. reliance on independent QA partners of speed. At 44%. such as: brittle scripts with high on tools and their usage. on automating what is required to meet current project and/ yet this year’s study participants claim just 29% of testing or release needs. Further.

Putting a central team cycle. service and execution in the development cycle. with significant cycle the test process. along with the means to measure the level time reduction. A reduction of test cycle time This is followed by better reuse of test cases (40%). Zero touch testing and execution. scriptless automation. as well as new test and automation testing. such as progressive automation. which again ties better control and transparency of test activities (38%). and life cycle automation for vastly improved outcomes. to compare data and results. and high license costs with low utilization and ROI. Other changes industry seeking engineers capable of moving beyond single to bring on board are the use of continuous integration competencies to be able to carry out manual testing. Figure 26 Test Automation Benefits 2016 2015 Benefit-value 2016 19% 21% 20% 22% 20% 18% % 80 72% 70% 69% 66% 67% 60 42% 40% 40% 39% 39% 38% 40 20 0 Better detection Better Test Better reuse Reduction of Reduction of Better control and of defects Coverage of Test Cases Test Cycle Time Test Costs transparency of test activities . in place will ensure best test automation practices and new A need for multi-skilling also presents a challenge. Those in with the overarching QA and Testing objectives. Benefits of test automation re-inforce desired outcomes of QA and Testing The most realized benefit of test automation is better were additionally asked to estimate the percentage of detection of defects. and (22% level of benefit) scored most highly. have the potential to deliver selection of a basic tool strategy and guidelines as part of 20-30% reduction in cost of quality. cited by 42% of the study respondents. WORLD QUALITY 42 REPORT 2016-17 heterogeneous tools and frameworks across the testing and effects of increased automation. including respondents claiming to have benefited from test automation contributing to business growth and outcomes. improvement experienced. and writing some code or tool strategies and solutions. with the technology developments can be leveraged. and Getting to grips with all of these challenges demands the automation as a utility service.

Current Trends in Quality Assurance & Testing 43 Robotics and machine-based learning indicate a shift to intelligence-led QA and Testing As Digital Transformation initiatives continue to roll out across Intelligent and smart bots can be self-aware. in reality. both robotics automation and test design automation were The high percentage for robotics automation is. helping to significantly participants which emerging automation techniques they lower operating costs with increased business agility. going beyond regression testing across QA and Testing to the next stage beyond mere to user acceptance testing and even progression testing. So it is not surprising that critically change test automation over the coming years. Figure 27 also illustrates business focused QA also demands higher levels of self. It industrialization towards an intelligence-led approach with makes sense for the world of core QA striving to reach the predictive analytics. self-learning technology will drive analytics and artificial intelligence. They can interact efficient. At the same time. foresaw using in the coming year. Intelligent. the modern IT landscape with its currently undertaken by humans. the potential uptake of other emerging techniques that will aware test automation solutions. once they’ve been shown what to do. It will become far easier to identify what to test and where Essentially. the top two techniques. cognitive automation is equally on the uptake. data-led testing. Robotics will bring the head count down next frontier of automation: a software defined self-learning in the QA and Testing function as these machines take on the test organization that is even less dependent on human more repetitive analysis and execution tasks and routine jobs effort. it marks an unmistakable movement process intensive QA world. At 42% of responses. rather than already fully Robotic automation has wide-scale applicability in the utilized. self-monitored. processing and interpreting the huge Robotics automation sees configurable software being set volumes of data that it is not humanly possible to analyze. and application agnostic. the World Quality Report 2016 survey asked cognitive. the robots more testing is required to bring down costs with more (‘bots’) can be left to get on with the work. all sectors. Figure 27 Emerging Automation Techniques 50 2016 42% 42% 41% 40% 40% 40% 40% 39% 40 30 20 10 0 Robotics Test Design Cognitive Test Data Machine Predictive Self-remediation Test Automation Automation Automation Automation learning analysis Environment Virtualization . with the system or application the same way a human does. more a reflection that this technique is so promising that it will be explored in the coming year. up to perform the tasks a human assigns and controls. Nonetheless.

with a view to do more for less. there is data-rich digital applications. there is a wave of script automation to scriptless automation and on to robotic existing outsourced and offshored traditional QA supported automation in the legacy world. There is an evolving trend by automation that is moving to higher levels of automation to higher levels of intelligent automation. analytics-driven automation. Banking. First. moving from a basic industrialization at two different levels. This is the future of testing. The biggest challenge in testing is identifying the right risks and identifying the measures to take for testing to cover those risks. All of these trends open up an a newer need from agile QA for initial outsourcing. 38% performing most test activities 0% 10% 20% 30% 40% 50% An intelligence-led future characterized by agile and DevOps We can see that testing is embarking on a new era of Automation as a science is also evolving. QA & Test Manager. self-remediating. largely driven by via robotics. Netherlands . Second. offshoring exciting and intriguing world of self-aware. and automation. WORLD QUALITY 44 REPORT 2016-17 Figure 28 Statement best reflecting your company’s testing activities 2016 We have a de-centralized Agile TCOE: decentralized test 42% teams that deliver test experts to agile teams We do not have a form of organization or governance 41% specific for the test function We have a hybrid TCOE model. and it is beginning right now. with independent test team per LoB 41% and a small central team for overall quality governance We have a central agile TCOE: a corporate function 40% that execute tests on agile delivered software We have outsourced most testing activities 40% to an independent QA partner We have multiple independently operating test 39% teams per Line of Business (LoB) We have outsourced most testing 39% activities to software vendors We have a central TCOE (Test Center of Excellence).

all of which demand level of speed and flexibility. for which testing is commonly confined equipped and able to adopt agile and DevOps practices at to this environment. Those organizations that invest in enhancing their is typical of the systems-of-record that form the backbone test environments. different types of test environment that have the potential for Figure 29 Test Environment Types 40 2016 28% 27% 24% 22% Mean percentage 20 10 0 % of our testing occurs in a % of our testing occurs % of our testing occurs in a % of our testing occurs in a permanent test environment in a cloud-based temporary virtualized test environment temporary test environment test environments that are not cloud-based . and testing in a temporary of testing. Current Trends in Quality Assurance & Testing 45 Test Environments and Test Data Management Key-enablers for quality in agile and DevOps AJay S Walgude. Some 28% of testing management (TDM) need to evolve as DevOps drives overall occurs in a permanent test environment. closely further virtualization and Digital Transformation makes itself followed by 27% in a cloud-based temporary environment. Financial Services SBU. it will be another So despite the relentless move to cloud. while virtualization enables easier and faster set-up a relatively even spread of testing in different types of of integrated system test environments. Capgemini Test environment management (TEM) and test data enhancement in one way or another. This is a reflection of the variety of systems major pre-requisites for an efficient test solution with a high in operation across IT landscapes today. Overall there is an increasing trend for enhancing and Just a few percentage points behind are: testing in a increasing the flexibility of test environments across all areas virtualized test environment (24%). we see that the use of cloud the level needed to secure business outcomes and drive and virtualized test environments in combination cover 50% customer value. This is re-assuring because cloud enables test environments to be spun up or deleted at an individual The World Quality Report 2016 study participants record level. are better of large enterprises. for example by adopting cloud. There is. although there are signs of increasing maturity in this skills area. This profession. of all test activities. felt. however. at a broader IT industry level and. Positively. a major part (28%) year or two before it takes on the mantle of a fully-fledged of testing still relies on a permanent test environment. This is one of the environments. still a lack of skillsets in TEM test environment that is not cloud based (22%). Vice President.

across multiple browsers and devices. organizations want to test these applications security testing (43%). this percentage could be higher because cloud-based We expected this trend to be more pronounced in the North environment solutions would be ideal for this type of testing America region and in Financial Services because of their in order to quickly generate an envionment that is a close initial enthusiasm and dependency on CSD applications. ERP and financial systems. The least testing of portals and environments is also understandable. but place on the cloud too.) wearables solutions . ERP. up a reflection of the increased use of cloud-based analytics from 44% in 2015. performed in the cloud are performance testing (44%) and Typically. solutions as portals and and Financial mobile device and Systems etc. the Surprisingly. Figure 30 Percentage of testing performed in a Cloud-based test environments 80 2016 2015 64% 60 50% 50% 48% 47% 44% 46% 44% 45% 43% 43% 42% 40 20 0 Functional testing of Functional testing Functional testing of Functional testing Performance testing Security Testing Business Intelligence/ of cloud services customer facing of core enterprise Business Analytics applications such packages (CRM. We believe that show a dip from their initial enthusiasm for cloud adoption. we note that the dip in cloud adoption is almost environments in the coming years. Last year’s most performed testing type solutions. However. copy of production. The high proportion of functional this is down from 64% in 2015 to 50% this year. 50% of respondents carry out functional testing of Business The high level for functional testing of BI/BA solutions is Intelligence (BI) and Business Analytics (BA) solutions. the percentage of perfomance tests on cloud- legacy and custom software development (CSD) applications based test environments is relatively low. Of the testing that is uniform (50%) across all sectors and all geographies. up from 43% in 2015 to 46% in 2016. currently performed in a cloud-based environment. the testing takes in the cloud was functional testing of cloud services. such as CRM. WORLD QUALITY 46 REPORT 2016-17 Cloud-based testing used across all applications Our key prediction is for greater use of cloud-based test However. for which maintaining physical environments and providing continuous upgrades We are also seeing an increase in cloud adoption for testing would be costly. As these systems run in the cloud. core enterprise packages.

claimed by 48% of with some interesting new candidates. are. however. This is worsened by the fact that in terms of their provisioning. working stations. understanding of the environment challenges.) at the right time 32% Inability to manage excess needs for 45% test environments (lack of data/network/ service virtualization facilities) 0% 20% 40% 60% . Current Trends in Quality Assurance & Testing 47 Challenges increase as need for visibility grows This year’s World Quality Report study records an overall 48% of organizations saying each challenge is an issue. In most cases In this complex landscape. In the future. organizations are looking for the underlying challenge is a lack of ownership and clearly greater visibility and automation across test environments assigned responsibilities. storage. No one challenge stands A lack of visibility to test environment availability in terms of out. cited by 46% its share of test environments and tools. with all issues being more pronounced this year. Instead there is a fairly even range of 45% to utilization and demand is also cited by 46% of respondents. The increase in the level of challenges with provisioning test most cited of these is the difficulty with having to maintain environments to test teams compared with 2015. in fact. The problems study participants. This is highlighted by the increased adoption of agile and DevOps leads to more second most encountered challenge. of respondents. 27% databases etc. Figure 31 Challenges with Test Environment Provisioning 2016 2015 2014 48% Having to maintain multiple versions of Test Environments 44% 45% 46% Lack of facilities to book and manage your own environments 46% Lack of visibility to test environment availability (utilization and demands) 46% Lack of availability of right tools for testing 30% 54% Lack of availability of right test environment 46% (servers. and up from 44% last year. along multiple versions of test environments. fairly basic and reflect the relative immaturity of test environment management overall. we expect to see an increase in cloud-based environments that can be provisioned on a It is interesting to note a growing awareness and one-to-one basis for each individual wanting to run a test. the lack of facilities multiple parallel testing activities. networks. each of which requires to book and manage their own environments.

the of test data. The most cited challenge of creating and regulations for test data is a leading challenge. which limit the use of production data for test purposes. it has jumped again in the 2016 study findings with 46% of It also indicates a better appreciation of the potential cost respondents citing it as a challenge. both cited by 47% of respondents. but this requires a clear picture of the the abundance of tools for testing and the proliferation of utilization level so that organizations can make intelligent cloud-based access to these tools. Two data management concerns a much better understanding of the challenges. This is strange given saving opportunity. Similarly. We see. for in particular stand out for the size of the increase over example. As test sets proliferate across the enterprise. Test data management challenges increase as data grows exponentially The challenges with environments extend to the management purposes. with an increase in the level of challenges issue of test data has come of age and organizations have reported across the board. alongside the maintaining test data that are not copies from production need to maintain the right test dataset versions with different data has risen to 48% from 29% last year. but to meet the escalating needs of agile and DevOps cycles. that complying with data security and data privacy the past year. security regulations. The data several challenges cited by 47% of respondents is that of security challenge is up 9% on last year. providing a clear creating and retaining useful copies from production data. WORLD QUALITY 48 REPORT 2016-17 This reflects the increased difficulty with controlling and decisions about environment optimization. A lack of the optimizing the growing number of environments being used right tools for testing appeared to be reducing last year. one of test versions. a sign of more awareness and scrutiny of privacy and data also up from 29% last year. These two big increases are indicative of a lot more focus being given to creating and maintaining data for test Figure 32 Challenges in managed test data 2016 2015 2014 48% Creating and maintaining test data which are 29% not copies from production data 35% 47% Complying with data security and data 38% privacy regulations for test data 41% 47% Creating and retaining useful copies 29% from production data 37% Having to maintain the right test data set versions 47% 44% with different test versions 41% 46% Lack of test data for complex integration testing across 32% systems and organizations 32% 46% Managing the size of test data sets 40% 43% 0% 20% 40% 60% .

Test data generation tools are on the rise Figure 33 Test Data Generation technique 2016 2015 2014 2013 18% Use purchased. are serious bottlenecks for larger groups of systems. This significant rise reflects the heightened need for far managed service contracts. as well as value before go-live. One example of this is the need to further optimization. From the continuous drive reaching integration testing. appreciate and acknowledge the importance of test data while a lack of test data for complex integration testing provisioning and TDM in general. custom-built automation tools 15% to generate new test data 16% 13% 16% Use spread-sheets to manually 14% generate new test data 14% 12% 15% Copy production data that we 16% anonymize before testing 14% 12% 13% Copy production data without 14% further manipulation 13% 11% Create data ‘as we go’ (though the GUI) 15% 16% 16% 0% 5% 10% 15% 20% 25% . On the other hand. which are often located/owned by increasing the productivity of their test teams. which is not surprising the board is a clear sign that organizations today better in light of the increase in test data. This increased awareness across systems and organizations (46%) is not the highest comes after an initial period during which the focus was on encountered data challenge. it is evident to many respondents for increased customer journey testing to secure customer that challenges with the test data provisioning. it is nonetheless up 14% on last optimized human resource capacity provisioning through year. Current Trends in Quality Assurance & Testing 49 46% of respondents say that the difficulty in managing the The increased reporting of test data challenges across size of test datasets is a challenge. different Lines of Business and developed and maintained by completely different teams. automation tools to 13% generate new test data 13% 13% 18% Restore backend with each iteration to 13% allow same test to be reused 13% 21% 17% Use in-house. This end-to-end testing stretches across the environment provisioning.

TEM and TDM will become ever more automated and virtualized. IT organizations have a thorough knowledge analytics companies to integrate testing logs with test data of how to manage their test infrastructure. this category. although “creating data as we go” has dropped from 15% to 11%. they are not yet analytics-driven decision making tools and work with niche widely used. using analytics and those unmasked. The first of the leading two is to purchase automated tools to generate Looking more closely at the highest used approaches in new test data. Low-quality applications turn off customers. Senior Test Engineer. going While solutions that integrate the testing function with test beyond integration to intelligent interactions and automated data provisioning systems and test environment provisioning decisioning. Also of the tools on the market. which leads us to conclude that this increased from 14% in 2015 to 16% this year. All four of the most of managing test data needs. up 2% to 17% in 2016. we believe that the core systems built to reliance on production data. These are welcome trends. those masked and especially support testing will become smarter. using automation and virtualization as custom-built automation tools to generate new test data. own test data sets. one reason lies in the increasing maturity this is up from the 13% recorded every year since 2013. risk damaging your brand and require significant development/testing. China . Instead. yet are not on and environment. test environment systems evolve. largely unchanged in terms of use. This is up from to the market and these are being snapped up by today’s 13% in 2014 and 2015. enabling automated intelligence-driven top of the proliferating test environments and test data. but is not at the 2013 level of 21%. At an average of 18% of all the methods used. many organizations shows. WORLD QUALITY 50 REPORT 2016-17 Only one third of this year’s respondents resort to data manually generate new test data. There is a clear trend towards using more tools for generating This is a welcome drop because it is the most primitive way test data for multiple iterations of testing. Teams without access to live-like test environments cannot test efficiently and comprehensively to ensure production-ready performance. Solution providers and tool at 18% is the approach to restore the backend with each vendors are bringing unique data management solutions iteration to allow the same test to be reused. but there is a long way to go to get the most out of test ecosystems. generation is on the rise. A gradually maturing profession is getting to grips with challenges around tools and resourcing. As Figure 33 is still not a prevailing method. often in partnership with external help. QA and Testing will tap into many of the systems are available to drive efficiencies. We see this as the future of QA and Testing. enterprise IT. Avionics & Aerospace Systems Manufacturing. decision making in the coming years. while the reliance on data self-managing capability for testing environments and data. there are several other approaches to generating continue to rely on copies of production data. used approaches show an increase over last year. it is positive to note that the Looking ahead. We also see that organizations have a much better understanding of how to create and manage their There have also been increases in the use of in-house. is going down. test data. which has surprisingly generation techniques. and in the use of spreadsheets to A need for better test ecosystem management in future As TEM and TDM mature.

the journey towards key recommendations last year. so the stabilization is an more efficiency should still be an important objective of encouraging development. even this 2015. This is slightly above the 2015 expectation of 39% Testing outstripping predictions. This is within design and development activities will add further leading to uncertainties about the options for controlling and cost pressures. while ongoing technology advancements. this year’s World Quality Testing budgets observed since 2012 has come to a halt. while this is a positive the QA and Testing functions in almost every organization. Current Trends in Quality Assurance & Testing 51 QA and Testing budgets Getting a grip on test spending Brian E F Shea. DevOps and Digital Transformation. despite this year’s stabilization. in spending. and 18% in 2012. However. year’s average is still too high and an indication that QA and Testing is not as efficient as it should be. Indeed. Sogeti UK The steady year-on-year growth of Quality Assurance and they are in full control. containing the QA and Testing budget due to the increased such as Internet of Things. So while the Better containment of QA and Testing costs was one of our reduction in budget is encouraging. it is clear that organizations are not yet confident that Figure 34 Budget allocation as percentage of IT spend 40 2016 30 % of IT spend 40% 39% 20 35% 31% 26% 10 23% 0 2019 2018 2016 2015 2014 2013 (predicted) (predicted) (realised) (realised) (realised) (realised) Uncertainty about budget needs as digital continues to disrupt One reason for the lack of confidence in the ability to stop pace of deliveries and continuous stream of updates with costs increasing is that more and more organizations are agile and DevOps. . 23% in 2013. Report study participants predict an upward move to 40% Where last year we were talking about IT spend on QA and by 2019. 26% in 2014. compared with 35% in At well above a reasonable 25% of the IT budget. CEO. increase the risk of failures. trend. this year has seen a decline by 2018. On average the industry is now spending 31% of its IT budget on QA and Testing. The growing role of QA and Testing looking at agile.

52 REPORT 2016-17

QA and Testing must aim to mitigate the impact of these In many cases, due to the rapid adoption rate for DevOps
failures. Organizations also recognize a need to invest in and the escalating volume of releases, the right frameworks
next generation automation tools. While this could add to manage this landscape have not yet been put in place.
initially to the QA and Testing spend, with sufficient return Organizations have had around 10 years to adopt agile,
on investment control this should actually lead to a decrease but the rate of DevOps adoption is clearly much faster,
in budgets thereafter. with just 12% of respondents to this year’s World Quality
Report survey saying they are not using DevOps. As such,
This year’s study findings reveal that the increased level we should not be surprised that this change comes with
of development and releases is deemed to have had the quality issues. Investment in the frameworks to continue
biggest impact on QA and Testing budgets, ranked 5.3 this journey, with governance checks, certifications, and
on a scale of 1-7 (with 7 being the biggest impact). This is adherence to regulation, will have to be put in place over
followed by increased challenges with test environments the coming years.
at 5.1, and the shift to agile and DevOps causing more test
iteration cycles at 4.9. Respondents clearly feel that the It is also interesting to note that, despite the ongoing
shorter and faster development cycles as a result of the investment in digital and Internet of Things technologies,
shift to agile and DevOps, and the corresponding need to the survey participants feel that an increase in the complexity
test more, are major contributors to the predicted increased of IT applications is having the least impact on their budgets
IT spend on QA and Testing. Shorter development cycles at present. Just 20% of respondents cited it as having a big
leave teams with less time for appropriate quality checks, impact. Clearly, the need for speed is perceived as making
leading to more test cycles than anticipated. a bigger impact on quality than having to work with more
complex and integrated technologies.

Figure 35

Aspects impacting the increase of QA and Testing budgets


Increased amount of developments and releases 52%

Shift to agile and DevOps causing more test iteration cycles 41%

Increased challenges with test environments 36%

Business demands higher IT quality 33%

We detect more defects which leads to more/longer test cycles 31%

Increased inefficiency of test activities 29%

More non-functional testing required 24%

Increased complexity of IT applications 20%

0% 20% 40% 60%

Current Trends in Quality Assurance & Testing 53

More efficient focus on human resourcing helps to reduce budget

The distribution of test budget between human resources, reveal that costs as a proportion of the budget are falling.
hardware and infrastructure, and tools (software and licenses) Spending on human resources has dropped from 33% to
is shifting. Spending on hardware and infrastructure has 31% over the past year, despite the need for new skillsets, for
increased from 37% in 2015 to 40% in 2016, and it remains example in the testing of Internet of Things technology. This
the biggest area of budget allocation. It had appeared to has been achieved with increased automation, by leveraging
stabilize in 2014 when it dropped back from the previous offshore resources, and by making use of flexible service
year’s 40% to 33%. However, it has since begun to climb in contracts. Furthermore, it continues the downward trend
tandem with a proliferation of test environments. that began in 2014.

Many organizations still have challenges with mastering their At 29% of the QA and Testing budget, spending on tools is
test environments. Although we predict that these challenges almost stable this year, down just 1% from 30% in 2015. This
will stabilize in the next few years, it is likely that the proportion follows a spike in spending activity in 2014, largely due to the
of the QA and Testing budget allocated to hardware and need for investment in digital tools. The reduction over the
infrastructure will continue its climb and reach 42% before past two years reflects the growing maturity in the market
that stabilization. In this landscape, increased focus on test and a greater adoption of open source test tools. It is possible
hardware and infrastructure remains a major opportunity for that spending in this area will go down still further in the
organizations to save cost on their QA and Testing spend. coming years. One reason for this is a continued shift from
Options for this include test infrastructure rationalization perpetual licenses to software-as-a-service. Additionally,
programs, central management and consolidation of best niche vendors offer alternative solutions at relatively lower
test practices, pay-per-use service models, and the adoption cost and this too brings down spending. There is also a
of cloud-based test environments. continued provisioning of open source solutions that have
become better, are more complete, and offer very good
While it is often difficult to find and rationalize the right value for money. So even on a corporate scale, this type
hardware and infrastructure, it is a different picture when it of tool is accepted as being part of the development and
comes to spending on human resources and tools. These testing team’s toolbox
aspects are easier to control and this year’s study findings

Figure 36

Split of QA and Testing Budgets



80 31% 33% 35%

29% 30% 32%


20 41% 37% 33%

2016 2015 2014
Human resources Tools (software licenses) Hardware and Infrastructure

54 REPORT 2016-17

Sector spending reflects emergence of Internet of Things

The highest sector spenders overall are in Financial Services Front office and mobile solutions together make up 30%
and Energy & Utilities, both spending 33% of their IT budget (down from last year’s 35%). Likewise, cloud is slightly down
on QA and Testing. They are followed by High Tech (31%) to 16% (from 18%). The slight decrease in these areas is
and Automotive (31%), with the least amount being spent by unsurprising with the growing adoption of Internet of Things,
Public Sector and Government (29%). The high percentage cloud and mobile. The percentage of the budget given over
in Energy and Utilities is surprising in an industry that is to Business Intelligence and Business Analytics is down
seeing significant cutbacks at present. This year’s spending slightly compared with last year to 15% (from 17%), while
percentage range of just 4%, compares with last year’s 8% back office legacy is also slightly down to 13% (from 16%).
range from 38% at the top end and 30% at the bottom end. The budget allocated to ERP back office solutions has
remained static at 13%.
As discussed in the chapter on Digital Transformation, the
impact of the Internet of Things is being felt across different Regionally, we observe a similar range of spending on testing
aspects of QA and Testing, including the budget. This wasn’t mobile solutions with the Asia Pacific region investing the
measured in 2015, so we cannot make a direct comparison most at 17% of the IT budget and Latin America the least,
this year. However, we can see that the budget distribution at 14%. Similarly, the budget allocated to Internet of Things
between different types of applications is slightly shifting to testing is fairly even, from Central Europe at the top end (14%
take account of the 13% average budget allocation to the of its IT budget) and Western Europe at the bottom end (12%).
Internet of Things.

Figure 37

Testing Budget Split 2016

16.5 41%

16 15.72


Mean %

13.49 13.48



Cloud Solutions Mobile Solutions Business Front Office Back Office ERP back
Intelligence Solutions Legacy solutions office solutions
and Business (Customer
Analytics Channels)

Current Trends in Quality Assurance & Testing 55

Good DevOps practise can be a lever for cost optimization

The uptake of Internet of Things technologies in tandem years. For traditional waterfall-based development, it makes
with the rapid move to DevOps is changing the shape of QA more sense to centralize and consolidate all three key cost
and Testing. Certain work previously carried out by testers drivers of testing, environment and tools within a test factory.
is now being performed by more generalist members of the However, with more and more agile and DevOps adoption,
DevOps team. This move to DevOps could easily result in we will need a lot more data analytics to drive decision
a lack of transparency and therefore control over QA and making within such an integrated test factory.
Testing budgets. However, if the move to more DevOps
practices is done well, the QA and Testing budget will go While some effort is being seen in further improving quality
down as testing shifts left (the fail forward principle) enabling and costs within the testing function, for example with vertical
organizations to find defects earlier. On the other hand, if analytics, such as predictive test management tools, more
done poorly, it will increase the budget as a result of many effort is also needed in horizontal analytics. This is the
more QA issues and test cycles being needed to reach the collection of test data relating to test requirements and test
right level of quality. Another key opportunity to optimize cases, test environments, test data, and resources. It uses
costs is a centralized focus and governance of the test highly intelligent analytics software to drive decision making
environments in use in the enterprise. This will not always be inside the test factory. We predict that those organizations
an easy task due to test environments in many enterprises best able to rise to the challenge of escalating QA and
being neglected for too long. But, as explained in the chapter Testing costs and budgets will be the ones investing in both
on test environments, there are huge opportunities to move horizontal and vertical analytics.
faster to cloud-based and virtualized environments. Potential
saving will easily be in the region of 10%-20% of the total Overall, this year’s prediction of an increase in spending
spend on QA and Testing. over the coming years demonstrates an understanding of
the implications of the changes in development cycles and
Nonetheless, while there is some control over spending on adoption of new technologies for the QA and Testing budget.
environments, the predictions of escalating costs show that The rationalization programs, best practice measures, new
confidence is not high. Thus there is still a need for tight focus service delivery models, and cloud-based testing options
on containing the cost of quality and it will be interesting recommended in this chapter, along with the use of intelligent
to track the topic of QA and Testing budgets in the coming test tools, will help to counter potential budget increases.

The biggest change in testing as a result of the transition to DevOps is
Automation, as it is the core of the successful DevOps cycle.
VP of Applications,
Logistics Provider, Italy


Sector Analysis 57 Sector Analysis 58 Automotive 60 Consumer Products. Retail and Distribution 62 Energy and Utilities 64 Financial Services 66 Healthcare and Life Sciences 68 High Tech 70 Public Sector 72 Telecom. Media and Entertainment .

but industrialization can take place. but mainly in their premium segments. and embedded software. in data. Europe currently shop floor. with the highest volume of unit sales globally. the sector is clearly ramping up a gear. Equipment and systems are becoming smart and remains relatively strong with improved sales. Automotive Sector. the other key digital trend all organizations. This development has come The World Quality Report 2016 study findings bear out earlier than many people anticipated although. with heavy investment in digital capabilities be incorporated in our roads and highways before full-scale underway. etc. The more connected the manufacturing plant becomes. are adapting ways in this sector is the advances being made in autonomous of operating and seeking to engage with the end customer driving. Some 43% of is likely that full-scale industrialization of autonomous cars Automotive participants say they have a dedicated Chief could take up to ten or more years. communication capabilities are driving an exponential growth For instance. Global Leader. no matter the country. established manufacturers already testing certain advanced features. although certainly not flat-lined. in this. making it no longer the preserve of tech companies such as Apple and Google. Capgemini Digital is having a big impact on the Automotive sector where all areas of the Automotive sector: original equipment companies are looking to get close to the end customer via manufacturers (OEMs). It is clear that Beyond digital manufacturing. This . In this landscape the record sales at the top end of the spectrum and Latin Internet of Things is increasingly a disruptive force on the America in heavy decline at the bottom. but just not at the same year-on-year rate as seen previously. China’s automotive sales continue to grow. Further. The uneven growth observed at a regional digital manufacturing is now becoming a reality as operations level last year continues. robotics and digital market growth has slowed. WORLD QUALITY 58 REPORT 2016-17 Automotive Digital capabilities continue to change the character of development and testing Kai Grambow. second only to Government and Public Sector lanes and smart technology to guide vehicles will have to (51%). with North America achieving and information technology converge. it the increasing digitalization of this sector. Nonetheless. digital means. supplier plants. Almost all manufacturers are now investing heavily in a more personalized manner. we are seeing nonetheless still has some way to go. Thus. Changes such as clear Digital Officer. The gradual uptake of digital technologies is making itself felt both on the shop floor and in the vehicles themselves. while Asian connected. the Digital Transformation projects are up and running across greater the requirement for QA and Testing services. assembly. in reality.

it is perhaps indicative of the industry’s their testing. This reverses management. the HQ-centric model of a few years ago. Offshore locations. While this predicted upward trajectory has to typically offshored their application management. which took too long has become a main theme for OEMs. that is more prevalent in the Automotive sector and China are now setting the scene for de-centralized than in many others is the need for heightened cybersecurity. Today. the cost and quality outcomes enthusiasm for emerging technologies and automation of handing testing to a Systems Integrator able to industrialize techniques. but is typical of the world. More respondents are using penetration testing offshoring trend. The Automotive sector powerhouses of America however. security testing activities. approach. whereby one system is rolled out all over approach to test data and test environment management. It QA and Testing to these markets to support development is no surprise. significantly more to the set-up of test operations. however. Some 44% of Automotive 43% expect to use robotics automation. One aspect of digitalization. but not stop at some time. Instead. QA partner. This is a shift for OEMs who have by 2019. the percentage of the Major testing initiatives have continued over the past twelve overall IT budget spent on QA and Testing this year has months and testing as a discipline is becoming more dropped. with systems and applications brought The development of connected car features. such as India. security testing. It is down from 35% in 2015. while there are many connected car agility and efficiency. This corresponds with findings is the move away from centralized HQ-induced system from across all sectors of an increasingly intelligence-led development. currently provide labor arbitrage benefits. up to 50% from 39% last year. Its application security testing. There raises the possibility of bringing QA and Testing functions has been a marked drop in the use of dynamic application closer to the main markets. Automotive companies give than any other objectives. Currently. In line with the all-sector average. (ethical hacking). an equally high importance ranking to the setting up of a Testing Center of Excellence (TCOE) in a hybrid (nearshore/ There is a prevailing nervousness in the industry regarding offshore) model and to a decentralized TCOE for improved hacking. down from 64% in 2015 to 44% this year. therefore. Elements of these developments are already evident. these are not as well protected as they should important by 52% of Automotive respondents respectively. the Automotive sector performs a range of industrialization. These are cited as important or very features. . employed by 53% of respondents potential to cut the cost of testing and development activity in this sector. but increasing Manual code review is also down. Still industrialized. and 40% foresee participants in the World Quality Report 2016 study say they the use of predictive analysis (automated identification of risk have outsourced most testing activities to an independent areas). from 59% last year to 43% automation will reduce this and could create a negative this year. not just in testing but in the wider IT function. Sector Analysis 59 is not restricted to the Automotive sector. are being sought. and requires deeper levels of engagement with local as an important or very important aspect of IT strategy by partners. The most used approach is static will change the landscape over the coming decade. with 47% of Automotive sector respondents saying they use predictive analytics to determine or optimize test coverage One aspect of QA and Testing operations that has emerged in a DevOps environment. This is reflected in our 2016 study findings relating 67% of study participants in this sector. applications. system development. be and this has increased the interest in the specialist area of cybersecurity QA and Testing. OEMs are responding to cost pressure with the Automotive sector study participants predict QA and both industrialization and offshoring featuring in their QA Testing spending to rise to 40% of the overall IT budget and Testing strategies. For example. driver behavior analysis and safety monitoring. which is just 1% down from last year. that enhancing security is ranked activity. who must respond to get innovations to the different global markets. to 32% in 2016. we are seeing a more market-specific today’s digital manufacturer. rather than for just for single sector foresee using cognitive automation in the coming year. It shifts to consumer concerns about data security and privacy. 48% of survey respondents in this the process for a whole domain. we believe that the use of automation and study findings. such as vehicle to market and tested in an agile DevOps model. As with previous years’ Looking ahead.

Thus the interoperability of devices will be CPRD survey participants rank enhancing security as 5. Clearly this user interface and ease of use. enabling the customer experience in this sector. Digital sizes (laptops. they are asking in turn what the Digital Transformation and the move to cloud platforms are CPRD companies can do for them. On a browsing on one platform. Indeed. all-sector average of 39%. and paying scale of 1 to 7. The task for CPRD companies is to provide a seamless QA and Testing is a key enabler in enhancing customer interface between all platforms.85. Unsurprisingly. tablets. Retail and Distribution Achieving customer intimacy across multiple channels is a growing challenge Sreenivasrao Baru. CPRD Sector. second only to security and is especially true for enterprises in the Consumer Products. with 7 being the most important. Global Head CPR Sector. this year’s on yet another. Retail and Distribution (CPRD) sector. Capgemini Theodore Levine. the most cited CPRD objective. WORLD QUALITY 60 REPORT 2016-17 Consumer Products. This is reflected in the leading QA interaction of browsing and buying. Corporate Vice President. enhancing user experience across all devices. In this landscape. technologies. and Testing objective amongst the CPRD companies taking difficulties in managing test data across the end-to-end part in the World Quality Report 2016 survey. . of test data for complex integration testing across systems and organizations is a challenge. Leader. Capgemini The trend for greater customer intimacy reported in 2015 has a challenge going forward. They insist on security also giving rise to growing concerns around the security of of their personal information and they expect a consistent transactions and customer data. efficiency/performance. ensuring end-user satisfaction is tools and platforms proliferate in the coming years. They are increasingly security remains the number one strategic IT priority. selecting on another. both cited by 46% of respondents. encompassing different screen become even more pronounced over the last year. Against an transaction will increase as multiple identities. with 48% saying it is either 46% of the CPRD participants in the 2016 study say a lack important or very important. mobile phones) and operating Transformation is part and parcel of this trend as it enables systems. 44% of CPRD companies in this year’s survey organizations to better understand their customers through say they focus their mobile applications testing efforts on the intelligence-led insight into their buying behaviors. As customers continue to share their personal information with CPRD companies.

there is less demand for big. applications Easter. CPRD teams in the CPRD sector. down from 26% last year to 17% this year. based delivery and digital enablement. Thanksgiving. on how to make the last mile pay while meeting customers’ As companies in this sector investigate as-a-service cloud- expectations. three distinct focus areas are emerging in the 2016 survey state that crowdsourcing in this way is the agile and DevOps era for the QA and Testing function: important or very important. the need for to reduce CapEx is placing further demands on the QA and complex systems integration testing will push testing costs Testing function. Also driven by customer demand. in higher at 55%. despite a slight drop over the interfaces to ensure they are protected and will work well past year. data across all layers. They claim that on average 36% of their applications are currently cloud based. centralized testing in terms of ensuring intimacy with customer needs. family and more. users of private cloud. When combined with on-premise cloud platforms. against a 38% all sector average. embracing the full technology by the timing of big seasonal events. and Black Friday. traditional development The frequency of releases. infrastructure. This is due to the escalating development of native in the cloud environment. With the shift away from large scale. . Sector Analysis 61 Security mechanisms in retail have evolved ahead of other In contrast to this. and a corresponding focus continue to be a factor in the CPRD development lifecycle. which demand and the customer experience. While development costs look set to come down due to The continuing investment in cloud platforms as a means agile and a further uptake of DevOps practices. in the general organization of test in this respect too as they build deep customer knowledge activities. relevant releases. The using agile development saying they work in a distributed specialist knowledge of testing teams remains important agile model. and corresponding changes to smart in-store reveal that CPRD organizations are the highest percentage formats. This year’s chapter on industrialization and automation discusses the rise of crowdsourcing as a model for providing While traditional testing issues with the likes of SAP and Java rapid test capacity. cloud applications and a focus on customer experience. making the move to cloud is similar to the transition from mainframe to client servers of a few Looking ahead. This is reflected in the the most used of which is automated test data generation. and cloud. especially dealing with cloud. It is important to identify and test existing will increase in the coming years. Another key trend characterizing the CPRD sector is the steady progress of new multi-channel business models. along with hybrid for some of their development testing. stack. and software-as-as-a service solutions in tandem with private this gives a different perspective to the focus on security. developments with the convergence of digital and in-store Nonetheless. An average 47% of all respondents to remain the same. particularly amongst retailers. as noted above. driven by customer demand for digital shopping drop in the percentage of hybrid cloud applications being and the need to safeguard multiple points of interaction used. again the highest across all sectors. this year’s World Quality Report study findings channels. cloud for certain aspects. They are evaluating third party other sectors. such as production environments. but the CPRD sector comes the digital customer experience. Despite this across a consumer’s life. however. with Agile development and the application of DevOps practices the likes of Uber setting the pace. Solutions being offered to organizations in this sector and more CPRD companies are moving to the cloud but will move to full-service end-to-end business transaction their migrations. with just 32% or respondents saying they use this approach. presents a challenge projects. will drive an increased need for QA and Testing. projects. with 41% of those CPRD respondents cited by 47% of CPRD respondents using DevOps. Further. Big Data. we see a growing companies employ a number of different approaches for demand for testing associated with platform migration testing in the DevOps environment to help them achieve this. we believe that spending on QA and Testing decades ago. For retailers in particular. 2016 survey findings. Instead. embracing lifestyles. such as Christmas. this figures rises to 49%. this sector records the lowest usage of a central of the way in which an application is used in order to create Testing Center of Excellence performing most test activities. more higher. they are seeking to leverage DevOps methods to help them on this journey. minimal disruptions and quiet times for production releases. are governed QA and Testing support. In one sense. we believe that businesses are continuing shoppers share with retailers is more personal than in most to look for hybrid solutions. Furthermore. the level of information recorded drop. CPRD survey participants record a big sectors.

a Chief Digital Officer (CDO). whereby a number of well- towards Digital Transformation. for taking responsibility for these transformation strategies. from the desire to embed customer convey a more modern corporate image. At 25% of this number of customers giving these apps a high satisfaction sector’s study participants. 43% of this year’s . Other leaders also have a remit While there is no clear ramp-up of these today due to the for digital. However. Greater API development. and 8% saying it resides with the will become increasingly important. we can see that this industry is One aspect of Digital Transformation that has taken off is the still slightly behind the curve. board the attributes that make a digital app user friendly. with 19% in this sector saying they don’t need relative digital immaturity of this sector. Companies need to reengineer sectors. on the other hand. Introducing digital capabilities into the business demands a This year’s World Quality Report study looked at the roles new QA and Testing skillset. 27% example. business leadership. which the QA and experience at the heart of business strategy. requires a different type of test tool. This Energy and Utilities (E&U) companies this year is a move aligns with an interesting trend. They must take on (10%) industry study respondents. while the move of the study participants from the E&U sector say they have to more agile development demands new testing services. Global Head of Sectors & Utilities. While a very high proportion of of companies saying they don’t have a CDO because Digital companies in this sector have launched a mobile app. While other sectors have been quick to adopt digital technologies. the Transformation is not part of their strategy. to the need to Testing function has a role in protecting. there is no business models and don’t reflect the more digital operating clear consensus in terms of which key Digital Transformation environment. The new sub-brands. as evidenced by the proportion use of mobile applications. deliver operational improvements. this is the highest across all rating is reportedly fairly low. with to retain the best talent? Interestingly. There has been a big known companies have created secondary brands to better increase in the number of organizations starting to embrace appeal to the market. clearly objectives are driving this. Capgemini The most significant trend shaping the IT landscape for a much higher rating than any of the other sectors. there is no doubt a CDO because Digital Transformation is in the hands of that both automation and industrialization of QA and Testing their IT leadership (CIO). What is it that the consumer is looking for? How can we Protecting the corporate image is cited by Energy and Utilities better engage with our workforce using digital apps in order companies as the leading objective of QA and Testing. The legacy names are tied to legacy digital with long-term digital strategies. and there is clearly a big difference when compared their processes and move away from viewing their digital with the likes of the Transportation (9%) and Automotive apps as they did their old technologies. WORLD QUALITY 62 REPORT 2016-17 Energy and Utilities Exciting transformation journey gathers pace Perry Stoneman.

This is the same as last conservatism isn’t preventing the industry investigating year but. Cyber attacks on energy grids have ensured This type of new capability will forge a more intelligent-led that security is not just a senior-level responsibility. In fact. threat from hackers. When testing move ahead with Digital Transformation. The new skillsets required are also lacking. 49% anticipate using cognitive automation undergoing massive QA and Testing change. the testing for the right testing process or method for mobile and multi. That’s the role of and aligns with the operational performance objective of QA and Testing and it will be an increasingly important role Digital Transformation. There is a of sensitive data on the device or over the air is an area long transformation journey underway and while the industry of focus. home security recognition amongst the industry’s CIOs of the need for firms and utility companies building up their capabilities in cloud strategies. Instead. the 43% saying they don’t have mobile testing experts. a strategy. Developments such as smart Another technology change associated with digital strategies home technology are emerging. in reality the meters themselves don’t 48% of respondents in this sector say that they don’t have yield that much sensitive information. With a strong track record of physical and perimeter security. including devices and wearables. but as yet no one sector is is cloud adoption. Testing is an important facilitator of end-user satisfaction. (automated capture. these is more likely to be around efficacy. this area. it is no surprise that once again security budget spent on QA and Testing this year remained static is the leading IT strategy priority this year. 52% of survey participants in this sector say that security whether that user is a customer or an employee. plants. it is this year’s study participants say they do not use DevOps no wonder that 17% of QA and Testing budgets is currently principles. companies in this sector predict a rise to 41% in review and static application security testing are the most three years’ time. While the percentage of the total IT this responsibility. but it will also make an to the very top of the organization. Looking to the future. While this might present a security is spent on mobile solutions. This spent on testing cloud solutions. This has moved forward in the past year. against an all industry average of 12%. while 32% say they expect to use predictive analysis as an for example in the protection and access control of nuclear automation technique. next step will be to enable customers to access data from their smart meters in real time. Customers simply channel. more than any other aspect of mobile testing. the E&U sector has also turned its attention to cybersecurity. With CEOs assuming impact on spending. Around 300 million smart meters have been . This aligns with the strategic decision to performed security testing activities overall. for which QA and mobile applications. with want their smart meters to work. Last year we reported that there was winning the prize amongst the broadcasters. it paints a picture of an industry For example. when combined with the 13% spent on testing and predicting the use of new and emerging capabilities. with 1 billion predicted to be in place are already testing the user interface and ease of use of between 2020 and 2025. it can’t protect This assurance of security is critical to giving a field-based itself against customer expectations for a seamless digital workforce remote access to customer and enterprise data interface to the services they consume. With cloud offering potential cost adoption has remained consistent this year. but goes approach to future test activities. is protected by a strict regulatory regime. There are challenges. Manual code at 33%. but now actively defining a timeline for migrating non-SCADA (supervisory control and data acquisition) The somewhat conservative approach to agile and DevOps systems to the cloud. as the pace of Digital Transformation escalates. however. Internet of Things solutions. and 17% of the QA and Testing budget that didn’t exist before. recovery and distribution of specific knowledge and operation instructions) in the coming year. Smart meter programs are being rolled out at pace in this industry. Sector Analysis 63 World Quality Report survey participants in the E&U sector deployed so far. It is nonetheless an evolving area for which QA with many IT leaders no longer simply planning to implement and Testing will become increasingly relevant. 15% of reductions of 30-40% in the provision of infrastructure. These are nodes on the system mobile applications.

Principal. Capgemini Shivakumar Balasubramaniyan. Senior Vice President. Capgemini Krithika Kumar. While the uptake of DevOps is slower in results in a shorter timeframe and in a more complex insurance. as well as for skills along with functional testing skills. there is a feeling that from 2017 onwards it will technology environment. The ongoing trend to market entrants. banks are converting internal assets into APIs for tasks such As insurance companies. to a new firms are already part of many innovative market-facing mindset that embraces a deeper understanding of the new experiments and they arebecoming further embedded in delivery paradigm. For example. companies are building more and more a common validation approach across the industry. Financial Services institutions are co-innovating teams in Financial Services beyond purely the tools and with the financial technology companies (FinTechs). application programming interfaces (APIs). QA are seizing the opportunities it brings for better customer and Testing professionals are required to possess developer engagement and new business models. in the use of agile and DevOps in banking requires rapid market goals. and limited growth. Capgemini In an industry beset by tough competition from new pushing workforce enablement. banks and capital markets as fraud management. These processes needed to get a new product to market. Co-developed with the FinTechs. . Financial Services SBU. Financial Services SBU. which have ambitious time-to. Shorter release cycles with an increase transformational programs. WORLD QUALITY 64 REPORT 2016-17 Financial Services A new approach to QA and Testing is needed for rapid development cycles Nilesh Vaidya. These APIs require a new approach firmscontinue to embrace Digital Transformation. take off. differentiate the customer experience. In banking the move of DevOps into mainstream we see Blockchain and digital payments as a hotbed development is resulting in a push for more test automation to of innovation in customer experience and transaction assure quality across the shrinking development timeframe. they to QA and Testing. develop front-end solutions for tablets and smartphones is organizations across the Financial Services sector are seizing shifting to the development of more functional applications the opportunity to disrupt with digital and use their data to to work on these different devices. Financial Services SBU. low interest rates. Given these trends. with a lot more service virtualization. Vice President. processing. Many Financial Services institutions are evaluating smart contracts with their industry peers and As the focus on customer experience is embedded further implementation of these standardized contracts will need into digital strategies. True agile delivery moves developers and QA and Testing Increasingly. QA today has to deliver QA and Testing.

mobile and while metadata concerning family members is used to cybersecurity. particular services. such as predictive analytics work for the QA and Testing function. We are also seeing posed by digital disruption. Again. such as a credit card. When coupled these technologies can be hard to find. whereby homeowners can manage Services respondents say a lack of test data for complex their home security remotely. For example. 42% of Financial automation solutions. customer value production and non-production platforms are fundamental analytics is used to create richer profiles of customers using in this. will also make an impact on integration testing across systems and organizations is a home insurance policies in the future. will be vital. In the profiling is being used in some car insurance instances. stability testing and compatibility testing yet. fitness trackers offer health insurers the potential to analyze fitness data In this data-rich landscape. For example. This is borne out by the solutions and tools. There’s a sound impact on QA and Testing budgets going forward. Continuity. Financial Services is amenable in tandem and specific security testing experience with to end-to-end and lifecycle automation. cited by 45% physical stores and customers’ digital ID to understand their in this sector. shopping patterns. this is an emerging. test environment and companies aggregating security products and taking over test data management will see a push to more open source the management of the service. to identify what needs to be tested (and what doesn’t) and an integrated tooling approach. rather than a mainstream technology. requirements and risk and to peak load requirements and risk. with 67% of Financial Services survey respondents saying Investment in future innovations such as this will have an it is either important or very important. both 53%. is being driven in part by cloud providers bringing improved and take an intelligence-led approach to claims. The challenge will be to make study say they pay special attention to security requirements this API testing. but the extent of it is. 56% of the Financial Services participants in our the process highly complex. Quality of service and service assurance for both of mainstream processing. Thus. credit challenge. This presents testers with a challenge. the available products evolve package-driven business. yet contain costs. products to the market. All of this demands a deep established strategy for this. many embracing predictive analytics. as well as the management and build of and risk. In addition. this type of solution does not exist and will require non- across multiple technology browsers and within concurrently traditional ways of testing with a big focus on automation. While currently there appears to be no embellish the customer profile. more than the level of attention paid to performance the functionality. Complying with data security and data privacy card issuers are developing and piloting solutions connecting regulations for test data is another challenge. a lot of customer experience API and third party software-as-a-service aspects of the IT development will involve several different providers. data strategies must incorporate and offer rewards to customers with a healthy lifestyle. . To ever before and it is not uncommon to see 100+ security meet these expectations. better understand policy risks. As a highly As the security threats evolve. big waves of technology focus. we see a with the fact that the sector has a mature QA approach to lot of QA being carried out by developers. In banking too. For example. Geo-presence capability The Internet of Things is another element of Digital now enables the location of a transaction to be identified. it looks set to be one of the next understanding of the underlying technology. This profiling is not new. 57% of companies in this sector who say they use managed application security testing services. there is a significant amount of and the use of new approaches. organization will need more developer-testers and system The entire security architecture is more multi-faceted than integration testing orchestrators. data analytics is now part adoption. Transformation. Home World Quality Report 2016 study findings. Enhancing security is the number one IT strategy objective. making landscape. capacity management and cost optimization largely define the agenda for cloud In banking and capital markets. Instead. Integration is complex and testing has to be carried out quickly to meet development Telematics in cars is already a reality and driver behavior demands. a seamless process. test automation products utilized in it. sitting alongside cloud. Further. running development programs. Further. Security in this sectoris incredibly complex. and they come at a cost. which will reduce license costs. demanding that significantly to neutralise the escalating cost challenges they do much of the security testing. insurers are analyzing customer data The rapidly increasing cloud adoption in Financial Services to tailor customized services. Sector Analysis 65 There is a need to understand the new architectures on which Security is also a priority test activity for the cloud-based to test these APIs. Data is increasingly being used across this sector at a number of levels. this can drive down cost firms make use of the product vendors. As integration testing. The QA reason for this.

Product and Engineering Services. with predictions a phone line. Cloud adoption demands new QA approaches. there is now hospitals. QA activities and Testing profession. with quality firmly embedded in the process of All of these developments make a big impact on the QA designing. to public health bodies and equipment and medical increasing confidence in cloud platforms for sharing and device manufacturers. WORLD QUALITY 66 REPORT 2016-17 Healthcare and Life Sciences Smart technologies and connected data streams help to push a patient-centric agenda Shakthi Kumar. with the QA and care units. Sogeti Digital technologies are changing the face of the medical One outcome of increasing digitalization is the ability to truly device industry and broader healthcare sector. but also integration. storing data. such as fitness trackers and service subscription-based delivery. The Internet put the patient at the heart of healthcare provision. enabling care providers to monitor the sickest Testing function reporting that 17% of the testing budget is of patients remotely. for example with major healthcare providers implementing tele-intensive This is having an impact on spending. growing interest in telemedicine is also evident. There is a push toward proactive. along with health portals connecting Life Sciences respondents to the World Quality Report 2016 patients to their clinicians. It is an increasingly strategic role. regression verifying a massively growing product set and proliferating and performance testing for all the different layers of these data channels. These connected data streams are enabling the that this will increase to 35% in the coming years. Indeed. We are rather than reactive healthcare. Where previously there were focus wholly on the patient. such as electronic medical records. increase in wearable devices. spent on cloud solutions. Associate Vice President and Head Product V&V practice. and home-based medical study claim that 84% of their applications are currently cloud monitoring whereby clinicians can receive patient data via based. The core new cloud platforms. while a delivery of increasingly individualized patient-centric care. the entire industry is coming together with a this evolving digital world. from healthcare providers and concerns about sharing data across networks. Capgemini Malavika Athavale. Some 28% are private cloud-based. which is tasked with validating and must cover not just validation. Healthcare and elderly care monitors. A further 24% are public cloud-based. For the of Things. cloud and transformation programs characterize first time ever. purpose of this sector is to improve the quality of life and . with software-as-a. For example. Senior Vice President. building and procuring for the cloud. with enabling technologies also seeing more and more large hospital systems being driving connected health provision. Life Sciences Sector Leader. we see an implemented in a cloud-based model.

it is of manual tests. is a noticeable trend that looks set support teams. Testing the customer method. say they utilize test-driven and BDD approaches. it other challenge. manufacturers are investigating smart device increased functional test automation combined with agile management and using remote capabilities to optimize lifecycle management tools at 44%. A third stage of maturity. Healthcare and Life Sciences to increase. agile development methods. but is about making that app as user friendly as However. Analytics are also shaping test approaches. is driving further change.85 ranking for implementing and using test tools for proactive activities are created on top of the standard customer experience testing. along with patient data a common platform is keeping the QA and Testing function security. The most In addition. Manufacturers are developing organizations in Healthcare and Life Sciences that use agile smart apps to enable clinicians to tap into this data remotely. access. while 39% of Healthcare and Life respondents to the World Quality Report 2016 survey claim Sciences respondents say they use continuous testing. Sector Analysis 67 healthcare for patients. that systems and wearables in the Internet of Things. 50% say test automation helps to experience is a challenge. have had to transform not only ways of working. whereby with a 4. with a further 17% being spent on mobile solutions. The medical device sector was For example. which is workstations is being overtaken by centralized data that is understandable for a product-centric industry. a to spend 16% of their testing budget on Internet of Things further 54% currently not using it say they plan to in future. the first level of automation maturity. Cited as important aspect of digital implementation. we are also seeing more use of challenge of designing test cases. equal only to the Beyond simple automation. Despite this. the second stage of maturity Data analytics has come into its own in this sector as more in automation. however. whereby software robots replicate human and more information is captured from connected devices. DevOps too. as Internet of Things functionality becomes used approach to testing in agile for this sector is that of widespread. where reduce costs. To this end. the most highly rated QA and Testing busy. The . rather than purely in this sector moves beyond simply standard automation on features. patient data that was previously stored in a pioneer of behavior-driven-development (BDD). The Healthcare and Life Sciences study no longer just a case of providing an app enabling remote participants claim that 27% of their test cases are automated. This is more than any or very important by 49% of respondents in this sector. and testing service costs for their health sector customers and their own in the DevOps lifecycle. Of course. This is driving of cognitive testing techniques. is also emerging. Interestingly. For example. but also the products they’re bringing to the market. Indeed. 41% of those accessible from anywhere. interaction with applications. a focus on user experience. niche product developers and tool vendors. including study respondents is ‘to assure minimal disruption to local. as the most challenging business operations by mitigating risks’. and that includes the efficacy and resulting mix of processes and tools all brought together on safety of equipment and devices. test automation is also an enabler of speed-to- In common with Digital Transformation programs across market and there is increasing maturity as the QA and Testing all sectors. This is closely followed autonomics. private and public cloud. and higher than the all-sector average of is significantly higher than the all-industry average of 39% 29%. It is perhaps why 34% of this sector’s World Quality objective amongst this year’s Healthcare and Life Sciences Report 2016 respondents rank integration services. with 34% a data-led approach to improving disease management. solutions. the Healthcare and Life Sciences study of defects. and vendors to build more analytical platforms and create new business models embracing ecosystems of solution vendors. in this sector say that in the coming year they foresee using robotics automation. of Healthcare and Life Sciences study participants saying they use predictive analytics to determine or optimize test In this increasingly digital landscape. with 41% for this objective. of the Healthcare and Life Sciences study participants saying Partnerships are being forged between healthcare providers they foresee using cognitive automation in the year ahead. while 47% say it results in better detection 7 is most challenging. and by establishing the test automated testing. device manufacturers coverage when testing in a DevOps environment. participants rank identifying the end user expectations and requirements as the top challenge at 4. Medical device The demand for ever faster releases of new products and applications previously built on proprietary hardware are enabling technologies has seen a high level of adoption for now being built on digital technologies and mobile apps.86. For example.80. this is not without its challenges and 41% in this possible so that the clinician can see more patients and sector claim they don’t have the right automated process/ use his or her time more effectively. On a scale of 1-7. 42% of those surveyed data for customer experience testing with a ranking of 4.

and with a high level of and volume.0 (the fourth industrial revolution). Big Data. robotics and the huge scale and complexity of many industrial projects mathematics. These are at the forefront Report study are: to increase quality awareness among all of the Digital Transformation programs making an impact disciplines. connected products. knowledge to the domains of business. assets and the High Tech sector. Sogeti High Tech Malavika Athavale. but about pleasure-of-use. This has an impact on Digital Manufacturing is the convergence of operational the user experience. technology solutions are becoming highly complex and operated in ever larger contexts. latency. Big Data. understandably. simulation is not new in this environment. factory flow giving deeper insight into product behavior through automation) with IT (business process automation). Associate Vice President and Head Product V&V practice. Testing Leader. to that of digital QA services. there is now an expectation of zero Also known as Industry 4. the scope of QA and Testing services outcome of this convergence is the ability to deliver new must be extended in terms of expertise. development and test lifecycle. aerospace firms. This explains the availability of enabling technologies and new capabilities. on economic landscapes across the hardware and device and to detect software defects before go-live (42%). has an impact in QA and Testing: simulation (virtualized testing). with improved omni-channel information technologies (industrial process automation. data. A more scientific approach is being adopted. However. defense contractors and large-scale consumer electronics companies that make up In this world of smart. In this context there are four trends emerging customization. such as those in aircrafts. We have moved from the provision of purely means simulation is even more vital today for controlling QA services. costs and preventing traditional QA and Testing efforts from . predictive analytics cited by the High Tech participants in this year’s World Quality and industrial systems security. Product and Engineering Services. and cybersecurity. with testers extending their Of course. plants. for consumer products. cited as important or very important by 44%. This speed. design. trains and power plants. WORLD QUALITY 68 REPORT 2016-17 High Tech New technologies enable adoption of Digital Manufacturing Jean-Pascal Duniau. robotics. we are no longer talking the journey to Digital Manufacturing has been triggered by about ease-of-use. on the QA and Testing function throughout the specification. The analytics. manufacturers. As a result. testing coverage products faster and more often. Sogeti The continuing uptake of Digital Manufacturing processes In factories and complex systems. why the two most important objectives of QA and Testing such as the Internet of Things. and approaches characterizes the High Tech sector this year.

Thus more and more companies are turning bench. new tools for efficiency. in essence. and with input from data analysts and data scientists.13). simulation. With penetration testing and design security. Test data from bench. where as scientists. devices and machines is pushed out Instead. rather than and security will push up QA and Testing costs over the near a drain on resources. making a massive impact on there is a real concern about the security of data amongst costs. perform complex tests. through different channels. for example a production line. In some instances.98. and an evolution in robotics. 80% of must be part of any Digital Transformation project. factory in a build-operate-transfer model at 5. co-workers. virtualization enables testing during specification. the study findings indicate that High Tech companies expect spending on QA and Testing to increase An acceleration of cybersecurity testing is another to 40% of the overall IT budget by 2019. smart sensors. Virtualization is also used in the verification of systems High Tech companies. For example. Traditionally. Indeed. that of IT cost optimization (46%). Some 65% of High can save many system integration-bench hours and deliver Tech participants say it is important or very important. cover more parameters. such as smart metering. In the High Tech Test industrialization and automation are critical aids to sector. With the made possible by the new digital tools and sophisticated convergence of IT and operational technology in the High approaches. smart building. Industrial networks wider adoption. Thus the growth in Big Data becomes a major creating real-time data. 57% of High Tech repetitive commissioning testing. The test activities can generate huge volumes of data spread efficiency offered by industrialization comes in the form of over thousands of parameters. These are already reducing test cycle times Tech sector. we believe consequence of the move to Digital Manufacturing. Internet of Things as data from industrial control systems. or the environment around important objective of IT strategy across all sectors covered the system under test. in this sector.000 different parameters. while others may be extended to with an expert QA vendor is rated 4. Robots are. there is an meeting the escalating need for QA and Testing services. This shift left to early validation of specification appliances. and home production. that this will start to reduce significantly thereafter as a result Awareness of the need for cybersecurity to safeguard data of greater efficiency in the verification and validation process and assets has been growing over recent years. This further cost savings. the focus in Testing Centers of Excellence (TCOEs) and this sector adopts terms of this data has been on the results themselves. increased enthusiasm for robotized testing. This reflects struggle with test data and environment availability. into IT infrastructure and systems. project basis. Some 42% of But there are challenges with automation. 42% High Tech participants in this year’s study foresee using of the High Tech participants in this year’s study say they robotics automation in the coming year. and previous testing results (predictive testing). there is heightened focus on product security. mathematicians and robot programmers. with security being cited as the most under test. horizon. the implications for more cost efficient QA that used to operate in isolation are more vulnerable in the and Testing are clear. at its a number of approaches: on a scale of 1 to 7 (with 7 being basic. A managed service model tests that are irrelevant. set up to handle to external. such Big Data is proliferating within High Tech companies. Internet of Things testing lever of cost reduction and increased revenue. and real systems can also be correlated to improve test Spending on test environment automation. Another lever for efficiency is robotics. where the use of robotics is not new. However. with robots able to operate in 42% say their delivery methodology doesn’t support test collaboration with a human operator on the same test automation. simulators. Now. it also opens a gate defects can make a significant contribution to faster time. with some simulators processing more is way ahead of the second most cited IT strategy priority than 100. They are fast becoming and 49% use external security testing teams on a project-by- an important asset for testing complex physical systems. with the ability to analyze most important) High Tech companies rank a captive TCOE data. followed the industry is starting to define test strategies based on by a hybrid near shore/offshore TCOE factory (5. More and more Internet of as well as in the system integration and validation stages. from three months-to-a-year down to just one week. While the omni-channel information flow gives and design and the ensuing early-stage identification of deeper insight into product behavior. It Things connectivity is appearing in omni-channel consumer makes it possible to simulate use cases without having to stop solutions. This by the World Quality Report 2016 study. . In fact. This gives companies cost effective access to the new skillsets now characterizing QA and Testing. managed service provision. such as an engine. Sector Analysis 69 becoming a barrier to the rapid release of new developments. however. and companies say they use managed application testing services quickly learn to recognize defects.17). This is why security testing to-market and cost reduction.27. use cases can be simulated. a pass or fail. This can detect an offshore TCOE factory (5.

creating the duality entirely iterative lifecycle. more than any other sector. the majority of governments online. Report study. CDO is now a core role to which business and IT look to testing in the legacy environment still operates using the deliver transformation. the pace is certainly ramping up. still be a requirement for traditional-style testing. however. . At the same time. There will. keep pace with the level of change demanded due to the centric design. Capgemini The focus in Government and the Public Sector is on risk-based testing and data creation tools in an attempt to engaging more closely with citizens. A further of automation and emergence of predictive analytics to 33% have assigned this role to other senior leaders. as fast as possible. While the citizen engagement focus remains unchanged as a driver. The determine or optimize test coverage. On the other hand. as well as on user. QA departments continue precedence over the Chief Information Officer (CIO). They are investing in cloud-based environment will increasingly be the norm. there Government needs to deliver better services more cost is a policy to get as much of government as is possible effectively. In general. to skip a generation of IT to move ahead rapidly with their Instead. the duality of IT landscapes requires A key measure of digital intent and maturity is whether different types of testing activity split between the two. some instances replaces the CIO all together. with tooling and testing shifting of digital and legacy that we noted last year. 51% of Government As an example. This focus is a key driver for the many digital projects underway across the world. in a DevOps development environment and Public Sector respondents to this year’s survey say more typical of greater digitalization. Offshore testing is often not governments not held back by large portfolios of legacy possible because data cannot easily be moved offshore in its applications are ahead of the game. and in to implement improvements around automation levels. even though many data masking tools are available. in which they in this fast-paced world is leading to a very different and have invested billions over the years. The impact on QA and Testing are still running legacy technology landscapes. Head of Public Sector. just as it was a year ago in our World Quality amount of digital front-end releases driven through DevOps. WORLD QUALITY 70 REPORT 2016-17 Government and Public Sector The pace of Digital Transformation activity increases Matt Howell. the CDO often takes systems on aging technologies. but digital technologies. of and open source systems to build business services on course. For the main. organizations have a Chief Digital Officer. Those few left to earlier in the process. the way in which systems are tested is changing. and have a chance entirety. we see high usage they have a CDO. With the move to cloud platforms traditional waterfall methods associated with complex and an increase in agile development. smaller testing teams embedded in the development digital engagement.

for example in the to be factored in to the hunt for digital talent. 29% of respondents in this sector report that it is very feature automation and industrialization of test activities. It supports increasingly the norm. However. in the the adoption of cloud in this sector initially. devices. area of focus. The opened up into them to facilitate citizen engagement. It is cited by 49% of opportunities are still being identified and built into future the Government and Public Sector study participants as an IT plans. more than any other area. while The importance of security is unsurprising as government a range of Testing Center of Excellence (TCOE) options are does all it can to make it easier for citizens to engage and used to drive industrialization. 28% of test cases in this sector are already automated. It is also important to recognize that will ultimately play a massive part in IT. This important. At the moment. The latter embracing an enterprise hybrid cloud model. Government and Public Sector organizations have a followed by a TCOE factory in a hybrid (near shore/offshore) duty to safeguard data. 59% of Government and Public Sector the cost and resource effort of delivering services. testing services. there is The caution associated with a need to safeguard data slowed a lack of internal resource in key areas. Sector Analysis 71 As the pace of Digital Transformation accelerates. This suppliers. The most used TCOE model for services to be delivered online. must create the right culture to attract new. however. To keep up Currently. and digital will be an enabler of this. 19% of the QA and Testing budget in Government with demand. but is also it needs. organizations are looking at tools that can analyze code for weaknesses at the earliest stage in the development lifecycle. Government has to deliver securing of data assets. 52% of cloud providers moving to operate “sovereign” data centers survey respondents in this sector say that one of their greatest in many of the bigger countries. a long way ahead of the second most cited area will be essential as digital programs roll out at pace. evident externally where the sheer volume of people required to run agile programs is lacking in the market. Specialized teams and agencies services to the citizens in the way they want to consume with responsibility for securing data on a policy basis are them. this sector Furthermore. organizations in this sector. Alongside penetration testing techniques that have been in use for some time. more compares with 14% on Internet of Things solutions. take-up will accelerate. For example. Enhancing security is also the most important aspect of IT strategy by a long This future QA and Testing landscape will also continue to margin. The opportunities for QA and Testing skills are part of the digital agenda and need Internet of Things capabilities are huge. will be respondents say they use an internal testing team with its a feature in the future. Indeed. The least used is an offshore TCOE (19%). it is early days and these channel applications is that of security. challenges is the lack of an in-house testing environment. According to the World Quality Report 2016 study. the implementation of new business models. The starting point should be to understand and manage the transformation of large elements of the what data there is and then ascertain the level of security IT environment. which digitally attuned talent. that have the potential to cut testing activities. engages both internal and external teams for such security such as a mobile workforce. requiring further investment in test own tools. For the QA and Testing function. health sector and hospitals where high volumes of assets can be monitored and patient health tracked via wearable Another matter for concern in testing mobile and multi. Further. This presents a risk internally. this the rapid creation of citizen-centric business services. while 57% utilize managed application security industrialization. for has heightened the focus on testing applications for security. that of IT cost optimization (14%). by Government and the Public Sector is a captive TCOE in a build-operate-transfer model (23% of respondents). no matter how many routes are model (21%). . as well as external and the Public Sector is spent on cloud solutions. it is likely that we will see cloud’s evolution often 51% cite a lack of internal mobile testing experts. which high volumes of QA and Testing activity is required. with some shortage is indicative of a wider resourcing challenge from data on the public cloud and more highly sensitive data in the top down and a struggle to find people able to scope a private cloud. with large testing of mobile and multi-channel applications. This industrialization offered by a TCOE will be essential as this is leading to significant investments in cybersecurity and the sector continues to transform. of strategic importance.

Media and Entertainment Market. Telecom. Cited by just 40%. Preventing customer churn. is testing (ethical hacking). Capgemini Customer experience sits at the heart of business and IT enhancing security. closely followed by dynamic application testing at last year. to be tested and fixed. with the application layer being a particular focus of cyber attack. WORLD QUALITY 72 REPORT 2016-17 Telecom. Media and Entertainment Testing the end-to-end customer experience enables market differentiation Dennis Hielkema. will drive At the network level. Media and Entertainment (TME) and cost optimization of IT (44%) are the second and third sector. Media and Entertainment Sector Leader. an even distribution of the leading security testing activities this year amongst the TME study participants. the least used activity is penetration or very important by 68% of TME survey participants. there are now many more vulnerabilities revenue and profitability. significant reputational and financial implications for both mobile. A malicious hack on a Telecom companies are also pushing ahead with quad. can the service be fully penetration tested for use the data for generating better customer insights and unauthorized access to prevent revenue leakage? There is making informed investment decisions. whilst attracting customers from the competition. Companies aim to play service. Smartphone penetration has that customers only use the services they pay for. Vice President. television and internet. Big investment is being made the telco operator and its enterprise customer. in the push to deliver television as part of a quad- an ever bigger volume of network data. cited as important 52%. Sogeti Sanjeev Arya. Enhancing customer experience (46%) strategy across the Telecom. Companies must constantly test not just the network This is a business necessity as growth in more developed and infrastructure. This markets flattens and more digitally aware customers demand calls for robust end-to-end QA across all planes of business. There is also in network innovation with a move towards intelligent and an interesting security challenge in terms of how to ensure software defined networks. such as a credit card operator. customers’ network. but the service and apps as well. As an grown faster than expected and with this growth comes example. Telecom. . the leading IT strategy priorities identified code review and static application security testing are used by the World Quality Report 2016 study are unchanged from by 53%. Testing the customer journey to ensure that every most cited IT objectives. Both manual In this landscape. By far and away the top priority. Vice President. services fit for their lifestyle. The huge expectation of telecom touchpoint is both seamless and customer centric is a key network security requires a continual process of QA and part of this as companies strive to differentiate themselves. Testing. has play strategies bringing together the four streams of fixed.

Sector Analysis 73 With a great customer service being essential for success in processes that govern them. engaging both the systems and the testing and validating actual or what-if system and parameter processes around them. with agile increasingly part of apps. Programs to leverage assets and integrate enterprise. Companies in this sector need to prepare for the appropriate level of test coverage due to potentially changing This test automation and the industrialization offered in a data protection laws. . for example with connected cars in the Automotive by challenges with identifying the right areas in which to test. albeit at an early robust business case for adopting and developing solutions stage. see a higher spend on the tools and systems that give this visibility. services and software in a transformed IT issue resolution. QA partner. and across domains and markets. game- a lack of appropriate test environments and data. with TME operators taking on Appropriate tools offer the potential for innovation around more of an orchestration role. as well as on and more about a complete service. while proliferate. but a gradual The macro situation continues to impact IT strategies. particularly for digital projects. In this model. and this because they support the rapid release of new. Some 48% of TME respondents say companies have struck partnerships with firms in other that early involvement of testers in the inception phase of sectors to develop new products and get ahead of the sprint planning is difficult. there has been a struggle to define a the development cycle. perhaps online or often in-store. This calls for a tailored telcos join forces with cable and content providers to expand personal experience for each customer. and to develop part of a continuous development cycle. leading to hybrid business models by which data traversing the network can models at play. and 47% don’t have is emerging and 39% of TME survey participants say that an in-house testing environment. while 24% say the use of agile and DevOps has use outsourced testing activities from an independent created a need for more predictive analysis skills. 47% are already using predictive analysis to determine or optimize test It is thus clear why 42% of TME companies say they coverage. policies. emotional analytics tools internal capabilities as new digital channels and devices to detect a customer’s disposition on a given day. A lack of mobile testing they foresee using it as an automated test technique in the experts is cited by 44% of TME participants. for example. This journey begins with a customer’s very Quality Report continues to be a factor in this sector. and continues quad-play strategy is one of the reasons for this as traditional at every stage of the relationship. We expect further growth in this respect. Nonetheless. and now DevOps. One reason for this is limited service effectiveness. test pricing has this sector. Companies changing products. ensuring accuracy in billing and promptness in architectures. the propensity to move away from daily and fixed-price there is evidence of TME companies proactively working contracts to managed service contracts. In a DevOps environment. At such times test is expected to be massive. sector. The first contact. This will have an impact on the location TCOE model is evolving. changes. and 40% outsource to software vendors. However. paving IT estates and multiple stakeholders to assure the quality the way for cross-selling and value-selling. A challenge emerges in this end-to-end stack are thus ongoing. The high level of with partners to create delivery models aligned with the mergers and acquisitions activity noted in last year’s World customer journey. A number of without its challenges. The shift to agile and short waterfall development projects is impact on IT of a fluid European eco-political landscape evident. including the business service bundles for enterprise and retail customers. Predictive analytics mobile and multi-channel applications. providing the network the optimum way to test and assure every touchpoint on the backbone required to connect the millions of devices and customer journey. becoming less about functionality of IT infrastructure. We expect to of full service. followed in equal measure at 47% game. Agile development and continuous QA are ideal for an inability to apply test automation at appropriate levels. in this sector still struggle to combine their traditional waterfall development with the newer agile and DevOps. A mix of both decentralized and central Testing Center of The TME sector was one of the first to pick up on the Internet Excellence (TCOE) models exists as companies determine of Things trend several years ago. be monetized. Some 47% of TME sector companies in this Big Data analytics can enable a more proactive service year’s study say that they don’t have the right tools to test built on predicted customer behavior. there is evidence of testing shifting back in-house as to support the Internet of Things explosion. coming year. A big challenge is to enable visibility process of how best to bring together the components of across all services. We are seeing much of the QA and Testing work being carried out by external parties. and flow of data. underpinned by both software and network. The QA and Testing function automation can be brought in to help support the more must step up as an enabler to help navigate ambiguity by rapid release cycle. whether retail or their offerings. and heavily impacted by the adoption of digital. Currently 32% of study respondents in this sector say they do not have a specific test strategy for their The adoption of agile and use of continuous testing is not products with Internet of Things capability.

WORLD QUALITY 74 REPORT 2016-17 About the Study .

us with valid trending data. the B2B market research conducted Research participants were selected so as to ensure for the World Quality Report 2016-17 is based on a sample sufficient coverage of different regions and vertical markets of 1.000 employees (41%). where the same question was asked. The quotations research study). During the interviews. two new questions were added this year The research sample consists mainly of senior-level IT and a couple of constructive changes were also made to executives as shown in Figure 41. Research. the answers he/she provided to previous questions where applicable. Survey Sample For this year’s research. About the Study 75 The World Quality Report 2016-17 is based on research subset of these questions depending on the interviewee’s findings from 1. rather than individuals. To ensure a robust and substantive market research study. The main themes for all management functions. working for companies and public survey questions remained the same. with the actual interview consisting of a manner by the interviewee. The 44 question survey covered a range of QA . for Automotive. in consultation with Coleman Parkes shown in the report are taken from these in-depth interviews. Sogeti and HPE (sponsors of the from the additional in-depth interviews. In business-to-business (B2B) market research study. The approach and sample size used for the With the inclusion of product heads/CTOs for the first time. As mentioned above. the average recommended sample size is 100 companies.000 to provide industry-specific insight into the QA and Testing employees (25%). the full 1.600 interviews carried out during April and role in the organization. In order Healthcare and Life Sciences and High Tech Sector. the base number of respondents The required sample size varies depending on the population for each survey question shown in the graphs is not always it represents – usually expressed as a ratio or incidence rate.000 employees (in the respondent’s national to-consumer (B2C) market research because whole market) – an approach used for the last two years to provide organizations are being researched. For this reason. the research questions asked of each the recruited sample must be statistically representative of participant were linked to the respondent’s job title and the population in terms of its size and demographic profile. enriched by qualitative data obtained experts in Capgemini. responses were also added for the first time this year Quality measures were put in place to ensure the questionnaire was This interviews this year were based on a questionnaire understood. was complemented by additional in-depth interviews to The average length of each interview was 30 minutes and provide greater insight into certain subject areas and to the interviewees were all senior executives in corporate IT inform the analysis and commentary. the answer options. research this year enables direct comparisons of the current we are able to bring in their views and insights in the space results to be made with previous research studies conducted of Product and Engineering Services (P&ES). though a few objective sector organizations across 32 countries.600 sample size. organizations with more than 5.000 issues within each sector. for the report. we selected only organizations with is lower than the average sample size used for business- more than 1. This Questionnaire and Methodology The survey questionnaire was devised by QA and Testing and Testing subjects. The quantitative research study May 2016 using CATI (Computer Aided Telephone Interviews). answered accurately and completed in a timely of 44 questions. to derive better insights and to ensure effective analysis of the survey results.600 interviews from enterprises with more than 1. employees (34%) and companies with more than 10.

WORLD QUALITY 76 REPORT 2016-17 Figure 38 1600 Total Interviews by Country Country Number of Country Number of Country Number of respondents respondents respondents USA 280 Denmark 25 China 60 Canada 20 Finland 25 Hong Kong 20 France 150 Italy 35 Singapore 20 Germany 125 Spain 35 Japan 35 Switzerland 20 Portugal 35 UAE (excluding Dubai. Abu Dhabi) 15 Netherlands 100 Poland 30 Qatar 10 Belgium and Hungary 30 Luxembourg 30 Dubai 10 Czech Republic 30 UK 125 Abu Dhabi 10 Brazil 80 Ireland 25 Saudi Arabia 10 New Zealand 10 Sweden 85 Jordan and Bahrain 5 Norway 30 Australia 80 Figure 39 Interviews by Region 425 105 165 90 Southern Europe Western Europe Nordics Eastern Europe 300 North America 60 100 35 Japan Middle Southeast East Asia Asia 80 Latin America 90 150 Australia & New Zealand UK & Ireland .

About the Study 77 Figure 40 1600 Total Interviews by Sector 129 318 116 5 Automotive Healthcare and 21 Life Sciences Financial Services Telecommunications. 151 Media & Entertainment 120 Consumer Products. Retail and Distribution 245 High Tech Energy and Utilities Public Sector/ 106 115 Transportation Government 85 Manufacturing Figure 41 1600 Total Interviews by Job Title 434 CIO 363 IT Directors 290 VP Applications 88 CTO/Product Head 98 CMO/CDO 327 QA/Testing Manager .

Learn more about us at: www. Mobile. combining world class methodologies and its global delivery model. The Group reported 2015 global the industry’s most comprehensive portfolio. Cloud and Cyber Security.hpe. Sogeti is a wholly- owned subsidiary of Cap Gemini S. enabling them to achieve make IT more efficient. its worldwide delivery model. Rightshore®. technology and technology and services help customers around the world digital solutions that fit their needs. innovation and competitiveness.sogeti.capgemini. specializing in Application. listed on the Paris Stock Exchange. WORLD QUALITY 78 REPORT 2016-17 About the Sponsors About Capgemini and Sogeti About Hewlett Packard Enterprise With more than 180. Sogeti is a leading provider of technology and software testing. faster. Infrastructure and Engineering .9 billion. spanning the revenues of EUR 11. the Collaborative Business ExperienceTM. our Capgemini creates and delivers working. Capgemini has developed its own way of More information for HPE is available at www. With outsourcing or www. technology and company that enables customers to go further.000 professionals in 15 countries and has a strong local presence in over 100 locations in Europe. and draws on Rightshore®. cloud to the data center to workplace applications. Business Intelligence & Analytics.A. A deeply multicultural organization. Hewlett Packard Enterprise is an industry leading technology Capgemini is a global leader in consulting.000 people in over 40 countries. more productive and more secure. Together with its clients. USA and India. Sogeti offers cutting-edge solutions around Testing. Sogeti brings together more than 20.

www. part of the Ngaire McKeown Capgemini Group. Rightshore® is a trademark belonging to Capgemini. TMap®. expertise and market experience. About the Sponsors 79 Thank you Capgemini. is a full member of the Market Research Society. TMap NEXT®.com All the business leaders and subject matter experts who provided valuable insight into their respective areas of ©2016 Capgemini.worldqualityreport. product and service names mentioned are the trademarks of their Main Report Authors respective owners and are used herein with no intention Mark Buenen and Govind Muthukrishnan of trademark infringement. Georgia McDermitt Market Research Stephen Saw. as well as the legal their responses remain confidential and are not available obligations under the Data Protection Act 1998. Program Manager Mitali Kini Project Manager Balaji Narasimhan Creative Design Partha Karmakar Content Proof Reading Charles Kronauer Partner Management Jean-Philippe Favrot (HPE). to the sponsors. are trademarks or registered trademarks of their respective companies. In accordance with the UK Market Research Society (MRS) All research carried out by Coleman Parkes Research is Code of Conduct (under which this survey was carried out) conducted in compliance with the Code of Conduct and the identity of the participants in the research study and guidelines set out by the MRS in the UK. used herein. and their respective marks and logos from Capgemini. All Rights Reserved. All other company. Rachel Robinson and Ian Parkes (Coleman Parkes Research)* Printing and Distribution David Cole and Gerry Court (Crucial Colour) . TPI® and Writer TPI NEXT® are registered trademarks of Sogeti. Jerry Lacasia. including the authors of country and industry sections and subject-matter experts Capgemini and HPE. Assistant Writer (Region/ Country Pullouts) No part of this document may be reproduced or copied in Rahul Mitra any form or by any means without written permission from Capgemini and HPE.600 IT executives who took part in the research *Ian Parkes. Sogeti and HPE. Research. Satish Varghese. Sogeti and HPE would like to thank The 1. Sogeti and HPE. Elliot Fliesler (HPE). CEO and co-founder of Coleman Parkes study this year for their time and contribution to the report.

North America Product and Engineering Services Paul Collins Bid Director Testing Jacko Smit Julian Clarke Director Principal Testing Leader.Capgemini Govind Muthukrishnan Bhavesh Thakur Senior Vice President Senior Vice President Testing Leader. Continental Europe jacko. QA and Testing Practice Head Product V&V Practice bhavesh.thakur@capgemini.collins@capgemini. Australia Testing .com Sogeti Mark Buenen Malavika Athavale Vice President Associate Vice President Global Leader.buenen@sogeti.smit@capgemini. Financial Services Testing HPE Jean-Philippe (JP) Favrot John Jeremiah Global Alliance Director Digital Research Team Leader jp. United Kingdom HPE SW julian.