DevOps - Shift Left

Shift Left to better business results

Glyn Rhodes – Product Manager

Continuous Testing
Delivering higher quality
software faster by shifting left

© 2013 IBM Corporation

The Continuous Testing Solution: RTW, RPTS, RTVS
 IBM Rational Test Workbench
– Enable functional, performance and
integration testing throughout your project
lifecycle
– With a scriptless, wizard-driven test
authoring environment and support for
more than 70 technologies and protocols
QA DEV
Functional Testing
 IBM Rational Performance Test Server Integration Testing
Test
– Save time and effort by reusing your Performance Testing
Workbench Regression Testing
functional integration tests for performance Mobile Testing
testing
– Combine your tests to model real-world Test Load Agents
scenarios and assess the impact of load on Servers Virtualization Agents
your integrated infrastructure.
 IBM Rational Test Virtualization Server
– Model real system behavior to eliminate
test dependencies and simulate production,
– Accelerate testing and reduce the setup
and infrastructure costs of traditional testing
environments. Test Lab
Infrastructure
1 © 2013 IBM Corporation
Why organizations need Continuous Testing
Accelerating the delivery of higher quality software
Better – Cheaper - Faster

Without continuous testing there

is no continuous delivery!
Continuous Testing:
•Helps extend agile and lean practices across the
delivery lifecycle
•Removes traditional testing bottlenecks such as
unavailable test environments to increase
efficiency
•Enables teams to test earlier and with greater
coverage at lower cost
•Provides project teams with continuous feedback
on software quality to reduce business risk
Resolve today’s testing
challenges with confidence.
•Avoid the big bang!

2 © 2013 IBM Corporation

Development/Test - Expectations continue to increase
Better – Cheaper - Faster

In the Past Today

•Organizations looked to find a
balance between speed, cost & risk.
•Development teams were not as
agile as they are today; testing
never seemed to have enough time.
•Organizations and customers are
demanding teams release higher
quality software faster.
•Development teams are becoming
more and more agile; testing still
never seems to have enough time.

3 © 2013 IBM Corporation

The reality is…
Current approaches are not helping

Testing Labor Test Lab Complexity Development Velocity

Public Cloud
Private Cloud
Traditional
Routing Web/
Service Collaboration Internet Content EJB Business Testing
Providers Partners
Third-party Shared
Archives Services
Testing costs Services Portals
Messaging
File Services
systems
Directory
Identity

Other costs Enterprise

Data Warehouse Mainframe Service Bus

Heterogeneous Environments

Outsourcing labor is no longer Product and application Test teams can no longer
a sustainable model as global complexity are driving keep up with agile
wages are increasing test lab cost increases development

4 © 2013 IBM Corporation

IBM DevOps – Continuous Testing
Adopting continuous testing principles provides early and constant feedback on
quality

Continuous
Business Planning
Accelerate
software delivery –
for faster time to value
Continuous Steer Collaborative
Customer Development
Feedback &
Balance speed, cost, Optimization
DevOps Develop/
quality and risk – Operate Continuous
Feedback
Test
for increased capacity
to innovate
Continuous Deploy Continuous
Monitoring Testing

Reduce time to
customer feedback –
for improved customer Continuous Release
experience and Deployment

Our DevOps POV is resonating with clients and they are delivering
measurable business outcomes with DevOps
5 © 2013 IBM Corporation
Dev/Test and Ops – mutual goals

6 © 2013 IBM Corporation

Dev and Test – mutual goals

7 © 2013 IBM Corporation

Business drivers for change

• Market opportunity

• Keeping ahead of the competition

• Customer satisfaction

• Technology platform for future growth opportunities

8 © 2013 IBM Corporation

The rise of complex, distributed systems

• Built for interoperability

• Various technologies

• Distributed (inc. 3 party)

• Contention

• We need to test them!

$5-30 million investment 30-50% average time

Typically required to build a single test Testing teams spend on setting up test
lab for a Fortune 500 company. Most environments, instead of testing
have dozens …

9 © 2013 IBM Corporation

What does Shift Left mean for testing?

Go Live

Development Released

When defects are

found

If you are testing then you have

already begun the journey

10 © 2013 IBM Corporation

The baseline

Development

When defects are

found

11 © 2013 IBM Corporation

 Today, integration of enterprise applications still
follow the “big-bang” approach …

Big Big
Bang! Bang!

Mobile
component

Integrate

Integrate
Web
component

Mainframe 1 week 1 week

component
typ. 6 Weeks typ. 6 Weeks

 Integration amongst different components of the enterprise application occur at the end of a
milestone in a “big-bang”
 Difficult to continuously integrate and deploy across different technology stacks and varying
delivery schedules. Very hard for different teams to collaborate together on interim builds inside a
milestone
12 © 2013 IBM Corporation
The Gap

Unit Testing Integration Testing E2E Testing

13 © 2013 IBM Corporation

New world of continuous testing with shift left

Unit Testing

Integration Testing E2E Testing UAT

Performance Testing

• Automated integration testing fill the gap…

• … and shifts to the left to execute just after unit testing
begins
• More on performance testing later

14 © 2013 IBM Corporation

The changing role of the tester

Performance Testing UI Testing

Unit Testing Executing Testing Frameworks
Building Testing Frameworks Acceptance Testing

Technical Tester/Dev Traditional Tester

DevOps Tester

Integration Testing
Service Virtualization
Continuous Testing

15 © 2013 IBM Corporation

Test Centers of Excellence need to change too

Project A Project C

TCoE

Project B Project D

• Promote culture of flexibility

• Eliminates barriers to progress
• Central asset repository
• Skills the workforce in automation
and service virtualization
16 © 2013 IBM Corporation
Shift Left with Automated Integration Testing

Tests can bypass the UI and execute

beneath the iceberg. Test while your
system is being built and at an
accelerated rate. Much greater coverage
= much greater confidence

17 © 2013 IBM Corporation

Shift Left with Service Virtualization
A shift in your test environment strategy

• IBM technology helps to eliminate project and environment

contention. Create virtual components that faithfully simulate
behavior of missing components
• Now you can test when environments are unavailable or even
before they have been built
• A fundamental change to the way you can approach your
testing Substitute with
virtual
component

Interfaces to
other
Your system
systems
(e.g. Billing)

Continue to test even

when all interfacing
systems are not available
18
This new testing approach is called
“Service Virtualization”
© 2013 IBM Corporation
An Example
Batch File

Finance
Transport
SAP RFC

Tibco
SAP RFC SAP IDOC

JDBC MQ HTTPS
MQ SAP BAPI

HTTP
SAP IDOC
SAP XI 3.0

HTTP

MQ SAP BAPI JMS Manufacturing

CRM

HR Sales JDBC
HTTP

HTTP
19 © 2013 IBM Corporation
Only one-tenth of an iceberg is above water

Traditional UI testing:

Presentation layer
Automated Integration Testing and
Service Virtualization: Integrations, data
and business logic

The majority of risk in modern systems is

20 seldom tested as it is unseen.
© 2013 IBM Corporation
Discover
What kind of services do you have?

Web
Config
Service
Here’s the list and how to work with
Client them. Server

Accelerator

WSDL SOAP JSON

Accelerator

XSD XML REST WADL

HTTP(s)
• Record transactions
• Or use accelerators such as WSDLs to short cut the process
• Build tests and stubs across a range of supported protocols and formats
21 © 2013 IBM Corporation
Example details

Mantras:
Project B
• Test what changes, virtualize what does not change
Transport • Prioritize the most technical tests
• Plan to be flexible

MQ Steps:
1. Discover
2. Build Tests and Virtualized Components
3. Execute
Project A
4. Incrementally increase scope
CRM
No Change Project C
HR Sales
HTTP

HTTP
22 © 2013 IBM Corporation
The Critical Path for Continuous Testing

• The critical path is 100 days – any delays from Projects B and C will delay the program
• The impact of late delivery from Project C is much greater as the burn rate from the
program is greater at that point
• Failure at a technical integration level between the projects is the greatest risk

2
Project C – 30 days
Project A – 100 days
Project B - 20 days
1

Burn rate Burn rate

$20k per $75k per
day day

1. Project B is at risk of running late. Test environments for integration between A and
B are not ready. Every day it runs late, $20k will be burned, and the critical path is
pushed out.
2. Project C is running late. They are only half-way through their testing and are not
able to integrate yet. Every day it runs late, $75k will be burned.
23 © 2013 IBM Corporation
Service Virtualization across the SDLC

Developer uses
virtual services to
unit test code on
Development and Test teams
their workstation –
uses virtual services for early
no elaborate
stage advanced testing
environment
required

Define constrained
systems and Unit Integration System System
SystemIntegration
Integration User Acceptance Operability
Operability
services for Performance
virtualization
Later in the cycle, final pre-
The full system or components production versions of services
of the system can be are swapped in as they
performance tested early in the become available. Final testing
cycle. Virtual services can be is done against real services
used when components are
not available

24 © 2013 IBM Corporation

 DevOps Automated Testing Maturity
For complex, distributed environments

Beginner Intermediate Advanced Extreme

- UI testing - Early and - Early and - E2E

- Some Integration continuous continuous orchestration
Testing (mainly integration testing performance between UI and
manual) - No “big bang” (1) testing integration testing
- Reactive service - Plan ahead with - Automated data
virtualization service provisioning
virtualization (2)
- Traditional
performance - Automated
testing environment
provisioning

1. Don’t wait until all components are ready before beginning

integration testing
2. Test Managers can plan ahead and mitigate the risk of
environments not being available
25 © 2013 IBM Corporation
Continuous Deployment and Testing

Change

Feedback Deploy real

into dev, test Automated and
Feedback
and ops Deployment virtualized
components

Automated
Testing
Integration
testing and
UI testing
26 © 2013 IBM Corporation
Continuous testing with virtualized services
Automating deployment and testing to production-like environments
Change

Rational Jenkins
Team Concert

Dev QA Application
changes
being tested

IBM UrbanCode Deploy

Test
Databases Mainframe Third-party
applications Services
Rational Test Workbench
virtualized services

Rational Test Virtualization Server

Feedback
27 © 2013 IBM Corporation
Continuous Performance Testing

PREDICTIVE
ASSESSMENT
MONITORING

Requirements Design Coding SIT UAT Production

ASSESSMENT

Low Intensity High Intensity

Performance Testing Performance Testing
(LIP) (HIP)

Incremental integration strategy defines HIP start point;

focus of HIP is informed by LIP results

28 © 2013 IBM Corporation

 Measuring success with defect removal efficiency

£1032

£16 Unit test

£97 Integration test
£290 E2E Test
Reduction £1032 in UAT
in cost of Source: SEI
fixing
defects £290 Big Bang

£97

Requirements Integration testing E2E testing UAT

29 © 2013 IBM Corporation

Summary

 Getting to market faster requires a collaborative and continuous

approach to testing
 Continuous testing aims to fill the gap with automated integration
testing…
 …and shift it to the left

 Discovery becomes a core skill for the DevOps tester

 Complex and high risk tests can be prioritized

 Plan to be flexible with service virtualization

 Continuous testing coupled with continuous deployment, checks

the software & validates the deployment process for ops

30 © 2013 IBM Corporation

Shift Left
Is your effort directed at your risk?
Automated Integration
Testing  Testing only at the UI level
provides little confidence for
deployment of complex
Traditional UI Tools
distributed systems

 Testing must be shifted left to

focus on the highest areas of
UNIT SERVICE UI risk

 Automated integration
massively increases testing
coverage and confidence

 Service virtualization can be

applied at all phases – even
Service Virtualization allowing integration testing to
begin in parallel with Unit
Testing.
Shift Left

31 © 2013 IBM Corporation

Additional Resources
Continue your learning journey as you shift left

ibm.co/agilefordummies ibm.co/ARDfordummies

ibm.co/devopsfordummies ibm.co/ServiceVirtualizationForDummies

32 © 2013 IBM Corporation

Additional Resources
Continue your learning journey as you shift left

http://www-01.ibm.com/software/rational/servicevirtualization/

YouTube playlist:

https://www.youtube.com/playlist?list=PLlhg84-xzjfwdUShtwq7sG4HZQpFUwwFM

Search for: IBM Service Virtualization

33 © 2013 IBM Corporation

 ibm.com/devops

34
Acknowledgements and disclaimers
Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries
in which IBM operates.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for
informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant.
While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without
warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this
presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or
representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of
IBM software.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have
achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to,
nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

© Copyright IBM Corporation 2012. All rights reserved.

– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products and
services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these
and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate
U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or
common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at
www.ibm.com/legal/copytrade.shtml

35 © 2013 IBM Corporation