You are on page 1of 19

The 3 Essential Building

Blocks of Continuous Testing


With a 5-Step Plan For Building a
Continuous Testing Foundation
Increasing Automation to Decrease Business Risk
Testing is essential to ensure that your software retains an
Agile, Continuous Testing &
acceptable quality level and provides a great experience for
your users. And automating as much of your testing strategy as Continuous Delivery
possible will help improve efficiencies across your entire DevOps Although enterprises are adopting Agile
pipeline. At first glance, “automated testing” and “continuous methodologies for their software development,
testing” might seem interchangeable, however, Continuous very few have actually reached a satisfactory
level of test automation.1 The philosophy of
Testing (CT), which descends from Continuous Integration (CI)
the Manifesto for Agile Software Development
and Continuous Delivery (CD) principles, isn’t primarily about is closely tied to the concept of CI, which
automation; it’s about business risk - especially the risk of suggests that code changes should be merged
losing your competitive advantage and loyal customer base. often, and to CD, which dictates that software
should be production-ready at all times. This
is where continuous testing – and risk – step
The key conceptual difference between test automation and CT in. When more changes are made more quickly,
is that releasing a sub-standard piece of software is bad for your bugs can creep in. CT is the methodology –
business. In the best case, customers are disappointed. In the and the mindset – of testing at regular points
worst case, they leave and don’t come back. Continuous testing throughout the DevOps pipeline to ensure
is all about making sure that despite quicker release schedules, that quality is high enough to minimize risk to
the business from releasing the software. As
your brand’s reputation and the user experience are protected. release schedules get tighter, automation is
Risk avoidance is at the heart of CT. critical to a successful CT strategy.

1
https://sdtimes.com/agile/report-testing-lags-agile-development-shops/

2
Test automation gives pass/fail validations of code at
various points in the pipeline

Continuous testing is the process of considering whether


the code is safe to release

3
The 3 Fundamentals of CT
Organizations – and developers – need to innovate to stay competitive, and with release schedules
shrinking all the time, it’s more likely that a business-threatening bug might slip through to production.

Successfully implementing CT, which aims to avoid business-killing software failures, depends on
continuous alignment between people, processes, and technology in the organization. Each of these
three are essential elements to a properly functioning CT organization.

People Challenges
When making the shift towards CT in your organization, people are crucial to your success.

In the context of CT, ”people” can be broken down into roles or personas within teams – executives,
developers, testers, operations, and security. Since each has a unique responsibility within the delivery
pipeline, it is critical that each has the proper skills for their role and that there is a high level of
coordination between them. Effective communication is key to having all parties on the same page.

Leadership needs to make sure that the various individuals work as a team and that they have the right set
of tools, requirements, and continuous measurements to work with. If there is a skill set issue, it must be
addressed so that there are no gaps due to lack of capabilities within teams.

4
Process
Continuously releasing value to the customers, with high quality, mitigated risks, and efficient feedback
and analytics that can be acted upon, is a challenging task for today’s DevOps teams.

Time constraints often get in the way of following common practices, such as:

• Treating staging code as production code, full version control, etc.

• Including test automation within the “definition of done” per iteration

• Committing new staging test code only when it shows consistent results

• Continuously re-factoring staging test code to make sure it is still relevant, and excluding
redundant tests from CI/regression suites

• Test automation practices, use of proper object identifier strategies, and page object model

5
Subhead 1 Open Sans Nunc neque tortor, consequat hendrerit massa
in, dictum porta nunc. Nam ultrices vehicula
Regular
Today, 18pt
there over
is a lot 22pts
of focus dapibus. Sed id porta urna. Curabitur eleifend
on percentage of test automation - as well as test automation suite size
justo eu elit elementum tincidunt. Class aptent
-Body
rather
copy than test dolor
lorem ipsum automation
sit amet, code quality, stability,
taciti sociosqu ad litoraand efficiency
torquent and its ability to provide valuable
per conubia
consectetur adipiscing elit. Maecenas viverra nostra, per inceptos himenaeos. Aliquam
feedback to tincidunt
risus eget dolor developers.
elementum. Curabitur accumsan ultricies elit, sit amet volutpat velit
massa mauris, malesuada ut pulvinar id, tincidunt at.
facilisis lobortis leo. Cras quis gravida purus.
To correct this, executives must promote change, from measurements and metrics, to identifying
Ut eu turpis id nulla rhoncus eleifend. Mauris
• The [bullet run-in character] for emphasis
issues across the various teams and ensuring
is sentenceuse
posuere felis in semper ultrices. Quisque in case.of the
This right tools
paragraph and practices, such as including
style is [body
eleifend ipsum, sit amet vestibulum ligula.
automation as part of the definition ofbullet
Maecenas dignissim lectus at sem pharetra
done (DoD).
first] and is sentence case, set in Open
Sans Regular 10/15pt.
fringilla.
• The [bullet run-in character] for emphasis
Leadership
Nunc neque tortor,should adopt
consequat a “start
hendrerit massa small, then
is sentence grow” methodology as a process. Just as teams
case.
develop theirnunc.
in, dictum porta code inultrices
Nam smallvehicula
chunks per––iteration and stabilize
The [body subbullet] them,
is sentence sointoo should testing grow and be
case, set
dapibus. Sed id porta urna. Curabitur eleifend
stabilized throughtincidunt.
justo eu elit elementum automation and CI. Only
Class aptent thenRegular
Open Sans can you grow
10/14pt. Sim the
ing eusuite.
etue
conulluptat wis ea acil iustrud essi ex enit
taciti sociosqu ad litora torquent per conubia
auguero od exer auguerostrud
nostra, per inceptos himenaeos. Aliquam
accumsan ultricies elit, sit amet volutpat –– The [body subbullet] is sentence case
velit tincidunt at. Nunc consequat ac purus • The [body 2nd subbullet] is sentence
a vehicula. Sed ullamcorper elit tortor, non case
vestibulum nulla tristique ac. Proin eu mauris in
nisl bibendum porttitor id quis tellus. Vivamus • The [body 2nd subbullet] is sentence
nec quam eleifend magna aliquet faucibus case consecte dolor suscilla adigna.
non quis metus. Vestibulum at augue diam. Atem iliquis dunt volutpat, vel ut aliq
Donec pretium dignissim nisi, sed laoreet dolor.
Quisque pellentesque venenatis dui, quis iaculis
purus aliquet eget.

6
Technology
Good leadership and practices on their own aren’t enough to successfully implement a CT strategy. You also
need alignment with your technologies and test labs. Whenever a test environment is unstable, insecure,
not scalable enough, and not easily integrated with other technologies and testing frameworks, teams can
fail to meet their DevOps objectives.

In an Agile environment, where various developers and testers use different technologies and platforms,
and sometimes are composed of remote teams, having a stable test environment upon which to base all
quality activities is a key for success. It is not just about matching the test lab and the environment to
the tools of the team, but also matching it to the teams’ skills and software methodologies, such as BDD,
ATDD, TDD, or others.

Test labs that cannot fit seamlessly into development CI processes cannot serve the requirements of
continuous testing. The stability and reliability of CI relies on the lab as its backbone.

7
In the era of DevOps and CT2, there is much more test data that is being generated per each execution.
Gaining insight into large amounts of test data and a deeper understanding of specific issues can be a key
enabler for implementing CT. Executives can’t reach a “go, no-go” decision without executive dashboards;
developers and release managers
struggle to see whether their CI
processes and builds are moving in
the right direction and remaining
within time constraints. In many
cases, pinpointing an issue feels like
finding a needle in a haystack, and
test engineers and test managers
might lack the necessary test
artifacts to provide as much feedback
as possible to developers.

Successfully implementing a CT approach is easier said than done. There are many moving parts in the
pipeline that can slow down or entirely disrupt your quality release activities.

2
Cloud Computing – making CI, CD and CT work together - https://www.cloudcomputing-news.net/news/2018/apr/11/
finding-right-agile-formula-making-ci-ct-and-cd-work-together/

8
Page Title Officina Sans Black
8 Considerations for Executives When Planning
headline 35pts over 43pts
for Continuous Testing
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
ipsum ut, pulvinar lobortis lectus. Phasellus mollis ornare euismod. Fusce semper nisi eget bibendum cursus.
When planning
Vivamus sit ametfor a continuous
semper nunc, sed testing
rutrumstrategy, leadership
sem. Ut sed should
lectus non have a sagittis
est pharetra comprehensive plan for
sed non enim. all teams
Curabitur
and individuals:
rutrum ante enim, nec porta mauris pharetra.

1. Number of projects and their type


Body copy Open Sans light 10pts over 14pts
justo (mobile,
eu elit web,
elementum responsive
tincidunt. web,ultricies
Class aptent progressive web,
elit, sit amet etc.)
volutpat velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
2. Team size – dev, test, ops, source
adipiscing elit. code
nostra, per management,
inceptos etc.
himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
3. Team skill sets – development
velitlanguages,
Duis lacus odio, pretium ac varius eget, testing
tincidunt at. Nunc framework
consequat ac purus familiarity porttitor id quis tellus. Vivamus nec quam
consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
4. Software development lifecycle methods
purus, dignissim sed ipsum ut, pulvinar lobortis
vestibulum – ATDD,
nulla tristique BDD,
ac. Proin eu etc
mauris in metus. Vestibulum at augue diam. Donec
lectus. Phasellus mollis ornare euismod. Fusce nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
5. Technology availability – which
semper nisi eget bibendum cursus. Vivamus tooleleifend
nec quam stackmagna
is currently in use and what is missing?
aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
6. Market trends and analyticsDonec
in order
lectus non est pharetra sagittis sed non enim. todignissim
pretium supportnisi,new features,
sed laoreet dolor.user stories, etc.
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
7. Lab sizing and coverage requirements
pharetra quis. purus aliquet eget.
Nunc neque tortor, consequat hendrerit massa
8. Metrics for success which are achievable,
elementum
in, dictum porta nunc. Nam ultrices vehicula
meaningful,
tincidunt. and clearly understood
Class aptent taciti
sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan
“Plan your work for today and every day, then work your plan.” ~ Margaret Thatcher

9
Page Title Officina Sans Black
Aheadline
5-Step Plan
35ptsfor Building
over 43pts your CT Foundation
Duis lacus
After odio,
setting pretium
goals ac variusobjectives
and quality eget, consequat pellentesque
for your quam.
projects, the Suspendisse
various nisl in
individuals purus,
yourdignissim
team mustsedhave
aipsum
solid ut, pulvinar lobortis
foundation lectus.inPhasellus
under them order to mollis ornaregoals.
meet these euismod.
ThisFusce semper relies
foundation nisi eget
on bibendum cursus.
a few important
Vivamus sit amet semper nunc, sed rutrum sem. Ut sed lectus non est pharetra sagittis sed non enim. Curabitur
elements:
rutrum ante enim, nec porta mauris pharetra.

Body copy Open Sans light 10pts over 14pts


• Test automation frameworkjusto
andeutest
Lorem ipsum dolor sit amet, consectetur
elit elementum tincidunt. Class aptent
authoring
taciti sociosqu ad litora torquent per conubia
ultricies elit, sit amet volutpat velit tincidunt
at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit.
• Test framework fit with yournostra,
teams' per inceptos himenaeos. Aliquam
requirements
accumsan ultricies elit, sit amet volutpat
ullamcorper elit tortor, non vestibulum nulla
tristique ac. Proin eu mauris in nisl bibendum
Duis lacus odio, pretium ac varius eget,
• Risk-based test automationvelit tincidunt at. Nunc consequat ac purus
strategy
consequat pellentesque quam. Suspendisse nisl
a vehicula. Sed ullamcorper elit tortor, non
porttitor id quis tellus. Vivamus nec quam
eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis
vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
• Stable CI maintained continuously
lectus. Phasellus mollis ornare euismod. Fusce
nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
• Support for test data generation/analytics
sit amet semper nunc, sed rutrum sem. Ut sed
non quis metus. Vestibulum at augue diam. purus aliquet eget.
lectus non est pharetra sagittis sed non enim. Donec pretium dignissim nisi, sed laoreet dolor.
• Stable lab and test environment
Curabitur rutrum ante enim, nec porta mauris
Quisque pellentesque venenatis dui, quis iaculis
pharetra quis. purus aliquet eget.
Nunc neque tortor, consequat hendrerit massa elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

10
Page Title Officina Sans Black
1. Test Automation Framework and Authoring
headline 35pts over 43pts
Authoring test automation code should be no different than developing and maintaining code for your
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
application features, such as login functionality, etc.
ipsum ut, pulvinar lobortis lectus. Phasellus mollis ornare euismod. Fusce semper nisi eget bibendum cursus.
Vivamus sit amet semper nunc, sed rutrum sem. Ut sed lectus non est pharetra sagittis sed non enim. Curabitur
As the product
rutrum evolves,
ante enim, so too
nec porta should
mauris your test scenarios. When the test code is not properly maintained
pharetra.
through source code management (SCM) tools, it is hard to make timely changes and build on top of them
since there is no proper documentation,
Body copy Open Sans light 10pts over 14pts
traceability, or history of the testultricies
justo eu elit elementum tincidunt. Class aptent
code.elit, sit amet volutpat velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
In addition, when a new functionality is built into your mobile application or your web site, this requires
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
Duis lacus odio, pretium ac varius eget,
changes to your test code that may include writing new tests, modifying existing test flows, retiring tests,
velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
or maybe
purus, even
dignissim sedmerging new tests
ipsum ut, pulvinar into one large test. To be able to continuously address product changes,
lobortis
vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
your
lectus.test codemollis
Phasellus always
ornareneeds toFusce
euismod. be in sync with the product’s evolution and carry with it relevant history and
nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
documentation.
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
lectus non est pharetra sagittis sed non enim. Donec pretium dignissim nisi, sed laoreet dolor.
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
pharetra quis. purus aliquet eget.
Nunc neque tortor, consequat hendrerit massa elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

11
Page Title Officina Sans Black
headline 35pts over 43pts
Just as you treat your production code with ongoing analysis, refactoring, and more, it is critical that you
do the same for your test code. This is an enabler for CT.
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
ipsum ut, pulvinar lobortis lectus. Phasellus mollis ornare euismod. Fusce semper nisi eget bibendum cursus.
Developing
Vivamus yoursemper
sit amet test code thesed
nunc, right waysem.
rutrum fromUtthe
sedstart can
lectus mean
non the difference
est pharetra sagittisbetween a high-
sed non enim. Curabitur
value test
rutrum ante suite
enim,and
nec aporta
time-consuming, flaky and inefficient one. As mentioned, treating test code as
mauris pharetra.
production code is recommended. Writing good, valuable tests is a different practice that involves
various
Body considerations,
copy Open including:
Sans light 10pts over 14pts justo eu elit elementum tincidunt. Class aptent ultricies elit, sit amet volutpat velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
• The right object identification strategy
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
Duis lacus odio, pretium ac varius eget, velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
• The right test framework to work with
consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
• Measuring test efficiency within the CI
lectus. Phasellus mollis ornare euismod. Fusce nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
• Taking a risk-based approach to test automation
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
lectus non est pharetra sagittis sed non enim. Donec pretium dignissim nisi, sed laoreet dolor.
• Continuous test data analysis and improvement
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
pharetra quis. purus aliquet eget.
Nunc neque tortor, consequat hendrerit massa elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

12
Page Title Officina Sans Black
2. Risk-Based
headline 35ptsTest Automation
over 43pts Strategy
A proper
Duis CT strategy
lacus odio, pretiumoffers feedback
ac varius and the best
eget, consequat possible experience
pellentesque to end users.
quam. Suspendisse However,
nisl purus, these
dignissim sedare
ipsum ut, pulvinar
only part lobortis lectus.
of the equation. Phasellus mollis
An organization mustornare euismod.
always Fusce semper
be considering nisi eget bibendum
the potential cursus.
consequences of
Vivamus sit software
releasing amet semper
thatnunc,
mightsed rutrumthe
damage sem. Ut sed lectususer
all-important non experience.
est pharetra sagittis
This is sed
the non
real enim. Curabitur
mission of CT.
rutrum ante enim, nec porta mauris pharetra.
One possible approach to developing a CT test automation strategy would be to embrace the best
Body copy Open Sans light 10pts3over 14pts
practices Angie Jones laid out: justo
Lorem ipsum dolor sit amet, consectetur
eu elit elementum tincidunt. Class aptent
taciti sociosqu ad litora torquent per conubia
ultricies elit, sit amet volutpat velit tincidunt
at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
- What's the test engineer’s gut feeling?
Duis lacus odio, pretium ac varius eget,
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
- Risk calculated as probability to occur and impact to customers
consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis
- Value – does the test provide new information and, if failed, how much time to fix?
lectus. Phasellus mollis ornare euismod. Fusce
vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
- Cost efficiency to develop – how long does it take to develop and how easy is it to script?
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
sit amet semper nunc, sed rutrum sem. Ut sed
- History of test – volume of historical failures in related areas and frequency of breaks
lectus non est pharetra sagittis sed non enim.
non quis metus. Vestibulum at augue diam. purus aliquet eget.
Donec pretium dignissim nisi, sed laoreet dolor.
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
pharetra quis.
It is important to use either this or another method that makes sense from quality, business, and time-
purus aliquet eget.

to-market
Nunc perspectives
neque tortor, to drive
consequat hendrerit efficient,
massa ongoing
elementum tincidunt.CT processes.
Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

3
Angie Jones slide share on what to automate - https://www.slideshare.net/saucelabs/which-tests-should-we-
automate-by-angie-jones

13
Page Title Officina Sans Black
3. A Stable CI
headline 35pts over 43pts
The essential components for achieving efficient DevOps are CI, CD, and CT.
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
ipsum ut, pulvinar lobortis lectus. Phasellus mollis ornare euismod. Fusce semper nisi eget bibendum cursus.
If Continuous
Vivamus sit ametDelivery
semper and
nunc,Continuous
sed rutrumTesting
sem. Utare
sedthe wheels
lectus on apharetra
non est car, then CI is sed
sagittis the non
engine that
enim. drives
Curabitur
them. ante
rutrum That’s whynec
enim, continuous integration
porta mauris pharetra.builds should always be green unless there is a real issue. To
make sure your CI is always green, teams need to focus and develop processes that decide what gets into
thatcopy
Body engine.
Open Sans light 10pts over 14pts justo eu elit elementum tincidunt. Class aptent ultricies elit, sit amet volutpat velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
Duis lacus odio, pretium ac varius eget, velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
lectus. Phasellus mollis ornare euismod. Fusce nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
lectus non est pharetra sagittis sed non enim. Donec pretium dignissim nisi, sed laoreet dolor.
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
pharetra quis. purus aliquet eget.
Nunc neque tortor, consequat hendrerit massa elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

14
Page
4. TestTitle DataOfficina
Generation Sans Black
headline 35pts over 43pts
Depending on the market segment or vertical, test data generation offers high value to both developers
andlacus
Duis testers.
odio,For banks,acthe
pretium ability
varius toconsequat
eget, continuously test against
pellentesque pre-generated
quam. banking
Suspendisse nisl purus,accounts
dignissimthat
sed
have various
ipsum profiles
ut, pulvinar is key
lobortis to validating
lectus. both functionality
Phasellus mollis ornare euismod.of their
Fusce apps as well
semper as performance
nisi eget in
bibendum cursus.
pulling sit
Vivamus account data. The
amet semper same
nunc, sedgoes forsem.
rutrum insurance
Ut sedapps
lectusand
nonhealthcare apps.
est pharetra Such
sagittis sedtest
nondata cannot
enim. be
Curabitur
actualante
rutrum production
enim, nec data formauris
porta a widepharetra.
variety of reasons. Therefore, as part of CT, teams need to have a way
to virtualize and use this test data. Also, when the product evolves and test data becomes outdated,
there
Body copymust be alight
Open Sans clear process
10pts for refreshing
over 14pts justo eu elitthe data to
elementum include
tincidunt. Classnew fields
aptent or other
ultricies elit, sitadvanced features.
amet volutpat velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
Duis lacus odio, pretium ac varius eget, velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam

5. 24/7 Stable Lab and Test Environment


consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
lectus. Phasellus mollis ornare euismod. Fusce nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
As the entire CI process relies on well-written test code, both test execution and CI stability depend
lectus non est pharetra sagittis sed non enim. Donec pretium dignissim nisi, sed laoreet dolor.
on the availability and stability of the test lab. A stable lab means that all platforms connected to it,
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
pharetra quis. purus aliquet eget.
together with supporting test environments, 3rd party tools, and APIs, are working 24/7 with near-zero
Nunc neque tortor, consequat hendrerit massa
downtime. To create such an environment,
in, dictum porta nunc. Nam ultrices vehicula
elementum intincidunt.
the DevOps 4
landscape,
Class aptent taciti there is no doubt that the cloud
sociosqu ad litora torquent per conubia nostra,
has proven
dapibus. to be
Sed id porta the
urna. only practical
Curabitur eleifend way to meet
per inceptos these Aliquam
himenaeos. objectives.
accumsan

4
Mabl blog – mapping the DevOps tool chain - https://www.mabl.com/blog/devops-tools-organized-by-category

15
Start
Page Small and GrowSans
Title Officina Your Black
CT
headline 35pts over 43pts
Incorporating CT into your DevOps pipeline means always having the right coverage at each stage, both
from a test scenario and a platform perspective.
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
Each
ipsumphase of DevOps
ut, pulvinar has lectus.
lobortis different objectives
Phasellus mollisand eacheuismod.
ornare of theseFusce
require different
semper coverage
nisi eget principles.
bibendum cursus.
The earlysit
Vivamus stages
amet of developing
semper a new
nunc, sed functionality
rutrum involve
sem. Ut sed lectusmany unit
non est test activities
pharetra andnon
sagittis sed back-and-forth
enim. Curabitur
debugging
rutrum antecycles
enim, by
necthe developer.
porta Typically, these activities happen on the developer’s workstation and,
mauris pharetra.
to get fast feedback, the developer doesn’t need a full-blown lab and will be satisfied with one or two
relevant platforms.
Body copy Open Sans light As development
10pts over 14pts of theeufeature(s)
justo elit elementum evolves, there
tincidunt. Class are different
aptent quality
ultricies elit, sit amet and development
volutpat velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
criteria which require an extended sets
adipiscing elit.
of test scenarios and platforms toullamcorper
nostra, per inceptos himenaeos. Aliquam
be executed; this requires a
elit tortor, non vestibulum nulla
scheduled CI workflow backed by a larger accumsantestultricies
lab (that
elit, sitwill
ametprobably
volutpat be cloud-based)
tristique ac. Proin toeuprovide support
mauris in nisl bibendum
Duis lacus odio, pretium ac varius eget,
for various
consequat users, various
pellentesque scales,nisl
quam. Suspendisse andvelit
a more complex
tincidunt test environment.
at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
lectus. Phasellus mollis ornare euismod. Fusce
Continuous testing within the DevOps pipeline should be built according to each different phase, from
semper nisi eget bibendum cursus. Vivamus
nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
early development through production monitoring. Each phase will include its relevant testing activities
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
lectus non est pharetra sagittis sed non enim.
which will be triggered based on the activity and quality objectives via the relevant environment (dev
Curabitur rutrum ante enim, nec porta mauris
Donec pretium dignissim nisi, sed laoreet dolor.
Quisque pellentesque venenatis dui, quis iaculis
workstation vs. CI server) – per commit, nightly regression, and ongoing monitoring. For each, and based
pharetra quis. purus aliquet eget.
on
Nuncthe team’s
neque tortor,process, skills, and
consequat hendrerit massaCT plan, teams will need to have the appropriate test framework for
elementum tincidunt. Class aptent taciti
coding
in, dictum(Espresso, Appium,
porta nunc. Nam XCUITest, sociosqu
ultrices vehicula Protractor, etc.)
ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

Also, CT triggers and environments do not define the testing scope; for this, each project needs a solid
test plan and quality criteria that strike the correct balance between API, unit, UI, functional, and non-
functional testing.

16
Page Title Officina Sans Black
headline 35pts
Minimize risk over 43pts
- Develop the CT Mindset Across the
Organization
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
ipsum ut, pulvinar lobortis lectus. Phasellus mollis ornare euismod. Fusce semper nisi eget bibendum cursus.
Vivamus sit amet semper nunc, sed rutrum sem. Ut sed lectus non est pharetra sagittis sed non enim. Curabitur
rutrum anteout
We’ve laid enim,
thenec porta mauris
background, pharetra.
theory, and some best practices for implementing CT in your organization.
Implementing CT is not an easy process. There’s not a button you can push to make it happen. If your
organization
Body isn’t
copy Open Sans lightthere yet,14pts
10pts over you’re not
justoalone. According
eu elit elementum to a Class
tincidunt. survey conducted
aptent at Cloud
ultricies elit, sit amet Expo
volutpatEurope, 43%
velit tincidunt
Lorem ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
of organizations
adipiscing elit. haven’t yet adoptednostra,DevOps. 5
Continuous
per inceptos himenaeos. testing
Aliquam is a mindset thatelitneeds
ullamcorper to vestibulum
tortor, non be fostered nulla
across your whole organization, from top to bottom.
Duis lacus odio, pretium ac varius eget,
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
consequat pellentesque quam. Suspendisse nisl a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis
CT is about incorporating a testing mentality at every stage of the SDLC pipeline in order to minimize
lectus. Phasellus mollis ornare euismod. Fusce
vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
risks to user experience, and at a more fundamental level, your company’s reputation. Continuous Testing
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
sit amet semper nunc, sed rutrum sem. Ut sed
is not something that you do when everything else is done. It needs to be part of the process. With
lectus non est pharetra sagittis sed non enim.
non quis metus. Vestibulum at augue diam. purus aliquet eget.
Donec pretium dignissim nisi, sed laoreet dolor.
integrations built for nearly every testing framework, unparalleled quick-view visual analytics, and built
Curabitur rutrum ante enim, nec porta mauris Quisque pellentesque venenatis dui, quis iaculis
into all major CI platforms, Perfecto has the tools to help ease your transition to a successful Continuous
pharetra quis. purus aliquet eget.

Testing
Nunc strategy.
neque tortor, consequat hendrerit massa elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

5
https://medium.com/ecs-digital/7-devops-adoption-statistics-for-2017-infographic-ca368065a7c7

17
Page Title Officina Sans Black
Appendix: Glossary of Terms6
headline 35pts over 43pts
• Test automation: Designed to produce a set of pass/fail data points correlated to user stories and
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
ipsumapplication requirements
ut, pulvinar lobortis lectus. Phasellus mollis ornare euismod. Fusce semper nisi eget bibendum cursus.
• Continuous
Vivamus integration:
sit amet semper nunc, sed
Therutrum sem.
practice of Ut sed lectus
merging non est pharetra
all developer workingsagittis
copies sed
tonon enim. mainline
a shared Curabitur
rutrum ante enim,
several timesnec porta mauris pharetra.
a day
• Continuous delivery: The evolution of continuous integration by always being able to put a product
Body copy Open Sans light 10pts over 14pts justo eu elit elementum tincidunt. Class aptent ultricies elit, sit amet volutpat velit tincidunt
Lorem into
ipsumproduction
dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
• Continuous deployment: Takingaccumsan continuousultriciesintegration and continuous
elit, sit amet volutpat delivery
tristique formauris
ac. Proin eu automatic
in nisl bibendum
Duis lacus odio, pretium ac varius eget,
consequat
deployment
pellentesqueto production
quam. Suspendisse nisl
velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
a vehicula. Sed ullamcorper elit tortor, non eleifend magna aliquet faucibus non quis
purus, dignissim sed ipsum ut, pulvinar lobortis
• Continuous development: The umbrella term of continuous integration, continuous delivery, and
lectus. Phasellus mollis ornare euismod. Fusce
vestibulum nulla tristique ac. Proin eu mauris in metus. Vestibulum at augue diam. Donec
nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
continuous deployment
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
• Continuous testing: The process of executing automated tests as part of the software delivery
lectus non est pharetra sagittis sed non enim. Donec pretium dignissim nisi, sed laoreet dolor.
Curabitur rutrum ante enim, nec porta mauris
pipeline in order to gain feedback on business risks with a software release candidate as rapidly
pharetra quis.
Quisque pellentesque venenatis dui, quis iaculis
purus aliquet eget.
Nunc neque
as possible
tortor, consequat hendrerit massa elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula
• DevOps: The software engineering practice that aims at unifying software development (Dev) and
dapibus. Sed id porta urna. Curabitur eleifend
sociosqu ad litora torquent per conubia nostra,
per inceptos himenaeos. Aliquam accumsan
software operation (Ops)

6
https://www.techwell.com/techwell-insights/2018/01/whats-winter-2018-issue-better-software-magazine

18
Page Title Officina Sans Black
headline 35pts over 43pts
About Perfecto
Duis lacus odio, pretium ac varius eget, consequat pellentesque quam. Suspendisse nisl purus, dignissim sed
ipsum ut, pulvinar
Perfecto lobortis
enables lectus. Phasellus
exceptional digitalmollis ornare euismod.
experiences. Fusce
We help semper
you nisi eget
transform bibendum
your businesscursus.
and
Vivamus sit amet every
strengthen semperdigital
nunc, sed rutrum sem.
interaction withUt sed lectus non estapproach
a quality–first pharetra sagittis sed non
to creating enim.
web andCurabitur
native
rutrum ante enim, nec porta mauris pharetra.
apps, through a cloud–based test environment called the Continuous Quality Lab™. The
CQ Lab is comprised of real devices and real end–user conditions, giving you the truest test
Body copy Open Sans light 10pts over 14pts justo eu elit elementum tincidunt. Class aptent ultricies elit, sit amet volutpat velit tincidunt
environment
Lorem available.
ipsum dolor sit amet, consectetur taciti sociosqu ad litora torquent per conubia at. Nunc consequat ac purus a vehicula. Sed
adipiscing elit. nostra, per inceptos himenaeos. Aliquam ullamcorper elit tortor, non vestibulum nulla
accumsan ultricies elit, sit amet volutpat tristique ac. Proin eu mauris in nisl bibendum
More than 1,500 customers, including 50% of the Fortune 500 across the banking, insurance,
Duis lacus odio, pretium ac varius eget, velit tincidunt at. Nunc consequat ac purus porttitor id quis tellus. Vivamus nec quam
consequat pellentesque quam. Suspendisse nisl
retail, telecommunications and media industries rely on Perfecto to deliver optimal mobile
purus, dignissim sed ipsum ut, pulvinar lobortis
a vehicula. Sed ullamcorper elit tortor, non
vestibulum nulla tristique ac. Proin eu mauris in
eleifend magna aliquet faucibus non quis
metus. Vestibulum at augue diam. Donec
app functionality and end user experiences, ensuring their brand’s reputation, establishing
lectus. Phasellus mollis ornare euismod. Fusce nisl bibendum porttitor id quis tellus. Vivamus pretium dignissim nisi, sed laoreet dolor.
semper nisi eget bibendum cursus. Vivamus nec quam eleifend magna aliquet faucibus Quisque pellentesque venenatis dui, quis iaculis
loyal customers, and continually attracting new users. For more information about Perfecto,
sit amet semper nunc, sed rutrum sem. Ut sed non quis metus. Vestibulum at augue diam. purus aliquet eget.
visit www.perfecto.io, join our community follow us on Twitter at @PerfectoMobile.
lectus non est pharetra sagittis sed non enim.
Curabitur rutrum ante enim, nec porta mauris
Donec pretium dignissim nisi, sed laoreet dolor.
Quisque pellentesque venenatis dui, quis iaculis
pharetra quis. purus aliquet eget.
Start your free trial with Perfecto today!
Nunc neque tortor, consequat hendrerit massa
elementum tincidunt. Class aptent taciti
in, dictum porta nunc. Nam ultrices vehicula sociosqu ad litora torquent per conubia nostra,
dapibus. Sed id porta urna. Curabitur eleifend per inceptos himenaeos. Aliquam accumsan

19

You might also like