Agile Methodology: The Complete Guide to Understanding Agile Testing

Over the past several years, a new way of creating software has taken the software development and
testing world by storm: Agile.
In fact, according to VersionOne’s State of Agile Report, as of 2018, 97% of organizations practice agile
in some form. However, respondents report that this adoption is not always widespread within their
organizations, which means there’s still a long way to go in terms of adoption and maturity.

So what exactly is agile and why has it become so popular so quickly?

Let’s explore exactly what agile methodologies entail and how to introduce it in your organization in
more detail. Specifically, we’ll cover:
 How does testing fit into agile methodologies?
 What are the different ways to test on an agile team?
 What’s next for the agile movement?

About Agile Methodology

Agile methodology has taken the software development world by storm and rapidly cemented its place as
“the gold standard.” Agile methodologies all started based on four core principles as outlined in the Agile
Manifesto. These methodologies are rooted in adaptive planning, early delivery and continuous
improvement, all with an eye toward being able to respond to change quickly and easily. As a result, it’s
no surprise that 88% of respondents in VersionOne’s 2017 State of Agile Report ranked “ability to adapt
to change” as the number one benefit of embracing agile.
However, as more and more development teams adopt an agile philosophy, testers have struggled to keep
pace. That’s because the widespread adoption of Agile has led teams to issue releases and totally
undocumented software on a more frequent basis. This frequency has forced testers to shift when they
conduct testing, how they work with developers and BAs and even what tests they conduct, all while
maintaining quality standards.
What Does it Mean to Test on an Agile Team?
Agile principles are all about being collaborative, flexible and adaptive. It’s built on the premise that the
world now changes regularly, and that means software teams no longer have years to bring new products
to market. In that time, competitor offerings or customer expectations can change, and the team risks
irrelevance. Agile minimizes this risk by helping teams collaborate together more by adapting to what the
team needs to be successful. It does this by encouraging teams to regularly show off their work and gather
feedback so that they can adapt to change quickly.

Narrowing in on testing, the fast pace of agile development has led to several imperatives for testers:

1. Prioritizing requirements based on risk since it’s not possible to test everything

2. Automating tests to increase efficiency

3. Increasing the use of exploratory testing to accelerate the time from code delivery to test
completion and to emphasize the need to create code that works

4. Adapting to changes from sprint to sprint

The fourth imperative — adaptability — is particularly important because it requires testers to have
broader, cross-functional testing skills, which represents a departure from the narrower testing skills often
needed in a waterfall environment. Additionally, unlike in a waterfall environment, testers following an
agile methodology need to stay in close contact with developers to collaborate on testing throughout the
entire software development lifecycle. In waterfall methodologies, there is often a large requirements
document that testers test against. That document doesn’t change frequently, so testers have been able to
exist fairly independently of developers. However, most Agile methodologies are light on documentation
and the requirements for a new feature may only be in a ticket in a requirements tracking system without
all the edge cases listed. The testers in these scenarios need to be highly communicative with the
development and business teams, as tests they wrote a few weeks ago may become obsolete quickly. To
succeed, testers need to be flexible and able to adapt to moving targets.
To succeed, testers need to be flexible and able to adapt to moving targets.
In general, there are four central tenets of the Agile Manifesto that are important for testers to remember:
1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Responding to change over following a plan
4. Collaborating with customers over contract negotiation
The bottom line in all of this is that everyone — testers, developers and beyond — must evolve in order
to embrace an agile style of working.

Agile is Not One-Size-Fits-All

Every organization is unique and faces different internal factors (i.e. organization size and stakeholders)
and external factors (i.e. customers and regulations). To help meet the varying needs of different
organizations, there are various agile methodologies and several different types of testing you can do
while working within one of those agile methodologies. Which mix is right for your team will depend on
your internal and external factors, needs and goals.
Let’s take a look at what some of the most popular agile methodologies and testing methods entail,
including:
 Agile Methodologies
o Scrum
o Kanban
 Testing Methods
o Behavior Driven Development (BDD)
o Acceptance Test Driven Development (ATDD)
o Exploratory Testing
o Session Based Testing
2 Agile Methodology Types
1) Scrum

What is it? One of the most popular software testing methodologies (used by 58% of organizations
that have embraced agile according to VersionOne), Scrum takes a highly iterative approach that focuses
on defining key features and objectives prior to each sprint. It is designed to reduce risk while providing
value quickly.
Scrum starts with a requirement or user story that outlines how features should perform and be tested. The
team then cycles through a series of sprints to provide small bursts of value quickly. To help the team
work in this flexible way and avoid shifting priorities, Scrum requires that questions be answered from
the very start.

How is it different from Waterfall? Whereas Waterfall includes several testing and bug fixing
cycles before releasing a product, Scrum is far more collaborative and iterative. One of the biggest
differences is that Waterfall calls for heavy documentation early on. This documentation makes it harder
to change features as the process goes on, which may be a negative in some environments (such as
consumer-grade software) and a positive in others (such as those where the team is trying to launch a
rocket, since no one wants requirements for something dangerous shifting frequently). That said, you
might think of Scrum like many “mini waterfalls,” since requirements are well defined at the start of each
sprint and shouldn’t shift within it. The difference is that detailed requirements for the next sprint are not
set months in advance.
Diving deeper, Scrum calls for more regular collaboration between testers, developers and BAs, typically
in the form of daily standups and sprint retrospectives, to ensure proper communication and alignment.
Additionally, there is a Scrum Master who helps keep the project on task by removing blockers from the
team to make sure they can be most effective. The Scrum Master can be anyone on the team, such as a
developer or a tester.

What does adoption entail? Scrum offers one of the easiest transitions for teams coming from a
Waterfall environment because it’s time-based with sprints and releases can still be planned out in
advance. That said, it does call for faster iterations and stronger collaboration.

Who is it for? Because of its fast iterations, Scrum is best suited for teams whose customers and
stakeholders want to be actively involved by regularly seeing working products at showcase meetings.
This collaboration allows the team to make changes for upcoming showcases. Key team members who
should be involved when taking a Scrum approach include:
 Product Owner
 Scrum Master
 Developers
 Automation Engineers
 Testers
 Stakeholders

What are best practices? In addition to strong communication, collaboration and adaptability, other
best practices for testers following a Scrum methodology include:
 Determining acceptance criteria based on communication (typically in the form of a user story)
from a sales representative or customer (note: this direct connection should help reduce
miscommunications)
 Using the acceptance criteria to develop code and ensuring team approval of that code
 Testing the code in sandbox-like environments as well as production-like environments prior to
deploying it to production

2) Kanban

What is it? Kanban is a very simple Agile based methodology rooted in manufacturing (it was
developed by Toyota to help increase productivity in factories). At it’s core, Kanban can be thought of as
a large, prioritized to-do list. Like with Scrum, requirements in Kanban are tracked by their current stage
in the process (to-do, in development, in test, done).
Unlike scrum, Kanban is not time-based. Rather, it is based solely on priority. When a developer is ready
for the next task, he/she pulls it from the to-do list. Since there are fewer planning meetings, this approach
means the team needs to be extremely close. In this type of environment, if developers work much faster
than the testers, bottlenecks will crop up. In these situations, anyone on the team should jump in and help
in different areas. Of course meeting this need requires a great deal of flexibility and adaptability.

How is it different from Waterfall? Kanban still has requirements like Waterfall, but the
requirements can change as the testing team doesn’t start to think about testing each requirement until the
developer selects it from the top of the backlog. In contrast, Waterfall is heavily time-based with a lot of
overhead in planning. The heavy planning that comes in a Waterfall environment is great in some cases,
like when building expensive things, but it’s not always necessary. With Kanban, releases still get
planned, but teams usually don’t promise anyone features by certain dates unless the item in question is
near the top of the backlog.

What does adoption entail? Kanban offers a simple transition for the right teams. To make a
smooth transition to Kanban, business analysts, developers, testers and stakeholders should sit close
together and communicate regularly. When transitioning to Kanban, it’s important to remember that this
methodology offers the quickest way to bring code to production, but the code is likely to have some
technical debt. That’s because developing without always knowing what’s next doesn’t necessarily lend
itself to producing the most reusable code.

Who is it for? Kanban is best suited for small teams or teams that don’t produce features for the public
and/or promise certain dates for releases. Additionally, it is a top methodology of choice for any products
or teams focused primarily on maintenance work since bugs are not always straightforward and often
require research to resolve, which makes time management challenging. Teams that cannot minimize the
amount of planning for issues are likely better off following a Scrum or Waterfall methodology.
Key team members who should be involved in a Kanban environment include:
 Product Owner
 Project Manager
 Developers
 Automation Engineers
 Testers

What are best practices? In addition to maintaining visibility and prioritizing collaboration, best
practices for testers following a Kanban methodology include:
 Keeping very open lines of communication between the business owners, developers and testers
 Ensuring the team has flexibility to take on other roles outside of their core responsibilities in
order to help clear bottlenecks
 Making everyone an owner of the product so that they care fully about the result
4 Agile Testing Methods
1) Behavior Driven Development (BDD)

What is it? Many people have heard of or used Test Driven Development (TDD). For example,
developers use TDD to write unit tests to fail before code has been written. BDD is based on the same
principles as TDD, but instead of unit tests, it calls for higher level tests at the business level. Instead of
starting with a technical-facing unit test as TDD does, BDD starts with an initial requirement based on
end user behavior and calls for tests that are “human readable” and can even replace some requirements
documentation. This requirement is based on behaviors that the product should exhibit, creating an
airtight guide for engineers to use as they develop tests.
For more information, read our article: “Why BDD is the Secret Sauce for Testing in DevOps.”
Specifically, BDD starts with a functional specification using the Gherkin Given/When/Then syntax. This
specification then guides developers, testers and product owners who move across features. As they do,
they use automated test functions to determine completeness, refining code until it passes the test, much
like in the TDD approach except at the team level. To ensure the test passes (and it typically requires
multiple tries), the developer should only refactor code, not add any new functionality.
In sum, BDD requires a “smart” automation strategy that drives a high level of efficiency. This strategy
sets BDD apart from other Agile methodologies.

How is it different from standard Waterfall testing? BDD is extremely different from
standard Waterfall testing because the former requires test cases to be written early against requirements
and calls for those tests to be executed toward the end of the development cycle. With BDD in an Agile
environment though, tests are not based on requirements and testing happens in conjunction with
development of the features.
Additionally, within a Waterfall methodology, testers are the ones writing the test cases. A BDD
approach, on the other hand, lends itself to the business owners writing the tests. This switch reduces
communication (or miscommunication) between the business analysts, developers and testers.

What does adoption entail? Changing to a BDD methodology can be challenging when the team is
used to a traditional style of testing. It requires a BA or tester to write tests up front and for the developers
to write the test specification in code to match. It’s a new type of coordination within the team, but it’s
extremely positive in that the team works together as one unit, business users included.

Who is it for? The BDD methodology is ideal for teams that work on feature-focused software and/or
teams that put the user experience first. Key team members who should be involved in a BDD
environment include:
 Product Owner/Business Analyst
 Project Manager
 Developers
 Automation Engineer/Testers

What are best practices? Best practices for testers following a BDD methodology include:
 Streamlining documentation in order to keep the entire process lean
 Embracing a “three amigos” model where the product owner, developer and tester form a
cohesive team
 Using a test framework like Cucumber to define criteria
 Building automated tests in a way that make them easy to reuse as much as possible
 Having business analysts learn the Gherkin syntax and write test cases directly

2) Acceptance Test Driven Development (ATDD)

What is it? ATDD is like BDD in that it requires tests to be created first and calls for the code to be
written to pass those tests. However, unlike in TDD where the tests are typically technical-facing unit
tests, in ATDD the tests are typically customer-facing acceptance tests.
The idea behind ATDD is that user perception of the product is just as important as functionality, so this
perception should drive product performance in order to help increase adoption. To bring this idea to life,
ATDD collects input from customers, uses that input to develop acceptance criteria, translates that criteria
into manual or automated acceptance tests and then develops code against those tests. Like TDD and
BDD, ATDD is a test-first methodology, not a requirements driven process.
Also like the TDD and BDD methodologies, ATDD helps eliminate potential areas for misunderstanding
by removing the need for developers to interpret how the product will be used. ATDD goes one step
further than TDD and BDD though because it goes directly to the source (aka the customer) to understand
how the product will be used. Ideally, this direct connection should help minimize the need to re-design
features in new releases.

How is it different from standard Waterfall testing? ATDD is different from standard
Waterfall testing because it is a test-first methodology. Standard Waterfall testing calls for test cases to be
written upfront based on requirements, whereas ATDD is not a requirements driven testing process.

What does adoption entail? Because ATDD represents such a departure from traditional methods,
going from one to the other isn’t easy for teams to do. To be in the best position to adopt an ATDD
methodology, teams need to get stakeholder buy-in, which can prove challenging at times.

Who is it for? Due to its emphasis on user perception, ATDD is best suited for teams that are focused
on the user experience, have goals around high rates of adoption and want to minimize the number of
feature changes in future releases. Key team members who should be involved in an ATDD environment
include:
 Customer/Customer Advocate
 Developer
 Product Owner/Business Analyst
 Automation Engineer/Testers
 Project Manager

What are best practices? Best practices for testers following an ATDD Agile methodology include:
 Interacting closely with customers, for example through focus groups, in order to determine
expectations
 Leaning on customer-facing team members, such as sales representative, customer service agents
and account managers, to understand customer expectations
 Developing acceptance criteria based on customer expectations
 Prioritizing two questions:
o Will customers use the system if it does X?
o How can we validate if the system does X?
3) Exploratory Testing

What is it? Next we have exploratory testing, which is actually a type of functional testing but is
important in Agile environments. Exploratory testing gives testers ownership over the code to test it in an
organized, chaotic way. In this case, testers are not following test steps, but rather using the software in
standard or clever ways to try to break it. Testers will document defects as usual, but detailed
documentation of what and how the application was tested is not always provided.
Exploratory testing is not scripted. Rather, it’s about developing the best tests based on each unique piece
of software. Because of its unscripted approach, exploratory testing often mimics how users will interact
with the software in real life.
Across the board, exploratory testing follows four key principles:
1. Parallel test planning, test design and test execution
2. Specific yet flexible
3. Aligned toward investigation of potential opportunities
4. Knowledge sharing

How is it different from standard Waterfall testing? Exploratory testing can actually be done
in both Waterfall and Agile environments, but the tight integration between testers and developers in an
Agile environment helps ease any bottlenecks that might surface while running exploratory testing in a
Waterfall environment.
Furthermore, in order to run exploratory testing in a Waterfall environment, documentation on test results
is a must and this documentation should be easy to trace back to requirements. Of course, this type of
documentation is always useful in any environment.

What does adoption entail? Embracing exploratory testing is relatively easy, as it is quick to
launch (and scale), simple to learn and provides benefits for the entire team. That said, it’s important to
keep in mind that it should not be the only form of testing (rather, it should inform what type of testing
happens next). Additionally, even though it’s unscripted, exploratory testing should not be unstructured
(testers still need to set a goal, log your activities and take on the mindset of a specific user persona).

Who is it for? Exploratory testing can help reduce time spent testing, find more defects and improve
code coverage. As a result, exploratory testing is best suited for teams that are under time constraints,
teams that need help identifying the best types of tests to run (especially in cases where there are no
specifications from developers) and teams that want to make sure they didn’t miss anything in previous
tests. Key team members who should be involved in exploratory testing include:
 Testers (although everyone on the team should participate in some way)

What are best practices? Best practices for testers using exploratory testing include:
 Organizing the functionality in the application using something like a Mindmap or a spreadsheet
 Focusing on certain areas or certain scenarios
 Tracking what gets tested to help reproduce any errors
 Documenting results in a tool like qTest Explorer so there is some accountability for what got
tested

Finally, Agile takes on a very customer-centric approach in order to ensure customers receive as much
value as possible as quickly and as early on as possible. Testers have a big role to play in delivering this
value, but it requires them to take on a business mindset so that they can understand customer
expectations, desires and concerns and develop their testing strategies accordingly.