To get all the benefits out of test automation, you need to have a right test

automation framework in place. Selecting a wrong one may lead to a waste of your
time and money.

Since each and every project is unique, the requirements, duration, and tool choice
will vary. That’s why your primary goal is to select an automation framework that
leads to smarter testing, productivity benefits, and better overall results in your
particular project.

So what should you consider to make the right choice?

Our quality assurance team asks this question each and every time they start
working on a new project. To save our time, we’ve worked out a step-by-step
checklist that guides our test automation framework choices. Now we are sharing it
with you.

This guide is designed to help you choose a right tool whether you are a QA or Test
Automation (TA) specialist, stakeholder, or business owner who takes part in the
development process:

Step 1. List out project requirements

Step 2. Define the budget for Test Automation

Step 3. Consider the tech stack

Step 4. Analyze and compare

Step 5. Verify your choice

Step 6. Put it all together and make the final decision

How to choose a top-flight test automation framework for your needs

To make the task easier, we’ll review the tool selection process in the context of
one of our projects.

It involves testing a private practice and clinic management software for our
client from Europe. The solution transforms how clinics operate, how clinical notes
are recorded, and gives insight into all parts of the processes.

Step 1. List out project requirements

Everything starts with the requirements. The first exercise for you is to reach a
clear understanding of expectations before applying any tool on the project. It
means that the testing framework should focus on a specific problem (s).

Each framework on the market is neither entirely good nor bad – everything depends
on your particular project, solution, and software development methodology.

Read next — All you need to know about types of software testing before product
launch

Among the aspects that’ll help you start is the type of system, level of software
testing (unit testing, integration testing, performance testing, load testing, or
other), an average project scope, test coverage, nice-to-have features like
reporting, etc.

So first of all, figure out your needs and write them down.

In our particular case, we had to automate a user interface (UI) testing for a web
application. Here is a short entry list of requirements:

cross-browser testing (Chrome, Safari, Edge);

full coverage of end-to-end regression scope;
fast execution;
simple reporting; and
continuous integration (CI).
Additionally we have certain “must have” add-ons for our TA project, like CI and
infrastructure for running tests, results visualization (reports), and test data
management via API.

Analyze your own project up and down and make a similar requirements list that will
become a foundation stone for your future automated testing framework.

Step 2. Define the budget for Test Automation

This is one of the most important aspects while selecting the automation test
frameworks. On this step you have three options for TA tools to consider:

open-source;
commercial; or
customized.
Open-source tools are free tools with an active community, free tutorials, and
meetups. Due to this, they are constantly evolving and you have a great opportunity
to make a contribution. But it has a flip side to the coin. Such solutions thrive
as long as the community supports and improves them. If it stops (for whatever
reason), it may become outdated or discontinued. Moreover, such tools often require
adjustments to fit your needs.

That is where customized solutions come into play.

In situations that the functionality of an open source tool is not enough, you can
build an improved solution based on the open-source framework. Sure, you’ll need
strong expertise, a sizable budget, and have no access to the community, but, at
the same time, you’ll build exactly what you need for your project.

Most commercial tools have a rich set of features out of the box, quick set up and
good support. They usually have paid tutorials and small communities. Along with
high cost, this makes them rather unpopular, except for large enterprise projects.
If you’re ready to spend money on a commercial tool, better to start a free trial
and experiment on a small project before buying the full package.

For those companies and teams who are only at the beginning of their automation
journey, open-source frameworks are often the best option to start with.

Read next — Usability testing: how to analyze your website for UX pitfalls

In any case, you should not expect to find the best tool at once. Think about the
“proof of concept phase” which will allow you to analyze the framework and
understand whether it is a good fit (we’ll talk about it further).

In our particular case for clinic management software, we’ve selected an open-
source tool. To know why let’s move to the next step.

Step 3. Consider the tech stack

The tool must support the programming languages used on the project as well as the
operating systems your testers are comfortable with (e.g. Windows, Linux, Mac,
etc.).

In case you work with mobile apps, you’ll also have to consider the supported
platforms. If you target two platforms, you can either choose a tool that supports
both iOS and Android (e.g. Selenium, Appium) or use separate testing suites (e.g.
Robotium for Android, UI Automation for iOS).

For programming languages, looking at modern trends in automation, it is evident

that most projects use JavaScript (JS) for different levels and types of testing.

If your project is done on JS, try to use it for test automation, too. If you have
a solution using another programming language, Java, for example, it’s better to
use that. It’s not critical, but this way doesn’t expand your tech stack and allows
you to ease into the process while helping to avoid unexpected issues.

In our example, we have a ReactJS web application. Due to this fact, our choice
came down to JavaScript frameworks for end-to-end (E2E) UI test automation.

In order to find the tools, you can do whatever you can: make personal research,
ask colleagues or friends, explore the most downloadable tools, etc.

For example, NPM Trends is a great tool to find out the popularity of open source
automation test frameworks.

test automation framework js

From the data, we found that Puppeteer, Protractor, Cypress and WebdriverIO are
some of the most downloaded testing frameworks over the last year. However, there
are also many other good frameworks you can consider like Testcafe, Nightwatchjs,
etc.

Read next — The role of an Agile mindset in times of turbulence

There is no clear winner on this step, as each of the frameworks required in-depth
analysis.

Step 4. Analyze and compare

To ensure the testing tool is beneficial for project needs, the QA team has to dig
deeper. A tool comparison matrix is a great way to make an informed decision on the
basis of your requirements.

Give your project requirements a second glance and decide the key characteristics
you are going to compare.

After we’ve analyzed our initial set of tools, we concluded that none of the top
four tools fully suited our project requirements.

test automation frameworks