For instance, Protractor is quite an old test runner that needs a lot of

customization. It was not actively updated for several years, meaning there are no
improvements, no bug fixes, no documentation updates. It has an over complicated
design and issues with debugging.

WebdriverIO requires control from a TA Engineer, including lack of built-in error

messages that need to be defined manually. Cypress, as well as Puppeteer, have no
cross-browser testing that is quite a problem in our particular case.

That is why we went back to the list of test automation frameworks and analyzed the
rest, less popular, testing tools in the same manner.

The more extended analysis brought us to CodeceptJS, which seemed to cover the
whole list of initial project requirements in one go.

CodeceptJS is an ‘all-in-one’ open-source framework meaning it is a wrap around

multiple testing tools. It allows cross-browser testing, code is easy to
understand, everything is well documented, there is even a test UI where you can
see snapshots of each test step and debug it easily.

Step 5. Verify your choice

When you have a preliminary winner among the automated test frameworks, don’t hurry
to implement it right from the start.

The more rational way to start introducing a new framework into your test process
is to make a proof of concept (POC). Most teams skip this phase, however, it may be
very useful in case you still have some doubt about the tool and want to make sure
it is the “one and only”.

A POC in a short period of time can provide you with valuable information about the
tool usability but also whether the right people are involved, and whether the
testing goals are clear, etc.

Moreover, it helps you identify the possible risks even if you are not familiar
with the new tool or technology.

The proof of concept should begin with setting immediate goals it has to meet. Keep
in mind that its main goal is not actual test automation implementation, but
defining what kind of tools, methods and resources you’ll need for a successful
test automation process. It should balance between the need to uncover the key
automation challenges and
demonstrating feasibility of the tool.

Most commonly, the outcome of the proof of concept phase is one of the three:

Tool is a good fit and meets the project requirements. In this case you have to
proceed with further details. If you plan to apply a commercial tool, consider
license fees, training & support costs, cost of implementation, etc. In case of
open source, investigate learning resources available, learning curve, support
available etc. Maintenance costs have to be considered too both for commercial and
open source tools.
Tool has certain limitations and does not meet the requirements. In such a case,
the tool is no longer considered as an option.
Tool meets the requirements but only partially. Check out whether there is any
other workaround with the same tool, or if any requirements can be covered in
another way, etc. If not, consider another framework with better ROI.
Turning to our case study, we tried CodeceptJS on our test environment and
confirmed it was a good fit to our project requirements.

Step 6. Put it all together and make the final decision

This is the closing stage of your selection process. Now your team has all the
necessary information to make an informed decision.

The QA team should have a strong awareness of the tests automation tool, understand
all the strong and weak points of the solution and keep the balance between cost
and benefit. This decision greatly influences the project, the testing process, and
the business goals, which is why you should spend a good time analyzing all the
information you obtained.

With CodeceptJS on our project, we’ve successfully run more than 1500 test cases
and revealed 500+ bugs in just twelve hours.

Ready to initiate a search for a test automation framework?

Selecting a test automation tool is tricky, as every tool is a good player on its
own ground.

But nothing is impossible when you know exactly what result you pursue. That is why
understanding your project objective from the start, recognizing your team’s
strengths and weaknesses and being ready to make in-depth analysis of the tools
will help you start the process on the right foot.

I hope the tips we’ve shared here will help you on the road to the best test
automation framework for your project needs.

But, if you’re still feeling a bit overwhelmed with the ocean of tools on the
market, don’t hesitate to drop us a line. We’d love to understand your challenges
and see how our team can help you on your software development journey.