Creating Calabash Tests

Testing Xamarin Mobile Applications with Calabash

Overview
Acceptance testing is an engineering practice that is used to prove that an application will work
according to predefined specifications. Acceptance testing is also be used to prove that changes to
an application have not unintentionally broken any of the existing functionality. This involves running
a suite of high-level tests on an application. Traditionally, acceptance testing has been a manual
process that involves testers running all of the test scenarios by hand against the application.
Manual testing is time consuming and expensive, and deters many developers.
Calabash is a framework, designed to automate the acceptance testing process for mobile
applications. Calabash is build on Cucumber, a popular open-source testing framework, and makes
it possible to script acceptance tests for mobile applications that can be submitted to Xamarins Test
Cloud and run against a large number of actual mobile devices. This large scale testing provides
valuable feedback on how an application performs on real devices, at a fraction of the cost and effort
of manually testing on devices.
Acceptance tests are not written in isolation. Good tests require that developers and subject matter
experts get together and discuss how the application should work. These discussions allow the
developer to capture the behavior of the software in a simple, business-readable text file. The
developer then writes the code so that these written tests pass. This style of software development is
known as behaviour driven development (BDD). There are three features that make BDD a powerful
technique:

The communication between developers and subject matter experts results in an ubiquitous
language that minimizes misunderstandings as the application is developed.
The acceptance tests provide clear and simple documentation about how the application should
work.
The application has tests that will notify developers if any future changes to the application
break existing functionality.

Software development can be broadly lumped into one of two scenarios:

1. Brownfield development involves existing or legacy applications that require new features. In
this scenario acceptance tests are written so that the new features do not introduce
regressions. New features would also be covered by acceptances tests.
2. Greenfield applications are new projects. Ideally, acceptance tests are written before each
feature is coded. Developers first implement the tests, and then write the application code to
make the tests pass.
The approach for writing acceptance tests in each scenario is similar; the biggest difference is that in
brownfield development, the code is written first. The upfront collaboration between a developer and
a subject matter expert does not occur.
In Cucumber, acceptance tests are known as features. Features are text files with the .feature
extension that are written in plain language understood by developers and business users alike. A
feature is used to test one particular function of an application. Features are broken down into one or

more scenarios. A scenario describes a test that describes when some functionality of an application
has been properly implemented. It exists within the context of the feature. A scenario is in turn
subdivided into steps a list of sentences that describe the pre-conditions for the scenario, the
actions a user would take, and the expected results of the users actions. Each step is matched to a
step definition - a piece of Ruby code that executes when the test runs.
The following diagram illustrates how all these pieces fit together:

Step definitions use the Calabash APIs to interact with the application while it is running on a device
or in the simulator/emulator. The APIs contain methods to simulate user actions such as touching
the screen, entering text into text fields, and more. Calabash also provides a rich query language to
locate and interact with objects on the screen.
This guide is broken up into two main sections. The first section covers the basics of creating an
acceptance test in Calabash. It will discuss the basic workflow and the steps involved in building
acceptance tests. It discusses how to use the predefined steps that ship with Calabash, as well as
how to write custom steps using the Calabash APIs and the Calabash query language.
The second part of this guide introduces some of the techniques and best practices for writing tests
for both iOS and Android applications.

Requirements
This guide builds on the material covered in several other Xamarin documents. It is crucial that you
read and understand the content in the Introduction To Calabash guide. This guide assumes that
Ruby, Cucumber, and Calabash are installed and configured as per the steps defined in the
Calabash Installation document.
Familiarity with the Ruby programming language and Cucumber is helpful.
A text editor such as Sublime Text or TextMate is required when developing the features and the
step definitions. Optionally, you can use RubyMine, a full-featured commercial Ruby IDE with built in
support for Cucumber.

Creating Tests with Calabash

Before we can start writing tests, we need to create the proper directory structure to house the tests
and the supporting files for the tests. Calabash provides some helper commands to create the basic
directory structure. These helpers are run at the command from within the solution directory. If we
want to start writing tests for an iOS project, we would run calabash-ios gen at the command line, as
illustrated by the following snippet:
$ calabash-ios gen ----------Question---------- I'm about to create a subdirectory called features.
features will contain all your calabash tests. Please hit return to confirm that's what you want. -------------------------- ----------Info---------- Features subdirectory created. Make sure you've
build your -cal scheme in XCode and try executing cucumber ---------------------------

This command creates the directory structure in the following screenshot:

created the features folder, a sample .feature file, and the folders step_definitions and
support. The tests are written using Gherkin, a plain text domain specific language (DSL) that can be
read and understood by anybody.
calabash-ios

The two subdirectories will hold Ruby source code files. The files in the step_definitions hold the
step definitions - the code snippets that make up the steps that make up the feature. The support
directory holds the source code that is shared amongst the step definitions.
The calabash-android gen command is the corresponding command for Android projects. It will create
the same directory structure for Android projects.
Once the Cucumber directory structure is created, we can begin writing the acceptance tests. Writing
tests follows this general workflow:
1. Write the feature As described in the overview, the developer and a subject matter expert
work together to write the Calabash feature. They create a .feature file that describes how the

scenarios should work.

2. Run the feature Next, the developer runs the feature that was created. It will fail because the
steps have not yet been defined. However, Cucumber will help us out by providing some code
snippets that we can use to create the step definitions.
3. Create the Step Definitions Using the snippets that Cucumber provided in step #2, we
create a Ruby source code file, and paste the snippet output into it.
4. Explore the Application using the Calabash Console Not strictly necessary, this step
involves starting up an instance of the Calabash console. The Calabash console is a command
line utility that allows us to issue commands to the Calabash test server communicating with
our app. We can use this to discover the Calabash queries necessary to interact with the UI
object in the application.
5. Update Step Definitions Once we have figured out the Calabash queries for locating and
manipulating the UI object, we can use these - along with the Calabash API - to implement the
step definitions.
6. Run the Feature When the step definitions are finished, we can run the features. If this is a
brownfield application, the functionality has already been coded and the tests should all pass. If
this is a greenfield project, the tests will not pass as there is no application code we will move
on to step #7 below.
7. Implement the Feature in the Application This step is only necessary in a greenfield
project. The developer will shift attention to the application and write the code to implement the
desired functionality, and make the test pass.
8. Repeat Once the feature is implemented with a passing test, it is done. Time to move on and
implement the next bit of functionality in the application.
With this understanding of the basics in place, lets examine each of these steps in a bit more detail
as part of a simple walkthrough.
Note: calabash-ios gen or calabash-android gen only need to run once. These commands will
terminate if they detect an existing features directory.

A Simple Walkthrough
To better appreciate the workflow described in the previous section, lets put them in action as part of
a walkthrough. In this walkthrough, we will write a feature on a simple mobile application that
performs some basic validation of a credit card number. In the application, the user enters a credit
card number into a text field, and clicks the Validate button. If the credit card number is invalid, the
application displays an error message on the screen for the user. The following low fidelity mockup
shows an example of what this application might look like on either iOS or Android:

There are many rules for credit card validation, but let's keep things simple so that its possible to
focus on learning how to write an acceptance test. This sample application will only have one rule: a
credit card must be exactly 16 digits long.

Getting Started
To save time, well use a pre-built application for testing. The CreditCardValidation.zip file contains a
functional Xamarin.iOS application and has been already been prepared to support Calabash testing
by ios-calabash gen. This is a working application, but there are no acceptance tests written for it at
this point.
Running the application, we should see the following:

So, with a working application and the basic file structure for the features in place, lets get started
with the feature.

Creating the Feature

Our first acceptance test will consist of one Cucumber feature with two scenarios:

Credit card is to short

Credit card is to long

Following the workflow of the previous section, add a text file to the features folder and name it
credit_card_validation.feature. Edit the file so that it contains the following feature and scenario:
Feature: Credit card validation. Credit card numbers must be exactly 16 digits. Scenario: Credit
card number is to short Given I use the native keyboard to enter "123456" into text field number 1
And I touch the "Validate" button Then I see the text "Credit card number is to short."

This first scenario is an example of using the pre-defined steps that come with Calabash iOS. Predefined steps are handy because they dont require us to write any step definitions. We can run this
test against the sample right now and this first feature should pass. Executing the test is as simple
as running Cucumber from the command line, as illustrated by the following snippet:
$ cucumber Feature: Credit card validation. Credit card numbers must be exactly 16 digits.
Sceanario: Credit Card is too short # features/credit_card_validation.feature:4 Given I use the
native keyboard to enter "123456" into text field number 1 # calabash-cucumber0.9.162/features/step_definitions/calabash_steps.rb:140 And I touch the "Validate" button #
calabash-cucumber-0.9.162/features/step_definitions/calabash_steps.rb:31 Then I see the text
"Credit card number is to short." # calabash-cucumber0.9.162/features/step_definitions/calabash_steps.rb:373 1 scenario (1 passed) 3 steps (3 passed)
0m18.035s

This is great! With minimal effort on our part we have our first test up and running. Lets move on to
the remaining scenarios and see how they would be implemented using custom steps.

Implementing the Second Scenario

The predefined steps are a great way to get started, but we cant rely on them for every possible
scenario for the following reasons:

There are a limited number of predefined steps its impossible for Xamarin or the Calabash
community to create steps for every possible scenario.
The predefined steps promote an imperative style of writing steps. This means that scenarios
are saying how to do something, as opposed to what to do. This will make the scenario hard to
read and difficult to maintain.
Many of the predefined steps have step definitions that use an older, deprecated Calabash API.

In this section, we will address these three points by creating custom steps.
Well implement the second scenario using custom steps. Open up the file
credit_card_validation.feature, and add the following scenario at the end of the file:
Scenario: Credit card number is to long Given I try to validate a credit card number that is 17
characters long Then I should see the error message "Credit card number is to long."

This is an example of a declarative style of step writing. Note that the scenario itself is not concerned
with what field to enter text into or what view to touch to validate the credit card. The scenario
explains what should be done and what the application should do. The details of how enter 17
characters onto the screen and what view should be touched are implementation details and are the
responsibility of the step definition, and we will get to that in a bit.
Before we dive into creating the step definitions, lets run the test again at the command line. We
should see the following output:
$ cucumber Feature: Credit card validation. Credit card numbers must be exactly 16 digits.
Scenario: Credit card number is to short # features/credit_card_validation.feature:4 Given I use
the native keyboard to enter "123456" into text field number 1 # calabash-cucumber0.9.162/features/step_definitions/calabash_steps.rb:140 And I touch the "Validate" button #
calabash-cucumber-0.9.162/features/step_definitions/calabash_steps.rb:31 Then I see the text
"Credit card number is to short." # calabash-cucumber0.9.162/features/step_definitions/calabash_steps.rb:373 Scenario: Credit card number is to long #
features/credit_card_validation.feature:9 Given I try to validate a credit card number that is 17
characters long # features/credit_card_validation.feature:10 Then I should see the error message
"Credit card number is to long." # features/credit_card_validation.feature:11 2 scenarios (1
undefined, 1 passed) 5 steps (2 undefined, 3 passed) 0m29.895s You can implement step definitions
for undefined steps with these snippets: Given(/^I try to validate a credit card number that is
(\d+) characters long$/) do |arg1| pending # express the regexp above with the code you wish you
had end Then(/^I should see the error message "(.*?)"$/) do |arg1| pending # express the regexp
above with the code you wish you had end

Notice that Cucumber realizes that we dont have any step definitions for the custom steps in our
second scenario, so it tries to help us out. It provides some snippets for the undefined steps. Lets
use these snippets to create the step definitions. Add a new file called
features/step_definitions/credit_card_validation_steps.rb. Edit that file and paste in the two
snippets Cucumber suggested. The step definition file should resemble the following bit of code:
Given(/^I try to validate a credit card number that is (\d+) characters long$/) do |arg1| pending #
express the regexp above with the code you wish you had end Then(/^I should see the error message
"(.*?)"$/) do |arg1| pending # express the regexp above with the code you wish you had end

These step definitions are stubbed in with one line and the pending keyword. Cucumber can run
them, but it will stop when it hits the pending keyword, raise a notification, and skip the remaining
steps.
The next thing we need to do is to actually implement the step definitions. The first step definition will
take care of entering text into the UITextView for the password and pressing the Validate button. Lets
do that now.
Creating a Step Definition
To create this step definition, we need to figure out the Calabash queries to locate the credit card
text field, the Validate button, and the error messages text field. We can perform this exploration
using the Calabash console.
To start up the Calabash console:

Drop to the command line.

Change the working directory to the solution directory.
Run calabash-ios console at the command line.

Start the Calabash test server on the device.

The following snippet shows an example of this:

$ calabash-ios console Running irb... irb(main):001:0> start_test_server_in_background

Calabash::Cucumber::Launcher: Launch Method instruments Log file:
/var/folders/61/bhbtr4_51tx9bl3l5lqzl87m0000gn/T/run_loop20131205-90049-c66i85/run_loop.out
irb(main):002:0>

At this point, the iOS simulator should be running and display the first view in the application. Our
application will respond to any Calabash commands that come in through the Calabash console.
The Calabash API provides the query function that will find objects on the screen. It takes a query
string and returns an array of hash objects with information about the objects that match the query.
The query string format is very similar to the CSS selector. To locate all objects on the screen, we
would invoke query like so:
irb(main):002:0> query "*" [ [0] { "class" => "UIView", "id" => nil, "rect" => { "center_x" => 160,
"y" => 0, "width" => 320, "x" => 0, "center_y" => 284, "height" => 568 }, "frame" => { "y" => 0,
"width" => 320, "x" => 0, "height" => 568 }, "label" => nil, "description" => "

The "*" query string is handy for when we dont know exactly what we're looking for on the screen. It
will dump a list of all objects on the screens. We can examine the properties of those objects and
use the details of each object to craft a query string.
It is possible to get very specific with the query string to locate individual objects on the screen. For
example, the Calabash query string to locate the Validate button would look something like the
following:
irb(main):003:0> query "button marked:'Validate'" [ [0] { "class" => "UIButton", "id" => nil,
"rect" => { "center_x" => 160, "y" => 242, "width" => 280, "x" => 20, "center_y" => 264, "height"
=> 44 }, "frame" => { "y" => 242, "width" => 280, "x" => 20, "height" => 44 }, "label" =>
"Validate", "description" => "

If we were to translate the query string to English, it would read something like Find all of the
UIButton objects on the screen that are marked with the text Validate. The marked keyword uses
an algorithm that examines the properties of an object and tries to find one that matches the string.
For more information about the Calabash query syntax, read the Calabash query language
documentation.
The following table lists the Calabash query strings to locate the various objects on the screen.
View Object
Credit Card Number text field
Validate button
Error Messages text view

Calabash Query String

textField marked:'CreditCardNumberField'
button marked:'Validate'
textView marked:'ErrorMessagesField'

Now that we have the Calabash query strings for the various UI objects on the screen, we can
implement the step definitions. Open up the file credit_card_validation_steps.rb, and change the
Given step definition to match the following code:
Given(/^I try to validate a credit card number that is (\d+) characters long$/) do
|number_of_digits| touch("textField marked:'CreditCardNumberField'") wait_for_keyboard
keyboard_enter_text("9" * number_of_digits.to_i) touch("button marked:'Validate'") end

This step definition makes extensive use of the Calabash API to interact with the view objects on the

screen. This step definition will touch the Credit Card text field, giving that view focus, and wait for
the keyboard to appear. Then the step definition uses the keyboard_enter_text function to input a
credit card number using the iOS keyboard. Once that is done, the touch method will simulate a user
tapping the Validate button.
The final step definition is responsible for ensuring that the correct error message is displayed on the
screen. Edit credit_card_validation_steps.rb so that it resembles the following code snippet:
Then(/^I should see the error message "(.*?)"$/) do |error_message| text_view = query("textView
marked:'ErrorMessagesField' {text CONTAINS '#{error_message}'}") raise "The error message
'#{error_message}' is not visible in the View." unless text_view.any? end

This step definition will query for a that contains a particular string using a predicate. The predicate
is enclosed in the curly brackets can be used to test properties on an object. In this example the
predicate will check to see if the text property of the text view contains a specific string. If the query
method is not able to find any objects that match the string then it will return an empty array. If there
is an empty array, the step definition will raise an error that will cause the test to fail.
At this point, the feature should be complete. If we run the tests, we should see the following output
in command line console:
$ cucumber Feature: Credit card validation. Credit card numbers must be exactly 16 digits.
Scenario: Credit card number is to short # features/credit_card_validation.feature:4 Given I use
the native keyboard to enter "123456" into text field number 1 # calabash-cucumber0.9.162/features/step_definitions/calabash_steps.rb:140 And I touch the "Validate" button #
calabash-cucumber-0.9.162/features/step_definitions/calabash_steps.rb:31 Then I see the text
"Credit card number is to short." # calabash-cucumber0.9.162/features/step_definitions/calabash_steps.rb:373 Scenario: Credit card number is to long #
features/credit_card_validation.feature:9 Given I try to validate a credit card number that is 17
characters long # features/step_definitions/credit_card_validation_steps.rb:1 Then I should see the
error message "Credit card number is to long." #
features/step_definitions/credit_card_validation_steps.rb:7 2 scenarios (2 passed) 5 steps (5
passed) 0m36.442s

If youve made it this far, then congratulations! You now have a working application that has some
acceptance tests to prove that the application works as desired.

Best Practices
Up to this point in the document, we were only concerned with using Calabash to test a mobile
application on one platform - iOS in the case of the previous walkthrough. Writing cross-platform
tests is also possible, but requires a bit more effort to abstract away the platform specific parts from
the shared parts.
Android and iOS applications are made up of different UI components with zero overlap in their APIs.
In order for the step definitions to be cross-platform, they cannot contain any code that is platform
specific. This means that we have to introduce another layer of abstraction. This layer will be used
by the step definitions and is responsible for driving the UI.
A well-established pattern has evolved in test engineering to support this abstraction - the PageObject Pattern (POP). The Page-Object Pattern maps a single screen (such as an Android Activity or
an iOS View) to a class. This class contains methods and properties that can be used to manipulate
the screen (such as pressing a button) or to query the state of the screen (checking to see if some

text is present). This pattern changes the focus of the test from how to drive the screen to how to
test a screen. In other words, our tests will follow a declarative style, while the page objects
themselves will become more imperative. This abstraction makes it possible to share the features,
scenarios, steps, and step definitions, because the platform specific details are encapsulated within
the page objects.
The Page-Object Pattern provides the following benefits:

Reuse If there are other screens that require a login, the details for doing so are encapsulated
in an easy to use class.
Encapsulation Different page objects can be created for Android and iOS. This allows the
steps and step definitions to remain the same across all platforms as the platform specific details
are in the page object.
Readability / Maintenance The code inside the step definitions is reduced. Step definitions
are no longer cluttered with implementation details such as how to scroll a view or touch a
button. This makes them easier to understand and maintain.

A good example of the Page Object Pattern is a login screen. A typical login screen contains a
username, password and a Login button. A page object for the login screen would contain a
login(username, password) method that would abstract the details of entering the login credentials and
pressing the Login button. The step definition calls the login method on the page object.
The following diagram illustrates how the various components the Page Object relate to each other:

Setting Up the Project Structure

The following screenshot shows a sample directory structure for cross-platform testing:

The features directory is inside the solution folder, which still contains the step_defintions and
support directories. There are two new directories, ios and android, which hold the platform-specific
code. The following table describes the content of these folders:
Directory Description
This directory holds platform specific shared code that would be used by the
helpers
page objects.
pages
This directory holds the page object implementations.
support This directory holds code used by Cucumber.
The config directory holds the Cucumber configuration file cucumber.yml. This file contains the
Cucumber profiles. A Cucumber profile is a shortcut to commonly used Cucumber command line
parameters. The profile will tell Cucumber what files to load before the tests run. See the Cucumber
wiki for more information about profiles.
Now that we have seen an example of how to create the directory structure for cross platform
testing, lets make some page objects.

Writing the Page Object in Calabash

Calabash provides a Page Object base class for both Android and iOS, Calabash::ABase and
Calabash::IBase. Each screen in the application should have one page object, which is a subclass of
one of these two classes.
A page object provides a method called trait. This method returns a Calabash query string that in
turn returns an object unique to this screen. The page object in Calabash calls this method when it
needs to determine if the current screen corresponds to a given page object
The following code snippet is an example of a page object called TaskDetailsScreen:
require 'calabash-android/abase' class TaskDetailsScreen

Apart from the mandatory trait method, the page object needs additional methods that return query
strings for the important objects on the screen. The page object should also have methods that
correspond to the actions a user might make on the screen.
For example, the user may touch a text field, enter some text, and then tap a save button. Including
these actions in the step definition will unnecessarily clutter the step definition. It is better to place all
these actions inside a method of the page object class. This allows the step definitions to be brief
and focus on what is being tested rather than how to do the test.
If you want to see an example of the Page Object pattern, check out the TaskyPro-Calabash
example on GitHub.

Summary
This guide began with an overview of what acceptance testing is. It then went on to provide a brief
explanation of what Calabash is, and how it can be used create acceptance tests. We created two
sample scenarios to test a sample application. In the first scenario, we used the predefined steps
that come with Calabash, while in the second scenario we created custom steps. In the process of
writing these tests, we introduced the Calabash API and Calabash Query language. We closed with
a description of the Page Object Pattern, and how to use it to write cross platform tests.