You are on page 1of 91

The

Current State of
Mobile Tes3ng
Stephen Janaway

Stephen Janaway 2014

Who am I?
Stephen Janaway

14 years experience in mobile soDware


tes3ng.
Mobile devices
Mobile networks
Mobile applica3ons
Have worked for companies such as
Ericsson, Motorola and Nokia.
Work for the NET-A-PORTER GROUP.


www.stephenjanaway.co.uk
@stephenjanaway
Stephen Janaway 2014

www.tes3nginthepub.com
@tes3nginthepub
Search for Tes3ng In The Pub on iTunes
Stephen Janaway 2014

Today

A bit of history.
An overview of mobile.
Why is mobile dierent?
Where are we now?
Whats next?
Your mobile tes3ng future.

Stephen Janaway 2014

A Bit of History

"Mobile radio telephone" by Original uploader was Hackgillam at en.wikipedia - Transferred from en.wikipedia; transferred to Commons by User:LiDarn using CommonsHelper.. Licensed under
Public domain via Wikimedia Commons - h`p://commons.wikimedia.org/wiki/File:Mobile_radio_telephone.jpg#mediaviewer/File:Mobile_radio_telephone.jpg

A Bit of History

"2007Computex e21-Mar3nCooper" by 2007Computex_e21Forum-Mar3nCooper.jpg: Rico Shenderiva3ve work: PowellS (talk) - 2007Computex_e21Forum-Mar3nCooper.jpg. Licensed under Crea3ve
Commons A`ribu3on-Share Alike 3.0 via Wikimedia Commons - h`p://commons.wikimedia.org/wiki/File:2007Computex_e21-Mar3nCooper.jpg#mediaviewer/File:2007Computex_e21-
Mar3nCooper.jpg

Mobile devices are

Smart Phones
Feature Phones
Tablets
E-readers
Big screens, small screens, touch screens and
keypads.

Stephen Janaway 2014

And now

Fitness devices.
Smart watches.
Wearable devices.
Beacons.
Household gadgets.

Stephen Janaway 2014

Mobile devices are much more

Alarm Clocks
Maps
Link to loved ones
Hand held shops
Tickets
Music Players
Books
Games Players
Cameras

These are key experiences.


Stephen Janaway 2014

WHY IS MOBILE DIFFERENT?

Stephen Janaway 2014

MOBILE USERS ARE MORE


EMOTIONALLY ATTACHED

Stephen Janaway 2014

MOBILE USERS ARE NOT


SECOND CLASS CITIZENS

?
Stephen Janaway 2014

?
Stephen Janaway 2014

BUT QUALITY IS CURRENTLY


STILL NOT GREAT
AND USERS INCREASINGLY DONT
TOLERATE THIS

MOBILE USERS CAN LEAVE BAD


FEEDBACK MUCH MORE QUICKLY

What Makes Mobile Tes3ng Dierent?

Many plaoorms.
Many OS versions in the eld.
Many dierent hardware congura3ons.
Devices with limited memory and processing
capability.
Limited ba`ery life.
Network connec3vity.
Applica3on stores.
Low maturity of automa3on solu3ons.
Stephen Janaway 2014

Func3onal Mobile Tes3ng

Selec3ng categories.
Accounts.
Shopping bag.
Opening and closing.
Images.

Stephen Janaway 2014

Usability Ma`ers
Usability is more important in the mobile world:
Smaller screen.
More dicult to interact with and typically used one
handed.

Understanding
the User

In app analy3cs.
Website analy3cs.
User personas.
Similar applica3ons.

Stephen Janaway 2014

Consistency
Think about what a mobile user would want.
Something that looks similar to desktop?
Takes advantage of mobile func3onality:
Loca3on.
Mobile search.
M-commerce.

A design that can be viewed on a small screen


and in bright light.
Stephen Janaway 2014

Engagement
People typically have more than one device.
They get distracted.
How engaging is your app or mobile website?

Stephen Janaway 2014

Understand Gestures

Stephen Janaway 2014

Mobile specics

Download and install.


Un-install.
Upgrade.
Rota3on.
Sensors.
Loca3on.
External media.
Stephen Janaway 2014

Interac3ons
While your applica3on runs, the phone is:
Talking to the network
Working out where you are
Maybe receiving calls, texts, Facebook and Twi`er
updates.

What happens when you have your applica3on


running, and then get a call, and a text, whilst
uploading video to YouTube? Test for it and nd
out.

Performance

Limited processor speeds.


Limited on-board memory.
Typically not turned o very oDen.
Dont forget - most apps are just clients to
web services. Use your web performance
tes3ng skills.

Stephen Janaway 2014

How Much Data?


Check for excessive data consump3on.
Using on device data measuring tools.
Using a proxy.

Stephen Janaway 2014

The Current State Of Mobile

Stephen Janaway 2014

Subscrip3ons vs. People

Stephen Janaway 2014

The mobile market is huge

Stephen Janaway 2014

Its Increasingly
Smartphones and Tablets

Stephen Janaway 2014

Dierent OSs

Stephen Janaway 2014

Dierent Mobile Browsers

Stephen Janaway 2014

The Future

Stephen Janaway 2014

More Devices = More Tes3ng (?)

Stephen Janaway 2014

More Wearables

Stephen Janaway 2014

OSs

Source: gs.statcounter.com
Stephen Janaway 2014

Android versions

Source: h`p://opensignal.com/reports/fragmenta3on-2013/

Screen Sizes

Stephen Janaway 2014

Deciding What to Test


Know your predicated customers (not your
current ones).
Find out what OS updates are coming up.
Get as much data on device usage that you
can:
In app analy3cs.
Web searches.
Similar apps.
Stephen Janaway 2014

Fragmenta3on
Lots of dierent hardware and soDware
congura3ons to support.
Op3ons:
Dont use them at all use simulators or user
agents.
Buy all or some of them.
Rent all or some of them.
Use a cloud based service (Perfecto Mobile,
DeviceAnywhere).
Stephen Janaway 2014

My Recommenda3on
Buy 2 from the top, 2 from the middle and 2 from
the bo`om.
Understand what your customers use or will use.
Test rst on simulators.
Use cloud services if needs be, to ll any gaps.
Never just use browser emula3on for mobile
websites.
Stephen Janaway 2014

Making More Use of Your Users

Stephen Janaway 2014

Dog fooding/ Alpha/ Beta


Get an internal user group going.
Distribute app via TestFlight, TestFairy or
similar.
Alpha/ Beta test via app stores.

Stephen Janaway 2014

Using Cloud Based


Tes3ng Services
Where Appropriate

Stephen Janaway 2014

Not Buying All The Devices

Stephen Janaway 2014

Open Device Labs


Borrow devices, normally
for free.
h`p://opendevicelab.com
Open Device
Labs = shared community
pools of internet
connected devices for
tes3ng purposes.

Stephen Janaway 2014

(Even) More Loca3on Awareness

Stephen Janaway 2014

More Financial Transac3ons Will


Happen On Mobile

Stephen Janaway 2014

M-Commerce

Google Wallet.
Paypal.
Scanning barcodes.
Contactless payments.
Displaying bar or QR codes.
Loca3on based.

Stephen Janaway 2014

Security
Big recent rise in the amount of malware targeted at
mobile devices.

Stephen Janaway 2014

Stephen Janaway 2014

Stephen Janaway 2014

Stephen Janaway 2014

Is This Appropriate?

Stephen Janaway 2014

Smartphones Everywhere
= Your App Everywhere

Stephen Janaway 2014

Accessibility
How readable is the app/ website with a large
font?
Is there voice ac3va3on / control?
Are there text alterna3ves to images?
Tools:
TalkBack (Android)
Voice Over (iOS)

Stephen Janaway 2014

Media Usage Is Changing

Stephen Janaway 2014

We Are Consuming Content Dierently

Stephen Janaway 2014

Voice Ac3va3on

Siri iOS
Google Now Android
Also on Windows Phone and Blackberry.
Is the app voice ac3vated?
What happens when you ac3vate voice with
the app running?

Stephen Janaway 2014

Its No Longer OK to
Have a Bad Mobile Presence

Stephen Janaway 2014

?
Stephen Janaway 2014

Responsive Design

h`p://designmodo.com/responsive-test/
Stephen Janaway 2014

Your Mobile Tes3ng Future


You Can Avoid The Common
Mobile Tes3ng Gotchas

Stephen Janaway 2014

Only Tes3ng On Simulators

Stephen Janaway 2014

Using Simulators
and Browser Emula3on
Simulators:
Normally found within OS SDKs.
Good to start tes3ng with.
Do not recreate exactly how a device func3ons.

Browser Emula3on:
User Agent Switching:
Within a browser (Chrome developer tool is best).
Changes how the browser renders the page.
Stephen Janaway 2014

Real Devices
Keep a device library or rent devices from one.
Cannot pick all devices.
Choose a few at the top and a few at the
bo`om device ranking.
Do not forget to nd out what is being used by
customers.
Can be expensive and needs maintenance.
Stephen Janaway 2014

Staying In The Oce

Stephen Janaway 2014

Dont Just Test Here

On Here, and At Here

Out Here

In Here

Forgeung About Ba`ery Life

Stephen Janaway 2014

Ba`ery Life

Stephen Janaway 2014

Forgeung That It Takes Time


To Submit Apps

Stephen Janaway 2014

Apps Stores
Submission criteria vary:
Store

Time

Details

Apple Apps Store

2 weeks

Signicant
checking/ tes3ng

Google Play

day

Limited tes3ng

Windows Phone
Store

4 days

Blackberry World

1 day

Amazon App Store

1-2 weeks

Stephen Janaway 2014

Ignoring OS Updates

Stephen Janaway 2014

OS Updates
Always regression test your app or website
before an OS update.
Get on the developer program for early OS
versions.
Users update more quickly on iOS than
Android.
Carriers oDen control update cycles.
Decide beforehand how many OS version you
will/ can support.
Stephen Janaway 2014

Confusing Dev With Live

Stephen Janaway 2014

Thinking Mobile Test Automa3on


Is Like the Web Test Automa3on

Stephen Janaway 2014

Automa3on
The holy grail to solving the fragmenta3on
tes3ng problem?
Caveat mobile test automa3on is like
desktop test automa3on 6 years ago.
You may want to consider pushing for more
unit and integra3on tests before spending
3me on mobile UI automa3on.
Stephen Janaway 2014

Some Op3ons
Tool

OS

Comments

Appium

iOS/ Android

Probably the best out


there, but s3ll aky

iOS Driver

iOS

OK but cannot test apps on


real devices

Frank

iOS

Cannot test Webviews

Zucchini

iOS

Cannot test Webviews

Calabash

iOS/ Android

Requires embedded server

Robo3um

Android

Android only

Selendroid

Android

Used at Facebook

Monkey Talk

iOS/ Android

Requires agent to be added


to app.

And many, many more, some of which are even s3ll supported.
Stephen Janaway 2014

Automa3on Sugges3ons
Determine if you require cross plaoorm
support.
Avoid tools that use embedded servers or
agents they alter the produc3on code.
Look for tools with ac3ve online communi3es.
Consider the future.

Stephen Janaway 2014

My Picks
Appium:

Cross plaoorm.
Well supported user community.
About to be more closely integrated in Selenium 3.

iOS Driver:

Well supported user community.


About to be more closely integrated in Selenium 3.

Calabash:

Well supported user community.


Heavily used at the BBC.
Stephen Janaway 2014

So

Mobile Test Strategy Top 5 Tips


1. Understand the poten3al user:

Which plaoorms and devices will they use?


How will they use the app or website?

2. Understand what OS versions are being used and


when new ones are coming.
3. Test using simulators, real devices and users.
4. Understand the cost of automa3on:

It is where desktop automa3on was 6 years ago.


Best combined with extensive unit and integra3on tes3ng.

5. Use an exploratory approach to manual tes3ng:


Mobile device projects are typically short.
Stephen Janaway 2014

Be proud, your work will soon


be in the hands of millions

Ques3ons?

Stephen Janaway

www.stephenjanaway.co.uk
@stephenjanaway
www.tes3nginthepub.com