Tutorial #1: Introduction To Mobile Application

Testing
Gone are the days when the telephone used to be an appliance that sat in a corner and
had to ring to get our attention or a computer was a machine only a few people used –
they are now an extension of our being- a window to the world and virtual servants that
do as they are told.

Computers were a rage and changed how we humans thought, behaved, learned and
existed.

Nowadays, Mobility solutions have taken over the market. People don’t want to switch
ON their laptops/PC for everything, rather they want their handheld devices to perform
everything quickly.

Hence the mobile solutions which we deliver to our clients should be tested very well.
This tutorial is intended for those people who are already in mobile testing or those who
have switched to it in recent times. As we already have many tutorials on definitions of
mobile testing related terminologies, we will be directly dealing with the scope of this
tutorial.

This tutorial will be both an introduction and your guide to Mobile Testing. So,
read through!

Types of Mobile Testing

There are broadly 2 kinds of testing that take place on mobile devices:

#1. Hardware testing:

The device including the internal processors, internal hardware, screen sizes, resolution,
space or memory, camera, radio, Bluetooth, WIFI etc. This is sometimes referred to as,
simple “Mobile Testing”.
#2. Software or Application testing:

The applications that work on mobile devices and their functionality are tested. It is
called the “Mobile Application Testing” to differentiate it from the earlier method. Even in
mobile applications, there are few basic differences that are important to understanding:
a) Native apps: A native application is created for use on a platform like mobile and
tablets.
b) Mobile web apps are server-side apps to access website/s on mobile using different
browsers like Chrome, Firefox by connecting to a mobile network or wireless network
like WIFI.
c) Hybrid apps are combinations of native app and web app. They run on devices or
offline and are written using web technologies like HTML5 and CSS.
There are few basic differences that set these apart:
 Native apps have single platform affinity while mobile web apps have the
cross-platform affinity.
 Native apps are written in platforms like SDKs while Mobile web apps are
written with web technologies like HTML, CSS, asp.net, Java, PHP.
  For a native app, installation is required but for mobile web apps, no
installation is required.
 A native app can be updated from the play store or app store while mobile
web apps are centralized updates.
 Many native apps don’t require an Internet connection but for mobile web
apps, it’s a must.
 Native app works faster when compared to mobile web apps.
 Native apps are installed from app stores like Google play store or app
store where mobile web are websites and are only accessible through the
Internet.
The rest of the article is going to be about Mobile Application Testing.

The significance of Mobile Application Testing

Testing applications on mobile devices is more challenging than testing web apps on the
desktop due to

 Different range of mobile devices with different screen sizes and

hardware configurations like a hard keypad, virtual keypad (touch screen)
and trackball etc.
 Wide varieties of mobile devices like HTC, Samsung, Apple and Nokia.
 Different mobile operating systems like Android, Symbian, Windows,
Blackberry and IOS.
 Different versions of operation system like iOS 5.x, iOS 6.x, BB5.x,
BB6.x etc.
 Different mobile network operators like GSM and CDMA.
  Frequent updates – (like Android- 4.2, 4.3, 4.4, iOS-5.x, 6.x) – with each
update a new testing cycle is recommended to make sure no application
functionality is impacted.
As with any application, Mobile application testing is also very important, as the clientele
is usually in millions for a certain product – and a product with bugs is never appreciated.
It often results in monetary losses, legal issue, and irreparable brand image damage.

Basic Difference Between Mobile and Desktop Application

Testing:
Few obvious aspects that set mobile app testing apart from the desktop testing

 On the desktop, the application is tested on a central processing unit. On

a mobile device, the application is tested on handsets like Samsung,
Nokia, Apple, and HTC.
 Mobile device screen size is smaller than a desktop.
 Mobile devices have less memory than a desktop.
 Mobiles use network connections like 2G, 3G, 4G or WIFI where desktop
use broadband or dial-up connections.
 The automation tool used for desktop application testing might not work on
mobile applications.
Types of Mobile App Testing:
To address all the above technical aspects, the following types of testing are
performed on Mobile applications.
  Usability testing– To make sure that the mobile app is easy to use and
provides a satisfactory user experience to the customers
 Compatibility testing– Testing of the application in different mobiles
devices, browsers, screen sizes and OS versions according to the
requirements.
 Interface testing– Testing of menu options, buttons, bookmarks, history,
settings, and navigation flow of the application.
 Services testing– Testing the services of the application online and
offline.
 Low-level resource testing: Testing of memory usage, auto-deletion of
temporary files, local database growing issues known as low-level
resource testing.
 Performance testing– Testing the performance of the application by
changing the connection from 2G, 3G to WIFI, sharing the documents,
battery consumption, etc.
 Operational testing– Testing of backups and recovery plan if a battery
goes down, or data loss while upgrading the application from a store.
 Installation tests– Validation of the application by installing /uninstalling it
on the devices.
 Security Testing– Testing an application to validate if the information
system protects data or not.
Mobile Application Testing Strategy
The Test strategy should make sure that all the quality and performance guidelines are
met. A few pointers in this area:

1) Selection of the devices – Analyze the market and choose the devices that are
widely used. (This decision mostly relies on the clients. The client or the app builders
consider the popularity factor of certain devices as well as the marketing needs for the
application to decide what handsets to use for testing.)

2) Emulators – The use of these is extremely useful in the initial stages of development,

as they allow quick and efficient checking of the app. The emulator is a system that runs
software from one environment to another environment without changing the software
itself. It duplicates the features and works on the real system.

Types of Mobile Emulators

 Device Emulator- provided by device manufacturers

 Browser Emulator- simulates mobile browser environments.
 Operating systems Emulator- Apple provides emulators for iPhones,
Microsoft for Windows phones and Google Android phones

List of few free and easy to use mobile device emulators

i. Mobile Phone Emulator – Used to test handsets like iPhone, Blackberry, HTC,
Samsung etc.
ii. MobiReady – With this, not only can we test the web app, we can also check the
code.
iii. Responsivepx – It checks the responses of the web pages, appearances, and
functionality of the websites.
iv. Screenfly – It is a customizable tool and used to test websites under different
categories.

3) After a satisfactory level of development is complete for the mobile app, you could
move to test on the physical devices for more real-life scenarios based testing.

4) Consider cloud computing based testing: Cloud computing is basically running

devices on multiple systems or networks via the Internet where applications can be
tested, updated and managed. For testing purposes, it creates the web-based mobile
environment on a simulator to access the mobile app.
Pros:

 Backup and recovery- Cloud computing automatically takes back up of

your data from remote location making recovery and restoring of data
easily. And also, the storage capacity is unlimited.
 Clouds can be accessed from different devices and anywhere.
 Cloud computing is cost-efficient, easy to use, maintain and update.
 Fast and quick deployment.
 Web-based interface.
 Can run the same script on several devices in parallel.
Cons

 Less control– Since the application runs on the remote or third-party

environment, the user has limited control and access to the functions.
 Internet connectivity issues– the setup is on the Internet. Network
issues affect the availability and functioning
 Security and privacy Issues– Cloud computing is an Internet computing
and nothing on the Internet is completing secure, so chances of data
hacking are more.
5) Automation vs. Manual testing
 If the application contains new functionality, test it manually.
 If the application requires testing once or twice, do it manually.
 Automate the scripts for regression test cases. If regression tests are
repeated, automated testing is perfect for that.
 Automate the scripts for complex scenarios which are time-consuming if
executed manually.
Two kinds of automation tools are available to test mobile apps:
Object-based mobile testing tools– automation by mapping elements on the device
screen into objects. This approach is independent of screen size and mainly used for
Android devices.

 Eg:- Ranorex, jamo solution

Image-based mobile testing tools– create automation scripts based on screen
coordinates of elements.

 Eg:- Sikuli, Egg Plant, RoutineBot

6) Network configuration is also the necessary part of mobile testing. It’s important to
validate the application on different networks like 2G, 3G, 4G or WIFI.

Test Cases for Testing a Mobile App

In addition to functionality based test cases, Mobile application testing requires special
test cases which should cover following scenarios.

 Battery usage– It’s important to keep a track of battery consumption while

running application on the mobile devices.
 The speed of the application- the response time on different devices,
with different memory parameters, with different network types etc.
 Data requirements – For installation as well as to verify if the user with
the limited data plan will able to download it.
 Memory requirement– again, to download, install and run
 The functionality of the application– make sure application is not
crashing due to network failure or anything else.
Feature Stor Test Summary Preconditi Executio Expected Actu Priorit Revie Executio Commen Teste Defec
Name y ID Case on n Steps Result al y w n Status ts r t#
ID Resu Status
lt
Install Mob- Mob- Verify that 1. Select 1. Click The              
1214 1214_ application the on install application
1 should be application button. should be
Installed from google 2. Installed
successfull play. Navigate successfull
y. to the y.
menu and
click on
the newly
installed
app.
Uninstall Mob- Mob- Verify that 1. Execute 1. Click The              
1214 1214_ application test case on application
2 should be ID: Mob- settings. should be
Uninstalled 1214_1 2. Select Uninstalled
successfull the newly successfull
y. added y.
applicatio
n on Mob-
1214_1.
3. Click
on
Uninstall
button.
4. Verify.
Interruption Mob- Mob- Verify that 1. Execute 1. Open User              
by Calls 1214 1214_ user test case the should
3 should ID: Mob- applicatio able to
able to 1214_1 n. accept
accept 2. Phone
Phone Navigate calls when
calls when here an application
application there for is running
is running a and should
and should moment. continue
 continue 3. Make a from the
from the call from same
same another point.
point. device to
the
device
where
you have
opened
the
applicatio
n.
4. Pick up
the call.
5. Now
disconnec
t it and
verify.
Interruption Mob- Mob- Verify that 1. Execute 1. Open User              
by 1214 1214_ user test case the should
Messages 4 should ID: Mob- applicatio able to
able to 1214_1 n. accept
accept 2. messages
messages Navigate when
when here an application
application there for is running
is running a and should
and should moment. continue
continue 3. Send a from the
from the message same point
same point from after
after another reading the
reading the device to message.
message. the
device
where
you have
opened
the
 applicatio
n.
4. Read
the
message.
5. Close
the
message
app and
verify.
Memory Mob- Mob- Verify that 1. The 1. Go to Application              
1214 1214_ user device the app should
5 should memory from display
able to see space google with proper
proper should not play error
error more than store. message
message 20 mb. 2. Click when
when 2. Execute on the device
device test case install memory is
memory is ID: Mob- button. low.
low. 1214_1 3. Wait till
Note: the
Application applicatio
memory n get
requirement installed
is 25 MB. and
verify.
Exit Mob- Mob- Verify that 1. Execute 1. Click User              
application 1214 1214_ user test case on app should
6 should ID: Mob- and open able to exit
able to exit 1214_1 it. from
from 2. Now application
application press the if we click
if we click end key on end
on end and key.
key. verify.
Battery Mob- Mob- Verify that 1. Execute 1. Click When              
1214 1214_ user test case on app battery is
7 should ID: Mob- and open low the
 able to see 1214_1 it. alert
the alert 2. Use should
when the display.
battery is applicatio
low. n till you
get the
low
battery
indication.
Battery Mob- Mob- Verify that 1. Execute 1. Click Application              
Consumpti 1214 1214_ application test case on app should not
on 8 should not ID: Mob- and open consume
consume 1214_1 it. more
more 2. Full 2. Use battery
battery charge your the
device. applicatio
n and
verify the
status of
the
battery in
15 mins
interval of
time.
Charge Mob- Mob- Verify that 1. Execute 1. Click Application              
1214 1214_ application test case on app should run
8 should run ID: Mob- and open when
when 1214_1 it. inserting
inserting 2. Insert the
the the charger. It
charger. It charging will not
will not pin in affect the
affect the between application
application running of
the
applicatio
n and
verify.
ypical activities and proceedings in Testing Mobile Application
The scope of the testing depends on a number of requirements to be checked or the
extent of changes made to the app. If the changes are few, a round of sanity testing will
do. In case of major and/or complex changes, a full regression is recommended.

An example application testing project: ILL (International Learn Lab) is an application

designed to help admin, publisher to create websites in collaboration. Using a web
browser, instructors choose from a set of features to create a class that meets their
requirements.

Mobile Testing process:

Step #1. Identify the types of testing: As an ILL application is applicable for browsers,
so it’s mandatory to test this application on all supported browsers using different mobile
devices. We need to do usability, functional and compatibility testing on different
browsers with the combinations of manual and automation test cases.
Step #2. Manual and Automated testing: The methodology followed for this project is
Agile with the iteration of two weeks. Every two weeks dev. team releases a new build
for testing team and testing team will run their test cases in QA environment. Automation
team creates scripts for the set of basic functionality and runs the scripts that help
determine if the new build is stable enough to test. The Manual testing team will test the
new functionality.
JIRA is used for writing of acceptance criteria; maintaining of test cases and logging /re-
verification of defects. Once the iteration gets over, iteration planning meeting held
where dev. The team, product owner, business analyst, and QA team discuss what
went well and what needs to improve.
Step #3. Beta Testing: Once the regression testing is completed by the QA team, the
build moves into UAT. User Acceptance Testing is done by the client. They re-verify all
the bugs to make sure every bug was fixed and the application is working as expected
on every approved browser.
Step #4. Performance test: Performance testing team tests the performance of the web
app using JMeter scripts and with different the loads on the application.
Step #5. Browser testing: The web app gets tested across multiple browsers- both
using different simulation tools as well as physically using real mobile devices.
Step #6. Launch plan: After every 4th week, the testing moves into staging, where a
final round of end to end testing on these devices is performed to make sure the product
is ready for production. And then, it goes Live!
*****************************************

How to Test Mobile Applications on Both Android and iOS

Platforms
It is very important for the testers who test their apps on both iOS and Android Platform
to know the difference between the both. iOS and Android have a lot of differences w.r.t
to the look and feel, app views, encoding standards, performance, etc.

Basic Difference between Android and iOS Testing

You might have gone through all the tutorials, I have put in some major
differences here, which in turn will help you as part of your testing:
#1)  As we have a lot of Android devices available in the market and all of them come
with different screen resolutions and sizes, hence this is one of the major difference.
For Example, Samsung S2 size is too small when compared with Nexus 6. There are
high possibilities that your app layout and design get distorted on one of the devices.
Probability is low in iOS as there are only countable devices available in the market and
out of those many phones have similar resolutions.
For Example, before iPhone 6 and above came into existence all the older versions had
the similar size only.
#2) Example to assert the above point is that in Android the developers must use
1x,2x,3x,4x and 5x images to support image resolutions for all devices whereas iOS
uses just 1x,2x and 3x. However, it becomes the tester’s responsibility to ensure that the
images and the other UI elements are displayed correctly on all devices.
You can refer to the below diagram to understand the concept of image
resolutions:
#3) As we have the market flooded with Android devices, the code must be written in
such a way in which the performance remains steady. So, it is quite probable that your
app may behave slowly on lower-end devices.
#4) Another issue with Android is that software upgrades are not available for all devices
at a go. Device manufacturers decide when to upgrade their devices. It becomes a very
difficult task to test everything both with the new OS and the old OS.
Also, it becomes a cumbersome task for the developers to modify their code to support
both the versions.

For Example, when Android 6.0 came, there was a major change as this OS started
supporting app level permissions. To clarify further, the user could change permissions
(location, contacts) at app level also.
Now the testing team owes the responsibility to make sure that showing permissions
screen on the app launch on Android 6.0 and above and not showing permission screen
on the lower versions.

#5) From the testing perspective, Pre-production build (i.e. beta version) testing is
different on both the platforms. In Android, if a user is added to beta users list then he
can see the updated beta build on the Play Store only if he is signed in into the play
store with the same email ID which is added as a beta user.
Key Factors in Mobile Testing
I have been working in Mobile Testing for the last 2 years on both iOS and Android
Platform and all the key points mentioned below in this tutorial are from my personal
experience and some got derived from the issues encountered in the project.
Define your own scope of Testing
Everyone has their own style of testing. Some testers just focus on what they see from
their eyes and the rest are passionate about everything that works behind the scenes of
any mobile application.

If you are an iOS/Android Tester, I would suggest you to at least get yourself familiar
with some common limitations/ basic functionalities of Android or iOS as it always adds
value to our style of testing. I know things are difficult to understand without citing
examples.

Given below are few Examples:

 We cannot change the permissions like camera, storage etc. on app level
in Android devices which are below 6.0.1 version.
 For iOS below 10.0 version, call kit was not there. Just to brief you in
simple words, call kit is used by a calling app and displays full-screen view
when a user is getting a call from the calling apps such as WhatsApp,
Skype etc. Whereas for iOS versions below 10.0 we see those calls as a
notification banner.
 Many of you might have come across problems in Paytm where your app
is not redirecting you to the payment page of the bank in case you want to
add money to your wallet. We think the above is an issue with our bank or
Paytm server but it is just that our AndroidSystemWebView is not updated.
Little knowledge about programming is always helpful for you and to share
with your team.
 In simple words, whenever an app is opening any web page in it, then
AndroidSystemWebView should be updated.

Do not Limit your Testing

Testing should not just be limited to exploring the mobile app and logging bugs. We, as a
QA should be aware of all the request that we hit our server and the response that we
get out of it.

Configure Putty to view logs or verify sumo logic for logs depending on what is being
used in your project. It not only helps you in knowing the End-to-End flow of the
application but also makes you a better tester as you get more ideas and scenarios now.
Reason: Nothing comes into this world without any reason. Any statement should have
a valid reason behind it. The reason behind analyzing the logs is that many exceptions
are observed in the logs but they don’t show any impact on the UI hence we don’t notice
it.
So, should we ignore it?
No, we shouldn’t. It doesn’t have any impact on the UI but it may be a futuristic concern.
We could potentially see our app crashing if these kinds of exception keep creeping. As
we have mentioned about App Crash in the last sentence, this leads the QA to have
access to crashlytics of the project.

Crashlytics is a tool where crashes are logged along with the time and device model.

Now the question over here is that if the tester has seen app crashing then why
does he need to bother about crashlytics?
The answer to this is quite interesting. There are some crashes which may not be visible
on the UI but they are logged on crashlytics. It could be out of memory crash or some
fatal exceptions which may impact the performance later.

Cross-Platform Testing
Cross-Platform Interaction Testing is very important.

Citing a simple Example, say you are working on a chat application like WhatsApp
which supports sending images and videos and the application is built on both iOS and
Android platforms(Development may or may not be going in sync)
Ensure to test the communication of Android and the iOS, the reason being that is iOS
uses “Objective C” whereas Android programming is Java-based and due to both of
them being built on different platforms sometimes extra fixes need to be made at the app
side to recognize strings coming from different language platforms.

Keep an eye on the size of your Mobile App

Another important advice for mobile testers – Please keep checking the size of your
app after each release.
We should ensure that the size of the app doesn’t reach a point where even we as an
end user won’t wish to download this app due to its large size.

Testing App Upgrade Scenarios

For mobile testers, app upgrade testing is very important. Ensure your app doesn’t
crash on upgrade as the dev team may have done mismatching of a version number.
Data retention is also equally important as in whatever preferences the user has saved
in the previous version should be retained when he upgrades the app.

For Example, a user might have saved his bank card details in apps like PayTm, etc.
Device OS may not Support App
Sounds Interesting?
Yeah, many devices may not support your app. Many of you must be knowing that
vendors write their own wrappers on the top of US and it could be possible that any SQL
query of your app is not compatible with the device and hence it throws an exception
and it may result in not even launching the app on that phone.

Point over here is – Do try to use your app on your own devices except for the ones you
use in the office. It is quite possible that you see some issues with your app.

App Permission Testing

Next on the list is Permission Testing of mobile apps. Almost every second app asks
its users for access to their phone’s contact, camera, Gallery, Location etc. I have seen
few testers who make a mistake by not testing the proper combinations of these
permissions.
I can recall a real-time Example when we were testing a chat app which had all the
features of sharing images and Audio files. Permission for Storage was set to NO.
Now, when a user would click on the Camera option it never opened till the permission
for storage is set to YES. The scenario was ignored as Android Marshmallow had this
functionality that if storage permission is set to NO, the camera cannot be used for that
app.

The scope extends further than what we have discussed in the above paragraph. We
should make sure that the app is not asking for any permissions which are not used.

Any end user familiar with the software industry may not download the app in which too
many permissions are asked. If you have removed any feature from your app, then
make sure to remove the permission screen for the same.
Compare with similar and popular Apps in the Market
Moral of the story – If ever you are in a doubt, then just don’t conclude it yourself.
Comparing with the other similar apps on the same platform can strengthen your
argument that the functionality under test will work or not.
Get an Overview of Apple’s Build Rejection Criterion
Lastly, a majority of you have might have come across situations where your builds got
rejected by Apple. I know this topic won’t interest major portion of the readers but it’s
always good to know the rejection policies of Apple.

As a tester, it becomes difficult for us to cater the technical aspects but still, there is
some rejection criterion which the testers can take care of.

For more information on this, please click here.

Always be on the Front Foot
Being a tester, don’t let things pass over to your court from the Dev Team/ Managers. If
you are passionate about testing then “Always be on Front Foot”. Try to engage
yourself in activities which take place well before the code comes to your bucket to test.
Most importantly, keep looking at JIRA, QC, MTM or whichever is used in your project
for all the latest updates on tickets from clients and the Business Analyst. Also, be ready
to share your views if you require modifications. This applies to all the testers who are
working on various domains and platforms.

Until and unless we don’t feel the product as our own, we never ought to give
suggestions for new improvements or changes to the existing functionality.

Keep your app in background for a long time (12-24 hours)

I know it sounds weird but there is much logic behind the scenes which all of us don’t
understand.

I am sharing this because I have seen the app crashing after launching it, say after
around 14 hours from background state. The reason could be anything depending on
how the developers have coded it.

Let me share a real-time Example:

In my case token expiration was the cause behind it. For one of the chat apps if
launched after 12-14 hours would be stuck on the connecting banner and would never
get connected until killed and relaunched. These kind of things are very difficult to catch
and in a way, it makes mobile testing more challenging and creative.

Performance Testing of your App

In the mobile world, the performance of your app impacts the extent to which your
application is getting recognized worldwide. As a testing team, it becomes too important
to check your app response and more importantly how it works when a large number of
users are using it all together.

Example:
Let’s talk about PayTm.
You all must have clicked on ADD MONEY option in the PayTm app, which then
displays the balance you have in your wallet. If we consider what is going on behind the
scenes, then it is a request which is going on to the server with the PayTm UserID and
the server sends back the response with the balance in your account.

The above case is only when one user has hit the server. We need to make sure that
even when 1000 users hit the server, they should get back the response well on time
because end-user usability is our prime goal.

Conclusion
I would conclude this tutorial by re-iterating that mobile testing seems to be very easy in
the beginning but as you keep digging in you will understand that it is not easy to ensure
that whatever developed will run smoothly on thousands of devices all over the world.

You would mostly see the apps that are supported on latest and last few versions of OS
only. However, it becomes the duty of the testers to ensure that they don’t miss out any
scenarios. They are many other points which need to be taken into consideration but I
haven’t mentioned those already iterated in the other tutorials.
Scenarios like battery consumption, interrupt testing, testing on different networks (3G,
Wi-Fi), testing while switching networks, monkey testing of mobile apps, etc are all
useful when it comes to mobile testing.

The attitude of testers matters a lot when it comes to the real testing environment. Until
and unless you love your job you won’t bother doing things which are mentioned in the
tutorial.

I have been in this field for around 6 years now and I’m very well aware that the tasks
get monotonous at times but there are many other things that we can do on our own to
make those monotonous tasks somewhat interesting.

Designing the right test strategy, choosing the right mobile simulators, devices, and
mobile testing tools can make sure that we have 100% test coverage and help us
include security, usability, performance, functionality, and compatibility based tests into
our test suites.