Challenges in A/B Testing Mobile Native Apps

Eric Florenzano RuPy Brno 2012

Why am I interested in this?

Started a company to make iOS apps Ended up building A/B testing tools Twitter acquired our IP All open sourced github.com/clutchio

Show of hands

Who does web stu?

Show of hands

Who does web stu? Keep your hand up if youve done A/B testing on the web.

Show of hands

Who does iOS or Android apps?

Show of hands

Who does iOS or Android apps? Keep your hand up if youve done A/B testing on these apps.

What is A/B Testing?

Show users dierent variations of your app Track those users and their actions Analyze their actions to nd the most eective variation

Examples of A/B Tests

Button text: do people understand Register better than Sign Up? On a mobile website, try replacing the e-mail eld with phone number eld and verify by text message. Does sign-up conversion go up or down if we give people more introductory reading material?

A/B Tests on the Web Today

Backend determines A/B test bucket Front-end changes display

Example of A/B Test on Web

{% ab user phone-signup %} <input name=phone type=tel placeholder=Phone Number /> {% else %} <input name=email type=email placeholder=E-Mail Address/> {% endab %}

Now Mobile

if(ab(user, @phone-signup)) { label.text = @Phone Number; } else { label.text = @E-Mail Address; }

However...

How does this ab() function work on mobile? What can our latency be on this call? Can it talk to a database? What happens when you are oine?

Solution: Manifest Upfront

Download manifest of all A/B tests on launch Client must be smart enough to make its own decision, immediately, oine

Problem: First Launch

Manifest downloads asynchronously at launch What about the rst launch? What about A/B tests on that rst screen? One possible answer: bundle a manifest with the app. Sucks. Adds extra step to build process.

Goal Tracking

How do we determine success or failure? When is a test completed?

Goal Tracking

goal_reached(user, phone-signup)

Goal Tracking

Must have capability on frontend and backend Frontend example: registration page completed successfully Backend example: e-mail veried successfully

Goal Tracking Mobile Gotcha

What happens when the phone is oine?

Goal Tracking on Mobile

Store all results in a local phone database Upload all of the information periodically If possible, upload everything when the app is quit

Note: Complications

May receive some data twice -- have to query to double-check rst May receive very old data -- working set of data now very wide How long is too long to wait for data to come back?

Side-Note: User/Test Consistency

Its important for a user to have a consistent experience Once a user is placed in a test bucket, they should remain there for that session How long is a session? On mobile, a good heuristic is forever, until they update their app.

So...Minimum Requirements

Download manifest up-front Make weighted random decisions oine Track goals and store progress in a local db Peg users to the same bucket during the session Periodically upload progress

Problem: Slow Release Cycle

Each release might take weeks or even months A/B testing at this time scale is frustrating Can anything be done to improve it?

Solution: Parameterized Tests

Instead of a simple boolean if-else, pass back data instead Now you can change the tests on the server Still need to think ahead, but this can add lots of exibility

Parameterized Test Example

[AB test:@"login" data:^(NSDictionary *data) { btn.title = [data objectForKey:@title]; }];

Note

Remember, this decision still needs to be made instantly, and o-line So all of this data now becomes part of the manifest Data must be kept compact and cant store e.g. a lot of binary

Interpreting the Data

We care about two things: Which variation is winning? How condent are we about it?

Which variation is winning?

Can easily calculate a ratio for each variation: How many people have seen this variation? How many of those people have reached the goal?

How condent are we about it?

Statistics. My worst subject in school. Must choose a p-value - the higher the value, the less results you need, but lower accuracy Now compute a condence interval Im told the Agresti-Coull Interval is a good choice for calculating condence interval Open source JavaScript ABBA library is great!

ABBA Example
http://www.thumbtack.com/labs/abba/

Questions?

@erico on Twitter github.com/clutchio