Brahmasta Adipradana

Produk Talks at
9 January 2021
Brahmasta Adipradana (Bram)

brahmasta@gmail.com
linkedin.com/in/brahmasta/

Education

● Informatics Engineering, Institut Teknologi Bandung (2003-2008)

● MSc Information System, Nanyang Technological University (2011-2013)

Experience

● IT Business Analyst, Supply Chain (Unilever Indonesia, 2008-2010)

● IT Business Analyst, Supply Chain (Unilever Asia, 2010-2015)
● IT Manager, Supply Chain + IT Manager, Korea (Unilever Asia, 2015-2018)
● Senior Product Manager, Economics (Grab, 2018-2019 )
● Lead Product Manager, Marketplace (Grab, 2019-…)
Look at a bit of theories of Take a look of what problem Few sample of experiments
why as PM we should run we are solving at Grab that we have run, and how
experiments to validate Marketplace we measure it
our hypotheses
to collect useful evidence, to make an
informed decision.

“... the biggest challenge facing product

managers is being able to conduct proper
market research to validate whether the market
truly needs what they’re building”*

Experimentations enable us to**:

1. Understand causation or correlation

2. Learn and Prioritize
3. Decentralize execution
* Mind The Product: “The biggest challenge for product managers?”
**SMoves The Needle: “Stop Building Products No One Wants”.
Problem Validation

“Do our users actually has this problem?”

“How painful it is for them?”

Feature Testing

“Will this feature actually help our users?”

“How many users will be interested to use this feature?”

Optimisations

“Is the pricing right?”

“Will users responds differently with this new offerings?”
A/B Testing

The gold standard of testing. Can be used for discovery or optimisations,

and typically easier to get statistically significant data

Pilot or Invite Only Testing

When the traffic is not enough or there are risks to do A/B test, then pilot can
be taken. However analysis can be more difficult to execute.
Choose your metrics carefully

Remember the objective of your test, and ensure you are monitoring
metrics that support validation of your hypotheses.

Avoid Self-Selection Bias!

Double check your treatment and control group. Ensure the profiles are the
same, and there is only single different conditions being applied.
Things to be ensured

● Representative of the audience we are targeting

● Both have the same characteristics
● The bigger the sample size is, the better (for statistical
significance)
● The longer the test run is, the better (for statistical
significance)

Quick “Crash Course” References:

● P-values and significance tests (Khan Academy)

● A Refresher on Statistical Significance (HBR)
Sample of checking treatment and
● Controlled Experiments (Crash Course Statistics) control user profiles
Transport Foods

Place an order

Eater Merchant
Passenger Driver
1. Accept order
1. Accept booking
2. Prepare order
Book a ride 2. Pick up passenger
3. Drop off passenger

Delivery driver
1. Accept job
2. Pick up order
3. Drop off order
Driver wants to have a good earning
by partnering with Grab,

Passenger / Eater / Sender /

Merchants want to have a high
quality drivers available nearby

Grab need to optimise the cost and

quality of service that is sustainable
for all business verticals

However, as driver constantly move,

the availability needs to be balanced
Guide Drivers to high Incentivise drivers to Make it fun and engaging
demand areas complete job or move
Get Insights from Qualitative Data
Focus Group Discussions, Surveys, Customer Support tickets, On the ground
feedback, User Research on specific customer problems

Monitor Quantitative Data

Fulfillment Rate, Driver earnings, Driver Quality Metrics, Utilisations, Supply
Demand imbalance, ...

+ Validate New Ideas through Experimentations

Note: Check our article about Supply Demand imbalance :)

Understanding Supply & Demand in Ride-hailing Through the Lens of Grab Data
1. Move Sticky Drivers?
2. Power of Community?
3. Incentive Guarantee
Problem
It was observed that there is a specific segment of drivers which remain in
the same area, and these Area-Sticky Drivers are much more idle as
compared to Area-independent Drivers.

Hypotheses
Area-Sticky Drivers does not know the demand outside of his zone. By
notifying drivers to move few hours before, drivers can prepare to catch
orders and have better earnings

Key Metrics
Utilisations (+ve), Average Rides (+ve)
Experiment Design
We designed various communications with the help of Behavioural
Scientist. Different drivers group were exposed to the different treatment
notifications for over a span of 30 days.

Treatment “What are you doing at 5pm today? Earn more by going to a nearby
Kebayoran Baru area! Drivers there get 20% more jobs during 5 pm to 7 pm”

“You can earn more! Drivers in nearby area Kebayoran Baru have 20% more
jobs during 5pm to 7pm”

Control (Work as per normal)

Findings Next Steps

Tested several 10+ areas. Continue testing it in other regions

Statistically significant movements: with same problem.

● Pasar Minggu → Pancoran Build this as contextual comms

● Pancoran → Setiabudi integrated with current driver
● Cilandak → Kebayoran Baru guidance

Key Metrics

Utilisations (+ve)
Average Rides (+ve)
Problem
Drivers to do not fully understand our incentive program and driver
guidance, except “lead drivers”. Many drivers learn to drive effectively from
their team lead in community.

Hypotheses
By enabling drivers to work in a team, it will help to improve other team
members to work more efficiently and earn more.

Key Metrics
Average Rides (+ve), Online Hours (+ve), Completion Rate (+ve), Driver
Earnings (+ve)
Lots of Driver communities at big cities at Indonesia, Thailand, There are many YouTube videos where
and Vietnam drivers are sharing how they can earn
incentives

"I love to be around fellow drivers. We have a spot that is shady and windy; we share information on
how to earn more, and more importantly, we share a laugh! At this spot, I know that I can get to nearby
hotspots quickly and pick up passengers at areas that we cannot wait at" – Grab Driver
Experiment Design

● Utilise the current gems backend to calculate

for all drivers in a team, and show in the
leaderboard (as widget)
● Approach predetermined subset of drivers
about Team missions, and randomly for
Treatment (team missions) and control
(additional incentives)
● Run the competitions for 2-4 weeks period
● Monitor drivers’ performance on the metrics
defined.
Results Next Steps

● High Engagements: drivers encouraged each enable more scalable

others (lend money for GF, community leader solutions, include chat
give award, motivations messages) and driver-managed
● WhatsApp Group being used extensively team, ensure competition
● Some got demotivated after knowing they are is not too long.
not able to win
● Issues on the app: lags, ops monitoring

Key Metrics

Average Rides (+ve), Online Hours (+ve),

Completion Rate (+ve), Driver Earnings (+ve),
Cost(-ve)
Problem
Many drivers switches off after reaching a tier target, resulting lack of
supply later at the day. This has been highlighted multiple times by lead
drivers because they do not want to lose their incentive by canceling odd
jobs.

Hypotheses
By guaranteeing incentives, drivers will be willing to work longer because
there is no risk for them to continue driving

Key Metrics
Online Hours (+ve), Fulfillment Rate (+ve), Driver Earnings (+ve)
Experiment Design

● Enable the incentive backend to lock the

payout although when the driver reach the
top tier.
● Enable the functionality to treatment group,
compare the results with control
● Monitor drivers’ performance on the metrics
defined for 2-4 weeks.
Results Next Steps

● Improvements on online hours visible, but Keep the feature on but

only to small subset of drivers. Network wise it do not invest big on it.
has minimum impact
● Driver satisfaction is increasing but only for
specific power users.

Key Metrics

Online Hours (+ve), Fulfillment Rate (+ve), Driver

Earnings (+ve), Cost(-ve)
● Experiments is important to collect useful evidence, to make an
informed decision.
● Use it for problem validation, feature testing, or optimisation
● Typically it is done through A/B test but there are other methods
possible
● Choose your metrics carefully and avoid selection bias
● Try to get insight from qualitative feedback to understand the
reasoning behind the results
Do you have any questions?

● Email: brahmasta@gmail.com
● Discuss: calendly.com/brahmasta

CREDITS: This presentation template was

created by Slidesgo, including icons by
Flaticon, infographics & images by Freepik
and illustrations by Stories

Please keep this slide for attribution