The Practice of

Chaos Engineering
Ana Medina
Chaos Engineer at Gremlin
@ana_m_medina
#reactive18 @ana_m_medina

@ana_m_medina

Gremlin

Uber
SFEFCU Google Quicken Loans
Stanford University Miami
Dade College
#reactive18 @ana_m_medina

How many of you have heard

of Chaos Engineering?
#reactive18 @ana_m_medina

How many of you have run a

Chaos Engineering
experiment?
#reactive18 @ana_m_medina

What is
Chaos Engineering?
#reactive18 @ana_m_medina

Chaos Engineering
Thoughtful, planned
experiments designed to reveal
the weakness in our systems.
#reactive18 @ana_m_medina

Chaos Engineering
Inject something harmful to
build an immunity.

-@KoltonAndrus
Gremlin Founder and CEO
#reactive18 @ana_m_medina

Why?
● Microservices
● Systems are scaling
fast
● Downtime is really
expensive
● Our dependencies
will fail
● Pager fatigue and
burnout really hurts
#reactive18 @ana_m_medina

Use Cases:
● Outage
reproduction
● On-call training
● Strengthen new
products
● Battle test new
infrastructure and
services
#reactive18 @ana_m_medina

What do you need before doing Chaos

Engineering?
● Monitoring/Observability
● On-Call and Incident Management
● Cost of Downtime Per Hour
#reactive18 @ana_m_medina

Chaos Engineering is not

● Unexpected or unmonitored experiments
● Creating outages
#reactive18 @ana_m_medina

“Chaos Engineering
Without Observability ...
Is Just Chaos”
-@mipsytipsy
Charity Majors
CEO of honeycomb
#reactive18 @ana_m_medina

Minimize the
Blast radius
#reactive18 @ana_m_medina

Level 0
VALUE PROVIDED

THE BEGINNING Prepare for host failures in the cloud

Chaos Monkey APPROACH TAKEN

Random

MATURITY REQUIRED

Low
#reactive18 @ana_m_medina

Level 1
VALUE PROVIDED
THE FIRST STEP
Prepare for host-level failures

Infrastructure APPROACH TAKEN

Failures Disciplined

MATURITY REQUIRED

Basic Operations
#reactive18 @ana_m_medina

Level 1.5
VALUE PROVIDED
INTERMEDIATE
Prepare for high impact events

Network Failures APPROACH TAKEN

Gameday

MATURITY REQUIRED

Networking expertise
#reactive18 @ana_m_medina

Level 2
VALUE PROVIDED
THE NEXT STEP
Safely validate the user experience

Application APPROACH TAKEN

Failures Precision Experiments

MATURITY REQUIRED

Advanced
Latency added to 50% of android traffic
Exceptions - 50% of android traffic failed
#reactive18 @ana_m_medina

You can and should inject chaos at every layer

of your stack

● Application
● API
● Caching
● Database
● Hardware
● Cloud Infrastructure / Bare metal
#reactive18 @ana_m_medina

Top places to inject chaos

#reactive18 @ana_m_medina
#reactive18 @ana_m_medina

https://www.gremlin.com/community/tutorials/what-i-learned-running-the-
chaos-lab-kafka-breaks/
#reactive18 @ana_m_medina

Getting Started:

● Identify top 5 critical systems

● Choose system
● Whiteboard the system
● Determine what experiment you want to run:
(resource, state, network)
● Determine Blast Radius
#reactive18 @ana_m_medina

Companies doing
Chaos Engineering
#reactive18 @ana_m_medina

Chaos Days
#reactive18 @ana_m_medina

Chaos Days: Dedicated day for your

entire company to focus on building
resilience instead of new products.

https://www.gremlin.com/community/tutorials/planning-your-own-chaos-day/
#reactive18 @ana_m_medina

“What could go wrong?”

“Do we know what will

happen if this breaks?”
#reactive18 @ana_m_medina

Chaos Day Crew:

VP Engineering / CTO / COO
Executive Assistant
Engineering Director / Manager
Senior Engineer
New Grad / Intern Engineer
#reactive18 @ana_m_medina

What experiments can you

run?
• Reproduce outage conditions
• Unpredictable circumstances
• Large traffic spikes
• Race conditions
• Datacenter failure
• Time travel - system clocks to be out of sync
• Network errors
• CPU overloads
#reactive18 @ana_m_medina
#reactive18 @ana_m_medina
#reactive18 @ana_m_medina
#reactive18 @ana_m_medina
#reactive18 @ana_m_medina
#reactive18 @ana_m_medina

gremlin.com/chaos-monkey/
#reactive18 @ana_m_medina

Learn more:
Join the Chaos, Join Slack:

bit.ly/chaos-eng-slack
1,900+ members across the world
#reactive18 @ana_m_medina

THANKS!

ana@gremlin.com
@ana_m_medina