SCALING YOUR DATABASE

@orendon

http://medellinrb.org
@RubyMedellin
 WHY DO I CARE ?
• High Availability

• Fault Tolerance

• Migration/Upgrades

• Load Balancing

• Increase Throughput
 WHY DO I CARE ?

• Support more users

• Provide better performance

• Geographic distribution

• Store bigger amounts of data

 YOU PROLLY
DON’T NEED TO
SCALE
(yet)
 CLEAN UP YOUR SH*T
• Database tuning *

• Check your relationships

• Check used/unused indexes

• Avoid N+1 queries

• Normalize or Denormalize

• Connection Pooling

• Monitor your App/BD (bottlenecks, cpu, memory, etc)

• Caching (redis, memcached, pgmemcache)

KNOW YOUR OPTIONS
Vertical vs Horizontal Scalability
 VERTICAL SCALING
Scale Up

• “Simpler”

• No extra maintenance

• Hardware is expensive*
 HORIZONTAL SCALING
Scale Out

• Hardware is cheaper

• Better availability / Fault tolerance

• “Harder”

• Higher maintenance cost

SCALING OUT POSTGRES
REPLICATION
(built-in)
 REPLICATION
• Write / Read-only

• Failover*

• Reporting

• SPoF*

• Different versions*
SHARDING
 SHARDING

• Large working sets

• Too many writes (IO/Slave exhausted)

• Unique identifiers (UUID, snowflake)

• “Complex” (backing up, indexes, schema)

http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
POSTGRES REPLICATION
ON RAILS
 INFRASTRUCTURE
(master)

CREATE USER rep REPLICATION LOGIN CONNECTION LIMIT 1

ENCRYPTED PASSWORD 'yourfancypasswordhere';

#pg_hba.conf

host replication rep 192.168.10.50/32 md5

#postgresql.conf

listen_addresses = ‘localhost,192.168.10.50’

wal_level = 'hot_standby'

hot_standby = on
 INFRASTRUCTURE
(slave)
#pg_hba.conf

host replication rep
192.168.10.50/32 md5

#postgresql.conf

listen_addresses = ‘localhost,
192.168.10.100’

wal_level = 'hot_standby'

hot_standby = on
 RUBY ON RAILS
(usage)
User.create(name: “Larry”)

User.find_by(name: “Moe”)

• Octopus

https://github.com/thiagopradi/octopus/

• Fully replicated by default


Writes => Master

Reads => Slave
https://github.com/thiagopradi/octopus/
 RUBY ON RAILS
(usage)

User.where(name: “Moe”)

.using(:slave_one)

Octopus.using(:slave_two) do

User.create(name: “Larry”)

User.create(name: “Curly”)

end
https://github.com/thiagopradi/octopus/
 RUBY ON RAILS
(Octopus config)
#database.yml (master database)

#shards.yml (slaves)

octopus:

replicated: true

fully_replicated: false

production:

slaveA:

host: 192.168.22.10

database: mydb_slaveA

slaveB:

host: 192.168.22.50

database: mydb_slaveB
 RECAP
• Avoid premature optimization

• Optimize everything else first

• KISS

• Hardware vs Maintenance costs

• Team Communication
 REFERENCES
• https://github.com/thiagopradi/octopus/

• https://devcenter.heroku.com/articles/heroku-postgres-follower-
databases

• https://devcenter.heroku.com/articles/heroku-postgres-ha

• http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-
at-instagram

• https://www.digitalocean.com/community/tutorials/how-to-set-up-
master-slave-replication-on-postgresql-on-an-ubuntu-12-04-vps
 <3
@orendon

http://medellinrb.org
@RubyMedellin