This action might not be possible to undo. Are you sure you want to continue?
Mike Perham http:/ /mikeperham.com
Problem: Scaling Past One Machine
Warning: This stuff is hard. Ph.D hard.
Asynchrony Locality Failure Byzantine?
Asynchronous execution Shared memory! Easy to discover sibling threads Synchronization primitives
Group Membership Consensus
Truly Anonymous is impossible Algorithm scalability? 10 nodes = 100 messages = 20kb 1000 nodes = 1,000,000 messages = 200mb! Can run often!
Now that everyone is in the same room...
the process of agreeing on one result among a group of participants. (wikipedia)
Breaking symmetry Algorithm scalability
Once we have consensus, everything else follows But at what cost?
Cold Reality: This stuff is unpredictable, hard to test, full of nasty edge cases.
So what do we do?
Get Real! Start making Trade-offs.
What are your actual reliability requirements?
Single Point of Failure? Oh noes!
http:/ /github.com/mperham/politics a Ruby library providing utilities and algorithms for solving common distributed computing problems.
Database frequently scanned for events of interest, don’t want to create event dupes Events broadcast to interested users, don’t want notiﬁcation dupes
Leader Election via Memcached
M processes, 1 becomes leader for the given time period and performs the work Only one process working at a time As reliable as memcached
4 module Aggregator 5 class Engine 6 include Politics::TokenWorker 7 8 def initialize 9 register_worker 'dash-aggregator', :iteration_length => 60.seconds, :servers => memcached_servers 10 end 11 12 def start 13 process do 14 MetricAggregator.new.aggregate 15 end 16 end
Have a known space to scan regularly, divisible into N parts Have 12 databases which need to be scanned for events every two minutes
Peer-to-Peer Work Coordination
Work is divided into N buckets M-1 processes can be working on those N buckets concurrently Leader hands out buckets Peers discover via Bonjour and communicate via DRb
What if you need the good stuff, with no SPOF?
a family of protocols for solving consensus in a network of unreliable processors. (wikipedia)
The Holy Grail of Distributed Algorithms Google has spent tens of man-years on their C++ implementation.
Step 1: Prepare
(a) A proposer selects a proposal number n and sends a prepare request with number n to a majority of acceptors. (b) If an acceptor receives a prepare request with number n greater than that of any prepare request to which it has already responded, then it responds to the request with a promise not to accept any more proposals numbered less than n and with the highest-numbered proposal (if any) that it has accepted.
Step 2: Accept
(a) If the proposer receives a response to its prepare requests (numbered n) from a majority of acceptors, then it sends an accept request to each of those acceptors for a proposal numbered n with a value v, where v is the value of the highest-numbered proposal among the responses, or is any value if the responses reported no proposals. (b) If an acceptor receives an accept request for a proposal numbered n, it accepts the proposal unless it has already responded to a prepare request having a number greater than n.
Happy path is easy. Recovering from failures is hard. Started implementing, quickly stopped.
Plan to continue work on Paxos, if there is interest!
Formal Group Membership and Consensus Informal TokenWorker and BucketWorker Paxos
github.com/mperham/politics mikeperham.com firstname.lastname@example.org