Lessons learned building a Web 2.

0
Application using MySQL
 Who are we?
• We’re missing a vowl (must be web
2.0!)
• 7 full-time employees
• Several ex-Andover.net (Slashdot)
guys including our CEO
• Boston based but distributed dev
• Been in existence about 2 years
 What do we do?
• We’re all about
• Create and share reading lists
• Merge and filter feeds
• Publish widgets
• Future: Do more advanced stuff
with feeds
 Why “Grazr”
“I’m actually coming to the conclusion that
the whole subscriptions mindset is a problem
and that in future we’ll ‘graze’‚ for the most
part instead of subscribing. As Zigbee
sensors, RFID chips and GPS trackers
proliferate we’ll be drowning in an RSS-
everywhere world if we don’t change our
approach.”
James Corbett “Eirepreneur”
http://eirepreneur.blogs.com
January 2006
http://grazr.com
The heart of our
system:
 Lots of Mistakes
• Warning: Some of these lessons
will seem obvious
• Hindsight is 20/20
• We *did* have reasons for many
of these at the time (good ones?)
• Tried a lot of experimental
configurations
 Lesson 1: Beware
•arch. momentum
Early system decisions affected
later architecture, even when
abandoned
• Careful about exotic w/out good
reason
• db read on our feed processing
web boxes
• The traditional setup, splitting
MySQL from Apache = system
much happier
Lesson 2: Scaling
•“Don’t worry about scaling” - 37
signals
• “You must build for scaling or die!”
- Friendster
• The truth? Somewhere in the
middle
• Understand growth and scaling
patterns but don’t build up front
• Your scaling plan: wrong in
some way
Overemphasis on
scaling
• Two hosted data centers + one
test/backup data center
• Geographically separated
• Over provisioned
• 18 servers, architecture mirrored
• Traffic could have been served
from two or three machines
“Skynet Jr.”
Lesson 3: Limits of
Testing
• Startup reality: not enough time for
thorough testing
• Replication testing and simulation
• Speed good enough, even cross
country
• Problem: real world system
behaved differently
 Lesson 4:
Replication is fast,
• We knew
until it isn’t
better: but empirical
testing seemed OK
• Asynchronous nature of repl.
sometimes hard to code, careful
with state
• Some of our code treated it as
synchronous (Fail!)
• Smarter code was slow (retries,
polls)
 Lesson 5:
Memcached is
your
• Excellent tool infriend
the scaling
toolbox
• Classic cache: limiting touching
your database is good!
• Added benefit: on top of repl.
synchronizer
• Good temporary storage for async.
proc.
 Lesson 6: Bulk
insert / lazy write
• Obvious: If you don’t need it now,
do it later
• Disconnected / async good in
these cases
• If you can do it later, glom many
together (bulk)
• Much better write perf.
 Lesson 7: User
experience vs.
• Scaling
Emphasis on scaling hurt user
experience
• Characterize lazy vs. user affecting
transactions
• Fast, data correct transactions =
single data store (no read/write io
split) or a sync buffer
(memcached)
 Lesson 8:
Instrumentation
• Visibility into system good
• More data = better
• Non live testing != reality
• Scaling is iterative process,
requires feedback loop
• Still somewhere we’re seriously
lacking (but working on it!)
Lesson 9: Try new
things
• Best practices are good, but new
ideas sometimes better!
• Memcached as write/bulk buffer,
good results
• UDF, clever replication uses,
triggers, virtual servers,
background async. daemons
 Lesson 10:
Everyone has the
same problems
• If you can, re-use!
• Obvious: MySQL, Apache (not re-
writing db’s, webservers)
• Less obvious: building batch job
processor, others have done this
better! (Gearman)
Lesson 11: Accept
change
• Architecture constantly in flux
• Think ahead, but don’t overthink,
design for now + 1, not now + 100
• Accept change, react, constantly
re-evaluate
Lesson 12: Listen

• Even amongst team, easy to be

dogmatic
• Listen to everyone’s ideas even
when it challenges your expertise
• Some of our best ideas: the
combination of several approaches
Questions?
Discussion?