You are on page 1of 165

Designing Scalable

Architectures
with MySQL Proxy

Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems

John Loehrer
Data Architect, Gaia Online

Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 1
Agenda
• Proxy concepts
• MySQL Proxy architecture
• Proxy with a single back-end
• Proxy with multiple back-ends
• broken?
• missing feature?
• not flexible?

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
 traditional way

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
 traditional way

1. file a bug report

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
 traditional way

1. file a bug report
2. wait

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
 traditional way

1. file a bug report
2. wait

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
• Open source way

source modify
code

new
compile source
code

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems

• creative (shortsighted) way

bring the logic
at application
level

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems

• creative (enlightened) way
set the logic at
server level
(stored
routines)

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems

• creative (more enlightened) way

set the logic at
protocol level
(proxy)

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
what can you do with MySQL Proxy
• create new commands
• filter queries (deny specific queries)
• collect statistics on usage
• implement usage quotas
• execute shell commands
• create customized logs
• implement server-side pivot tables
• start/stop a MySQL server remotely
• play movies (seriously!)
• make coffee (really?)
• sharding
• load balancing servers
Basic principles
Basic principles

PROXY
CORE Lua script
connection function
function
hook function

read query function
hook

read result function
hook
??
Lua

{
Perl ?
PHP?
Why not ...
Javascript?
[whatever]?
Lua

• SMALL ( < 200 KB)
• DESIGNED for
EMBEDDED systems
• Widely used (lighttpd)
lighttpd, like MySQL
Proxy, was created by
Jan Kneschke
Lua
Very popular among
game writers
Proxy overview
global Lua script
context
connect_server
session
context
session read_handshake
context
session

read_auth
context
session
context
session
context
session
context
session
read_auth_result
context
read_query

read_query_result
disconnect_client
Proxy overview
/usr/local/sbin/mysql-proxy \
--proxy-lua-script=/path/name.lua

IMPORTANT!
THE SCRIPT DOES NOT START UNTIL THE FIRST
CLIENT CONNECTION
read_query and read_query_result
client

query

result
function

function MySQL read_query_result
read_query Proxy

query if a query is passed directly to the
result server, its result is NOT evaluated
by read_query_result
SERVER
read_query and read_query_result
client

query

result

function
read_query_result
function MySQL
read_query Proxy only if a query is
query queue
added to the query
query
query
query queue, its result is
result
evaluated by
read_query_result
SERVER
An example: user quotas
• You want to limit user access
• When the user reaches a given amount of data,
further queries should be rejected
• the quota must work across sessions
(code online)
http://forge.mysql.com/tools/tool.php?id=111
User quotas
-- some global variables

-- proxy.global.bandwidth
-- will sum up the used bytes

proxy.global.bandwidth
= proxy.global.bandwidth or {}

-- session_user will identify the user
-- throughout the session
local session_user
User quotas
-- session_user needs to be initialized
-- when the user information is passed
-- (during the authentication)

function read_auth( auth )
session_user = auth.username
proxy.global.bandwidth[session_user]
= proxy.global.bandwidth[session_user]
or 0
end
BREAK (a handy Lua idiom)
-- simple assignment

a = a or 0

-- corresponds to:

if a == nil if a ~= nil
then then
a = 0 a = a
else else
a = a a = 0
end end
BREAK (a handy Proxy Lua function )
-- returns an error to the client

function error_result (msg)
proxy.response = {
type =
proxy.MYSQLD_PACKET_ERR,
errmsg = msg,
errcode = 7777,
sqlstate = 'X7777',
}
return proxy.PROXY_SEND_RESULT
end
User quotas
-- read_query (1)
-- checking if the quota has been
-- exceeded
function read_query (packet )
if proxy.global.bandwidth[session_user]
> 10000
and session_user ~= 'root'
then
return error_result(
'you have exceeded your query quota')
end
-- ...
User quotas
-- read_query (2)
-- adding to the totalizer

-- ...
proxy.global.bandwidth[session_user ]
=
proxy.global.bandwidth[session_user]
+ packet:len()
proxy.queries:append(1, packet )
return proxy.PROXY_SEND_QUERY
end
User quotas
-- read_query_result (1)
-- adding row headers to the totalizer

function read_query_result(inj)
local fields = inj.resultset.fields
local rows = inj.resultset.rows
if fields then
for i = 1, #fields do
proxy.global.bandwidth[session_user]
=
proxy.global.bandwidth[session_user]
+
(fields[i] and fields[i].name:len() or 0)
end
User quotas
-- read_query_result (2)
-- adding rows contents to the totalizer

if rows then
for row in rows do
for i = 1, #fields do
proxy.global.bandwidth[session_user]
=
proxy.global.bandwidth[session_user]
+
(row[i] and row[i]:len() or 0)
end
end
end
end
User quotas
-- read_query_result (3)
-- displaying the current bandwidth

print (session_user .. ' -> ' ..
proxy.global.bandwidth[session_user])
end
User quotas - what the user sees
mysql> select repeat('a', 10000);
ERROR 7777 (X7777): you have exceeded your
query quota
User quotas (VERY advanced)
-- you can create another module
-- to be loaded at run time
-- (this is really advanced)
-- and in such module you define
-- a SHOW QUOTAS command
User quotas (VERY advanced)
-- load_multi
User quotas (VERY advanced)
mysql> pload show_quotas.lua;
+--------------------------------+
| info |
+--------------------------------+
| module "show_quota.lua" loaded |
+--------------------------------+
mysql> show quotas;
+----------+-------+
| name | quota |
+----------+-------+
| simple | 3578 |
| root | 2111 |
| msandbox | 102 |
+----------+-------+
Designing Scalable
Architectures
with MySQL Proxy

Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems

John Loehrer
Data Architect, Gaia Online

Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 36
quick-start guide to mysql muxing
ME
online hangout
for teens
5 million unique users per month
1.3 billion forum entries
How does Gaia use the proxy?
connection multiplexer
turning 7,500 connections ...
into 100 MySQL connections
MySQL Proxy

7,500 connections
100 connections

MySQL
PHP Server
Why MUX?
a little history ...
Gaia’s data center in 2003
our server farm...
as our site grew ...
some scaling issues
added more servers ...
site explodes
diagnose the problem
GAIA, under the hood
typical LAMP stack
Linux
Share-Nothing Architecture
each apache server = 50 connections
Parent
Process

httpd

worker thread
child httpd
worker thread

child httpd ...

listener thread
...
50 connections * 150 servers = stampede
close connections each time
setup / tear-down is expensive
connecting to MySQL
read write read
connect challenge auth auth OK
read write read
connect challenge auth auth OK write query read result
75% 19% 6%

read write read
connect challenge auth auth OK write query read result close
MySQL Thread spawning
Client
Client MySQL Server

connect
Client MySQL Server

connect
Client MySQL Server

Thread
connect

new
Client MySQL Server

Thread
connect

new

accept
threads stack up
MySQL I/O
listen

accept
listen

close
accept
end program

NEW THREAD
listen

close
accept
end program

NEW THREAD

blocking read
listen

close
accept
end program

NEW THREAD

blocking read

data

blocking write
listen

close
accept
end program

NEW THREAD

blocking read

eof data

close blocking write

close thread
talking to multiple databases ...
Forum Page Session DB User DB Forum DB

start session

get userdata

get topic

get related users

save session
idle threads
we need a connection pool
ideally ...
Event Queue Thread Pool

Thread
allocate thread

process query

free
release the thread after each query
it’s like a limo ...
... vs. a taxi
decouple threads from connections
Libevent’s callback I/O very sweet!

listen

register callback

event dispatch

active descriptor Trigger callback
epoll/kqueue rulez! (proxy uses it)
MySQL 6.0 taking the right steps ...
In the meantime ...
by default, mysql-proxy isn’t a pooler
lua hooks are awesome!
hack some lua scripts ...
goal: recycle back-end connections
keep clients connected
proxy stays transparent
how to twist the proxy into a pooler
connecting: find an idle back-end
Proxy

client
Proxy

client

connect ahead idle pool
authentication
Client Proxy Back-End Pool

MySQL Server
Client Proxy Back-End Pool

MySQL Server
Client Proxy Back-End Pool

connect MySQL Server
Client Proxy Back-End Pool

connect MySQL Server
allocate
Client Proxy Back-End Pool

connect MySQL Server
allocate
Client Proxy Back-End Pool

connect MySQL Server
allocate

com-change-user
Client Proxy Back-End Pool

connect MySQL Server
allocate

com-change-user

challenge
Client Proxy Back-End Pool

connect MySQL Server
allocate

com-change-user

challenge

authenticate
Client Proxy Back-End Pool

connect MySQL Server
allocate

com-change-user

challenge

authenticate

OK
Client Proxy Back-End Pool

connect MySQL Server
allocate

com-change-user

challenge

authenticate

OK

free back-end
run the query
don’t be afraid to let go
Client Proxy

query
Client Proxy Back-End Pool

query
allocate
Client Proxy Back-End Pool

query MySQL Server
allocate

process query
Client Proxy Back-End Pool

query MySQL Server
allocate

process query

free back-end
connections scale!
Designing Scalable
Architectures
with MySQL Proxy
Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems

John Loehrer
Data Architect, Gaiaonline

Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 160
MySQL Load Balancer
MySQL software which can route database queries to
the appropriate database in a scale-out environment.

• Intelligently route reads across slaves
• Use database least behind, least loaded
• Remove latent slaves from read rotation
• Other distribution algorithms

• Improves/scales throughput of reads for online applications
• Helps customers reduce cost of adding slaves

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 161
MySQL Query Analyzer
MySQL Enterprise Monitor feature that allows users
to trace, monitor, and analyze MySQL query activity
for specific servers, users, and applications.

• Adaptive “Evil” query collection/tracing
• Historical browsing/analysis
• “Needle in a haystack” identification of worst
queries
• Worst execution times, # of execs, etc.

SQL code is the #2 cause of performance issues
97% of those surveyed will use this

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 162
MySQL Query Analyzer

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 163
Next Steps
MySQL Proxy Forum
http://forums.mysql.com/list.php?146

Documentation – MySQL Proxy
http://dev.mysql.com/doc/refman/5.1/en/mysql-proxy.html

Documentation – MySQL Load Balancer
http://dev.mysql.com/doc/refman/5.1/en/load-balancer.html

Beta Testing MySQL Proxy-Enabled Products
enterprise-beta@mysql.com

MySQL Proxy Online Poll

http://dev.mysql.com/tech-resources/quickpolls/

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 164
Questions?

Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems

John Loehrer
Data Architect, Gaiaonline

Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 165