Migration From Oracle to MySQL

An NPR Case Study

By Joanne Garlow

npr.org
 Overview
 Background
 Database Architecture
 SQL Differences
 Concurrency Issues
 Useful MySQL Tools
 Encoding Gotchas
 Background
 NPR (National Public Radio)
 Leading producer and distributor of radio programming
 All Things Considered, Morning Edition, Fresh Air, Wait, Wait,
Don’t Tell Me, etc.
 Broadcasted on over 800 local radio stations nationwide
 NPR Digital Media
 Website (NPR.org) with audio content from radio programs
 Web-Only content including blogs, slideshows, editorial columns
 About 250 produced podcasts, with over 600 in directory
 Mobile apps and sites
 Syndication
High-Level System Architecture
Limitations of the Oracle Architecture
 Reached capacity of single system to support our
load
 Replication outside our budget
 Databases crashes were becoming frequent
 Database Architecture Goals
 Redundancy
 Scalability
 Load balancing
 Separation of concerns
 Better security
High-Level System Architecture
 Database Architecture

Content
Mgmt System Main
RO slave
Main Web Servers
• Read and updated
InnoDBonly by our website
• Low resource contention • Updated
Mainby a nightly script
• Small tables or log tables • Read-only
RO slave by our Content
• Short Transactions Management System
AMG • Need fast full text queries
STATIONS PUBLIC
MyISAM (replacing Oracle Text)InnoDB InnoDB
• Large tables
• Isolation
Updatedbyfrom
• Updated bya our
main
Content
website
quarterly Management System
script
• Transaction
Read-onlyfrom
• Read-only byOriented
our
ourwebservers
website
• Horizontally
Resource
• Some Contention
log type scalable
information written Scripts Backup
• Highly
• Low Normalized
resource contention RO slave
• No transactions
 Issues When Converting SQL
 MySQL is case sensitive
 Oracle outer join syntax (+) -> OUTER JOIN clause
 Oracle returns a zero to indicate zero rows updated –
MySQL returns TRUE (1) to indicate it successfully
updated 0 rows
 MySQL sorts null to the top, Oracle sorts null to the
bottom
Use “order by – colName desc” for sorting asc with nulls at
bottom
 MySQL has Limit clause – YAY!
 Replacing Oracle Sequences
 Initialize a table with a single row:
CREATE TABLE our_seq (
id INT NOT NULL
);
INSERT INTO our_seq (id) VALUES (120000000);

 Do the following to get the next number in the “sequence”:

UPDATE our_seq SET id=LAST_INSERT_ID(id+1);

SELECT LAST_INSERT_ID();
 Replacing Oracle Sequences
 For updating many rows at once, get the total number of unique IDs you need first:
SELECT @totalRows := COUNT(*) FROM...

 Then update npr_seq by that many rows:

UPDATE npr_seq SET id=LAST_INSERT_ID(id+@totalRows);

 and store that ID into another variable:

SELECT @lastSeqId := LAST_INSERT_ID();

 Then use the whole rownum workaround described above to get a unique value for
each row:
INSERT INTO my_table (my_primary_id . . . ) SELECT
@lastSeqId - (@rownum:=@rownum+1), . . . FROM (SELECT
@rownum:=-1) r, . . .
 Converting Functions
 NVL() -> IFNULL() or COALESCE()
 DECODE() -> CASE() or IF()
 Concatenating strings || -> CONCAT()
 ‘test’ || null returns ‘test’ in Oracle
 CONCAT(‘test’,null) returns null in MySQL
 LTRIM and RTRIM -> TRIM()
 INSTR() works differently.
 Use LOCATE() for Oracle’s INSTR() with occurrences = 1.
 SUBSTRING_INDEX() and REVERSE() might also work.
 Converting Dates
 sysdate -> now()
 Adding or subtracting
 In Oracle “– 1” subtracts a day
 In MySQL “- 1” subtracts a milisecond – must use
“interval”
 TRUNC() -> DATE()
 TO_DATE and TO_CHAR -> STR_TO_DATE and
DATE_FORMAT
 Update Differences
 You can't update a table that is used in the WHERE
clause for the update (usually in an "EXISTS" or a
subselect) in mysql.
UPDATE tableA SET tableA.col1 = NULL
WHERE tableA.col2 IN
(SELECT tableA.col2
FROM tableA A2, tableB
WHERE tableB.col3 = A2.col3 AND
tableB.col4 = 123456);

 You can join tables in an update like this (Much

easier!):
UPDATE tableA
INNER JOIN tableB ON tableB.col3 = tableA.col3
SET tableA.col1 = NULL
WHERE tableB.col4 = 123456;
 RANK() and DENSE_RANK()
 We really found no good MySQL equivalent for these
functions
 We used GROUP_CONCAT() with an ORDER BY
and GROUP BY to get a list in a single column over a
window of data
 Collation
 You can set collation at the server, database, table or
column level.
 Changing the collation at a higher level (say on the
database) won’t change the collation for preexisting
tables or column.
 Backups will use the original collation unless you
specify all the way down to column level.
 Concurrency Issues
 In our first round of concurrency testing, our system
ground to a halt!
 Deadlocks
 Slow Queries
 MySQL configuration
 sync_binlog = 1 // sync to disk, slow but safe
 innodb_flush_log_at_trx_commit = 1 // write each
commit
 transaction_isolation = READ-COMMITTED
 Useful MySQL Tools
 MySQL Enterprise Monitor
http://www.mysql.com/products/enterprise/

 MySQL GUI Tools Bundle:

http://dev.mysql.com/downloads/gui-tools/5.0.html
 MySQL Query Browser similar to Oracle’s SQL
Developer
 MySQL Administrator
 Innotop and innoDB Status
 innotop
http://code.google.com/p/innotop
 Helped us identify deadlocks and slow queries (don’t
forget the slow query log!)

 In mysql, use
show engine innodb status\G;
 Useful for contention and locking issues
 Query Profiling
 Try the Query Profiler with Explain Plan when
debugging slow queries
http://dev.mysql.com/tech-resources/articles/using-new-
query-profiler.html
 Concurrency Solution
 Tuning our SQL and our server configuration helped

 Turns out that the RAID card we were using had no

write cache at all. Fixing that allowed us to go live.
 Encoding Gotcha’s
 Switched from ISO-8859-1 to UTF-8
 Migration Tool
 Issues with characters that actually were not ISO-8859-
1 in our Oracle database
 Lack of documentation for the LUA script produced by
the migration GUI
 Update encoding end to end
 JSPs, scripts (Perl), PHP, tomcat (Java)
 Continuing Issues
 Bugs with innodb locking specific records (as
opposed to gaps before records)
 Uncommitted but timed out transactions
 Use innotop or “show engine innodb status\G; “ and
look for threads waiting for a lock but no locks blocking
them
 Requires MySQL reboot
 Questions?
 Joanne Garlow
 jgarlow@npr.org
 http://www.npr.org/blogs/inside