Data warehousing with MySQL

MySQL

MS-SQL

Oracle

MySQL

DB2

Flat Files

Free and Open Source Software

MySQL is licensed under

GPL.
The GPL is a Free and Open Source
Software (FOSS) license that grants
licensees many rights to the software
under the condition that, if they
choose to share the software, or
software built with GPL-licensed
software, they share it under the
same liberal terms.

Free and Open Source Software

Advantages of Open Source
MySQL has 5 million plus active
installation base.
New releases immediately
downloaded by users providing early
feedback on bugs and features.
Access to source code
Write your own features/proprietary
Storage Engine
Freedom !

Data Warehousing application

Data Warehouse is a relational
database.
It is designed for query and analysis
rather than for transaction
processing.
It enables an organization to
consolidate data from several
resources.

Extraction ,Transformation and Loading

Data
Source

Staging
Tables

MERGE
& BULK
INSERT

MERGE
Tables

Indexes,
Memory

Views,
Summary

SWH

AWH

HEAP

Extract

Load

Transform Storage

Performance

OLTP/ BI

Users

Extraction ,Transformation and

Loading
Staging database
LOAD DATA INFILE . Command.
Merging of SQLs
Segregating Informations
View enhancements
Index Enhancement
Memory Manipulation

Extraction, Transformation and Loading

Staging Area and its benefits
Relational Table structures are
flattened to support extract processes
in Staging Area.
First data is loaded into the temporary
table and then to the main DB tables.
Reduces the required space during ETL.
Data can be distributed to any number
of data marts

Partitioning and Storage Engine

The MERGE Table
A collection of identical
MyISAM tables used as one
You can use SELECT,
DELETE, UPDATE, and
INSERT on the collection of
tables.
Use it when having large
tables
DROP the MERGE table, you
drop only the MERGE spec.
Advantage : manageability
and performance

MERGE SALES
Table

Sales
for
Yr04
Aug04

Oct04
Sep04

Partitioning and Storage Engine

MERGING based on month as Range
JUN2004

JUL2004

JUN2004
OCT2004

AUG2004

SEP2004

OCT2004

Partitioning and Storage Engine

MERGE Table Example
mysql> CREATE TABLE jan04 ( -> a INT
NOT NULL AUTO_INCREMENT PRIMARY
KEY, -> message CHAR(20));
mysql> CREATE TABLE feb04 ( -> a INT
NOT NULL AUTO_INCREMENT PRIMARY
KEY, -> message CHAR(20));
mysql> CREATE TABLE year04 ( -> a
INT NOT NULL AUTO_INCREMENT, ->
message CHAR(20), INDEX(a)) ->
TYPE=MERGE UNION=(jan04,feb04)
INSERT_METHOD=LAST;

Partitioning and Storage Engine

MyISAM Storage Engine

Supports MERGE table.

Support fulltext indexing
INSERT DELAYED ... option very useful
when clients can't wait for the INSERT
to complete. Many client bundled
together and written in one block
Compress MyISAM tables with
myisampack to take up much less
space.
Benefit from higher performance on
SELECT statements

Partitioning and Storage Engine

Restrictions on MERGE tables
You can use only identical MyISAM tables
for a MERGE table.
MERGE tables use more file descriptors. If
10 clients are using a MERGE table that
maps to 10 tables, the server uses (10*10)
+ 10 file descriptors.
Key reads are slower. When you read a key,
the MERGE storage engine needs to issue a
read on all underlying tables to check
which one most closely matches the given
key.

Partitioning and Storage Engine

my.cnf parameters for DWH (example)

key_buffer =
1G

myisam_sort_buffer_size =
sort_buffer =

256M

5M

query_cache_type

query_cache_size

100M

key_buffer is the important one, this tells

mysql how much memory to cap itself

Business Intelligence
Using MySQL database server
Drastically reduce information retrieval by
distributing data into replicated clusters.
This enables parallel processing.
Tighter storage format (3 TB squeezed to
1TB)
Aggregate huge amount of data and deliver
reports for OLAP
Relieve overloaded OLTP databases
Availability, scalability and throughput for
the most demanding applications, and of
course affordability

Summary

Free and Open Source under GPL

MyISAM
Storage Engine
No Transactional Overhead
MERGE Table
Tighter storage format
Highly efficient