DB-20: Highly Parallel Dump & Load

Tom Bascom VP Technology White Star Software tom@wss.com

Agenda
   

Why Dump and Load? What Methods are Available? Using a Highly Parallel Approach Demo!

DB-20: Highly Parallel Dump and Load - 2

Why Dump and Load?

  

Because it is X months since the last time. Because the vendor said so. Because we must in order to upgrade Progress. Because it will improve performance.

DB-20: Highly Parallel Dump and Load - 3

Why Dump and Load?

  

Because it is X months since the last time. Because the vendor said so. Because we must in order to upgrade Progress. Because it will improve performance.

DB-20: Highly Parallel Dump and Load - 4

Good Reasons to D&L

    

Improve sequential data access (reports). Consolidate fragmented records. To recover corrupted data. In order to move to a new platform. In order to reconfigure a storage management decision. In order to take advantage of hot new db features!

DB-20: Highly Parallel Dump and Load - 5

Hot Features That Need D&L

   

Variable Block Sizes Storage Areas Variable Rows Per Block Data Clusters (Type II Areas)

DB-20: Highly Parallel Dump and Load - 6

Some Dump & Load Choices

   

Classic Dictionary Dump & Load Bulk Loader Binary Dump & Load Vendor Supplied Utilities Parallelization
Many Benefits Very difficult to effectively balance manually.

Automating the process

DB-20: Highly Parallel Dump and Load - 7

Classic Dictionary D&L

1st the dump
Cust Order-line order hist

and then the load.

cust Order-line order

DB-20: Highly Parallel Dump and Load - 8

Classic Dictionary D&L

This can take DAYS (or even WEEKS!)

DB-20: Highly Parallel Dump and Load - 9

Multi-threaded D&L
1st the dump
cust order hist

Order-line

and then the load.

cust Order-line order

DB-20: Highly Parallel Dump and Load - 10

Multi-threaded D&L
 

Much more reasonable than classic. Generally can be done in a weekend. But still too long for many situations:
Threads arent easily balanced. Longest thread determines total time. Timing varies it is hard to predict.

DB-20: Highly Parallel Dump and Load - 11

Overlapping, Parallel, Balanced D&L

Both the dump
cust order hist cust Order-line order

Order-line

and the load in parallel, multi-threaded and load-balanced.

DB-20: Highly Parallel Dump and Load - 12

Parallel Balanced D&L

 

Usually a big improvement. Often can be done in hours. But difficult to configure and manage:
You need an intelligent and dynamic control program.

DB-20: Highly Parallel Dump and Load - 13

Highly Parallel D&L

DB-20: Highly Parallel Dump and Load - 14

Demo !!!
DB-20: Highly Parallel Dump and Load - 15

Resources


SUSE Linux 9 Server 2 x 2ghz (Xeon w/HT) 512 MB RAM 3 internal disks Various External Disks 5GB Demo Database 85 Tables 250 Indexes Various table and record sizes

 

DB-20: Highly Parallel Dump and Load - 16

The Storage Area Design

 

Separate Tables from Indexes. Use a single Type 1 Area for tiny miscellaneous tables that have low activity. Isolate Small, but very active, tables in Type 2 Areas (or standalone areas if v9). Organize Large Tables by RPB in Type 2 Areas
Any table over 1 million rows Any table over 1GB

Isolate very large indexes

DB-20: Highly Parallel Dump and Load - 17

The Storage Area Design

b exch06.b1 f 384000 # d "Schema Area":6,64 . # d "Misc":10,256 exch06_10.d1 f 768 # d "SmallTables":12,128;8 exch06_12.d1 f 9120 # d "RPB256":14,256;512 exch06_14.d1 f 221280 # d "RPB128":16,128;512 exch06_16.d1 f 768000 # d "RPB64":18,64;512 exch06_18.d1 f 1536000 d "RPB64":18,64;512 exch06_18.d2 f 1536000 # d "RPB32":20,32;512 exch06_20.d1 f 2048000 d "RPB32":20,32;512 exch06_20.d2 f 2048000 # d "RPB16":22,16;512 exch06_22.d1 f 768000 # d "Indexes":99,8 exch06_99.d1 f 1024000
DB-20: Highly Parallel Dump and Load - 18

Highly Parallel Dump & Load

   

90 threads, 16 concurrent. Dump threads are the same as load threads. Easy monitoring and control. No latency the load finishes when the dump finishes. No intermediate disk IO:
More efficient usage of disk resources. Higher overall throughput.

 

No index rebuild. No need for table analysis.

DB-20: Highly Parallel Dump and Load - 19

The 80/20 Rule

  

98% of the tables are done very quickly 2% of the tables take most of the time and nearly all of the space These are the tables that we target with multiple threads!

DB-20: Highly Parallel Dump and Load - 20

Multiple Threads
Table t Criteria

document document document document document document document LOGS LOGS LOGS LOGS LOGS LOGS LOGS LOGS LOGS LOGS

1 2 3 4 5 6 7 1 6 7 8 9 10 14 15 16 17

where where where where where where where where where where where where where where where where where

src.document.app# src.document.app# src.document.app# src.document.app# src.document.app# src.document.app# src.document.app# type type type type type type type type type type = = = = = = = = = = "P" "P" "P" "P" "P" "P" "L" "L" "M" "U" and and and and and and and and ( ( ( ( ( ( ( (

< >= >= >= >= >= >=

1100000 1100000 1200000 1300000 1400000 1500000 1600000 <= >= >= >= >= >= <= >

and and and and and

src.document.app# src.document.app# src.document.app# src.document.app# src.document.app#

< < < < <

1200000 1300000 1400000 1500000 1600000

logdate logdate logdate logdate logdate logdate logdate logdate

12/31/2003 01/01/2004 07/01/2004 01/01/2005 07/01/2005 01/01/2006 08/01/2002 08/01/2002

) and and and and ) ) )

logdate logdate logdate logdate

<= <= <= <=

06/30/2004 12/31/2004 06/30/2005 12/31/2005

) ) ) )

DB-20: Highly Parallel Dump and Load - 21

Dump Factors


Major factors that contribute to dump time:

Data distribution (scatter factor) Index Selection -B, -Bp, -RO Storage Area Configuration (dedicated areas and type 2 areas can be very, very fast) Disk Configuration and Throughput

proutil dbname C tabanalys (when doing a binary d&l)

DB-20: Highly Parallel Dump and Load - 22

Concurrency -- Dumping
Dump Time
1,800 Time (seconds) 1,500 1,200 900 600 300 0
5 10 15 20 30 40 50 90

Threads

DB-20: Highly Parallel Dump and Load - 23

Take a Load Off



Factors that contribute to load time:

Progress version Build indexes -i, -bibufs, -bigrow, -B, -TB, -TM, -SG Storage Area Configuration Disk Configuration and Throughput -T files -noautoreslist (aka forward-only)

The importance of testing

DB-20: Highly Parallel Dump and Load - 24

Concurrency -- Loading
Dump & Load Time
3,600 3,300 3,000 2,700 2,400 2,100 1,800 1,500 1,200 900 600 300 0
5 10 15 20 30 40 50

Time (seconds)

Threads
DB-20: Highly Parallel Dump and Load - 25

90

Buffer-Copy & Raw-Transfer

  

Very Fast Eliminates The Middle Man (temp file IO operations) Provides fine-grained, ABL level of control
Allows on the fly data manipulation Useful when merging databases

Can use with remote connections to bridge version numbers. In OE10.1 performance is essentially equal Cannot use RAW-TRANSFER with -RO

 

DB-20: Highly Parallel Dump and Load - 26

The Code
    

start.p dlctl.p dlclear.p dlwrap.p dlx.p

DB-20: Highly Parallel Dump and Load - 27

start.p
/* start.p * */ run ./dotp/dlclear.p. pause 1 no-message. run ./dotp/dlctl.p. pause 1 no-message. run ./dotp/dlmon.p. return.

DB-20: Highly Parallel Dump and Load - 28

dlclear.p
/* dlclear.p * */ define new global shared variable dlstart as character no-undo format "x(20)". dlstart = "". for each dlctl exclusive-lock: assign dlctl.dumped = 0 dlctl.loaded = 0 dlctl.err_count = 0 dlctl.note = "" dlctl.xstatus = "" . end. return.
DB-20: Highly Parallel Dump and Load - 29

dlctl.p
/* dlctl.p * */ for each dlctl no-lock where dlctl.active = yes by dlctl.expected descending: os-command silent value( "mbpro -pf dl.pf -p ./dotp/dlwrap.p -param " + '"' + string( dlctl.thread ) + "|" + dlctl.tblname + "|" + dlctl.criteria +"|" + dlctl.pname + '"' + " >> /tmp/dl/error.log 2>&1 ~&" ). end. return.
DB-20: Highly Parallel Dump and Load - 30

dlwrap.p
/* dlwrap.p * */ define define define define thr tbl cri pnm = = = = variable variable variable variable 1, 2, 3, 4, thr tbl cri pnm as as as as character character character character no-undo. no-undo. no-undo. no-undo. "|" "|" "|" "|" ). ). ). ).

entry( entry( entry( entry(

session:parameter, session:parameter, session:parameter, session:parameter,

if pnm = "" then pnm = "dlx". pnm = "./dotp/" + pnm + ".p". run value( pnm ) value( thr ) value( tbl ) value( cri ). return.
DB-20: Highly Parallel Dump and Load - 31

dlx.p - part 1
/* {1} = thread number * {2} = table name * {3} = WHERE clause */ define variable flgname as char no-undo initial /tmp/dl/{2}.{1}.flg". define variable logname as char no-undo initial "/tmp/dl/{2}.{1}.log". define stream unloaded. output stream unloaded to value( "/tmp/dl/{2}.{1}.d" ). output to value( logname ) unbuffered. put today " " string( time, "hh:mm:ss" ) " {2} {1}" skip. disable triggers for dump of src.{2}. disable triggers for load of dst.{2}.

define query q for src.{2}. open query q for each src.{2} no-lock {3}.

DB-20: Highly Parallel Dump and Load - 32

dlx.p - part 2
outer_loop: do for dlctl, dst.{2} while true transaction: inner_loop: do while true: get next q no-lock. if not available src.{2} then leave outer_loop. /* end of table */ create dst.{2}. buffer-copy src.{2} to dst.{2} no-error. d = d + 1. if error-status:num-messages = 0 then l = l + 1. else do: delete dst.{2}. e = e + 1. export stream unloaded src.{2}. find dl.dlctl exclusive-lock where dlctl.tblname = "{2}" and dlctl.thread = {1}. dlctl.err_count = e. next inner_loop. end.
DB-20: Highly Parallel Dump and Load - 33

dlx.p - part 3
if d modulo 100 = 0 then do: find dl.dlctl exclusive-lock where dlctl.tblname = "{2}" and dlctl.thread = {1}. assign dlctl.dumped = d dlctl.loaded = l . file-info:file-name = flgname. if file-info:full-pathname = ? then do: dlctl.note = "stopped". leave inner_loop. end. leave inner_loop. end. end. /* inner_loop */ end. /* outer_loop */

DB-20: Highly Parallel Dump and Load - 34

dlx.p - part 4
do for dl.dlctl transaction: find dl.dlctl exclusive-lock where dlctl.tblname = "{2}" and dlctl.thread = {1}. assign dlctl.dumped = d dlctl.loaded = l dlctl.err_count = e dlctl.xstatus = "done" dlctl.note = ( if dlctl.note <> "stopped" then "complete" else "stopped" ) . if dlctl.note <> "stopped" then do: if d < dlctl.expected then dlctl.note = "short". if d > dlctl.expected then dlctl.note = "long". end. if e > 0 then dlctl.note = dlctl.note + "+errors". end.
DB-20: Highly Parallel Dump and Load - 35

Validation


Belt & Braces!!!

Compare Record Counts


dlcompare.p (for binary d&l)

grep i f error.list

Check Logs for Known & Unknown Errors



fail, error, (1124)

grep 0 errors /tmp/dl/*.log | wc l grep dumped /tmp/dl/*.log grep loaded /tmp/dl/*.log ls l /tmp/dl/*.d

Check for Success

   

DB-20: Highly Parallel Dump and Load - 36

Post Load Monitoring



Watch for very rapid growth:

May be a rows-per-block assignment error (i.e. data areas with RPB = 1). May be a cluster-size error (i.e. index areas with 512 blocks per cluster). May be an unannounced application change.

It is not unusual for there to be somewhat elevated IO for a short period of time (perhaps a week or two).

DB-20: Highly Parallel Dump and Load - 37

Tip Sheet!
 

Dont rely on announcements to keep users out of your system! Shut down unnecessary services.
Dont forget cron jobs!

   

Disable remote access. Rename the database, the directory, the server Take the server off the network. Always perform NON-DESTRUCTIVE Dump & Load!

DB-20: Highly Parallel Dump and Load - 38

Recap


Excellent Reasons to Dump & Load:

Move to a new platform! Reconfigure a storage configuration!! Take advantage of a hot new db features!!! Especially Type 2 Areas!!!!

 

How To Get There Safely and Quickly!

DB-20: Highly Parallel Dump and Load - 39

Questions
http://www.greenfieldtech.com/downloads.shtml

DB-20: Highly Parallel Dump and Load - 40