FBI Full Refresh @ Restore

POC Findings
Table of Contents

Full Refresh - As is state

Options for improvement
POC plan for Option2
POC findings
DAC Migration: As is
DAC Migration: To be
HDE: As is
HDE: Thru DAC

2
As is state

OBIU1
Source Test Environment Run OOTB Full Load
(SDUAT) is restored with
Production Data around OBID1
every two months Run OOTB Full Load

OBIT1
Run OOTB Full Load

•As the data volume has increased from December 09 go live (Ex: WAXF
currently has around 300m Vs around 80m during go live). During Aug’10 the
U1 Full refresh took around 5 day (120 hrs). For D1 it took 7days (188 hrs).

•Instance non availability for long periods of time as per above is hampering
development and delivery efforts around every two months.

3
Options for improvement

Option 1 : Logical Database Backup using expdp(Data

Pump).
Pros: We will have flexibility to copy partial set of tables or single schema. No need to touch DAC_861 and
INFM_861.
Cons: Just database backup and restore might take 16 hours or more.

Option 2 : Physical Cold Backup Cloning

Pros: It might take less time. Database and restore will take around 6 hours. No dependency to start POC.
Cons: There are some known and unknown issues ex. Sequence Generator, Last extract date. Need to backup
and restore DAC_861 and INFM_861

Option 3 : Create one more database environment

Pros and cons: Difficult to define at this point as this requires further discussion whether we can create new
environments. We cannot start POC as we need new environments.

Option 4 : Create two more database environments ex.

U2 and D2.
Pros and cons: Difficult to define at this point as this requires further discussion whether we can create new
environments. We cannot start POC as we need new environments .

4
POC plan for Option 2

What is Option 2? Whenever Source Environment (SDUAT) restores

with Production Data…
1)Backup Target environment (T1) DAC & INFA REP.
2)Restore FBI Environments with Production cold backup.
3)Restore Target environment DAC & INFA REP.
4)Apply the Delta DB changes
5)Find out how to handle Sequence#, Refresh dates & unknown issues.
6)Start the incremental & HDE (Historical Data Enhancements).

OBIP1 Target environment

MDW DB +DAC REP + INFA REP

MDW DB
Consists of OBIP1 +
+ DAC REP
(Delta DB = MR +Projects + Tuning Tasks)
+ INFA REP

5
POC Findings

Step 1, 2, 3, took 5hrs - No issues.

For Step 4- We need to refine the current DB migration process. We should maintain
Delta DB changes & Environment specific PROCs properly. So that we can handle this
with minimum time. In D1, each individual should take care their development related DM
changes once D1 is restored.
Step 5-
•Sequence number is maintained at INFA level, so migrated INFA object from P1 to target
environment. Please make sure to mark All the Sequence transformations as reusable at
the time of development.
•When you export DAC logical data and user data only, you don’t mess up target
environment refresh dates.
• When core table has no Last_Update_Date, then OOTB used control table
(W_PROJ_ETL_PS) to track incremental changes based on core table sequence ID
column. So please truncate the W_PROJ_ETL_PS table to extract.

Step6–Turn around time to complete the Full Refresh (including DB delta changes).
POC Metrics:
•In T1 - 1st iteration 24hrs , 2nd iteration 12hrs

•Projected time in D1 - 24hrs (To apply DB Delta changes will take longer than U1
due to more activities going on, & Replace stored procedures specific to D1)
•Projected time In U1 it is 12 Hrs, because no purge required. Delta DB applied as
per CRs.
6
DAC Migration : As is

Currently Developers preparing migration document with an average of 14,

15 pages for MR. DBAs are not migrating DAC changes, they are creating
new/modifying tasks, adding code, synchronizing, building and assembling.
No test run for the changes the DBAs are implementing or migrating.
This is more labor intense or time consuming above all error prone approach.
We are highly dependent on of migration doc and DBAs manual setup.

Current OBID1

MDW DB
+ DAC REP For Dev
+ INFA REP

7
DAC Migration : To Be

One time Effort: Create New instance in OBID1

On going Efforts:
Instead of creating around 16 pages MD. Developers creates tasks again in
DAC migration repository that are actually going for migration. Also we can do
one INCR run with changes to make sure changes made properly.
Exporting importing completes in 30 min. Doesn’t require Assemble or build to
run INCR. Migration efforts reduced from 6hrs to 30 min.

Future OBID1

MDW DB
+ DAC REP For Dev
+DAC Staging REP
+ INFA REP

8
HDE: As is

Typical HDE (Historical Data Enhancement) process .

1) Wait for INCR run to complete after successful migration, sanity check.
2) Run Pre-SQL ,either to drop/recreate the FACT table or modify ETL PROC and Aux
Dates so that the SIL can insert or update the target table.
3) Run FULL SDE tasks manually (thru INFA)
4) If necessary do the gather stats
5) Run SIL Full for Fact or INCR Task to load Dimension table.
6) Run Post SQL to build index or gather stats.

This is manual process, some DB and INFA tasks involved so we need one DBA and
one ETL developer required to perform the HDE efforts.

Issues that we faced:

We didn’t calculate the dependency properly.
(Ex: For CR 2193, Ran W_SUPPLIER_ACCOUNT_D HDE first then W_PARTY_D HDE. It should
be other way around.)
Our current HDE process is longer good for FACT tables.
(Ex: We ran GL_Balance_Fact HDE after one day of our INCR. This is triggered stale dimension .
Even our recent firefix 2258 (merge update ) couldn’t fix the issue properly.

9
HDE: Thru DAC

1) Run Pre-SQL ,either to drop/recreate the FACT table or modify ETL PROC
and Aux Dates so that the SIL can insert or update the target table.
2) Set the Task target table refresh date is null in DAC. Then DAC run
Full tasks along with your INCR run. Tested CR 2193 both PARTY
HDE and Supplier Account HDE using this option with any INFA
changes.

We can’t standardize HDE efforts. Since each HDE effort is different and
depends on situation. However developer should try all HDE efforts thru
DAC first and compare the cost.

10
FBI Full Refresh @ Restore - POC Findings

Thank you!
Q&A?
Next Steps...

11