Professional Documents
Culture Documents
20 Tips and Tricks To Improve Data Load Performance: Jesper Christensen Comerit
20 Tips and Tricks To Improve Data Load Performance: Jesper Christensen Comerit
to Improve Data
Load Performance
Jesper Christensen
COMERIT
2012 Wellesley Information Services. All rights reserved.
In This Session
Source: SAP
Source: help.sap.com
6
Extraction time
DB time to select the data to be extracted
Logic applied during extraction such as joins, lookups, and
filtering
Middleware and network time
The time used to transfer the data from the source system to the
target SAP NetWeaver BW system
Interface types such as Web services and Universal Data (UD)
Connect are good for small amounts of data and cannot handle
large volumes
Fixed format files are larger to transfer but faster to load into SAP
NetWeaver BW
WAN Network time can become a bottleneck during peak hours
9
Transformation Types
13
16
Remember to set
the accuracy to
Low
Run
transaction
code RSA3
Note: SE30 can also be used for transformations by simulating the DTP
run
17
19
Key DB parameters
SAP has recommended some parameter values for SAP
NetWeaver BW that usually improve performance
Expect to evaluate these parameter settings frequently, though,
to ensure that the DB operates optimally
See three key SAP Notes:
830576 Parameter recommendations for Oracle 10g
387946 Use of locally managed tablespaces for
BW systems
1044441 Basis parameterization for NW 7.0 BI systems
20
21
22
23
Code the extractor user exits so that they call a dynamic program
per DataSource
Isolate the code per DataSource in a self-contained program
Minimize risk that a syntax error in code for one DataSource
impacts extraction from all other DataSources
Example
Program name = ZBW + <DataSource name>
Form name = DOZBW + <DataSource name>
This same technique can be used with customer exit variable
code
24
25
26
REPORT YBWZDS_AGR_USER.
****************************************************************
** Form called dynamically must start with DOYBW + <DataSource>
****************************************************************
*
REPORT YBWZDS_AGR_USER.
****************************************************************
** Form called dynamically must start with DOZBW + <DataSource>
****************************************************************
*
FORM DOYBWZDS_AGR_USER
TABLES C_T_DATA STRUCTURE ZOXBWD0001.
FORM DOYBWZDS_AGR_USER
TABLES C_T_DATA STRUCTURE ZOXBWD0001.
ENDFORM.
ENDFORM.
27
28
REPORT YBW2LIS_13_VDITM.
****************************************************************
** Form called dynamically must start with DOYBW + <DataSource>
****************************************************************
*
REPORT YBWZDS_AGR_USER.
****************************************************************
** Form called dynamically must start with DOZBW + <DataSource>
****************************************************************
*
FORM DOYBW2LIS_13_VDITM
TABLES C_T_DATA STRUCTURE ZOXBWD0001.
FORM DOYBWZDS_AGR_USER
TABLES C_T_DATA STRUCTURE ZOXBWD0001.
ENDFORM.
ENDFORM.
29
30
31
32
Program include
FORM startup
TABLES
MONITOR STRUCTURE RSMONITOR "user defined monitoring
MONITOR_RECNO STRUCTURE RSMONITORS
DATA_PACKAGE STRUCTURE DATA_PACKAGE
USING
RECORD_ALL LIKE SY-TABIX
SOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYS
CHANGING ABORT LIKE SY-SUBRC. "set ABORT <> 0 to cancel update
*
*$*$ begin of routine - insert your code only below this line **
****************************************************************
*
* INITIALIZATION (ONE-TIME PER DATA PACKET)
*********************
* TO READ FROM DATABASE (ALL RECORDS FOR DATA PACKAGE)
**********
****************************************************************
*
* FORM READ_USR02_TO_MEMORY_FOR_0BWTC_C02
*--------------------------------------------------------------*
Form READ_USR02_TO_MEMORY_FOR_0BWTC_C02
TABLES
MONITOR
STRUCTURE RSMONITOR
DATA_PACKAGE STRUCTURE /BIC/CS80BWTC_C02
USING
RECORD_ALL
LIKE
SY-TABIX
SOURCE_SYSTEM LIKE
RSUPDSIMULH-LOGSYS
CHANGING ABORT
LIKE
SY-SUBRC.
perform READ_USR02_TO_MEMORY_FOR_0BWTC_C02
TABLES
MONITOR
DATA_PACKAGE
USING
RECORD_ALL
SOURCE_SYSTEM
CHANGING ABORT.
*-
34
Program include
FORM compute_key_field
TABLES
MONITOR STRUCTURE RSMONITOR "user defined monitoring
USING
COMM_STRUCTURE LIKE /BIC/CS0BWTC_C02
RECORD_NO LIKE SY-TABIX
RECORD_ALL LIKE SY-TABIX
SOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYS
CHANGING RESULT LIKE /BI0/V0BWTC_C02T-USERGROUP
RETURNCODE LIKE SY-SUBRC
ABORT LIKE SY-SUBRC. "set ABORT <> 0 to cancel update
*
*$*$ begin of routine - insert your code only below this line*-*
* fill the internal table "MONITOR", to make monitor entries
PERFORM READ_GT_USR02
USING
COMM_STRUCTURE-TCTUSERNM
RECORD_NO
RECORD_ALL
SOURCE_SYSTEM
CHANGING GS_USR02
ABORT.
****************************************************************
** RECORD PROCESSING (RUN PER RECORD)
****************************
* TO READ FROM MEMORY (ONE RECORD)
******************************
****************************************************************
** FORM READ_GT_USR02
*--------------------------------------------------------------*
FORM READ_GT_USR02
USING
TCTUSERNM
LIKE USR02-BNAME
RECORD_NO
LIKE SY-TABIX
RECORD_ALL
LIKE SY-TABIX
SOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYS
CHANGING GS_USR02
ABORT
LIKE SY-SUBRC. "ABORT<>0 cancels update
RESULT = GS_USR02-CLASS.
*if abort is not equal zero, the update process will be canceled
*$*$ end of routine - insert your code only before this line *-*
ENDFORM.
ENDIF.
ENDFORM.
"READ_GT_USR02
35
36
37
What is it?
An architectural design to enable parallel data loading and
query execution
Partitioning criteria: Year, Region, or Actual/Plan
Source: SAP
39
History
(Summarized)
ALL years
Write-Optimized (No SIDs)
Ex: Current
Current
Current
Current
Current
Source: SAP
Year
Year
Year
Year
Year
Current Year 3
Current Year - 2
Current Year - 1
Current Year
Current Year + 1
Current Year 3
Current Year - 2
Current Year - 1
Current Year
Current Year + 1
+ 1 =
=
- 1 =
- 2 =
- 3 =
2010
2009
2008
2007
2006
DataSource
41
42
Switch off SID determination for DSOs that are not used in
reporting
SID determination is required only for report DSOs and take up
40-70% of the activation time
43
Parallel processing is
possible for most steps
in SAP NetWeaver BW
DTP Parallel
Processing
DSO settings
Transaction code
RSODSO_SETTINGS
44
45
The number range tables (NRIV) are called for every new distinct
record that is loaded to SAP NetWeaver BW as either master data
or dimension in an InfoCube
The NRIV table is accessed with a select for update statement,
which can be quite slow
Buffering should be done as follows:
Determine the large number ranges (Document numbers,
Dimensions with documents or many distinct values)
Goto t-code SNRO and set up buffering
46
47
2010
CPU
0.05
253.31
MIPS/$
MIPS/$
Memory
0.02
50.15
MB/$
MB/$
Addressable
Memory
216
264
Network
Speed
Disk
Data Transfer
100
100
Mbps
Gbps
600
MBPS
MBPS
Architectural Drivers
Improvement
5066x
2502x
248x
1000 x
120x
1990
2010
Disk-based data
storage
In-memory data
stores
Simple
consumption of
apps (fat client
UI, EDI)
Multi-channel
UI, high event
volume, cross
industry value
chains
Generalpurpose,
applicationagnostic
database
Applicationaware and
intelligent data
management
48
Source: SAP
49
50
51
53
Resources
Training
BW360 BW Performance and Administration class
54
Use the SAP NetWeaver BW statistics to find data loads that require
optimization target to optimize top 5-10 every month
Use SE30 to analyze ABAP runtime for DataSources and
transformations
Review and implement the recommended database parameters for
SAP NetWeaver BW
Ensure that all SQL statements used in the data loading process are
using indices and that statistics are calculated for the tables
Make sure that the ABAP coding used in extraction exits and
transformation is optimized
Review and optimize the data models to avoid unnecessary
processing
Use parallel processing during data loading and updates
55
Your Turn!
Disclaimer
SAP, R/3, mySAP, mySAP.com, SAP NetWeaver, Duet, PartnerEdge, and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product
and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by
SAP.
57