Know-How Network: SAP BW Data Load Performance Analysis and Tuning

Ron Silberstein
Platinum Consultant Netweaver RIG US – Business Intelligence SAP Labs, LLC

Welcome SAP Developer Network!

http://www.sdn.sap.com

 2003 SAP Labs, LLC,Know-How Netowke , Ron Silberstein2

Agenda
Data Load Overview Extraction Performance PSA, Transfer Rules, Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance

3

 2003 SAP Labs, LLC,Know-How Netowke , Ron Silberstein3

Agenda
Data Load Overview Extraction Performance PSA, Transfer Rules, Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance

4

 2003 SAP Labs, LLC,Know-How Netowke , Ron Silberstein4

Overview: Data Load Process
Goals of performance optimization: First tune the individual single execution and then the whole load processes.
Eliminating unnecessary processes Reducing data volume to be processed Deploying parallelism on all available levels

5

Parallel processes are fully scalable
The typical BW data load process :
ALE Scheduler ALE BW BW S-API S-API

Loading Process

Extractor Extractor

Business Information Warehouse

Source System
ALE ALE BW BW S-API S-API IDOC

InfoCube ODS

Update Rules

Transfer Rules

PSA

tRFC

IDOC

 2003 SAP Labs, LLC,Know-How Netowke , Ron Silberstein5

Ron Silberstein6 . LLC.Know-How Netowke .0 Communication Structure Communication Structure Transfer Rules transfer structure transfer structure Communication Structure Communication Structure Transfer Rules transfer structure transfer structure Source System BW S-API transfer structure transfer structure transfer structure transfer structure Extraction Source Structure Extraction Source Structure BW S-API Extraction Source Structure Extraction Source Structure Header Transaction Data Item DataSource Master Data ATTR TXT DataSource  2003 SAP Labs.SAP Service API: Extraction/Load Mechanism 6 BW InfoCube Update rules BW S-API ODS Master Data Attributes Texts Update rules NEW BW 3.

LLC. Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance 7  2003 SAP Labs.Know-How Netowke . Ron Silberstein7 .Agenda Data Load Overview Extraction Performance PSA. Transfer Rules.

How to identify high Extraction Time ? Determine the extraction time: 8 Extraction S-API ALE ALE Extractor Extractor Source System Loading Process Scheduler Business Information Warehouse Data load Monitor : ALE ALE Transaction RSMO InfoCube ODS Update rules Transfer rules PSA IDOC tRFC IDOC S-API  2003 SAP Labs. LLC.Know-How Netowke . Ron Silberstein8 .

Monitoring Extraction: Resource Utilization Further Analysis in case of Resource problems when extracting data. LLC. Ron Silberstein9 .. 9 Look for Many Long-Running Processes Check SM51 / SM50 in the source system  2003 SAP Labs.Know-How Netowke ..

Extraction Time is too high ? Further Analysis in case of PERFORMANCE problems extracting data. Ron Silberstein10 .. 10  2003 SAP Labs..Know-How Netowke . refer to relevant SAP notes and apply when necessary. LLC. Transaction: RSA3 For specific application areas specific notes exist.

Extraction Time is too high ? Analyze high ABAP Runtime: Particularly Useful for User Exits Use SE30 Trace option “in parallel session“.Know-How Netowke . Select corresponding Work process with extraction job 11  2003 SAP Labs. Ron Silberstein11 . LLC.

ALEREMOTE). LLC.Extraction Time is too high ? 12 Identify expensive SQL Statements Use ST05 Trace with Filter on the extraction user (e.Know-How Netowke .  2003 SAP Labs. Make sure that no concurrent extracting jobs run at the same time with this execution.g. Ron Silberstein12 .

XML Data loads: HTTP/HTTPS processes can be allocated to specific server groups Expected Results: Avoid CPU/Memory bottlenecks on one server Greater Throughput: Faster time to completion per request  2003 SAP Labs. event chains and Process Chains: all can be processed on specified server groups.Extraction Tuning: Load Balancing Parallel processes: distribute to different servers avoid bottlenecks on one server Config in table ROIDOCPRMS 13 RFC destinations (SM59) Example: RFC connection from BW to R/3 and R/3 to BW InfoPackages. Ron Silberstein13 .Know-How Netowke . LLC.

Know-How Netowke . reduce DataPackage size In larger systems. Ron Silberstein14 . Package size specified in table ROIDOCPRMS and/or InfoPackages 14 Scenario: Set up the parameters according to the recommendations. but take care not to impact communication process and unnecessarily hold work processes in SAP source system. Expected results: In a resource constrained systems. the contents and structure of records in the extracted datasets. LLC. Greater throughput = Faster time to completion per request  2003 SAP Labs. if upload performance is not improved. increasing the package size to speed collection.Extraction Tuning: Configuring DataPackage Size Size of DataPackages: Influencing Factors Specific to application datasource. SAP OSS note 417307: Extractor Packet Size Collective Note for SAP Applications Consider both the source system and the BW system (RSADMINC). Package size: impacts frequency of COMMITs in DB . try to find other values that fit exactly your requirements.

. OSS note 409641 for details Double Check ROIDOCPRMS Settings If no entry was maintained. the data is transferred with a standard setting of 10000 kbyte and 1 “Info IDOC” for each data packet.Know-How Netowke .. Ron Silberstein15 .  2003 SAP Labs. LLC.Tuning Extraction: Configuration (Slide 1) 15 Further Analysis in case of Resource problems when extracting data.

16 Consider Decreased Data Package Size for an specific InfoPackage ? ?  2003 SAP Labs... Ron Silberstein16 .Tuning Extraction: Configuration (Slide 2) Further Analysis in case of Resource problems when extracting data. LLC.Know-How Netowke .

Know-How Netowke . Ron Silberstein17 . 17 Use Selection Criteria ? ? Consider building indices on DataSource Tables based on selection criteria  2003 SAP Labs.Tuning Extraction: Configuration (Slide 3) Further Analysis in case of Resource problems when extracting data... LLC.

Know-How Netowke . Ron Silberstein18 . LLC.Extractor Tuning: SAP and Generic DataSources SAP Content extraction Convert old LIS extractors to Logistics Extraction Cockpit V3 Collection jobs for different DataSources can be executed in parallel Tune customer exit coding 18 Generic extractors: Collector jobs can be executed in parallel InfoPackages executed in parallel to extract data Not possible for delta extracts from one generic data source Investigate Secondary indexes on fields used for selection Optimize custom ‘collector’ ABAP coding  2003 SAP Labs.

Know-How Netowke .Extraction / Load Tuning: Flat Files 19 Use a predefined record length (ASCII file) File should reside on the application server not on the client PC Avoid large loads across a network. LLC. Avoid reading load files from tape (copy to disk first) Avoid placing input load files on high I/O disks Example: same disk drives or controllers as the DB tables being loaded.  2003 SAP Labs. Ron Silberstein19 .

LLC. Ron Silberstein20 .Agenda Data Load Overview Extraction Performance PSA.Know-How Netowke . Transfer Rules. Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance 20  2003 SAP Labs.

Ron Silberstein21 . LLC.Know-How Netowke .Analyze high PSA Upload Times S-API ALE ALE 21 Loading Process Scheduler Business Information Warehouse Extractor Extractor Source System ALE ALE IDOC S-API InfoCube ODS Update rules Transfer PSA rules IDOC tRFC Load Into PSA Data load Monitor : Transaction RSMO  2003 SAP Labs.

it probably will not work as expected. value in number of records Transaction SPRO or RSCUSTV6 : From SPRO Business Information Warehouse > Links to Other Systems > Maintain Control Parameters for the Data Transfer Note: If you start more than one load process at a time expecting to have each request in a separate partition. Ron Silberstein22 . the PSA threshold is not yet reached when the second process starts writing into PSA  2003 SAP Labs.PSA Partitioning PSA Data Load 22 PSA Table Partitions Size of each PSA partition.Know-How Netowke . LLC.

Ron Silberstein23 .Know-How Netowke .Data Processing-Transfer Rules S-API ALE ALE 23 Loading Process Scheduler Business Information Warehouse Extractor Extractor Source System ALE ALE IDOC S-API InfoCube ODS Update rules Transfer PSA rules IDOC tRFC Data load Monitor : Transaction RSMO Transfer Rules  2003 SAP Labs. LLC.

LLC.Know-How Netowke .Data Processing-Update Rules S-API ALE ALE 24 Loading Process Scheduler Business Information Warehouse Extractor Extractor Source System ALE ALE IDOC S-API InfoCube ODS Update rules Transfer PSA rules IDOC tRFC Update Rules Data load Monitor : Transaction RSMO  2003 SAP Labs. Ron Silberstein24 .

LLC.Know-How Netowke .x) Debugging and tuning Update and Transfer rules: Simple tool for debugging of transfer or update rules Improves error search and analysis – together with the enhanced error messages 25  2003 SAP Labs. Ron Silberstein25 .Tuning Transfer and Update Rules (BW 3.

0  2003 SAP Labs. LLC.Routines: Potential Performance Bottlenecks Identify the expensive update/transfer rules rules: Debug one update/transfer rule to the next update rule for each InfoObject.Know-How Netowke . 26 Also use ST05 or SM30 Recommendations: SINGLE SELECTs are one of the performance “killers” within these codings. as they are interpreted at runtime (not compiled like routines) The transformation engine or library is new in BW 3. Avoid too many library transformations. Ron Silberstein26 . use buffers (such as internal tables) and array operations instead.

Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance 27  2003 SAP Labs. Transfer Rules.Know-How Netowke .Agenda Data Load Overview Extraction Performance PSA. Ron Silberstein27 . LLC.

Ron Silberstein28 .Know-How Netowke .Data Load: Data Targets S-API ALE ALE 28 Loading Process Scheduler Business Information Warehouse Extractor Extractor Source System ALE ALE IDOC S-API InfoCube ODS Update rules Transfer PSA rules IDOC tRFC Load Into Data Targets Data Load Monitor .RSMO  2003 SAP Labs. LLC.

Expected Results: Accelerates data load performance per load request.Initial and Large Data load volume tuning Buffering Number Range (InfoCube): Activate The number range buffer for the dimension ID’s Reduces application server access to Database. reset the number ranges buffer to its original state: minimize unnecessary memory allocation. Note: After the load. Ron Silberstein29 . set the number range buffer for one dimension to 500.Know-How Netowke .  2003 SAP Labs. the system will keep 500 sequential numbers in memory SAP OSS note 130253: Notes on upload of transaction data into BW 29 Scenario: High volumes of transaction data: significant DB access (NRIV table) to fulfill number range requests.g. Ie. LLC.

LLC.  2003 SAP Labs. Ron Silberstein30 .Transactional Data Load Performance Tuning Load Master data before transaction data Creates all SIDs and populates the master data tables (attributes and/or texts). and need not be determined during transactional data load (large overhead).Know-How Netowke . SAP OSS note 130253: Notes on upload of transaction data into BW 30 Scenario: Always load master data before transaction data (ODS and InfoCube). When completely replacing existing data. delete before load! Expected Results: Accelerates transaction data load performance: all master data SIDs are created prior to transaction load.

.Know-How Netowke . delete before load! Expected Results: Data deleted from PSA can reduce PSA read times Data deleted from InfoCube reduces deletion and compression time.. “Drop partition.“ DDL statement instead of “delete from table.. LLC. Ron Silberstein31 .Transactional Data Load Performance “Snapshot” Reporting: Data Deletion Some reporting scenarios require no historical data 31 Scenario: When completely replacing existing data.“ DML statement only takes seconds Deleting Data also speeds data availability (aggregates. etc)  2003 SAP Labs..

1. active data change log Activation Activation New/modified data Activation queue Req1 Req2 Req1 Sequential load Req2 Req3 Req 3. LLC. 32 Doc-No.x Data Packets / Requests can not be loaded into an ODS object in parallel overwriting functionality Locking on Activation table Doc-No.2 Req3 Staging Engine Staging Engine  2003 SAP Labs. Ron Silberstein32 .Know-How Netowke .ODS Activation in BW 2.

No Activation Activation Doc-No. LLC.ID I Rec.Know-How Netowke . New/modified data Req1 Req2 Req3 Staging Engine Staging Engine Activation queue Req1 Req2 Req3 Req. active data change log Req.ID I Pack.ID I Rec.x 33 New queuing mechanism replacing previous Maintenance (M)table Doc-No. Ron Silberstein33 .ODS Activation in BW 3.No Parallel load  2003 SAP Labs.ID I Pack.

ID I Rec.1I4711I-10 ODSRy I P 1 I Rec. Req1 Before.No I Value 4711 I 10 4711 I 30 34 Change log Req.0) Active data Doc. After update.Know-How Netowke .ID I Pack.2I4711I+30 Upload to Activation queue Data from different requests are uploaded in parallel to the activation queue Activation Activation queue REQU1 I P 1 I Rec.and After Image Request ID in activation queue and change log differ from each other. Ron Silberstein34 Req2 Req3 Staging Engine Staging Engine . This guarantees the correct sequence of the records and allows inserts instead of table locks.1I4711I 30 Activation During activation the data is sorted by the logical key of active data plus change log key.1I4711I 10 REQU2 I P 1 I Rec. the data in the activation queue is deleted. LLC.  2003 SAP Labs.1I4711I 10 ODSRy I P 1 I Rec.ODS Activation example (BW 3.No ODSRx I P 1 I Rec.

for ODS Activation Server Group for RFC Call when Activating Data in ODS Controls data packet size utilized during parallel update/activation and number and allocation of work processes. of parallel Dialog work processes Min. Wait Time in secs. of recs per package Max.Know-How Netowke .  2003 SAP Labs.Control of ODS Data load Packaging and Activation 35 Transaction RSCUSTA2 Max. LLC. Ron Silberstein35 . no. no.

Ron Silberstein36 .ODS Load/Activation Tuning Tips Non-Reporting ODS Objects: BEx Flag: Computation of SIDs for the ODS can be switched off 36 BEx-flag must be switched on if BEx-reporting on the ODS is executed Loads are faster as Master Data SID tables do not have to be read and linked to the ODS data  2003 SAP Labs.Know-How Netowke . LLC.

only inserts Note: Data owner is responsible for uniqueness! Index maintenance Indexes speed querying Slow down activation Parallel SID creation SIDs are created per package Multiple packages are handled in parallel by separate dialog processes  2003 SAP Labs.x)37 Update of ODS object with unique records Significantly simplifies activation process No lookup of existing key values No updates in active table. LLC. Ron Silberstein37 .Further ODS Loading/Activation enhancements (3.Know-How Netowke .

Know-How Netowke . LLC.Using InfoCube Data Load Performance Tools Admin WB > Modeling > InfoCube Manage > Performance Tab 38 Recommendation: Drop secondary Indexes for large InfoCube data loads Create Index button: set automatic index drop / rebuild Statistics Structure button: set automatic DB statistics run after a data load  2003 SAP Labs. Ron Silberstein38 .

Agenda Data Load Overview Extraction Performance PSA. LLC. Ron Silberstein39 . Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance 39  2003 SAP Labs. Transfer Rules.Know-How Netowke .

Know-How Netowke .Parallel Data Load System-controlled parallelism Parallel upload by packaging the source data Packets are created during the extraction and sent simultaneously to BW Packet size for flat files definable in table RSADMINC (IDOCPACKSIZE) Packet size for mySAP source systems definable in table ROIDOCPRMS (MAXLINES) 40 Administrator controlled parallel data load InfoPackages Loading from the same or different data source(s) with different selection criteria simultaneously Enables parallel PSA posted in parallel Data Target process as PSA partitions can be Initial fill of aggregates Use aggregate hierarchy to schedule parallel filling jobs  2003 SAP Labs. LLC. Ron Silberstein40 .

LLC. packets are generated and sent simultaneously Packets are updated to data targets in parallel Extraction Staging Engine ODS Object Data Packets  2003 SAP Labs. Ron Silberstein41 .Packaging of Data Load 41 During extraction.Know-How Netowke .

User-controlled Parallelism / Splitting 42 Multiple processes can be started in parallel manually.Know-How Netowke . LLC. for example: InfoPackages (loading from multiple data sources or from the same data source with different selection criteria simultaneously) Initial filling of multiple aggregates 1 2 3  2003 SAP Labs. Ron Silberstein42 .

and their influence on parallelization and usage of dialog or batch process for loading from PSA to data targets n DIA 43 Option 1 Option 4 1 BTC n DIA Option 2 1 BTC n DIA Option 3 Option 5  2003 SAP Labs. Ron Silberstein43 .. LLC.Know-How Netowke .Processing Options in the InfoPackage ..

X Process Chains Replacement for Event chains 44 Transaction RSPC Process type : Delete Index Generate Index Auto suggestion depending on InfoPackage selected.Know-How Netowke . Ron Silberstein44 .Using InfoCube Data Load Performance Tools New in BW 3. Process Chain TechEd session: BW 210 BW Automation with Process Chains  2003 SAP Labs. LLC.

Know-How Netowke .Process Chains Parallel Data Load 45  2003 SAP Labs. Ron Silberstein45 . LLC.

LLC. Transfer Rules.Know-How Netowke . Ron Silberstein46 . Update Rules Loading InfoCubes and ODS Objects Parallel Data Load Aggregate Maintenance 46  2003 SAP Labs.Agenda Data Load Overview Extraction Performance PSA.

Ron Silberstein47 .Know-How Netowke . hence. to improve the roll-up performance Aggregate hierarchy is determined automatically General Guideline: define few basis (large) aggregates and many small aggregates that can be built from the hierarchy level before Display aggregate hierarchy  2003 SAP Labs. LLC.e.Aggregate Roll-Up Roll-up The roll-up process populates all aggregates of an InfoCube with the newly loaded delta load Basic Rule: InfoCube and all depending aggregates must be in-sync. i. you must see the same data no matter if it has been derived from the InfoCube or the aggregate Newly loaded data is not available for reporting until it has been rolled up into the aggregates 47 Aggregate Hierarchy Aggregates can be built out of other aggregates to reduce the amount of data to be read and.

Ron Silberstein48 . Material Group. Day} Many small aggregates {Material Group. Day} Few large basis aggregates {Material Group.Aggregate Hierarchy Example: Optimized Aggregate Hierarchy Example 48 Basic InfoCube {Material. LLC.Know-How Netowke . Customer. Month}  2003 SAP Labs. Customer.

i. LLC.Change Run Change Run Aggregates can contain Dimension Characteristics Navigational Attributes Hierarchy Levels 49 When master data changes. see SAP note 534630 for more details Check aggregate hierarchy (see Roll-Up for more details) Try to build basis aggregates that are not affected by the change run. Ron Silberstein49 . the changes of the navigational attributes/hierarchies must be applied to the depending aggregates.e.Know-How Netowke . this process is called change run Newly loaded master data is not active until the change run has been applied the changes to all aggregates Threshold for delta and new build-up in customizing The Change Run can be parallelized across InfoCubes. no navigational attributes nor hierarchy levels The following slides show details on the process itself …  2003 SAP Labs.

Inc. used under license for technology invented and implemented by Netscape. IBM®. DHTML. MVS/ESA. JAVA® is a registered trademark of Sun Microsystems. LLC. the Citrix logo. All other product and service names mentioned are trademarks of their respective companies. JAVASCRIPT® is a registered trademark of Sun Microsystems. AIX®. MarketSet and Enterprise Buyer are jointly owned trademarks of SAP AG and Commerce One. Program Neighborhood®. All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. mySAP. z/OS. VideoFrame®. EXCEL®. iSeries. DB2 Universal Database. AS/400®. ORACLE® is a registered trademark of ORACLE Corporation.. Netfinity®. S/390®. WinFrame®. XML. WINDOWS®. The information contained herein may be changed without prior notice. mySAP. PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation. and Motif® are registered trademarks of the Open Group. Massachusetts Institute of Technology. OS/400®. Inc. World Wide Web Consortium. OS/390®. WebSphere®. AFP. Tivoli®. Citrix®.Know-How Netowke . SAP Logo.  2003 SAP Labs. SAP. Word®. HTML. DB2®. Informix and Informix® Dynamic ServerTM are trademarks of IBM Corporation in USA and/or other countries. XHTML are trademarks or registered trademarks of W3C®. R/3.com and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. UNIX®. zSeries. Inc.Copyright 2003 SAP AG. Intelligent Miner. xSeries. MultiWin® and other Citrix product names referenced herein are trademarks of Citrix Systems. ICA®. Ron Silberstein50 . Parallel Sysplex®. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. X/Open®. OS/2®. Microsoft®. OSF/1®. R/2. NT®. MetaFrame®. pSeries.

Sign up to vote on this title
UsefulNot useful