Data Aging Strategies in SAP Business Warehouse BW 7.

3
Rainer Uhle, SAP Product Manager Dr. Peter Zimmerer, SAP Development Architect
Mannheim, Rosengarten - June 22, 2011

Disclaimer
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.

© 2011 SAP AG. All rights reserved.

2

You Need Complete and Trusted Information to Make Good Business Decisions

“ “

90% of upper level management feel they don’t have the necessary information for critical business decisions; 50% of them are afraid they are making poor decisions because of it.”

BI strategies are deemed to fail without a trusted data foundation

The #1 risk for building a data mart or data warehouse is data quality

© 2011 SAP AG. All rights reserved.

3

How Good is the Data Behind My Dashboard?

Where did these numbers come from? Are we considering all our relevant sources?

Are these terms consistent with our business definitions?

How current is this data? When was it last updated?

Can I trust this data enough to make my critical decisions? Has the data passed all our business rule checks?
© 2011 SAP AG. All rights reserved. 4

Enterprise Data Warehouse (EDW) Characteristics and Requirements

© 2011 SAP AG. All rights reserved.

5

SAP NetWeaver Business Warehouse Strong EDW capabilities Integrated. All rights reserved. Authorization and Identity Handling  High availability Enable sophisticated lifecycle management at different levels:  System  Meta Data  Lifecycle Management Data (Nearline storage. data base abstraction. archiving) © 2011 SAP AG. data base neutral  Sophisticated Security.  scalable Enterprise Data Warehouse (EDW) platform EDW = DBMS + X Business Content Fast. sustainable implementation through   Modeling Patterns Business Content Openness and data quality through Reliable Data Acquisition   Out-of-the box integration for data originating in SAP systems Integrated with SAP BusinessObjects Data Services (Data Integrator and Data Quality Management) Streamlined Operations Efficient data management through:  Management of data consistency. 6 .

All rights reserved. 7 .What does BW know about my Business? © 2011 SAP AG.

The LSA is an architecture that is applied in the entire BW system. regions and scenarios. All rights reserved. At the center is the service idea of the reference architecture: Each layer provides a service that can be used.Introduction into the term "Layered. Scalable Architecture (LSA)" The Layered. 8 Scalable Architecture © 2011 SAP AG. The LSA is a Reference Architecture and not only a data model. The data model is scalable and can be enhanced for example by other source systems. unified understanding. . Scalable Architecture (LSA) is a standard term for SAP for common. Layered Layer-based data model in which each layer performs a specific task.

integrated and independent data Business Transformation Layer Data Propagation Layer Harmonisation Layer Corporate Memory Application of Business Logic for the applications Operational Data Store LSA Source system close structure. temporary storage 9 . 1:1 mapping. complete storage of history as granular as possible. close to operational reporting Reporting Layer Easily digestible. granular. securing data quality. reusable. All rights reserved. Extractor inbox. “Master the Unknown” Data Acquisition Layer EDW Layer (Single Point of truth.The LSA Reference Architecture layers Reporting BI Applications (Architected Data Mart Layer) Layer optimized for reporting (consists of InfoCubes and MultiProviders) Near real-time reporting. complete history) Harmonization. consumable . plausibility Data Sources © 2011 SAP AG.

10 . All rights reserved.LSA Data Flow Templates as Content © 2011 SAP AG.

948 15.910 14.000 15.500 14.SAP NetWeaver BW adoption Productive SAP NetWeaver BW systems – constant growth  Adoption of SAP NetWeaver BW constantly growing  Unaffected by economic down-turn in 2009  More than 12000 customers referring to more than 15000 productive systems 16.728 13.000 14.000 Q1 09 13. Large installed Base.500 15. Stable Product.500 12.446 Q1 10 © 2011 SAP AG.687 13.359 14.500 13.000 12.238 Q4 10 11 13. Constant Growth Q2 09 Q3 09 Q4 09 Q2 10 Q3 10 .000 14.214 14. All rights reserved.

All rights reserved.Analyst Opinions Forrester 2011 © 2011 SAP AG. 12 .

SAP BW EDW and Reality „60 TB Proof of Concept‟ on RDBMS (IBM/ DB2) Discussions about corporate DWH architectures (EDW) are frequently driven by fears and prejudices. 40... This results in vague questions like: Can BW handle 30. 13 .60TB Proof of Concept © 2011 SAP AG. 100 Terabyte ? The answer: SAP BW .. All rights reserved..

14 SAP NetWeaver BW Accelerator Query & .BW Accelerator Query Run Time Merging and results preparation for BI queries SAP NetWeaver 7.0 Business Intelligence Information BW Analytical Engine Aggregation “on the fly” Response InfoCube Indexing (*) property setting („load index into main memory‟) or schedule program RSDDTREX_INDEX_LOAD_UNLOAD © 2011 SAP AG. All rights reserved.

000 reports / h 4. 15 . All rights reserved.BWA Linear Scalability .1 TB / h 101.6 TB / h 100.2 sec 37 M records 1. Resources (25 TB Showcase 2009) 25 TB 1.000 reports / h 4. # records touched per report 81 blades 135 blades Total DB Size 15 TB 5 TB 27 blades BWA Resources © 2011 SAP AG. report response time avg.2 sec 22 M records 0.5 sec 6 M records Legend: Index creation throughput Multiuser reporting throughput avg.Data Volume vs.2 TB / h 101.000 reports / h 4.

Source:Bill Inmon 16 .Bill Inmon‟s Corporate Information Factory & Nearline Storage DSS Applications Departmental Data Marts Acctg Finance Marketing Sales ERP ERP ERP CRM ETL Staging Area Changed Data eComm. All rights reserved. Int. Mart Bus. ERP Corporate Applications Exploration warehouse/ data mining local ODS Granularity Manager Session Analysis Dialogue Manager Cookie Cognition Preformatted dialogues Cross media Storage Management Near line Storage Archives Web Logs Internet © 2011 SAP AG. EDW Global ODS Oper.

Data-Aging Strategies for Volume Performance Storage Type / Nearline Storage Classic Archive Information Lifecycle according to Importance/Age: Online Database Data Category (read only) (read only) Frequently read / changed data (actual) Infrequently read data (mature) Very rarely read data (aged)       © 2011 SAP AG. All rights reserved. 17 .

Key facts about SAP NLS Data consistency guaranteed before deleting the data from source NLS should be a part of an Information Lifecycle Management (ILM) strategy Based on wellestablished SAP / SAP BW archiving concepts High compression rate (up to 95%) NLS is an application from a third party vendor. running on a separate system Process Chain support Increases retention period for analysis data Data archived in NLS can be incorporated into reporting Supports archiving of InfoCubes and DataStore Objects Mainly timebased archiving. 18 . All rights reserved. yet can also be based on other characteristics Saves storage costs and other system resources Lock of the archived data slice in the original InfoProviders Scheduling and Monitoring of archiving sessions from SAP BW system Copes with changes in the meta data to the BW objects of the archived data Included in the query statistic data collection (RSRT) © 2011 SAP AG.

30  Support for accessing Nearline-Storage data for MultiProviders  Feature to allow archiving from uncompressed InfoCubes  Archiving of Semantic Partioned Objects (SPO) with SP1  Automatic rebuild of BW Accelerator index possible © 2011 SAP AG. 19 .01 (EhP1)  Support of write-optimized DataStore Objects for ADK archiving and the NearlineStorage interface  Request based Archiving  Enhanced status and job monitoring within InfoProvider management view SAP NetWeaver BW 7. All rights reserved.00  Enhanced Look-Up API  Suspension and selective continuation of archiving processes within Process Chains  Restore of an archiving request with all successors  Smaller Data Object size for ADK-based Nearline Solution without semantic grouping SAP NetWeaver BW 7.Evolution by SAP NetWeaver BW Releases SAP NetWeaver BW 7.

SAND Technology DB2 Viper 9. 20 .0 yes yes 7.epx ) © 2011 SAP AG.IBM DataVard OutBoard 1.0 (see also http://www.01 SP6 yes     CBW® – PBS Software Dynamic NearLine Access® . All rights reserved.sap.com/ecosystem/customers/directories/SearchSolution.5® .The Nearline Storage Solution for SAP NetWeaver BW Based on the Nearline Storage Interface Development Partners can implement their Solutions for Archiving and NLS into the SAP BW 3rd Party NLS Solutions      are implemented within the SAP BW ABAP Stack in partner specific namespaces have to pass a certification process can offer specific Application Area in the SAP Support Portal have to be licensed in addition to SAP licenses can have a different release cycle compared to SAP NetWeaver BW NLS Partner Solution Present development partners (in alphabetical order of their products) Certified since SAP BW 7.

BW Archiving and Nearline Storage (based on 895 customer messages) © 2011 SAP AG. All rights reserved. 21 .Customer Adoption .

to file-based solutions. All rights reserved.sap.Data analysis and assistance for ROI analysis  Sizing of Nearline Storage solutions:  Hardware sizing of the NearLine-Storage solution has to be done by the vendor   Different Nearline Storage technologies on the market From database solutions.com/dvm  Deliver a thorough analysis of BW objects distribution  Can help on estimating the data volume that may be archived / transferred to NLS for the largest InfoProviders within the system  Considers only “technical facts” (and not the customer’s “business requirements”) © 2011 SAP AG. 22 . to column-based storage solutions  Data volume services by SAP Active Global Support (AGS)  http://service.

Data Management with Nearline Storage Implementation Aspects 1 2 3 4 Create a Data Archiving Process Create and schedule archiving requests Restore archiving requests Load data to subsequent Data Targets MultiProvider  Look-up during Transformation 5 6  Query Settings 7  MultiProvider Settings Reporting Layer SAP Sales InfoCube (Architected Data Marts) 6 Data Propagation Layer Nearline Storage 7 Nearline Storage 4 DTP Corporate Memory DTP LSA DTP 2 3Nearline Storage DAP DTP 5 Data Acquisition Layer DTP InfoSource 1 PSA InfoPackage DataSource © 2011 SAP AG. 23 . All rights reserved.

very frequently © 2011 SAP AG. All rights reserved. frequently not frequently rarely 24 .Design Aspects – Nearline Storage (NLS) vs. BW Accelerator (BWA) BI InfoMarts (InfoCube) ADK Archive BWA Acceleration Acquisition Archiving Nearline Storage RDBMS Access .

25 . All rights reserved. such as 1. 2. Data resides in the InfoProvider in the database Data resides in a classical Aggregate Data resides in the BW Accelerator Index Data resides in an NLS Partition Aggregate Types • BW Accelerator Index • NLS Partition © 2011 SAP AG.Data Management at Query Runtime The Data Manager identifies the availability of alternative data storage of any kind. 4. 3.

All rights reserved. 26 .NLS Related MultiProvider Settings Nearline read mode • disabled at all • enabled at all • InfoProvider settings © 2011 SAP AG.

MultiProvider: Query Runtime Statistics Listing of Basis Providers and NLS partitions used during Query execution © 2011 SAP AG. 27 . All rights reserved.

NLS Related Query Designer Settings Reporting Fixed NLS Settings • read NLS • do not read NLS • see InfoProvider settings © 2011 SAP AG. 28 . All rights reserved.

All rights reserved. 29 .NLS Related Query Designer Settings: Variable Variable NLS Settings (Dialog) • read NLS • do not read NLS • see InfoProvider settings © 2011 SAP AG.

InfoCube: Archiving of Uncompressed Data Central setting in Data Archiving Process (DAP)  Valid for all archiving requests und DAP-Variants  Can be changed during operation  Prerequisite: only already processed requests (aggregates. 30 . All rights reserved. Delta DTP) Allow Archiving for noncompressed data © 2011 SAP AG.

Data Management at Archiving Runtime During the delete phase of the archiving request the new setup of the BWA index is offered in the dialog. BWA consistence reflected during DAP processing © 2011 SAP AG. 31 . All rights reserved.

Optimized Support for Navigational Attributes Optimized Support for navigational attributes during Query processing on NLS  Navigational attributes are master data attributes that can be used to navigate/filter in queries. © 2011 SAP AG. 100 characteristic values) The attribute selection is replaced by this characteristic selection in the query selection. 32 . All rights reserved. Previous solution: – – –  Selections for navigational attributes were not transferred to NLS as selections … The attribute values were assigned subsequently and filtered in the result set Performance problems for highly selective attribute values  Improvement: – – Selections for navigational attributes are converted first to a selection for the characteristic bearing attributes (max. Master data attributes are located outside the InfoCube persistence in the extended star schema and thus are not a component of the NLS data stock.

33 .DSO Lookup for „nearlined‟ Partitions SAP NetWeaver BW 7. a DSO lookup In case a NLS solution is attached to the BW system. © 2011 SAP AG. All rights reserved.30 will come up with a separate transformation rule type. the lookup will automatically read from both the “online” and “near lined” data partitions.

Data Access within the APD With SAP NetWeaver BW 7. the Analysis Process Designer will be enabled to read from Nearline-Storage also for the source type “Read data from InfoProvider” Option to allow reading from NLS for InfoProvider sources © 2011 SAP AG. 34 .30. All rights reserved.

Reload data from both Online and Nearline partitions for InfoCubes Option to extract data from both the Online and Nearline Partition in a single DTP © 2011 SAP AG. 35 . All rights reserved.

36 .Transaction LISTCUBE Read data from NLS combined © 2011 SAP AG. All rights reserved.

All rights reserved. 37 . The DAP itself has the same options / settings as a regular InfoProvider. As a consequence. Semantic Partitioning criterion © 2011 SAP AG. or restore for a dedicated Semantic Partition. there is not a set of tables / files created in the NLS system per PartProvider but only a set of tables / files per SPO. the DAP must contain the logical partitioning criterion as additional archiving criterion so that data can be archived. reloaded. However.Archiving of Semantic Partitioned Objects Facts: Semantic Partitioning possible for InfoCubes (only standard InfoCubes) and DSOs (standard and write-optimized) There is not a DAP per PartProvider but only one DAP for the entire SPO.

All rights reserved. Instead. there is not “Archive” tab within the administration user interface. 38 .Archiving of Semantic Partitioned Objects Since archiving is not carried out per PartProvider. an archiving request can be scheduled by means of a dedicated / global button. Maintain Archiving © 2011 SAP AG.

Archiving of Semantic Partitioned Objects Since archiving is not carried out per PartProvider. All rights reserved. there is not “Archive” tab within the administration user interface. an archiving request can be scheduled by means of a dedicated / global button. An archiving request can be schedule to archive data from all available partitions or only from a dedicated partitions (which is equal to an archiving run being restricted to the semantic partition) Cross-partition archiving or only for a specific partition © 2011 SAP AG. 39 . Instead.

© 2011 SAP AG.30 data contained within a Nearline-Storage system can be read with a query being directly flagged to read data from NLS (query properties to read NLS data do no longer have to be maintained via transaction RSRT) Query can be set to read or to not read data from a NLS. which can also be taken into consideration. Furthermore.Reading data from SPOs Query In SAP NetWeaver BW 7. All rights reserved. 40 . it is possible to specify the same on InfoProvider level.

NLS support for DSO lookup within transformations (DSO lookup feature to be released with SAP NetWeaver BW 7.30) Combined DTP extraction from online and archive partition of an InfoCube (BW 7.30 with lookup for online data only) Master Data deletion to consider data within NLS Medium term NLS support for BW 7. as well as InfoSets and HybridProviders Archiving of master data and hierarchies Archiving with free selection criteria (not only time slice archiving) © 2011 SAP AG. All rights reserved.30) Enhanced NLS support for Semantically Partitioned Objects (SPO) based on standard InfoCubes and standard DSOs (BW 7.Summary and Outlook Latest Enhancements Enhanced lookup support especially for temporal lookups (non-equal lookup conditions) Request-based archiving for InfoCubes (avoid compression before archiving) (BW 7. 41 . NLS support for SPOs based on write-optimized DSOs is available with SP3.3 running on HANA In-Memory Physical deletion of NLS requests from the nearline Storage (BW 7.30 SP5) Long term Archiving of InfoCubes with non-cumulative key figures.30 SP 1).

3 SPnn  BW running on HANA as the underlying In-Memory DB Platform  In-Memory for Enterprise Data Warehousing  Integrated Planning InMemory enabled BW 7. eliminates need for separate database  Models for SAP business content enabling new applications SAP NetWeaver BW evolving to a fully In-Memory enabled EDW solution on top of HANA  Real-time operational analytics on mass data  Rapid creation of agile data marts  Non disruptive deployments of HANA side by side ERP and/or BW © 2011 SAP AG.01) BW 7.Planned Roadmap HANA & SAP NetWeaver BW BW 7.0 SPSnn  Additional calculation capabilities  Primary persistence layer under BW.2  Major step on Enterprise Data Warehousing scalability and flexibility  BW Accelerator: additional performance  Integration Improvements with SAP BusinessObjects Data Services BW 7.0 / BWA 7.0  Major release  BW Accelerator  New features and improvements across all components  Go-to release for integration with SAP Business Objects BI 2006 2009 2010 HANA V1.0 2011 Future direction HANA V1. 42 .0 EhP1 (7. All rights reserved.3 / BWA 7.

All rights reserved. 43 .Data-Aging Strategies: Nearline Storage Only Storage Type / Data Category Online Database Nearline Storage (read only) Classic Archive (read only) Frequently read / Information changed data (actual) Infrequently read data (mature) Very rarely read data (aged) Lifecycle according to Importance/Age: Archive       Current Situation  Nearline Storage is the leading and only persistency  No isolated Delete from Nearline Storage possible  Workaround: Restore to Online Database and delete from there © 2011 SAP AG.

44 . All rights reserved.Data-Aging Strategies: Classic Archive + Nearline Storage Storage Type / Data Category Online Database Nearline Storage (read only) Classic Archive (read only) Frequently read / Information changed data (actual) Infrequently read data (mature) Very rarely read data (aged) Lifecycle according to Importance/Age: Archive (ADK …      … + NLS)  Current Situation  ADK (Classic) Archive is the leading persistency  Nearline Storage is filled from ADK Archive during Verification Phase  Nearline Storage is strictly coupled to ADK Archive (no independent Delete) © 2011 SAP AG.

Data in ADK stays untouched  ADK delete is not supported from NLS Dialog (see SAP Data Life Cycle/ Retention concepts in ERP)  Later Restore from ADK to NLS supported © 2011 SAP AG. 45 .Details for the planned NLS Deletion Features (for SAP BW 7. SP05) 1) Data resides in NLS only (without ADK)  First step "logical" Deletion of NLS Data (set NLS Request to "Invalid" ) NLS Status in NLS Archiving-Request-List will be set to „Marked for Deletion“/ "Deleted"  NLS Data will be deleted asynchronously using a Clean-Up Job or (later) a Process Chain  Time slices will remain locked 2) Data resides in NLS and ADK  Request can only be deleted from NLS.3. All rights reserved.

All rights reserved. 46 .Data resides in NLS (only) (Final) Deletion of Nearline Request © 2011 SAP AG.

Data resides in NLS only Three Alternatives lead to Nearline Request Status "Deleted"  Finally Deleted from NLS (after successful archiving)  Restored (Deleted from NLS but stored in Online-DB again)  Invalidated (never deleted from Online-DB) © 2011 SAP AG. All rights reserved. 47 .

Data resides in ADK and NLS Restore deleted Nearline Request from ADK © 2011 SAP AG. All rights reserved. 48 .

All rights reserved.Data resides in ADK and NLS New Nearline Request after Restore from ADK © 2011 SAP AG. 49 .

com SAP NW BW PM SAP AG .uhle@sap.Thank You! Contact information: rainer.Walldorf .

Sign up to vote on this title
UsefulNot useful