Real Time Data Acquisition RDA - Powerpoint Presentation

Andreas Keppler RIG EMEA BI,
SAP AG
SAP Net Weaver BI
Real Ti me Dat a Ac qui si t i on
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 2
Ent er pr i se Dat a War ehousi ng Var i ant s & Pr oc esses
Modeling the EDW
Data Modeling
Meta Data
&
Document
Manage-
ment
Data Acquisition
Transformation
Data Distribution
Running the EDW
Administration & Monitoring
Data Flow Control
Performance Optimization
Information Lifecycle Management
User
Manage-
ment
Overview
Real-time Data Acquisition via the Service API
Real-time Data Acquisition via Web Services
Real-time Data Acquisition Daemon Monitor
Mot i vat i on f or r eal -t i me
Upload frequency for regular staging not sufficient
Number of requests can not be handled by BI system
Reporting requirements of data with sub 1 hours actuality
Operational reporting
The clear distinction between tools for analytical applications and tools for
operational applications is more and more difficult.
See transactional data in reporting as it is created in source system ( post & see )
Availability of data in source system not known
Use Push mechanism (e.g. SAP NW XI)
Polling or
Trigger data load process via event from source system
Remote access of data not feasible due to resource consumption in
source system (and cross application reporting requirements)
Splitting of staging processes (EDW) General business rules
are processed during the day whereas scenario specific rules (defined
by application, region, time zone, etc.) are processed at customer
defined times
What i s r eal -t i me ?
Real-time data warehousing is a framework for deriving
information from data as the data becomes available:
Lower time scale than for scheduled/batch data acquisition
Stream oriented
Near immediate availability for reporting
In general, Real-time data warehousing supports tactical decision-
making
Simon Terr: Real-time Data Warehousing 101, DM Review Online October 2003
External
System
Web Service
Push
Real -t i me Dat a Ac qui si t i on (RDA)
SAP Source System
Service API
Application Delta Queue
Real-time
Update
SAP NetWeaver Business Intelligence
DataSource
Data Warehouse Layer
Operational Data
Store
Architected Data Marts
Daemon
Pull
~ 1/min
InfoPackage for Real-
Time Data Acquisition
DataTransfer Process for
Real-time Data Acquisition
Scheduled Data
Transfer Process
Scheduled Data
Transfer Process
Real-time
Update
PSA
DataStore Objects
RDA Sc enar i os
Real-time data acquisition can be used in two primary scenarios:
via the Service API (SAPI)
Incorporates usage of InfoPackage for Real-time Data Acquisition
(source to PSA)
Then leverages Data Transfer Process for Real-time Data Acquisition
(PSA to DataStore Object)
via a Web Service
Incorporates usage of Web Services to populate the PSA
Then leverages the Real-time DTP to transfer data to the DataStore
Object
A system Daemon is used to initiate and control data transfer in
both scenarios.
What i s a Daemon ?
Daemon
System process fulfils a specific task at regular intervals
The SAP NetWeaver BI Daemon performs three steps (depending on
the scenario chosen):
Initiate Service-API (SAPI) data pull via InfoPackage for Real-
time Data Acquisition into PSA
Track status of data transfer from source system
Initiate update of DataStore Object via DTP
Successful execution of each step is tracked in a control table
Allows restarting if necessary
Restart can be initiated so it starts at the next step after the last
successfully executed step.
Daemon Moni t or i ng
Maintenance screen for demon control can be called from the Data
Warehousing Workbench
Transaction RSRDA
Overview
Sc enar i o I : RDA vi a SAP appl i c at i ons
SAP Source System
Service API
Application Delta Queue
Real-time
Update
SAP NetWeaver BI
DataSource
Operational Data
Store
Daemon
Pull
~ 1/min
InfoPackage for Real-time
Data Acquisition
Data Transfer Process for
PSA
DataStore Objects
Service API-based real-time data
acquisition is a two-stage
process:
Data is pulled into PSA by Real-
time InfoPackage
Data is transferred to DataStore
Object via Real-time DTP
In both cases the process of data
movement is initiated by the
system daemon
Appl i c at i on Ac t i vi t i es on t he Sour c e Syst em Si de
Source system
(ERP, CRM ...)
Application
Service API
1 2 3
Time 03:10:09
Delta Queue
Daemon Tr i gger s RDA Pr oc essi ng
Source system
(ERP, CRM ...)
Application
Service API
BI
1 2 3
Daemon
pull ~ 1/min
PSA
Time 03:11:00
Confirmation Table
DataStore Object
Delta Queue
Have a look
Data
transfer
TID 1
TID 2
TID 3
Log entries
Update PSA
1
2
3
TID 1 ok
TID 2 ok
TID 3 ok
Commit Commit
1 2 3
DataStore
DTP
Initiate
Step 1 completed
Step 2 compl.
Runt i me St eps I
Step 1
Call source system for new records
Status update of transferred records in confirmation table
Update PSA
Commit
Update confirmation table and update PSA in one step
Guarantees synchronization
Runt i me St eps I I
Step 2
Check for records in confirmation table
If records are available in confirmation table the corresponding records
exist as well in PSA
Daemon flags records in source system
Reply confirmation sent to SAP NetWeaver BI
Entries are flagged as processed in confirmation table
Initiate DTP after the records are confirmed in confirmation table
Commit
If not successful in BI, records will stay in confirmation table
Guarantees restart of step 2 even when the update in the source system
was successful
Daemon Tr i gger s RDA Pr oc essi ng ever y 3 mi nut es
Source system
(ERP, CRM ...)
Application
Service API
BI
1 2 3
Daemon
pull ~ 1/min
PSA
Time 03:14:00
Confirmation Table
DataStore Object
Delta Queue
TID 1
TID 2
TID 3
TID 1 ok
TID 2 ok
TID 3 ok
1 2 3
DataStore
DTP
Initiate
4 5
Next
data load cycle
1
2
3
4
5
1
2
3
4 5
TID 1 ok
TID 2 ok
TID 3 ok
TID 4 ok
TID 5 ok
TID 1 ok
TID 2 ok
TID 3 ok
Cl osur e of open r equest
Not executed on every data load cycle
After completion of data upload into PSA:
Entries in confirmation table are deleted once the request is closed
(threshold for number of records per request / time exceeded or daemon
stopped).
Before the next data load cycle is started:
The next request is opened for the next data load from source system
(determined by update period of the daemon)
Requests of InfoPackage and DTPs are synchronized via daemon
(i.e. request in PSA and DataStore Object are aligned).
Daemon c l oses r equest
BI
Daemon
pull ~ 1/min
PSA
Time 03:17:00
DataStore Object
DataStore
DTP
Initiate
Step 0 completed
cont. with load
1
2
3
4
5
1
2
3
Confirmation Table
TID 1 ok
TID 2 ok
TID 3 ok
TID 4 ok
TID 5 ok
Request closed due to
time or n of records
Request 1
Source system
(ERP, CRM ...)
Application
Service API
1 2 3
Delta Queue
1 2 3 4 5 4 5
I nf oPac k age f or Real -t i me Dat a Ac qui si t i on Cr eat i on
Real-time flag must be selected for the InfoPackage which the
daemon will monitor.
This option is only available for one InfoPackage attached to the
DataSource
Note: If you want to use a DataSource with real-time capabilities,
you have to install the PI_BASIS 2005.1 in the source system
I nf oPac k age f or Real -t i me Dat a Ac qui si t i on Def i ni t i on
Real-time extraction must be selected in the Adapter section of
the InfoPackage settings
I nf oPac k age f or Real -t i me Dat a Ac qui si t i on Daemon
Set t i ngs
Automatic Request Closure by # records
Periodicity is determined by Update Periods setting via Real Time Monitor (RSRDA).
Data Package Size: No. of Rows
Allows control of number of records per data package (multiple data packages per request possible)
The deamon uploads at least one data package (even if number of actual records is smaller than
setting)
Request Size: Number of Rows
Allows control of packets for downstream processing based on time
Automatic closure of Request by number of records (1,000,000 shown here)
Max number of failed attempts
Max number of times Daemon attempts to request/commit data before throwing error condition
I nf oPac k age f or Real -t i me Dat a Ac qui si t i on Updat e:
I nher ent l y Del t a
Only Delta Processing is
available for Real-time
updates (because the delta
queue is leveraged for data
staging)
On the Schedule tab you can jump
via Assign to the RDA Monitor.
There, you can assign the
InfoPackage to a deamon.
Overview
External
System
Web Service
Push
Sc enar i o I I : RDA vi a Web Ser vi c es
SAP NetWeaver Business Intelligence
DataSource
Operational Data
Store
Daemon
Pull
~ 1/min
Real-time
Update
PSA
DataStore Objects
InfoPackage for Real-time
Data Acquisition
Data Transfer Process for
Web Service-based Real-time data acquisition is also a two-stage
process:
Data is pushed into the PSA via a Web Service
Data is transferred to DataStore Object via Real-time DTP
The daemon monitors the PSA and initiates the DTP on a regular
basis (via settings in Real-time InfoPackage)
Web Ser vi c e Dat a Sour c e
I nf oPac k age f or Real -t i me Dat a Ac qui si t i on vi a Web Ser vi c e
An InfoPackage is needed for the Web Service to define
parameters such as time / record thresholds for request closure
and to allow the assignment of a daemon via the Real-time
Monitor.
Real -t i me r equest s i n Dat aSt or e Obj ec t
DataSource / PSA
Activation
Queue
Active
Data
Change
Log
Per.1 Per. 2 Per.3
Req.ID, Pack ID, Rec. No.
Req.SID, Pack ID, Rec. No.
Document No.
Latency between source system transaction and availability in DataStore Object is
minimal due to Lean Staging .
Lean Staging means that the DataStore Object handles the data differently
internally less logging and activation activity is allowed when the DTP for Real-
time Data Acquisition is connected to the DataStore Object
From an administrative point of view, the DataStore Object is a standard one.
The DataStore Object also handles the availability of this real-time data (e.g. data
whose request has not been closed) for reporting.
How t o embed RDA i n your dat a f l ow
DataSource
PSA
Regular
DataStore Object
OLTP
BI
Delta Queue
2LIS_02_VAITM
Application
real-time
Update
Using one DataSource
DataSource
PSA
RDA
DataStore Object
Regular
DataStore Object
OLTP
BI
Daemon
Pull
~5/min
Delta Queue
2LIS_02_VAHDR
2LIS_02_VAITM
Periodic
Scheduling
2LIS_02_VAITM
Application
real-time
Update
Using two DataSources
Daemon
Pull
~5/min
Well established Data Flow can
can be used
Separate (potentially) leaner
DataStore Object can be used
Reporting has to be enabled via
MultiProvider
Data has to be deleted frequently
from RDA DataStore Object to
avoid redundant data
Old data flow has to be replaced
by new real-time data flow
No issues with data alignment
from different DataSources
Can facilitate upload of huge
data volumes by spreading
data load over many smaller
data loads
Periodic
Scheduling
2LIS_02_VAITM
Extraction done via RDA and InfoPackage scheduling
( Normal Delta ) can currently not be done in parallel
DataSource can only use a single extraction mechanism
Overview
Daemon Moni t or
The RDA Daemon monitor provides an overview on the status of each daemon
and attached
InfoPackage(s) for RDA
Data Transfer Process(es) for RDA
Important to note that both loading to the PSA and the DataStore Object are
monitored within the Daemon Monitor
Transaction RSRDA
Real-time
DTP
Real-time
InfoPackage
DataSource
Daemon St at us
The Daemons can have the following statuses (identified by the icon in front):
- Daemon active and running
- Daemon not active
- Daemon has an error
- Daemon is stopped (changes its status to not active or error)
St andar d BI Moni t or f or I nf oPac k ages
Granular details
of data
movement look
similar to Data
Load Monitor in
prior releases
DTP f or RDA Moni t or
Detailed data transfer process steps in DTP Monitor
Similar to details provided by the Data Load Monitor
Copyr i ght 2004 SAP AG. Al l Ri ght s Reser ved
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express
permission of SAP AG. The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other
software vendors.
Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries,
pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or
registered trademarks of IBM Corporation in the United States and/or other countries.
Oracle is a registered trademark of Oracle Corporation.
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered
trademarks of Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium,
Massachusetts Institute of Technology.
J ava is a registered trademark of Sun Microsystems, Inc.
J avaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and
implemented by Netscape.
MaxDB is a trademark of MySQL AB, Sweden.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver and other SAP products and services mentioned herein
as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other
countries all over the world. All other product and service names mentioned are the trademarks of their respective
companies. Data contained in this document serves informational purposes only. National product specifications may vary.
These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated
companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group
shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and
services are those that are set forth in the express warranty statements accompanying such products and services, if any.
Nothing herein should be construed as constituting an additional warranty.

Real Time Data Acquisition RDA - Powerpoint Presentation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Real Time Data Acquisition RDA - Powerpoint Presentation

Uploaded by

Copyright:

Available Formats

Andreas Keppler RIG EMEA BI,

You might also like