James Hanck, Cheri Mallory, Paul Médaille Data Provisioning and Cleansing with SAP HANA’ SDI and SAP HANA’* SDQ © Rheinwerk® Bonn + BostonWhat You'll Learn Get your data in tip-top shape with SAP HANA smart data integration and SAP HANA smart data quality! Learn to set up data provisioning adapters to read data from flat files, SAP ECC, Twitter, and more. Utilize flow- graphs and transforms to move, cleanse, and manipulate your data, and see how SAP HANA's development tools can improve your data quality. 4 Introduction : 5 2. Primary Functions and Use Cases . . 8 2.4. Functions of SDI and SDQ 9 22 Use CASES eee cece eee ce eee eees 12 3 Architecture 6... 0. o eee eee eee eee e eee eee 16 4 Development Tools for SAPHANA ...... 02-02. 0c cece eee 20 4.1 SAP HANA Studio Overview 21 4.2 SAP Web IDE : 23 5 Data Provisioning Adapters ......... soteeeee 24 5.1 File Adapters 24 5.2 Log Reader Adapters 00... 0.00. oe cece veeeeeeee 31 5.3 SAP HANA Adapters 35 5.4 Hive Adapters . 39 5.5 Twitter Adapters ....... . 44 5.6 SAP ECC Adapters .... : . 47 5.7 OData Adapters 52 6 Flowgraphs and Transforms . . 58 6.1. Flowgraphs 59 6.2 Transforms : 7 7 What's Next? 0... e eee cece e eee eee e eee eee eees 924 Introduction SAP released the first version of the SAP HANA database in 2010, a signif- icant shift in focus for a traditional ERP company. Since then, the SAP community has embraced the SAP HANA approach to managing data, as SAP has expanded the SAP HANA database into the SAP HANA database platform. The SAP HANA platform offers more functionality than tradi- tional database systems and has a wealth of functionality and tools, which grow with each release. For example, in a recent release, SAP expanded the number of prepackaged predictive algorithms, improved tools to access and process manufacturing machine data, and enhanced text ana- lytics and text mining. Similarly, with each release, SAP adds more and more data integration functionality This E-Bite covers native SAP HANA functionality that provide data pro- visioning and data quality capabilities within the SAP HANA database platform, The types of data integration functionality that were once only available in standalone tools and complicated operating systems scripts are now native to SAP HANA. Since the early years of data processing, data integration has consisted of 1) creating a connection between two sources, 2) setting up security, and 3) acknowledging data or message receipt. This process hasn't fundamen- tally changed, though it has become a bit more complex. What has changed over time is the ease with which developers and system admin- istrators can create and manage data integration. The advent of custom tools was a huge leap in productivity and quality, and the SAP HANA database platform approach is at least as innovative and impactful. Within most application projects and implementations, the “data work" is handled by a team who are experts in whatever toolset is used. You may have a team of extract, transform, load (ETL) programmers or data quality developers. You may have a data migration team or a data quality team. Data integration is often viewed as a dependency rather than an integral part of the project. Overly complex designs may identify where data transformations will happen and who will code them, for example, the