Flat File Upload - Considerations - Preliminary Program

Flat File Upload Considerations Preliminary Program
Applies to:
SAP Business Intelligence 7.0. For more information, visit the Business Intelligence homepage.
Summary
The advantage of uploading flat files is the flexibility. This means that all kinds of sources of data can be used to provide data for the SAP BI application. Flat files have also some particularities. Amongst them are easy changes in the file structure, redundant data, lines without data and information in the filename. To handle these particularities some considerations can be useful. Author: Guy Mathys
Company: Aureus Financials Created on: 15 August 2009
Author Bio
Guy Mathys enables companies to obtain information from SAP Business Intelligence (SAP NetWeaver BW - SAP BI) for the last five years.
SAP COMMUNITY NETWORK 2009 SAP AG
SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com 1
Flat File Upload Considerations - Preliminary Program
Table of Contents
Reason of a preliminary program ....................................................................................................................... 3 Functionalities ..................................................................................................................................................... 3 Details of the functionalities ................................................................................................................................ 4 Get a file. ......................................................................................................................................................... 4 Read data record ............................................................................................................................................ 5 Define delimiter ............................................................................................................................................... 5 Check on allowed characters .......................................................................................................................... 5 Check on records with all fields empty............................................................................................................ 5 Check on comma as field separator and as decimal. ..................................................................................... 6 Read data of the record into a table................................................................................................................ 6 Check field types ............................................................................................................................................. 7 Include filename in a field ............................................................................................................................... 7 Write the record............................................................................................................................................... 8 Other considerations........................................................................................................................................... 8 Related Content .................................................................................................................................................. 9 Help.sap.com .................................................................................................................................................. 9 Articles............................................................................................................................................................. 9 Blogs ............................................................................................................................................................... 9 Disclaimer and Liability Notice .......................................................................................................................... 10
Reason of a preliminary program

A program before starting the loads prevents some errors during the loading process. This is important because an error can stop the activation of the loads of succeeding files in the same Data Store Object. A check program can also generate alerts.
Functionalities
A preliminary can handle several functionalities. The first group is getting the data. Then there is a checking and converting the characters, fields and their separators and checking and skipping records. A whole file can also be rejected. The last group is writing an improved file ready for upload. This all can be done generating error messages. 1. Get a file. This is to get a file based on the directory WA_DRCTR and file mask WA_FLMSK. 2. Read data record Opens the dataset reads a record, closes the dataset . 3. define delimiter The delimiter can be used to add fields later. 4. Check on allowed characters Put the allowed characters in a work area. Replaces the not allowed character by a 0. 5. Check on records with all fields empty Checks if the first 5 characters contains not a delimiter. If it does not contain delimiters it continues. 6. Check on comma as field separator and as decimal. Some csv files have comma as a field separator. on top of that the comma is also used a the decimal sign. For instance some records are like Field1,123,45,Field3 This removes the inverted comma and changes the decimal comma in a decimal point. 7. Read data of the record into a table This is useful to manipulate data in a table. This can be the check of field types. 8. Check field types This checks the field type date , numeric, character of a field based on parameters in an internal table. 9. Include filename in a field This enables to load the filename into a DSO and use it or part of it; For instance to exploit the date timestamp that is in it. 10. Write the record This writes a check record in the new file. This new file will be used to load.
Details of the functionalities

The basic idea is that the preliminary program takes the input file, checks and modifies it and writes it to another file. This other is used to load in SAP BI. These parts of ABAP are not real code. They are meant to explain the functionalities. Get a file. This is to get a file based on the directory WA_DRCTR and file mask WA_FLMSK.
* The first file that matches the pattern CALL 'C_SAPGPARAM' ID 'NAME' FIELD WA_DRCTR ID 'VALUE' FIELD WA_DIR_VALUE. CALL 'C_DIR_READ_FINISH' " just to be sure ID 'ERRNO' FIELD WA_FILE_ERRNO ID 'ERRMSG' FIELD WA_FILE_ERRMSG. CALL 'C_DIR_READ_START' ID 'DIR' FIELD WA_DRCTR ID 'FILE' FIELD WA_FLMSK ID 'ERRNO' FIELD WA_FILE_ERRNO ID 'ERRMSG' FIELD WA_FILE_ERRMSG. IF SY-SUBRC <> 0. ENDIF. DO. CALL 'C_DIR_READ_NEXT' ID 'TYPE' FIELD WA_FILE_TYPE ID 'NAME' FIELD WA_FILENAME ID 'LEN' FIELD WA_FILE_LEN ID 'OWNER' FIELD WA_FILE_OWNER ID 'MTIME' FIELD WA_FILE_MTIME ID 'MODE' FIELD WA_FILE_MODE ID 'ERRNO' FIELD WA_FILE_ERRNO ID 'ERRMSG' FIELD WA_FILE_ERRMSG. IF SY-SUBRC = 0. IF WA_FILE_TYPE(1) = 'f' OR " regular file WA_FILE_TYPE(1) = 'F'. EXIT. ENDIF. ELSEIF SY-SUBRC = 1. RAISE FILE_NOT_FOUND. EXIT. ELSE. ENDIF. ENDDO.
Read data record Opens the dataset , reads a record, closes the dataset .
OPEN DATASET WA_DIRFILENAME FOR INPUT IN TEXT MODE ENCODING DEFAULT. DO. CATCH SYSTEM-EXCEPTIONS CONVT_CODEPAGE = 1. READ DATASET WA_DIRFILENAME INTO WA_LINE. IF SY-SUBRC <> 0. EXIT. ENDIF. ENDCATCH. ENDDO CLOSE DATASET WA_DIRFILENAME.
Define delimiter The delimiter can be used to add fields later.

IF WA_SPRTR IS INITIAL. IF WA_LINE CA C_SMCLMN. WA_SPRTR = C_SMCLMN. ELSE. WA_SPRTR = C_CMM. ENDIF. ENDIF.
Check on allowed characters Put the allowed characters in a work area. Replaces the not allowed character by a 0.
CALL FUNCTION 'RSKC_ALLOWED_CHAR_GET' IMPORTING E_ALLOWED_CHAR = WA_ALLOWED_CHAR EXCEPTIONS OTHERS = 1. * check on allowed characters TRANSLATE WA_LINE TO UPPER CASE. DO. IF WA_LINE cn WA_ALLOWED_CHAR. WA_LINE+sy-fdpos(1) = 0. ELSE. EXIT. ENDIF. ENDDO.
Check on records with all fields empty Checks if the first 5 characters contains not a delimiter. If it does not contain delimiters it continues. The case is that there are records with only delimiters and no data.
* check on records with all fields empty IF WA_LINE(5) CN WA_SPRTR. . Write new data record
ENDIF.
Check on comma as field separator and as decimal. Some csv files have comma as a field separator. on top of that the comma is also used a the decimal sign. For instance some records are like Field1,123,45,Field3 This removes the inverted comma and changes the decimal comma in a decimal point.
* check on " and change decimal comma in decimal point WA_SWNVRTDCMM = 0. DESCRIBE FIELD WA_LINE :LENGTH WA_CLEN IN CHARACTER MODE. WA_COUNT3 = 0. DO WA_CLEN TIMES. IF WA_LINE+WA_COUNT3(1) = C_INVERTED_COMMA. WA_LINE+WA_COUNT3(1) = SPACE. IF WA_SWNVRTDCMM = 0. WA_SWNVRTDCMM = 1. ELSE. WA_SWNVRTDCMM = 0. ENDIF. ENDIF. IF WA_LINE+WA_COUNT3(1) = C_COMMA AND WA_SWNVRTDCMM = 1. WA_LINE+WA_COUNT3(1) = C_POINT. ENDIF. WA_COUNT3 = WA_COUNT3 EXIT. ENDIF. ENDDO. + 1. IF WA_LINE+WA_COUNT3(10) = SPACE.
Read data of the record into a table This is useful to manipulate data in a table; This can be the check of field types.
* read into table WA_RECORD = WA_LINE. CALL FUNCTION 'RSDS_CONVERT_CSV' EXPORTING I_DATA_SEP I_ESC_CHAR I_RECORD I_FIELD_COUNT IMPORTING E_T_DATA EXCEPTIONS = WA_DATA = WA_SPRTR = C_SCP = WA_RECORD = WA_FLDCNT
ESCAPE_NO_CLOSE ESCAPE_IMPROPER
= 1 = 2
CONVERSION_ERROR = 3.
Check field types This checks the field type date , numeric, character of a field based on parameters in an internal table.
* check field types IF WA_SWITHBEGIN <> C_B. WA_COUNT = 1. WHILE WA_COUNT <= WA_FLDCNT. LOOP AT IT_PARAMETER_S ASSIGNING <FS_PARAMETER>. IF <FS_PARAMETER>-/BIC/FLDNM = WA_COUNT. READ TABLE WA_DATA INDEX WA_COUNT INTO WA_DATA2. CASE <FS_PARAMETER>-/BIC/TPFLD. WHEN C_NUM. IF WA_DATA2 CN C_NUMERIC. PERFORM FILE_STRUCTURE_ERROR. ENDIF. WHEN C_DATS. IF WA_DATA2(4) < C_1900 AND WA_DATA2+6(4) < C_1900. PERFORM FILE_STRUCTURE_ERROR. ENDIF. ENDCASE. ENDIF. ENDLOOP. WA_COUNT = WA_COUNT + 1. ENDWHILE. ENDIF. WA_SWITHBEGIN = C_C.
Include filename in a field This enables to load the filename into a DSO and use it or part of it. For instance to exploit the date timestamp that is in it.
* add filename into data CONCATENATE WA_LINE WA_SPRTR WA_FILENAME INTO WA_LINE.
Write the record This writes a check record in the new file. This new file will be used to load.
* write new data file TRANSFER WA_LINE to WA_DIRFILENAMENEW.
.
CLOSE DATASET WA_DIRFILENAMENEW.
Other considerations
Amongst other considerations that will be explained later are: Using DSO for redundant data Making data uniform Using parameters Using file masks Sizing of data packages Limiting number of processes Converting xml to an internal table.
Related Content
Help.sap.com Creating DataSources for File Source Systems http://help.sap.com/saphelp_nw2004s/helpdata/en/43/01ed2fe3811a77e10000000a422035/frameset.htm Articles Gaurav Paul , Data Upload from Flat File to Business Intelligence 7.0 System https://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/60debae1-84dd-2b10-e7bfbdedf1eabdf9&overridelayout=true Aveek Ghose ,Flat File Interfaces in SAP https://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/c0341dc1-515c-2910-f99bb575e9f74671&overridelayout=true
Anurag Krishna Dev , Making Flat File Data Sources Delta Capable By Using Routine In Info package Selection.
https://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/a0b3f0e2-832d-2c10-1ca9d909ca40b54e&overridelayout=true Blogs Loading Text Files(.TXT) into BW with HEX option https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4676 Hierarchy Upload from Flat files https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/3113 Binary File in BW: what I can do with control file https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4803 How-to... Load a File into SAP NetWeaver BI Integrated Planning (Part 1) https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6671 How-to... Load a File into SAP NetWeaver BI Integrated Planning (Part 2) https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/7708
Disclaimer and Liability Notice

This document may discuss sample coding or other information that does not include SAP official interfaces and therefore is not supported by SAP. Changes made based on this information are not supported and can be overwritten during an upgrade. SAP will not be held liable for any damages caused by using or misusing the information, code or methods suggested in this document, and anyone using these methods does so at his/her own risk. SAP offers no guarantees and assumes no responsibility or liability of any type with respect to the content of this technical article or code sample, including any liability resulting from incompatibility between the content within this document and the materials and services offered by SAP. You agree that you will not hold, or seek to hold, SAP responsible or liable with respect to the content of this document.

Flat File Upload - Considerations - Preliminary Program

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Flat File Upload - Considerations - Preliminary Program

Uploaded by

Copyright:

Available Formats

Flat File Upload Considerations Preliminary Program

Company: Aureus Financials Created on: 15 August 2009

SAP COMMUNITY NETWORK 2009 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com 1