2/7/13

different formats of the flat file for the same target
Register | Login

Not endorsed by or affiliated w ith SAP

De live ring Be tte r Busine ss De cisions (O pe ns a ne w window)

Index | About

Sponsors | C ontact Us | C alendar | FAQ | Rules and Guidelines | Privacy Policy | Search

General Notice: We are proud to introduce the new Blogger Link Program for BOB. If you write a blog related to business intelligence and would like to register for this program, read the details here. Thanks. General Notice: Upcoming Events: ASUG Ontario: Feb 12. There are 5 ASUG sponsored events in the next 45 days, see calendar for details.

different formats of the flat file for the same target
Register or Login to Post Author bobj1008
Forum Me m be r

Search this topic... | Search DI: Designer and Job Design... | Search Box

Forum Index -> Data Integrator -> DI: Designer and Job Design Message

Poste d: Tue Aug 17, 2010 1:24 pm Post subje ct: diffe re nt form ats of the flat file for the sam e targe t

Joine d: 17 Aug 2010 Posts: 1

In our deployment, we use plugin code to extract the csv files in the required format. The customers are on same version of datamart, but they are on different versions of source database - from 3.x to 4.5 depending on which version of application they are using. In 4.0, we introduced a new column email in the user table in the source database. Accordingly, plugin will add the field in the csv file. But not all the customers will get the upgraded version of plugin at the same time. So ETL code needs to decide which data flow to process depending on the format of the csv file to load data to the same target table. I made the email field in the target table nullable but it still expects the same format of the csv file with delimiter for null value. Need help to achieve this. Can I read the structure of the flat file in DS or get the count of delimiters so that I can use a conditional to use different data flow based on the format of the flat files. Can I make the email column in the flat file optional? Thanks much in advance.

Back to top Cirno
Poste d: W e d Aug 25, 2010 9:26 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Se nior Me m be r

For delimited file formats you can set the 'Adaptable schema' flag and it won't complain if you don't have all the columns. It also won't complain if you are missing a big piece of a line so keep that in mind.
Joine d: 25 Jul 2008 Posts: 53 Location: Unde fine d Fantastic O bje ct

Back to top CLS69
Poste d: Mon Aug 30, 2010 5:16 pm Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

In order to understand which version of flat file you are reading, you might also do the following: 1. make an extra file format where you read the whole line in one field 2. read only the first line and perform a set of different "match_regex" of the line so to understand which fields it contains.
Joine d: 11 Jun 2009 Posts: 194 Location: Italy

Once you got the version, you know how to process it. Ciao Claudio

Back to top Shivbaba
Poste d: Fri Se p 10, 2010 1:34 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

Hi,

www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files

1/3

All suggestions invited.php?t=161304&highlight=csv+files 2/3 .. you might check the first row against a regex like the following (assuming tab as column separator): <first col name>\t<second col name>\t. The order and name of the columns defined in the file format must match with order and name of the columns in the input file. 2010 7:53 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t Principal Me m be r Hi. different formats of the flat file for the same target Joine d: 01 Jun 2010 Posts: 132 I am also facing the same situation. I should get some error message out and terminate the job immediately. you need an additional file format where the whole row is read in one field... Thanks for your reply. Back to top CLS69 Poste d: Fri Se p 10. Joine d: 01 Jun 2010 Posts: 132 Back to top CLS69 Poste d: Fri Se p 10. hth Claudio Back to top Shivbaba Poste d: Sun Se p 12.....Can you please give the exact syntax. I need to compare the header (1st row of the file ) with the columns defined in the input format. NULL) the third parameter set to NULL will fit mostly all the times Joine d: 11 Jun 2009 Posts: 194 Location: Italy of course. I will try it out and revert back..... Thanks for the reply.. If it doesn't match. in order to match the header. The files havent got unique www.com/busobj/viewtopic. 2010 7:38 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t Principal Me m be r Considering that you the file format definition.2/7/13 Hi.. Joine d: 01 Jun 2010 Posts: 132 Back to top FreekIM Poste d: Tue Se p 21.\t<last col name> Joine d: 11 Jun 2009 Posts: 194 Location: Italy and the file will match if and only if the outcome is 1. In my case. Can you please elaborate on how to use this function. 2010 11:31 pm Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t Principal Me m be r OK.. 2010 12:36 pm Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t Principal Me m be r match_regex(string to match. I have got 5 different Excel versions that need to be loaded into 1 source.regex...... 2010 2:52 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t Forum Me m be r I am new to data integrator and i am facing a similar problem. Back to top Shivbaba Poste d: Fri Se p 10.forumtopics.. I am sorry but I don't know the syntax of match_regex function. Thanks in advance.

I have loaded all files into a flat table.7976 seconds using 16 queries. like Gecko) Chrome/24.com Shameless plug for MomentsOfLight.forumtopics.17 (KHTML.php?t=161304&highlight=csv+files 3/3 .com/busobj/viewtopic. (SQL 0. WOW64) AppleWebKit/537.17 Hosted by ForumTopics.0.1312.7319 Other 0.0 (Window s NT 6.56 Safari/537.5 Hours Go Jum p to: • DI: Designer and Job Design Go Index | About | FAQ | RAG | Privacy | Search | Register | Login Get community updates via Twitter: Follow @BOB_Board 443 follow ers Not endorsed by or affiliated w ith SAP Pow ered by phpBB © phpBB Group Generated in 0. now based on some unique characteristics within the file i must determine what format to use for further handling.0123 Parse 0. So the different versions must be determined based on the content of the files.1.0534) Mozilla/5.2/7/13 Joine d: 21 Se p 2010 Posts: 2 different formats of the flat file for the same target characteristics in the filename.com www. How can i make this work? Back to top Display posts from pre vious: All Posts Oldest First Go Register or Login to Post Page 1 of 1 Select a search Forum Index -> Data Integrator -> DI: Designer and Job Design All tim e s are GMT .com | Terms of Service phpBB Customizations by the phpBBDoctor.

Sign up to vote on this title
UsefulNot useful