Teradata Utilities: FastExport

Reprinted for KV Satish Kumar, IBM kvskumar@in.ibm.com Reprinted with permission as a subscription benefit of Books24x7, http://www.books24x7.com/

Table of Contents
Chapter 6: FastExport.....................................................................................................................1 Why it is Called "FAST" Export..............................................................................................1 How FastExport Works....................................................................................................1 FastExport Fundamentals......................................................................................................2 FastExport Supported Operating Systems .......................................................................2 Maximum of 15 Loads ............................................................................................................3 FastExport Support and Task Commands.............................................................................3 Support Environment Commands..........................................................................................3 Task Commands....................................................................................................................4 FastExport Supported SQL Commands................................................................................5 SQL Commands ...............................................................................................................5 A FastExport in its Simplest Form ..........................................................................................5 Sample FastExport Script......................................................................................................6 FastExport Modes and Formats .............................................................................................7 Modes..............................................................................................................................7 Formats............................................................................................................................8 A FastExport Script using Binary Mode.................................................................................8

Chapter 6: FastExport
"An invasion of armies can be resisted, but not an idea whose time has come." - Victor Hugo

Why it is Called "FAST" Export

FastExport is known for its lightning speed when it comes to exporting vast amounts of data from Teradata and transferring the data into flat files on either a mainframe or network-attached computer. In addition, FastExport has the ability to use OUTMOD routines, which provide the user the capability to write, select, validate, and preprocess the exported data. Part of this speed is achieved because FastExport takes full advantage of Teradata's parallelism. In this book, we have already discovered how BTEQ can be utilized to export data from Teradata in a variety of formats. As the demand increases to store data, the ever-growing requirement for tools to export massive amounts of data also increases. This is the reason why FastExport (FEXP) is brilliant by design. A good rule of thumb is that if you have more than half a million rows of data to export to either a flat file format or with NULL indicators, then FastExport is the best choice to accomplish this task. Keep in mind that FastExport is designed as a one-way utility that is, the sole purpose of FastExport is to move data out of Teradata. It does this by harnessing the parallelism that Teradata provides. FastExport is extremely attractive for exporting data because it takes full advantage of multiple sessions, which leverages Teradata parallelism. By default, Fastexport will take up 4 sessions that will be running on the local computer. FastExport can also export from multiple tables during a single operation. In addition, FastExport utilizes the Support Environment, which provides a job restart capability from a checkpoint if an error occurs during the process of executing an export job.

How FastExport Works

When FastExport is invoked, the utility logs onto the Teradata database and retrieves the rows that are specified in the SELECT statement and puts them into SPOOL. From there, it must build blocks to send back to the client. In comparison, BTEQ starts sending rows immediately for storage into a file. If the output data is sorted, FastExport may be required to redistribute the selected data two times across the AMP processors in order to build the blocks in the correct sequence. Remember, a lot of rows fit into a 64K block and both the rows and the blocks must be sequenced. While all of this redistribution is occurring, BTEQ continues to send rows. FastExport is getting behind in the processing. However, when FastExport starts sending the rows back a block at a time, it quickly overtakes and passes BTEQ's row at time processing. The other advantage is that if BTEQ terminates abnormally, all of your rows (which are in SPOOL) are discarded. You must rerun the BTEQ script from the beginning. However, if FastExport terminates abnormally, all the selected rows are in worktables and it can continue sending them where it left off in a very smart and very fast manner! Also, if there is a requirement to manipulate the data before storing it on the computer's hard drive, an OUTMOD routine can be written to modify the result set after it is sent back to the client on either the mainframe or LAN. Just like the BASF commercial states, "We don't make the products you buy, we make the products you buy better". FastExport is designed off the same premise, it does not make the SQL SELECT statement faster, but it does take the SQL SELECT statement and processes the request with lighting fast parallel processing!

Reprinted for ibmkvskumar@in.ibm.com, IBM

Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

FastExport Fundamentals
#1: FastExport EXPORTS data from Teradata. The reason they call it FastExport is because it takes data off of Teradata (Exports Data). FastExport does not import data into Teradata. Additionally, like BTEQ it can output multiple files in a single run. #2: FastExport only supports the SELECT statement. The only DML statement that FastExport understands is SELECT. You SELECT the data you want exported and FastExport will take care of the rest. #3: Choose FastExport over BTEQ when Exporting Data of more than half a million+ rows. When a large amount of data is being exported, FastExport is recommended over BTEQ Export. The only drawback is the total number of FastLoads, FastExports, and MultiLoads that can run at the same time, which is limited to 15. BTEQ Export does not have this restriction. Of course, FastExport will work with less data, but the speed may not be much faster than BTEQ. #4: FastExport supports multiple SELECT statements and multiple tables in a single run. You can have multiple SELECT statements with FastExport and each SELECT can join information up to 64 tables. #5: FastExport supports conditional logic, conditional expressions, arithmetic calculations, and data conversions. FastExport is flexible and supports the above conditions, calculations, and conversions. #6: FastExport does NOT support error files or error limits. FastExport does not record particular error types in a table. The FastExport utility will terminate after a certain number of errors have been encountered. #7: FastExport supports user-written routines INMODs and OUTMODs. FastExport allows you write INMOD and OUTMOD routines so you can select, validate and preprocess the exported data

FastExport Supported Operating Systems

The FastExport utility is supported on either the mainframe or on LAN. The information below illustrates which operating systems are supported for each environment: The LAN environment supports the following Operating Systems: UNIX MP-RAS Windows 2000 Windows 95/98/XP Windows NT/2000 UNIX HP-UX AIX Solaris SPARC Solaris Intel
Reprinted for ibmkvskumar@in.ibm.com, IBM Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

The Mainframe (Channel Attached) environment supports the following Operating Systems:

MVS VM

Maximum of 15 Loads
The Teradata RDBMS will only support a maximum of 15 simultaneous FastLoad, MultiLoad, or FastExport utility jobs. This maximum value is determined and configured in the DBS Control record. This value can be set from 0 to 15. When Teradata is initially installed, this value is set at 5. The reason for this limitation is that FastLoad, MultiLoad, and FastExport all use large blocks to transfer data. If more then 15 simultaneous jobs were supported, a saturation point could be reached on the availability of resources. In this case, Teradata does an excellent job of protecting system resources by queuing up additional FastLoad, MultiLoad, and FastExport jobs that are attempting to connect. For example, if the maximum number of utilities on the Teradata system is reached and another job attempts to run, that job will not start. This limitation should be viewed as a safety control feature. A tip for remembering how the load limit applies is this, "If the name of the load utility contains either the word "Fast" or the word "Load", then there can be only a total of fifteen of them running at any one time". BTEQ does not have this load limitation. FastExport is clearly the better choice when exporting data. However, if two many load jobs are running. BTEQ is an alternate choice for exporting data.

FastExport Support and Task Commands

FastExport accepts both FastExport commands and a subset of SQL statements. The FastExport commands can be broken down into support and task activities. The table below highlights the key FastExport commands and their definitions. These commands provide flexibility and control during the export process.

Support Environment Commands

(see Support Environment chapter for details)

Reprinted for ibmkvskumar@in.ibm.com, IBM

Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

ACCEPT DATEFORM DISPLAY ELSE ENDIF IF LOGOFF LOGON LOGTABLE ROUTE MESSAGES RUN FILE SET SYSTEM

Allows the value of utility variables to be accepted directly from a file or from environmental variables. Specifies the style of the DATE data types for FastExport. Writes messages to the specific location. Used in conjunction with the IF statement. ELSE commands and statements will execute when a proceeding IF condition is false. Used in conjunction with the IF or ELSE statements. Delimits the commands that were subject to previous IF or ELSE conditions. Introduces a conditional expression. If true then execution of subsequent commands will happen. Disconnects all FastExport active sessions and terminates FastExport. LOGON command or string used to connect sessions established through the FastExport utility. FastExport utilizes this to specify a restart log table. The purpose is for FastExport checkpoint information. Will route FastExport messages to an alternate destination. Used to point to a file that FastExport is to use as standard input. This will Invoke the specified external file as the current source of utility and Teradata SQL commands. Assigns a data type and value to a variable. Suspends the FastExport utility temporarily and executes any valid local operating system command before returning.

Figure 3-1

Task Commands
BEGIN EXPORT END EXPORT EXPORT Begins the export task and sets the specifications for the number of sessions with Teradata. Ends the export task and initiates processing by Teradata. Provides two things which are: The client destination and file format specifications for the export data retrieved from Teradata A generated MultiLoad script file that can be used later to reload the export data back into Teradata FIELD FILLER IMPORT LAYOUT Constitutes a field in the input record section that provides data values for the SELECT statement. Specifies a field in the input record that will not be sent to Teradata for processing. It is part of the input record to provide data values for the SELECT statement. Defines the file that provides the USING data values for the SELECT. Specifies the data layout for a file. It contains a sequence of FIELD and FILLER commands. This is used to describe the import file that can optionally provide data values for the SELECT.

Figure 3-2
Reprinted for ibmkvskumar@in.ibm.com, IBM Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

FastExport Supported SQL Commands

FastExport accepts the following Teradata SQL statements. Each has been placed in alphabetic order for your convenience.

SQL Commands
ALTER TABLE Change a column or table options of a table. CHECKPOINT Add a checkpoint entry in the journal table. COLLECT STATISTICS Collect statistics for one or more columns or indexes in a table. COMMENT Store or retrieve a comment string for a particular object. CREATE DATABASE Creates a new database. CREATE TABLE Creates a new table. CREATE VIEW Creates a new view. CREATE MACRO Creates a new macro. DATABASE Specify a default database for the session. DELETE Delete rows from a table. DELETE DATABASE Removes all tables, views, macros, and stored procedures from a database. DROP DATABASE Drops a database. GIVE Transfer ownership of a database or user to another user. GRANT Grant access privileges to an object. MODIFY DATABASE Change the options for a database. RENAME Change the name of a table, view, or macro. REPLACE MACRO Change a macro. REPLACE VIEW Change a view. REVOKE Revoke privileges to an object. SET SESSION Override the collation specification during the current COLLATION session. UPDATE Change a column value of an existing row or rows in a table.

Figure 3-3

A FastExport in its Simplest Form

The hobby of racecar driving can be extremely frustrating, challenging, and rewarding all at the same time. I always remember my driving instructor coaching me during a practice session in a new car around a road course racetrack. He said to me, "Before you can learn to run, you need to learn how to walk." This same philosophy can be applied when working with FastExport. If FastExport is broken into steps, then several things that appear to be complicated are really very simple. With this being stated, FastExport can be broken into the following steps: Logging onto Teradata Retrieves the rows you specify in your SELECT statement

Reprinted for ibmkvskumar@in.ibm.com, IBM

Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

Exports the data to the specified file or OUTMOD routine Logs off of Teradata

/* Created by CoffingDW */ /* Setup the Fast Export Parameters */

Creates the logtable -Required

.LOGTABLE sql01.SWA_Log;

Logon to Teradata
.LOGON CDW/sql01,whynot; .BEGIN EXPORT SESSIONS 12;

.EXPORT OUTFILE Student.txt MODE RECORD FORMAT TEXT;

Begin the Export and set the number of sessions on Teradata Defines the output file name. In addition, specifies the output mode and format (LAN - ONLY) The SELECT defines the columns used to create the exported file.

/* Get Data From the Student Table */ SELECT Student_ID (CHAR(11)), Last_name (CHAR(20)), First_name (CHAR(14)), Class_code (CHAR(2)), Grade_Pt (CHAR(9)) FROM SQL_CLASS.Student_Table;

NOTE: The selected columns for the export are being converted to character types. This will simplify the importing process into a different database. End the Export and logoff /* Finish the Export Job and Write to File */ Teradata.
.END EXPORT; .LOGOFF;

Figure 3-4

Sample FastExport Script

Now that the first steps have been taken to understand FastExport, the next step is to journey forward and review another example that shows builds upon what we have learned. In the script below, Teradata comment lines have been placed inside the script [/*. . . . */]. In addition, FastExport and SQL commands are written in upper case in order to highlight them. Another note is that the column names are listed vertically. The recommendation is to place the comma separator in front of the following column. Coding this way makes reading or debugging the script easier to accomplish.

Reprinted for ibmkvskumar@in.ibm.com, IBM

Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

/* ---------------------------------------------------*/ /*@(#) FASTEXPORT SCRIPT */ /*@(#) Version 1.1 /*@(#) Created by CoffingDW */ /* ---------------------------------------------------*/

ALWAYS GOOD TO IDENTIFY THE */ SCRIPT AND AUTHOR IN COMMENTS CREATE LOGTABLE AND LOGON; BEGIN EXPORT STATEMENT. SESSIONS 12; DEFINES THE OUTPUT FILE NAME. IN ADDITION, SPECIFIES THE OUTPUT MODE AND FORMAT(LAN - ONLY) MODE RECORD FORMAT TEXT; THE SELECT PULLS DATA FROM TWO TABLES. IT IS GOOD TO QUALILY WHEN DOING A TWO-TABLE JOIN. END THE JOB AND LOGOFF TERADATA;

/* Setup the Fast Export Parameters */ .LOGTABLE SQL01.CDW_Log; .LOGON CDW/SQL01,whynot; .BEGIN EXPORT SESSIONS 12;

.EXPORT OUTFILE Join_Export.txt MODE RECORD FORMAT TEXT;

/* Join Data From the Employee and Department Table */ SELECT EMP.Employee_No (CHAR(11)) ,EMP.First_Name (CHAR(14)) ,EMP.Last_Name (CHAR(20)) ,DEPT.Dept_No (CHAR(6)) ,DEPT.Dept_name (CHAR(20)) FROM SQL_CLASS.Employee_table AS EMP INNER JOIN SQL_CLASS.Department_Table AS DEPT ON EMP.Dept_No = DEPT.Dept_No ;

/* Finish the Export Job and Write to File */ .END EXPORT; .LOGOFF;

Figure 3-5

FastExport Modes and Formats

Modes
FastExport has two modes: RECORD or INDICATOR. In the mainframe world, only use RECORD mode. In the UNIX or LAN environment, INDICATOR mode is the default, but you can use RECORD mode if desired. The difference between the two modes is INDICATOR mode will set the indicator bits to 1 for column values containing NULLS.

Reprinted for ibmkvskumar@in.ibm.com, IBM

Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

Both modes return data in a client internal format with variable-length records. Each individual record has a value for all of the columns specified by the SELECT statement. All variable-length columns are preceded by a two-byte control value indicating the length of the column data. NULL columns have a value that is appropriate for the column data type. Remember, INDICATOR mode will set bit flags that identify the columns that have a null value.

Formats
FastExport has many possible formats in the UNIX or LAN environment. The FORMAT statement specifies the format for each record being exported which are: FASTLOAD BINARY TEXT UNFORMAT

The default FORMAT is FASTLOAD in a UNIX or LAN environment. FASTLOAD format has a two-byte integer, followed by the data, followed by an end-of-record marker. It is called FASTLOAD because the data is exported in a format ready for FASTLOAD. BINARY format is a two-byte integer, followed by data. TEXT format is an arbitrary number of bytes followed by an end-of-record marker. UNFORMAT format is exactly as received from CLIv2 without any client modifications.

A FastExport Script using Binary Mode

COMMENTS
/*-------------------------------------------------/*@(#)FASTEXPORT SCRIPT - SWA /*@(#)Version 1.1 /*@(#)Created by CoffingDW /*-------------------------------------------------/* Setup the Fast Export Parameters */ .LOGTABLE SQL01.SWA_LOG; .LOGON CDW/sql01,whynot; .BEGIN EXPORT SESSIONS 12; .EXPORT OUTFILE Cdw_import.txt FORMAT BINARY; /* Get Data From the Student Table */ SELECT Student_ID */ */ */ */ */

CREATE LOGTABLE AND LOGON TO TERADATA; BEGIN EXPORT STATEMENT; NAME THE OUTPUT FILE AND SET THE FORMAT TO BINARY; GET THE DATA FROM THE STUDENT TABLE;
Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited

(CHAR(11)),

Reprinted for ibmkvskumar@in.ibm.com, IBM

Teradata Utilities: BTEQ, FastLoad, MultiLoad, TPump, and FastExport, Second Edition

Last_name (CHAR(20)), First_name (CHAR(14)), Class_code (CHAR(2)), Grade_Pt (CHAR(8)) FROM SQL_CLASS.Student_Table; /* Finish the Export Job and Write to File */

END THE JOB

.END EXPORT; .LOGOFF;

Figure 3-6

Reprinted for ibmkvskumar@in.ibm.com, IBM

Coffing Data Warehousing, Coffing Publishing (c) 2005, Copying Prohibited