DFSORT Beyond Sorting

Reformatting Features

BUILD, OVERLAY and IFTHEN

Using Build, we can reformat each record by specifying all

of its item one by one. We can delete, rearrange and insert fields and constants. An example of reformatting with OUTFIL and BUILD: OUTFIL FNAMES=OUT1,
BUILD=(21,20,5,3,PD,ADD,+20,TO=PD,LENGTH=3, 6C*',40:X)

COND.
Using OVERLAY, we can reformat each record by specifying just the items that overlay specific columns.
With this we can change specific existing columns without affecting the entire record. An example of reformatting with OUTREC and OVERLAY: OUTREC OVERLAY=(45:45,8,TRAN=LTOU)

COND
Using IFTHEN clauses, we can reformat different records

in different ways by specifying how BUILD or OVERLAY items are applied to records that meet given criteria. In IFTHEN clauses we can use conditional logic to choose different record types.
INREC IFTHEN=(WHEN=INIT,OVERLAY=(73:SEQNUM,8,ZD)),

IFTHEN=(WHEN=(4,5,CH,EQ,C'TYPE1'), BUILD=(1,40,61:C001',73:73,8)), IFTHEN=(WHEN=(4,5,CH,EQ,C'TYPE2'), BUILD=(1,30,61:C002',73:73,8)), IFTHEN=(WHEN=NONE,BUILD=(1,50,61:C000',73:73,8))

Fields and Constants

An example of rearranging fields and adding constants:
OUTREC BUILD=(5,2,5C*',18,6,C'ABC',+30,TO=BI, LENGTH=4,X01, 80:X)

The output records for the SORTOUT data set will contain the following: In output positions 1-2, the characters from input positions 5-6. In output positions 3-7, five asterisks. In output positions 8-13, the characters from input positions 18-23. In output positions 14-16, the characters ABC. In output positions 17-20, the decimal number +30 converted to a 4-byte

binary value (X'0000001E'). In output position 21, the hexadecimal value 01. Blanks in output positions 22-80.

Timestamps
An example of timestamps: INREC IFTHEN=(WHEN=(5,1,CH,EQ,C'A'),

BUILD=(2X,DATE=(MD4/),X,TIME,X,1,20)), IFTHEN=(WHEN=(5,1,CH,EQ,C'B'), BUILD=(2X,DATE=(DM4-),X,TIME=(24.),X,1,20))

The output records for the SORTOUT data set will contain the following: Blanks in positions 1-2. In output positions 3-12, the current date in the form 'mm/dd/yyyy' (e.g.

'03/28/2005') if position 5 has 'A', or in the form 'dd-mm-yyyy' (e.g. '28-032005') if position 5 has 'B'. A blank in position 13. In output positions 14-21, the current time in the form 'hh:mm:ss' (e.g. '11:05:31') if position 5 has 'A', or in the form 'hh.mm.ss' (e.g. '11.05.31') if position 5 has 'B'. A blank in position 22. In output positions 23-42, the characters from input positions 1-20.

Hexadecimal Display
An example of Hexadecimal display:

OUTFIL BUILD=(1,16,HEX,C**',17,16,HEX)
The output records for the SORTOUT data set will contain the following: In output positions 1-16, the hexadecimal representation of input positions 1-16.

Two asterisks in output positions 17-18.

In output positions 19-34, the hexadecimal representation of input positions 17-32.

You can even display an entire variable-length record, including the

variable part, in hexadecimal. OUTREC BUILD=(1,4,1,4,HEX,5,HEX)

The output records for the SORTOUT data set will contain the following: In output positions 1-4, the record descriptor word (RDW). In output positions 5-8, the hexadecimal representation of the RDW. Starting in output position 9, the hexadecimal representation of the variable bytes starting in input position 5.

Translation
Lowercase to uppercase translation:
INREC OVERLAY=(21:21,11,TRAN=LTOU,48:48,8, TRAN=LTOU)

Uppercase to lowercase translation:

OUTREC BUILD=(1,4,5,TRAN=UTOL)

Translation using an ALTSEQ table:

ALTSEQ CODE=(0040,5C40) OUTFIL BUILD=(1,20,21,30,TRAN=ALTSEQ,C'END')

Writing Constants
Character Strings

The format for writing a character string is:

Cx...x where x is an EBCDIC character. For example, C'FERN'. Hexadecimal Strings

The format for writing a hexadecimal string is:

X'yy...yy' where yy is a pair of hexadecimal digits. For ex: X'7FB0'. Decimal Numbers

The format for writing a decimal number is: n...n or n...n where n is a decimal digit. Examples are 24, +24, and -24.

Substring Search
Substring Search for INCLUDE and OMIT:

INCLUDE COND=(106,5,CH,EQ,CBIOL,OR, 106,5,CH,EQ,CHIST,OR, 106,5,CH,EQ,CBUSIN,OR,

106,5,CH,EQ,CPSYCH) Substring Search (SS format): INCLUDE COND=(106,5,SS,EQ,CBIOL ,HIST ,BUSIN,PSYCH) With substring search (SS format), you only write the field once and

write the character constant so it includes all of the strings you want to search for. If the value in the field matches any of the strings (for example, BUSIN), the record is included. If the value in the field does not match any of the strings, the record is omitted.

VB Data Set Considerations

Short control fields
If you know you have VB records with short compare fields, you can specify the VLSCMP option, if appropriate, to prevent DFSORT from terminating. For example: OPTION COPY,VLSCMP INCLUDE COND=(21,8,CH,EQ,CType 200)
VLSCMP tells DFSORT that you want to temporarily replace any missing compare field bytes with binary zeros, thus allowing the short fields to be validly compared (the zeros are not kept for the output records). In the example above, records less than 28 bytes are not included because the binary zeros added for the missing bytes in the field prevent it from being equal to 'Type 200'. Another way you can prevent DFSORT from terminating for VB records with short compare fields, if appropriate, is by specifying the VLSHRT option. For example:

OPTION COPY,VLSHRT INCLUDE COND=(21,8,CH,EQ,CType 200) VLSHRT tells DFSORT to leave records with short summary fields unsummed. That is, when one of the two records involved in a summary operation has a short summary field, the records are kept unchanged; neither record is deleted. In the example above, any records less than 28 bytes are not included.

Numeric Editing
You can use the following OUTREC statement to make

the PD values meaningful: OUTREC FIELDS=(1,8, 5X, 11,6,PD,M4, 5X, 31,7,PD,EDIT=(SI,III,III,IIT.TTT),SIGNS=(,-), 5X, 41,5,PD,EDIT=(TT-TT-TTTT))

COND
M4 is one of DFSORTs 27 pre-defined edit masks. It

edits a numeric field according to the pattern SI,III,III,III,IIT.TT. EDIT=(SI,III,III,IIT.TTT) is a userdefined edit mask. It edits a numeric field according to the pattern SI,III,III,IIT.TTT. EDIT=(TT-TT-TTTT) is another user-defined edit mask. It edits a numeric field according to the pattern TT-TT-TTTT.

COND
In the patterns: I indicates a leading insignificant digit to be displayed as

1-9, or as blank for a leading 0. T indicates a significant digit to be displayed as 0-9. S before the digits indicates a leading sign. S after the digits indicates a trailing sign. For M4, the leading sign is to be displayed as + for a positive value or as for a negative value. For EDIT=(SI,III,III,IIT.TTT),SIGNS=(,), the leading sign is to be displayed as blank for a positive value or as for a negative value. Any other character (for example, comma, decimal point or hyphen) is just displayed as appropriate. The results produced for the OUTREC statement are:

Numeric Conversion
An example of numeric conversion:
OUTREC BUILD=(21,5,ZD,TO=PDF,8,4,ZD,TO=FI,LENGTH=2)

The zoned decimal value in positions 21-25 of the

input record is converted to a packed decimal value in positions 1-3 of the output record (with an F for a positive sign or a D for a negative sign). The zoned decimal value in positions 8-11 of the input record is converted to a fixed-point value in positions 4-5 of the output record. .

Sequence Numbers
INREC, OUTREC and OUTFIL let us generate BI, PD, ZD or FS

sequence numbers in the output records. Starting values and increment values can be specified or defaulted.
An example of sequence numbers assigned before sorting with INREC:

INREC OVERLAY=(73:SEQNUM,8,ZD,START=1000,INCR=1000) SORT FIELDS=(25,5,CH,A) An example of sequence numbers assigned after sorting with OUTREC: SORT FIELDS=(25,5,CH,A) OUTREC OVERLAY=(73:SEQNUM,8,ZD,START=1000,INCR=100) We can also restart the sequence numbers at the START value each time the value in a specified field changes. An example with OUTFIL: OUTFIL OVERLAY=(81:SEQNUM,5,ZD,RESTART=(5,3))

Justifying(JFY)
Left-Justify:
OPTION COPY OUTREC FIELDS=(1,30,JFY=(SHIFT=LEFT))

Right-Justify:
OPTION COPY OUTREC FIELDS=(1,30,JFY=(SHIFT=RIGHT))

Leading, Trailing String:

OPTION COPY OUTREC FIELDS=(1,30,JFY=(SHIFT=LEFT,LEAD=C'<',TRAIL=C'>')) OPTION COPY OUTREC FIELDS=(1,30,JFY=(SHIFT=RIGHT,LEAD=C'<',TRAIL=C'>'))

Squeezing(SQZ)
Left Squeeze:

For a left-squeezed field, all blanks are removed and the

characters from the first nonblank to the last nonblank are shifted left, with blanks inserted on the right if needed. OPTION COPY OUTREC FIELDS=(1,40,SQZ=(SHIFT=LEFT))
OPTION COPY

OUTREC FIELDS=(1,40,SQZ=(SHIFT=LEFT, LEAD=C'<tag1>,TRAIL=C'</tag1>'))

Squeezing(SQZ)
Right Squeeze:

For a right-squeezed field, all blanks are removed and the

characters from the last nonblank to the first nonblank are shifted right, with blanks inserted on the left if needed. OPTION COPY OUTREC FIELDS=(1,40,SQZ=(SHIFT=RIGHT))
OPTION COPY

OUTREC FIELDS=(1,40,SQZ=(SHIFT=RIGHT, LEAD=C'<tag1>,TRAIL=C'</tag1>'))

Parsing Delimited Fields

This very powerful DFSORT function can be used to extract

delimited variable position/length data in many different forms into %nn fields, and then edit, convert, justify, squeeze, translate, lookup and change, or do arithmetic with the %nn fields.
Here's an example of parsing and reformatting comma separated values

(CSV): OUTREC PARSE=(%01=(ENDBEFR=C',',FIXLEN=8), %=(ENDBEFR=C','), %03=(ENDBEFR=C',',FIXLEN=5), %04=(FIXLEN=12)), BUILD=(%01,14:%03,TRAN=LTOU,25:%04,UFF,TO=FS,LENGTH=15)

Numeric and Non-Numeric Tests (NUM)

We can test a field for numerics (field,EQ,NUM) or

non-numerics (field,NE,NUM) in character (FS), zoned decimal (ZD) or packed decimal (PD) format. Examples: INCLUDE COND=(18,4,FS,NE,NUM) INCLUDE COND=(22,6,PD,EQ,NUM, AND,28,6,PD,EQ,NUM)

Splitting a File Contiguously (SPLIT1R)

SPLIT and SPLITBY=n rotate many times among the

output data sets, SPLIT1R=n only rotates once among the output data sets, resulting in contiguous records in each output data set. Examples: OUTFIL FNAMES=(OUT1,OUT2,OUT3),SPLIT OUTFIL FNAMES=(OUT1,OUT2,OUT3),SPLITBY=5 OUTFIL FNAMES=(OUT1,OUT2,OUT3),SPLIT1R=5

Symbols for Columns (sym:)

... //SYMNAMES DD

Col1,5 Col2,25 Col3,45 End_rec,80 First,28,10 Last,7,15 Amount,45,4,FS /* //SYSIN DD OPTION COPY OUTFIL REMOVECC, HEADER2=(Col1:'First',Col2:'Last',Col3:'Amount',/, Col1:10'-',Col2:15'-',Col3:6'-'), BUILD=(Col1:First,Col2:Last, Col3:Amount,EDIT=(IIT.TT),End_rec:X), TRAILER2=(Col1:5'=',Col3:6'=',/, Col1:'Total',Col3:TOT=(Amount,EDIT=(IIT.TT))) /*

ICEGENER: Make IEBGENER Jobs Fly

DFSORTs ICEGENER facility is a replacement for

IEBGENER. ICEGENER as a direct replacement for IEBGENER, so ICEGENER is then called automatically whenever IEBGENER is requested either directly or from a program. ICEGENER uses DFSORT to process IEBGENER jobs when possible and transfers control to IEBGENER when DFSORT can't be used. DFSORT will issue information such as the number of records copied and the RECFM, LRECL, and BLKSIZE.

Multiple Output (OUTFIL): Slash Elapsed Time, EXCPs and CPU Time
With OUTFIL, sort, merge and copy applications can

create multiple output data sets containing unedited or edited records, different ranges, subsets or samples of records, reports, and so on, from a single pass over one or more input data sets. We can see range of Performance improvements with DFSORT using a single sort step with OUTFIL vs DFSORT using a sort step followed by multiple copy steps

ICETOOL

INTRODUCTION
ICETOOL, a versatile data set processing and reporting

utility, provides an easy-to-use batch front-end for DFSORT. ICETOOL combines new features with previously available DFSORT features to perform complex sorting, copying, reporting and analytical tasks using multiple data sets in a single job step. The thirteen ICETOOL operators are: COPY, COUNT, DEFAULTS, DISPLAY, MODE, OCCUR, RANGE, SELECT, SORT, SPLICE, STATS, UNIQUE, and VERIFY. By using combinations of these thirteen operators, we can easily create applications that perform many complex tasks. By supplying these operators in a parameter list, we can use the information returned by ICETOOL in your program.

ICETOOL Return Codes

ICETOOL sets a return code for each operation it performs.

The return codes are:

0 - Successful completion. No errors were detected. 4 - Successful completion. DFSORT detected one or more warning

conditions.

12 - Unsuccessful completion. ICETOOL detected one or more

errors. Can also be set if the record count meets a specified criteria (for example, a data set is empty, or a data set contains more
than 50000 records).

16 - Unsuccessful completion. DFSORT detected one or more

errors. 20 - Message data set error. The TOOLMSG DD statement was not present or the TOOLMSG data set was not opened. 24 - Unsupported operating system. The operating system was not z/OS or OS/390.