Awk Programming

Awk
Programming
WHAT IS AWK?
created by: Aho, Weinberger, and Kernighan
scripting language used for manipulating data
and generating reports
versions of awk
awk,
nawk, mawk, pgawk,

GNU awk: gawk
WHAT CAN YOU DO WITH AWK?
awk operation:
scans
a file line by line

splits each input line into fields
compares input line/fields to pattern
performs action(s) on matched lines
Useful for:
transform
data files
produce formatted reports
Programming constructs:
format
output lines
arithmetic and string operations
conditionals and loops
THE COMMAND: AWK
BASIC AWK SYNTAX
awk [options] script file(s)
awk [options] f scriptfile file(s)
Options:
-F to change input field separator
-f to name script file
BASIC AWK PROGRAM
consists of patterns & actions:

pattern {action}
if
pattern is missing, action is applied to all lines

if action is missing, the matched line is printed
must have either pattern or action
Example:
awk '/for/' testfile
prints all lines containing string for in testfile
6
BASIC TERMINOLOGY: INPUT FILE

A field is a unit of data in a line
Each field is separated from the other fields by
the field separator
default
field separator is whitespace
A record is the collection of fields in a line

A data file is made up of records
EXAMPLE INPUT FILE
BUFFERS
awk supports two types of buffers:

record and field
field buffer:
one
for each fields in the current record.

names: $1, $2,
record buffer :
$0
holds the entire record

9
SOME SYSTEM VARIABLES

FS
RS
Field separator (default=whitespace)

Record separator (default=\n)
NF
NR
Number of fields in current record

Number of the current record
OFS
ORS
Output field separator (default=space)

Output record separator (default=\n)
FILENAME Current filename
10
EXAMPLE: RECORDS AND FIELDS

% cat emps
Tom Jones
Mary Adams
Sally Chang
Billy Black
%
1
2
3
4
4424
5346
1654
1683
5/12/66
11/4/63
7/22/54
9/23/44
543354
28765
650000
336500
awk '{print NR, $0}' emps

Tom Jones
4424
5/12/66
Mary Adams
5346
11/4/63
Sally Chang
1654
7/22/54
Billy Black
1683
9/23/44
543354
28765
650000
336500
11
EXAMPLE: SPACE AS FIELD

SEPARATOR
% cat emps
Tom Jones
Mary Adams
Sally Chang
Billy Black
%
1
2
3
4
4424
5346
1654
1683
5/12/66
11/4/63
7/22/54
9/23/44
543354
28765
650000
336500
awk '{print NR, $1, $2, $5}' emps

Tom Jones 543354
Mary Adams 28765
Sally Chang 650000
Billy Black 336500
12
EXAMPLE: COLON AS FIELD

SEPARATOR
% cat em2
Tom Jones:4424:5/12/66:543354
Mary Adams:5346:11/4/63:28765
Sally Chang:1654:7/22/54:650000
Billy Black:1683:9/23/44:336500
% awk -F: '/Jones/{print $1, $2}' em2
Tom Jones 4424
13
AWK SCRIPTS
awk scripts are divided into three major parts:
comment lines start with #
CSCI 330 - The UNIX System
14
AWK SCRIPTS
BEGIN: pre-processing
processing that must be completed before

the file processing starts (i.e., before awk starts
reading records from the input file)
useful for initialization tasks such as to initialize
variables and to create report headings
performs
15
AWK SCRIPTS
BODY: Processing
main processing logic to be applied to input
records
like a loop that processes input data one record at a
time:
if a file contains 100 records, the body will be executed 100

times, one for each record
contains
16
AWK SCRIPTS
END: post-processing
logic to be executed after all input data have

been processed
logic such as printing report grand total should be
performed in this part of the script
contains
17
PATTERN / ACTION SYNTAX
18
CATEGORIES OF PATTERNS
19
EXPRESSION PATTERN TYPES
match
input record
regular expression enclosed by /s
explicit pattern-matching expressions
~ (match), !~ (not match)
expression operators
entire
arithmetic
relational
logical
20
EXAMPLE: MATCH INPUT RECORD
% cat employees2
Tom Jones:4424:5/12/66:543354
Mary Adams:5346:11/4/63:28765
Sally Chang:1654:7/22/54:650000
Billy Black:1683:9/23/44:336500
% awk F: '/00$/' employees2
Sally Chang:1654:7/22/54:650000
Billy Black:1683:9/23/44:336500
21
EXAMPLE: EXPLICIT MATCH

% cat datafile
Charles Main
3.0
.98
34
western
Sharon Gray
5.3
.97
23
southwest SW
Lewis Dalsass
2.7
.8
18
southern
Suan Chin
5.1
.95
15
southeast SE
Patricia Hemenway
4.0
.7
17
eastern
TB Savage
4.4
.84
20
northeast NE
AM Main
5.1
.94
13
north
NO
Margot Weber
4.5
.89
central
CT
Ann Stephens
5.7
.94
13
WE
SO
EA
northwest NW
% awk '$5 ~ /\.[7-9]+/' datafile

southwest SW
Lewis Dalsass
2.7
.8
18
central
Ann Stephens
5.7
.94
13
CT
22
EXAMPLES: MATCHING WITH RES
% awk '/^[ns]/{print $1}' datafile

northwest
southwest
southern
southeast
northeast
north
% awk '$2 !~ /E/{print $1, $2}' datafile

northwest NW
southwest SW
southern SO
north NO
central CT
23
ARITHMETIC OPERATORS
Meaning
Add
Subtract
Multiply
Divide
Modulus
Exponential
Example
x+y
xy
x*y
x/y
x%y
x^y
Example:
% awk '$3 * $4 > 500 {print $0}' file
Operator
+
*
/
%
^
24
RELATIONAL OPERATORS
Meaning
Less than
Less than or equal
Equal to
Not equal to
Greater than
Greater than or equal to
Matched by reg exp
Not matched by req exp
Example
x<y
x<=y
x == y
x != y
x>y
x>=y
x ~ /y/
x !~ /y/
Operator
<
<=
==
!=
>
>=
~
!~
25
LOGICAL OPERATORS
Meaning
Logical AND
Logical OR
NOT
Example
a && b
a || b
!a
Examples:
% awk '($2 > 5) && ($2 <= 15)
{print $0}' file
% awk '$3 == 100 || $4 > 50' file
Operator
&&
||
!
26
RANGE PATTERNS
Matches ranges of consecutive input lines
pattern can be any simple pattern

pattern1 turns action on
pattern2 turns action off
Syntax:
pattern1 , pattern2 {action}
27
RANGE PATTERN EXAMPLE
28
AWK ACTIONS
29
AWK EXPRESSIONS
Expression is evaluated and returns value
Can involve variables
As part of expression evaluation

As target of assignment
consists of any combination of numeric and string

constants, variables, operators, functions, and
regular expressions
30
AWK VARIABLES
A user can define any number of variables within
an awk script
The variables can be numbers, strings, or arrays
Variable names start with a letter, followed by
letters, digits, and underscore
Variables come into existence the first time they
are referenced; therefore, they do not need to be
declared before use
All variables are initially created as strings and
initialized to a null string
31
AWK VARIABLES
Examples:
% awk '$1 ~ /Tom/
{wage = $3 * $4; print wage}'
filename
% awk '$4 == "CA"
{$4 = "California"; print $0}'
filename
Format:
variable = expression
32
AWK ASSIGNMENT OPERATORS

=
++
-+=
-=
*=
/=
%=
^=
assign result of right-hand-side expression to

left-hand-side variable
Add 1 to variable
Subtract 1 from variable
Assign result of addition
Assign result of subtraction
Assign result of multiplication
Assign result of division
Assign result of modulo
Assign result of exponentiation
33
AWK EXAMPLE
File: grades
john 85 92 78 94 88
andrea 89 90 75 90 86
jasper 84 88 80 92 84
awk script: average
# average five grades
{ total = $2 + $3 + $4 + $5 + $6
avg = total / 5
print $1, avg }
Run as:
awk f average grades
34
OUTPUT STATEMENTS
print
print easy and simple output
printf
print formatted (similar to C printf)
sprintf
format string (similar to C sprintf)
35
FUNCTION: PRINT
Writes to standard output
Output is terminated by ORS
ORS is newline
If called with no parameter, it will print $0

Printed parameters are separated by OFS,
default
OFS is blank
Print control characters are allowed:

\n
default
\f \a \t \\
36
PRINT EXAMPLE
% awk '{print $0}' grades

john 85 92 78 94 88
andrea 89 90 75 90 86
% awk '{print}' grades

john 85 92 78 94 88
andrea 89 90 75 90 86
% awk '{print($0)}' grades

john 85 92 78 94 88
andrea 89 90 75 90 86
37
PRINT EXAMPLE
% awk '{print $1 "," $2}' grades

john,85
andrea,89
% awk '{print $1, $2}' grades

john 85
andrea 89
38
PRINT EXAMPLE
% awk '{OFS="-";print $1 "," $2}' grades

john,85
andrea,89
% awk '{OFS="-";print $1 , $2}' grades

john-85
andrea-89
39
REDIRECTING PRINT OUTPUT

Print output goes to standard output
unless redirected via:
will open file or command only once

subsequent redirections append to already open
stream
> file
>> file
| command
40
PRINT EXAMPLE
% awk '{print $1 , $2 > "file"}' grades
% cat file
john 85
andrea 89
jasper 84
41
PRINT EXAMPLE
john 85
% awk '{print $1,$2 | "sort k 2"}' grades
jasper 84
john 85
andrea 89
% awk '{print $1,$2 | "sort"}' grades

andrea 89
jasper 84
42
PRINT EXAMPLE
% date |
awk '{print "Month: " $2 "\nYear: ", $6}'
Month: Nov
Year: 2008
% date
Wed Nov 19 14:40:07 CST 2008
43
PRINTF: FORMATTING OUTPUT

Syntax:
works
like C printf
each format specifier in format-string requires
argument of matching type
printf(format-string, var1, var2, )
44
FORMAT SPECIFIERS
decimal integer
single character
string of characters
floating point number
octal number
hexadecimal number
scientific floating point notation
the letter %
%d, %i
%c
%s
%f
%o
%x
%e
%%
45
FORMAT SPECIFIER EXAMPLES

Given: x = A, y = 15, z = 2.3, and $1 = Bob Smith
What it Does
%c
printf("The character is %c \n", x)

output: The character is A
%d
printf("The boy is %d years old \n", y)

output: The boy is 15 years old
%s
printf("My name is %s \n", $1)

output: My name is Bob Smith
%f
printf("z is %5.3f \n", z)

output: z is 2.300
Printf Format
Specifier
46
FORMAT SPECIFIER MODIFIERS

between % and letter
%10s
%7d
%10.4f
%-20s
meaning:
of field, field is printed right justified

precision: number of digits after decimal point
- will left justify
width
47
SPRINTF: FORMATTING TEXT

Syntax:
Works
like printf, but does not produce output

Instead it returns formatted string
Example:
{
sprintf(format-string, var1, var2, )
text = sprintf("1: %d 2: %d", $1, $2)

print text
}
48
AWK BUILTIN FUNCTIONS
Example: tolower("MiXeD cAsE 123")

returns "mixed case 123"
toupper(string)
returns a copy of string, with each lower-case
character converted to upper-case.
tolower(string)
returns a copy of string, with each upper-case
character converted to lower-case. Nonalphabetic
characters are left unchanged.
49
AWK EXAMPLE: LIST OF PRODUCTS
CSCI 330 The UNIX System
103:sway bar:49.99
101:propeller:104.99
104:fishing line:0.99
113:premium fish bait:1.00
106:cup holder:2.49
107:cooler:14.89
112:boat cover:120.00
109:transom:199.00
110:pulley:9.88
105:mirror:4.99
108:wheel:49.99
111:lock:31.00
102:trailer hitch:97.95
50
AWK EXAMPLE: OUTPUT
Marine Parts R Us
Main catalog
Part-id name
price
======================================
101
propeller
104.99
102
trailer hitch
97.95
103
sway bar
49.99
104
fishing line
0.99
105
mirror
4.99
106
cup holder
2.49
107
cooler
14.89
108
wheel
49.99
109
transom
199.00
110
pulley
9.88
111
lock
31.00
112
boat cover
120.00
113
premium fish bait
1.00
======================================
Catalog has 13 parts
51
AWK EXAMPLE: COMPLETE

BEGIN {
}
{
printf("%3d\t%-20s\t%6.2f\n", $1, $2, $3)
count++
}
END {
FS= ":"
print "Marine Parts R Us"
print "Main catalog"
print "Part-id\tname\t\t\t price"
print "======================================"
is output sorted ?
print "======================================"
print "Catalog has " count " parts"
}
52
AWK ARRAY
awk allows one-dimensional arrays
to store strings or numbers
index can be number or string
array need not be declared

its
size
its elements
array elements are created when first used

initialized
to 0 or
53
ARRAYS IN AWK
Examples:
list[1] = "one"
list[2] = "three"
Syntax:
arrayName[index] = value
list["other"] = "oh my !"

54
ILLUSTRATION: ASSOCIATIVE
ARRAYS
awk arrays can use string as index

55
AWK BUILTIN SPLIT FUNCTION

split(string, array, fieldsep)
string into pieces separated by fieldsep, and
stores the pieces in array
if the fieldsep is omitted, the value of FS is used.
Example:
split("auto-da-fe", a, "-")
sets the contents of the array a as follows:
a[1] = "auto"
a[2] = "da"
a[3] = "fe"
divides
56
EXAMPLE: PROCESS SALES DATA

input file:
output:
summary
of category sales
57
ILLUSTRATION: PROCESS EACH

INPUT LINE
58
ILLUSTRATION: PROCESS EACH

INPUT LINE
59
SUMMARY: AWK PROGRAM
60
EXAMPLE: COMPLETE PROGRAM
% cat sales.awk
{
deptSales[$2] += $3
}
END {
for (x in deptSales)
print x, deptSales[x]
}
% awk f sales.awk sales
61
DELETE ARRAY ENTRY
Format:
delete array_name [index]
Example:
The delete function can be used to delete an

element from an array.
delete deptSales["supplies"]
62
AWK CONTROL STRUCTURES
Conditional
Repetition
for
with counter
with array index
while
if-else
do-while
also:
break, continue
63
IF STATEMENT
Syntax:
Example:
if ( NR < 3 )
print $2
else
print $3
if (conditional expression)
statement-1
else
statement-2
64
FOR LOOP
Syntax:
Example:
for (i = 1; i <= NR; i++)
{
total += $i
count++
}
for (initialization; limit-test; update)

statement
65
FOR LOOP FOR ARRAYS

Syntax:
Example:
for (x in deptSales)
{
print x, deptSales[x]
}
for (var in array)

statement
66
WHILE LOOP
Syntax:
Example:
i = 1
while (i <= NF)
{
print i, $i
i++
}
while (logical expression)

statement
67
DO-WHILE LOOP
Syntax:
statement
while (condition)
statement is executed at least once, even if

condition is false at the beginning
Example:
i = 1
do {
print $0
i++
} while (i <= 10)
do
68
LOOP CONTROL STATEMENTS

break
exits loop
continue
skips rest of current iteration, continues with
next iteration
69
LOOP CONTROL EXAMPLE

20; x++) {
> 100) continue
x
< 0 ) break
for (x = 0; x <
if ( array[x]
printf "%d ",
if ( array[x]
}
70
EXAMPLE: SENSOR DATA

Temperature
Rainfall
Snowfall
Windspeed
Winddirection
also: sensor readings
Plan: print average readings in descending order
1
2
3
4
5
71
EXAMPLE: SENSOR READINGS
2008-10-01/1/68
2008-10-02/2/6
2007-10-03/3/4
2008-10-04/4/25
2008-10-05/5/120
2008-10-01/1/89
2007-10-01/4/35
2008-11-01/5/360
2008-10-01/1/45
2007-12-01/1/61
2008-10-10/1/32
72
EXAMPLE: PRINT SENSOR DATA
BEGIN {
printf("id\tSensor\n")
printf("----------------------\n")
}
{
printf("%d\t%s\n", $1, $2)
}
73
EXAMPLE: PRINT SENSOR

READINGS
BEGIN {
FS="/"
printf(" Date\t\tValue\n
printf("---------------------\n")
}
{
printf("%s
%7.2f\n", $1, $3)
}
74

SUMMARY
BEGIN {
FS="/"
}
{
sum[$2] += $3;
count[$2]++;
}
END {
for (i in sum) {
printf("%d %7.2f\n",i,sum[i]/count[i])
}
}
75
EXAMPLE: REMAINING TASKS

awk f sense.awk sensors readings
Sensor Average
2 input files
----------------------Winddirection 240.00
Temperature
59.00
Windspeed
30.00
sorted
Rainfall
6.00
Snowfall
4.00
sensor names
76

AVERAGES
Remaining tasks:
nature of input data
use: number of fields in record
substitute
sensor id with sensor name

use: associative array
recognize
sort
readings
use: sort gr k 2
77
EXAMPLE: SENSE.AWK
NF > 1 {
name[$1] = $2
}
NF < 2 {
split($0,fields,"/")
sum[fields[2]] += fields[3];
count[fields[2]]++;
}
END {
for (i in sum) {
printf("%15s %7.2f\n", name[i],
sum[i]/count[i]) | "sort -gr -k 2"
}
}
78

AVERAGES
Remaining tasks:
use: sort -gr
Substitute sensor id with sensor name
1. use:
join -j 1 sensor-data sensor-averages
Sort
2. within awk
79
EXAMPLE: SOLUTION 1 (1/3)
#! /bin/bash
trap '/bin/rm /tmp/report-*-$$; exit' 1 2 3
cat << HERE > /tmp/report-awk-1-$$
BEGIN {FS="/"}
{
sum[\$2] += \$3;
count[\$2]++;
}
END {
for (i in sum) {
printf("%d %7.2f\n", i, sum[i]/count[i])
}
}
HERE
80

BEGIN {
printf(" Sensor Average\n")
printf("-----------------------\n")
}
{
printf("%15s %7.2f\n", \$2, \$3)
}
HERE
81
join j 1 sensor-data /tmp/report-r-$$

> /tmp/report-t-$$
awk -f /tmp/report-awk-1-$$
sensor-readings |
sort > /tmp/report-r-$$
sort -gr -k 3 /tmp/report-t-$$ |

awk -f /tmp/report-awk-2-$$
/bin/rm /tmp/report-*-$$
82
EXAMPLE: OUTPUT
Sensor Average
----------------------Winddirection 240.00
Temperature
59.00
Windspeed
30.00
Rainfall
6.00
Snowfall
4.00
83

#! /bin/bash
trap '/bin/rm /tmp/report-*$$; exit' 1 2 3
NF > 1 {
name[\$1] = \$2
}
NF < 2 {
split(\$0,fields,"/")
sum[fields[2]] += fields[3];
count[fields[2]]++;
}
84
END {
for (i in sum) {
printf("%15s %7.2f\n", name[i],
sum[i]/count[i])
}
}
HERE
echo "
Sensor Average"
echo "-----------------------"
awk -f /tmp/report-awk-3-$$ sensor-data
sensor-readings | sort -gr -k 2
/bin/rm /tmp/report-*$$
85

Awk Programming

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Awk Programming

Uploaded by

Copyright:

Available Formats

Awk

nawk, mawk, pgawk,

WHAT CAN YOU DO WITH AWK?

a file line by line

THE COMMAND: AWK

BASIC AWK SYNTAX

awk [options] script file(s)

awk [options] f scriptfile file(s)

BASIC AWK PROGRAM

consists of patterns & actions:

pattern is missing, action is applied to all lines

BASIC TERMINOLOGY: INPUT FILE

field separator is whitespace

A record is the collection of fields in a line

EXAMPLE INPUT FILE

awk supports two types of buffers:

for each fields in the current record.

holds the entire record

SOME SYSTEM VARIABLES

Field separator (default=whitespace)

Number of fields in current record

Output field separator (default=space)

FILENAME Current filename

EXAMPLE: RECORDS AND FIELDS

awk '{print NR, $0}' emps

EXAMPLE: SPACE AS FIELD

awk '{print NR, $1, $2, $5}' emps

EXAMPLE: COLON AS FIELD

comment lines start with #

CSCI 330 - The UNIX System

processing that must be completed before

CSCI 330 - The UNIX System

main processing logic to be applied to input

if a file contains 100 records, the body will be executed 100

CSCI 330 - The UNIX System

logic to be executed after all input data have

CSCI 330 - The UNIX System

PATTERN / ACTION SYNTAX

CSCI 330 - The UNIX System

CSCI 330 - The UNIX System

EXPRESSION PATTERN TYPES

CSCI 330 - The UNIX System

EXAMPLE: MATCH INPUT RECORD

CSCI 330 - The UNIX System

EXAMPLE: EXPLICIT MATCH

CSCI 330 - The UNIX System

% awk '$5 ~ /\.[7-9]+/' datafile

EXAMPLES: MATCHING WITH RES

% awk '/^[ns]/{print $1}' datafile

CSCI 330 - The UNIX System

% awk '$2 !~ /E/{print $1, $2}' datafile

CSCI 330 - The UNIX System

CSCI 330 - The UNIX System

CSCI 330 - The UNIX System

Matches ranges of consecutive input lines

pattern can be any simple pattern

CSCI 330 - The UNIX System

RANGE PATTERN EXAMPLE

CSCI 330 - The UNIX System

CSCI 330 - The UNIX System

Expression is evaluated and returns value

Can involve variables

As part of expression evaluation

CSCI 330 - The UNIX System