11/26/23, 7:10 PM A practical guide to learning awk | Opensource.

com

LO G I N 

A practical guide to learning awk

Get a better handle on the awk command by downloading our
free eBook.

By Seth Kenlon (Team, Red Hat)

September 3, 2020 | 1 Comment | 7 min read 129 readers like this.

Image by: Image from Unsplash.com, Creative Commons Zero

Of all the Linux commands out there (and there are many), the three most
quintessential seem to be sed, awk, and grep. Maybe it's the arcane sound of their
https://opensource.com/article/20/9/awk-ebook 1/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

names, or the breadth of their potential use, or just their age, but when someone's
giving an example of a "Linuxy" command, it's usually one of those three. And while
sed and grep have several simple one-line standards, the less prestigious awk
remains persistently prominent for being particularly puzzling.

You're likely to use sed for a quick string replacement or grep to filter for a pattern on
a daily basis. You're far less likely to compose an awk command. I often wonder why
this is, and I attribute it to a few things. First of all, many of us barely use sed and
grep for anything but some variation upon these two commands:

$ sed -e 's/foo/bar/g' file.txt

$ grep foo file.txt

More Linux resources

Linux commands cheat sheet

Advanced Linux commands cheat sheet

Free online course: RHEL Technical Overview

Linux networking cheat sheet

SELinux cheat sheet

Linux common commands cheat sheet

What are Linux containers?

Our latest Linux articles

So, even though you might feel more comfortable with sed and grep, you may not
use their full potential. Of course, there's no obligation to learn more about sed or
grep, but I sometimes wonder about the way I "learn" commands. Instead of learning
how a command works, I often learn a specific incantation that includes a command.
As a result, I often feel a false familiarity with the command. I think I know a command

https://opensource.com/article/20/9/awk-ebook 2/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

because I can name three or four options off the top of my head, even though I don't
know what the options do and can't quite put my finger on the syntax.

And that's the problem, I believe, that many people face when confronted with the
power and flexibility of awk.

Learning awk to use awk

The basics of awk are surprisingly simple. It's often noted that awk is a programming
language, and although it's a relatively basic one, it's true. This means you can learn
awk the same way you learn a new coding language: learn its syntax using some basic
commands, learn its vocabulary so you can build up to complex actions, and then
practice, practice, practice.

How awk parses input

Awk sees input, essentially, as an array. When awk scans over a text file, it treats each
line, individually and in succession, as a record. Each record is broken into fields. Of
course, awk must keep track of this information, and you can see that data using the
NR (number of records) and NF (number of fields) built-in variables. For example, this
gives you the line count of a file:

$ awk 'END { print NR;}' example.txt

36

This also reveals something about awk syntax. Whether you're writing awk as a one-
liner or as a self-contained script, the structure of an awk instruction is:

pattern or keyword { actions }

In this example, the word END is a special, reserved keyword rather than a pattern. A
similar keyword is BEGIN. With both of these keywords, awk just executes the action in
braces at the start or end of parsing data.

You can use a pattern as a filter or qualifier so that awk only executes a given action
when it is able to match your pattern to the current record. For instance, suppose you
https://opensource.com/article/20/9/awk-ebook 3/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

want to use awk, much as you would grep, to find the word Linux in a file of text:

$ awk '/Linux/ { print $0; }' os.txt

OS: CentOS Linux (10.1.1.8)
OS: CentOS Linux (10.1.1.9)
OS: Red Hat Enterprise Linux (RHEL) (10.1.1.11)
OS: Elementary Linux (10.1.2.4)
OS: Elementary Linux (10.1.2.5)
OS: Elementary Linux (10.1.2.6)

For awk, each line in the file is a record, and each word in a record is a field. By default,
fields are separated by a space. You can change that with the --field-separator
option, which sets the FS (field separator) variable to whatever you want it to be:

$ awk --field-separator ':' '/Linux/ { print $2; }' os.txt

CentOS Linux (10.1.1.8)
CentOS Linux (10.1.1.9)
Red Hat Enterprise Linux (RHEL) (10.1.1.11)
Elementary Linux (10.1.2.4)
Elementary Linux (10.1.2.5)
Elementary Linux (10.1.2.6)

In this sample, there's an empty space before each listing because there's a blank
space after each colon (:) in the source text. This isn't cut, though, so the field
separator needn't be limited to one character:

$ awk --field-separator ': ' '/Linux/ { print $2; }' os.txt

CentOS Linux (10.1.1.8)
CentOS Linux (10.1.1.9)
Red Hat Enterprise Linux (RHEL) (10.1.1.11)
Elementary Linux (10.1.2.4)
Elementary Linux (10.1.2.5)
Elementary Linux (10.1.2.6)

Functions in awk
You can build your own functions in awk using this syntax:
https://opensource.com/article/20/9/awk-ebook 4/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

name(parameters) { actions }

Functions are important because they allow you to write code once and reuse it
throughout your work. When constructing one-liners, custom functions are a little
less useful than they are in scripts, but awk defines many functions for you already.
They work basically the same as any function in any other language or spreadsheet:
You learn the order that the function needs information from you, and you can feed it
whatever you want to get the results.

There are functions to perform mathematical operations and string processing. The
math ones are often fairly straightforward. You provide a number, and it crunches it:

$ awk 'BEGIN { print sqrt(1764); }'

42

String functions can be more complex but are well documented in the GNU awk
manual. For example, the split function takes an entity that awk views as a single
field and splits it into different parts. It requires a field, a variable to use as an array
containing each part of the split, and the character you want to use as the delimiter.

Using the output of the previous examples, I know that there's an IP address at the
very end of each record. In this case, I can send just the last field of a record to the
split function by referencing the variable NF because it contains the number of
fields (and the final field must be the highest number):

$ awk --field-separator ': ' '/Linux/ { split($NF, IP, "."); p

subnet: 1
subnet: 1
subnet: 1
subnet: 2
subnet: 2
subnet: 2

https://opensource.com/article/20/9/awk-ebook 5/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

There are many more functions, and there's no reason to limit yourself to one per
block of awk code. You can construct complex pipelines with awk in your terminal, or
you can write awk scripts to define and utilize your own functions.

Download the eBook

Learning awk is mostly a matter of using awk. Use it even if it means duplicating
functionality you already have with sed or grep or cut or tr or any other perfectly
valid commands. Once you get comfortable with it, you can write Bash functions that
invoke your custom awk commands for easier use. And eventually, you'll be able to
write scripts to parse complex datasets.

Download our eBook to learn everything you need to know about awk, and start
using it today.

What to read next

Drinking coffee with AWK

Keep track of what your office mates owe for the coffee they drink with a simple AWK
program.

https://opensource.com/article/20/9/awk-ebook 6/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

Moshe Zadka (Correspondent)

What's in an open source name?

Ever wonder where the names of your favorite open source projects or programming
languages came from? Get the origin stories behind popular tech nomenclature from
A to Z.

Lauren Pritchett (Team, Red Hat)

Don Watkins (Correspondent)

Joshua Allen Holm (Alumni)

Mike Bursell (Alumni)

https://opensource.com/article/20/9/awk-ebook 7/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

20 Linux commands every sysadmin should know

If your application isn't working—or you're just looking for more information—these 20
commands will come in handy.

Rosemary Wang

Tags: LINUX PROGRAMMING

Seth Kenlon
Seth Kenlon is a UNIX geek, free culture advocate,
independent multimedia artist, and D&D nerd. He has
worked in the film and computing industry, often at the
same time.

https://opensource.com/article/20/9/awk-ebook 8/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

More about me

1 Comment
These comments are closed.

Stuart Ferguson | September 3, 2020

No readers like this yet.

Thanks a lot for the article. I think Awk is more useful (and powerful) than
many people realise, especially since the release of GNU Awk 5.0 (Gawk).
The addition of namespaces in Gawk means that it is now easy to construct
function libraries to be called as part of large, well-structured Awk programs.
Awk is a seductive language, once you spend a little time learning it. For me
as an engineer, Awk makes reading in data from files -- even those with
millions of lines -- so simple that I can focus my time on developing the
algorithms for data processing. I probably do more programming in Awk than
any other language these days, which will probably raise some eyebrows. If
required, the translation from Awk to JavaScript (especially) is fairly trivial --
almost entirely achievable using some simple regex replacements.
Awk might be old but I think it has been rejuvenated by the improvements in
Gawk during the last few years. I have grown to love Awk!

Related Content

https://opensource.com/article/20/9/awk-ebook 9/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

What's new in 5 reasons virtual Remove the

GNOME 44? machines still matter background from an
image with this Linux
command

This work is licensed under a Creative Commons Attribution-Share Alike

4.0 International License.

ABOUT THIS SITE

The opinions expressed on this website are those of each author, not of the author's
employer or of Red Hat.

Opensource.com aspires to publish all content under a Creative Commons

license but may not be able to do so in all cases. You are responsible for ensuring
that you have the necessary permission to reuse any work on this site. Red Hat and
the Red Hat logo are trademarks of Red Hat, Inc., registered in the United States
and other countries.

A note on advertising: Opensource.com does not sell advertising on the site or in

any of its newsletters.

Copyright ©2023 Red Hat, Inc.

https://opensource.com/article/20/9/awk-ebook 10/11
11/26/23, 7:10 PM A practical guide to learning awk | Opensource.com

Privacy Policy

Terms of use

https://opensource.com/article/20/9/awk-ebook 11/11