UNIX Shell scripting

Netsoc
Stephen Shaw <stesh@netsoc.tcd.ie>

2011

Getting started

SSH to one of our servers PuTTY: Enter login.netsoc.tcd.ie as the hostname Real operating systems: $ ssh you@login.netsoc.tcd.ie NX to cube if you want - all you need is a shell though No netsoc account? CS: macneill.scss.tcd.ie Maths servers? Talk to an admin before you leave so you have an account for next time

UNIX

Multi-user, multi-tasking operating system Origins in the late 60s - UNICS The ancestor of many modern operating systems: BSD AIX Solaris Mac OS X

Kernels

In most operating systems, the kernel acts as an interface

between the machines hardware and the application software running on it

Linux is a kernel which was developed in the early 90s to

provide a free alternative to proprietary kernels

Generally the user doesnt interact directly with the kernel

Shells

A shell is a user-friendly, high-level wrapper around the

kernel Some shells:

sh bash ksh tcsh csh

bash is one of the more popular shells This talk will be based on
bash

chsh

Are you using bash?

echo $SHELL bash,

If youre not using

chsh -s /bin/bash

you can switch to it by running

Log out, then log back in again

Your prompt
You should see something like
1

stesh@cube :~$

This is called the prompt stesh - username cube - hostname ~ - current working directory $ - privilege level The format of the prompt is maintained in a variable called
PS1:
1 2

stesh@cube :~$ echo $PS1 ${debian_chroot :+( $debian_chroot )}\u@\h:\w\$ $

Well use

as a shorthand for the prompt

Variables
All variables in bash are strings This is both a blessing and a curse Variables are assigned values with Variables are evaluated with
1 2 3 4 5 6 7

$ foo=bar $ echo $foo bar $ echo zanzi${foo} zanzibar $ echo "zanzi$foo" zanzibar

No spaces around the equals - otherwise its ambiguous

(how?)

Special variables

$RANDOM: $$: $?: $!: $@:

random integer

current PID exit status of last process exited PID of last fork argv 0th to 9th argument current shell current user number of arguments

$0$9: $#:

$SHELL: $USER:

Quotes
Quotes are very important in shell scripts Single quotes mean literally:
1 2 3

$ foo='bar ' $ echo '$foo ' $foo

Double quotes cause variable names in strings to be replaced

with their values:

1 2 3

$ today="Monday" $ echo "todayis$today" today is Monday

This opens up interesting security issues

Backticks

Enclose a string in backticks, and bash will execute it and

return a result:
1 2 3 4

echo $(whoami) stesh echo `uptime ` 05:07:59 up 120 days , 16:17 , 115 users , 0.46, 0.47, 0.42 $()

load average:

can be easier to read

But many older versions of many shells dont support it

stdin, stdout, stderr

Three standard data streams

stdin: stdout: stderr:

Data going in (buered) Data coming out (buered) Warnings coming out (not buered)

cat
concatenate Copy
stdin

to

stdout

Specify lenames as arguments, and cat will copy them to

stdout one by one

Use it to concatenate les together On some systems,
cat -n

adds line numbers to each line

printed on

1 2 3

stdout

tac

is like

cat,

but it prints in reverse order:

$ echo "Stephen\nShaw" | tac Shaw Stephen

Pipes

Pipes make shell scripts really powerful connect

1 2 3 4 5 6 7 8 9

stdout

of one process to

stdin

of another

$ ls /home | sort | head -n 5 # First five home folders by alphabetical order alxsoky andyrew arboroia at_god baran $ ps -ef | grep emacs | grep -v grep | wc -l # How many emacs users are there ? 4

Redirects

1 2

< foo > foo

feeds

stdin

from

foo foo foo

redirects

stdout

to

2> foo 2>&1

redirects

stderr

to

redirects

stderr

to

stdout

$ mysql < my_database_backup .mysql $ top > running_processes .mysql

Fun with redirects

Silence error messages: Record error messages:

find . 2> /dev/null find . 2>&1 | less

Writing our rst script quickly:

1 2 3 4 5 6

$ cat <<EOF > myscript.sh echo 'Hello netsoc!' EOF $ chmod +x myscript .sh $ ./ myscript.sh Hello netsoc!

Conditional execution

&&

for conjunction and

||

for disjunction

shells are like most programming languages in that they

shortcut boolean expressions F n pi F no matter what each pi is i=0

T n pi T no matter what each pi is i=0 Abuse this to do conditional execution:
1 2 3 4 5 6 7

$ t r u e && echo "Hi$USER" Hi stesh $ f a l s e && echo "Hi$USER" $ t r u e || echo "Hi$USER" $ f a l s e || echo "Hi$USER" Hi stesh $ ./ configure && make

if

and exit codes

Processes have exit codes They tell you something about the status of the process when

it ended
Success? Failure? You exit a script with
1 2 3

exit

exit exit

followed by zero is true followed by a non-zero positive integer is false

$ i f ( e x i t 0); then echo 'yay!'; f i yay! $ i f ( e x i t 1); then echo 'yay!'; f i

Traditional-style conditionals

Programs have exit codes So why not write a program which turns condition tests into

exit codes?

1 2 3 4 5

is such a program. It tests conditions on strings, as well as characteristics of les

[ $ i f [ -e /home/stesh ]; then > ls /home/stesh > else > echo "ohno!myhomedirectoryisgone!" > fi

[ and logic

condition $p $p -a $q $p -o $q -z $str -n $str $a = $b $a != $b

true if $p is not true $p is true and $q is true $p is true or $q is true length of $str is zero length of $str is greater than zero $a and $b are equal $a and $b dier

[ and les
condition -e file -f file -d file -r file -w file -x file -p file true if file exists file exists file exists file exists file exists file exists file exists

and and and and and and

is is is is is is

a regular le a directory readable by me writable by me executable by me a pipe

You have to be careful using these le tests The condition is true as of when it was evaluated Race conditions

Looping

1 2 3

while: w h i l e [ -e $lock ]; do > sleep 1 > done for

iterates over arguments separated by spaces

$()

use
1 2 3 4 5 6 7 8

to make things more readable

f o r i in 1 2 3; do > echo $i > done 1 2 f o r file in $(ls); do > du -sh $i > done

Vocabulary

Now lets run through some fun programs we can glue

together into scripts

who
who is logged in, and from where
1 2 3 4 5 6 7 8

$ who jgilbert bunburya stesh stesh scott arboroia ...

pts /225 pts /38 :1010 pts /231 :1006 :1016

2011 -10 -26 2011 -09 -27 2011 -07 -10 2011 -10 -26 2011 -08 -30 2011 -10 -25

21:41 23:58 19:37 00:11 16:56 14:51

(46.7.75.138) (:pts /14:S.0) (spoon:s.0) (:1026.0) (89.126.1.54) (10.6.17.72)

When did we last boot?

1 2

$ who -b system boot 2011 -06 -27 12:51

How many people are logged in?

1 2

$ who -q | grep "#" # users =130

who is logged in, and what are they running?

1 2 3 4

$ w stesh pts /199 89.100.25.137 20:12 0.00s 0.06s 0.00s tmux a stesh pts /228 :1026.0 Tue23 24:14m 0.67s 0.61s ssh spoon stesh pts /230 :1026.0 Tue23 24:31m 0.06s 0.06s zsh

dierent on BSD Unixes and solaris:

1 2 3 4

$ w USER TTY FROM LOGIN@ IDLE WHAT stesh console - Mer18 6:53 stesh s000 - Mer19 1 ssh cube w -h

removes the header

last

Login histories
1 2 3 4 5 6 7

$ who mloc pts /129 104.76.534.53 Thu Oct 27 00:01 still logged in bunburya pts /222 88.151.27.232 Wed Oct 26 23:17 still logged in mloc pts /193 202.17.56.53 Wed Oct 26 21:35 gone - no logout scott pts /58 89.116.2.54 Wed Oct 26 21:12 - 00:30 (02:12) t1 pts /129 109.76.162.99 Wed Oct 26 22:16 - 00:06 (01:50) ... /var/log/wtmp

If

isnt world-readable, this wont work without

root

finger
Look up information about a user
1 2 3 4 5 6 7 8 9 10

$ finger stesh Login: stesh Name: Stephen Shaw Directory: /home/stesh Shell: /usr/bin/ zsh ... $ finger finger Login: finger Name: Kieran Manning Directory: /home/finger Shell: /bin/bash ... $ finger stephen # finger everyone called 'Stephen ' $ finger -m stesh # finger stesh in more detail touch ~/.nofinger

run

to prevent yourself getting ngered1

Some servers still allow ngers across the network:

1 2

$ finger @maths.tcd.ie User Real Name Console Location

1

What

Idle

TTY

Host

but who doesnt want to get ngered?

uptime

How long weve been up, and what the load averages are
1 2

$ uptime 00:44:03 up 121 days , 11:53 , 130 users , 0.71, 0.62, 0.56

load average:

ps
Get information about the processes that are currently running
ps

varies widely between operating systems

GNU ps:
1 2 3

$ ps -e # all processes $ ps -U stesh # all stesh 's processes $ ps -f # full format

BSD ps:
1 2

$ ps aux # all processes $ ps x # all my processes

Example: harvest passwords from silly people who place them

on the command line:

1 2

$ w h i l e t r u e ;do ps -ef;done|grep "password"| grep -v grep mysql -u sillyperson --password=RxFLo3YpEd

xargs

Read command-line arguments from

stdin

and pass them to

the specied program

1 2 3

$ ls ~ | xargs du -h # calculate sizes for my files $ find /srv/webspace/$USER - type d | xargs chmod 755 # fix webspace permissions $ find /srv/webspace/$USER - type f | xargs chmod 644 # fix webspace permissions

if you dont specify a program, prints an argument list on

stdout

cp

Copy a le
1 2 3 4

$ cp /etc/motd.tail /etc/motd $ cp -r /etc /var/backups/etc # recursively copy a directory $ cp -a ~/ Docs mnt/spoon # preserve access times and ownership $ cp -v /home /mnt/backupdrive # notify on stderr when a copy is made

mv

Move a le
1 2 3 4

$ $ $ $

mv mv mv mv

/var/log/auth.log /var/log/auth.log.1 -i /etc/profile /etc/passwd # confirm before moving -n new.txt old.txt # don 't move if old.txt exists -v # notify on stderr when a move is made

rm, rmdir

remove a le or directory
1 2 3 4

$ $ $ $

rm /bin/rm # oops rm -r ~/. Trash # recursively remove a directory rmdir ~/. Trash # remove a directory , fails if non -empty rm -rf --preserve -root / # Refuse to destroy slash

grep,fgrep

Print lines in a le which match a regular expression

1 2 3 4 5 6 7 8

$ grep root /etc/passwd root:x:0:0: root :/ root :/bin/bash $ ps -e | grep tmux 3279 ? 00:06:15 tmux 4888 pts /183 00:00:00 tmux $ fgrep -i fail /var/log/auth.log # ignore case $ last | grep -v netsoc # reverse the match $ last | grep -e '(\d+) \.(\d+) \.(\d+) \.(\d+)' # use extended regexes

wc

Count things in a le
1 2 3 4 5

$ $ $ $ $

wc -l /var/log/sshd.log # count lines wc -m myfile.txt # count characters wc -b myfile.txt # count bytes mv -w myfile.txt # count tokens grep ":0:0" /etc/passwd | wc -l # toor?

Archiving and compressing

1 2 3 4 5 6 7

tar

- tape archive

$ tar -cf homebackup.tar /home/stesh # archive my home directory $ tar -czf homebackup.tar /home/stesh # same , but with compression $ tar -xf homebackup.tar # restore from an archive $ gzip access.log # compress a file $ gzip -9 access.log # highest compression level ( between 1 and 9) $ gunzip access.log.gz # decompress $ zcat access.log.gz # decompress and output to stdout

pv

Pipe viewer Just like

cat

except it draws a progress bar on

stderr

Monitor the ow of data through a pipe:

1 2

$ pv backup.tgz | tar x 0O 0:00:05 [ 0B/s] [<=>

sed and tr
sed - Stream editor modify input line-by-line a silly example: replace all the colons in
/etc/passwd

with

hyphens:
1

$ cat /etc/passwd | sed "s/:/-/g"

tr - Transliterator modify input character-by-character

1 2

$ cat ls /home | tr '\n' ' ' # replace newlines with spaces $ finger stephen | tr -s ' ' # 'squeeze ' multiple spaces into one

head and tail

Output the rst and last few lines of a le

1 2 3 4

$ $ $ $

man ssh | head head -n 5 /etc/shadow # first 5 lines last | tail -n 10 # last 10 lines tail -f /var/log/userweb.log # watch for new writes

sort

Sort lines of input

1 2 3 4 5

$ $ $ $ $

who | sort sort -g myfile # sort numerically sort -r myfile # reverse order sort -u myfile # don 't print duplicates df -h | sort -h # sort human - readable quantities (1G, 2K , etc .)

shuf

Shue lines of input

1 2 3

$ who | sort $ shuf /etc/passwd | head -n 1 | cut -d ':' -f 1 | # a random user $ shuf /usr/share/dict/words | head -n 1 # a random word from the dictionary

cut

Tokenize lines of data on a given delimiter modify input character-by-character

1 2 3

$ cut -d ':' -f 1 /etc/passwd # list the usernames in /etc / passwd $ ps -ef | cut -d ' ' -f 2,3,4 # the second , third , and forth space - delimited tokens $ cut -c 100 ~/. plan # the first 100 characters

comm, diff, uniq

1 2 3

comm diff uniq

prints lines common to two les shows the dierences between two les shows the unique lines in a le

$ ps -ef | cut -d ' ' -f 1 | sort | uniq $ comm /etc/ssh/ssh_config ~/. ssh/config $ diff myfile.txt myfile.txt.old comm

and

diff

work on adjacent lines only

You get unexpected results if the input lines are not sorted

perl

Perl is a general-purpose, interpreted programming language It is used a lot in text processing and system administration Very powerful regular expressions

Regular expressions for mathematicians

Formal language theory Mathematicians and computational linguists use regular

expressions to dene regular sets

The same expressive power as regular grammmars All regular expressions have a generatively-equivalent

nite-state automaton
This is usually irrelevant for the purpose of shell scripting Use to match patterns in text Can also perform limited amounts of parsing

Some regular expressions

Expression a . a* a+ a|b ab ab?

Recognizes a single occurrence of a a single occurrence of any character zero or more occurrences of a one or more occurrences of a a single occurrence a or of b (but not both) a single a followed by a single b a single a, optionally followed by a single b

cron

1 2 3 4 5 6 7 8

cron lets crontab crontab crontab

you schedule tasks to run at particular times -l to view your cron table -e to edit your cron table -lu user to view users cron table (requires root)
command

$ crontab -l # m h dom mon dow

# hourly backups to spoon @hourly /home/stesh/bin/hourly -backups # daily backups from CS 30 4 * * * /home/stesh/bin/daily -backups

Its often good to end a cron entry with 2>&1 >/dev/null Otherwise cron daemon will send emails about your cronjob It is good manners not to schedule a big cron job during peak

hours
Notice how my big daily backup job runs at 4:30 in the

morning

nc

netcat copy
1 2 3 4

stdin

to

stdout

over a network

$ cat myfile.txt | nc -lp 9999 # serve myfile .txt on port 9999 $ nc localhost 9999 > myfile.txt.copy $ nc -z spoon.netsoc.tcd.ie 22 # is port 22 open on spoon? $ nc -z spoon 1 -1000 # which ports between 1 and 1000 are open on spoon? nc

is useful in all sorts of situations

the TCP/IP swiss army knife

Example: backup

I want to upgrade a lot of packages on spoon, so I should take

a backup of

/etc/

in case something goes wrong.

I need to store the backup on a remote machine The remote machine isnt as physically secure as spoon.

Example: backup

Use

tar and gzip to consolidate compress it.

/etc

into an archive and

Encrypt the archive using the GNU privacy guard (gpg) Use
ssh

to transfer the le securely to the remote machine

we can write a script to automate this

Example: backup

1 2 3 4 5 6 7 8 9 10 11

#!/ bin/bash s e t -e # die if any call exists with an exception ln -s $$ lock || e x i t 1 i f [ ! -e etcbackup.tgz ]; then tar -czf etcbackup.tgz /etc gpg -c etcbackup.tgz scp etcbackup.tgz.gpg prime.netsoc.tcd.ie: fi rm lock

thoughts?

Example: backup
thoughts? locking is important in admin-style scripts, especially cronjobs make sure at most one instance of the script can run at any

one time
Be careful when using
[

le tests

This implementation creates a few unnecessary les We can condense it down to one line:
1 2

#!/ bin/bash tar -c /etc | gzip --best | gpg -c | ssh prime.netsoc.tcd. ie ">etcbackup.tgz.gpg" stdin

We can have SSH accept

and pass it to stdout on the

remote end