UNIX Shell scripting

Netsoc
Stephen Shaw <stesh@netsoc.tcd.ie>

2011

Getting started

• SSH to one of our servers • PuTTY: Enter login.netsoc.tcd.ie as the hostname • Real operating systems: $ ssh you@login.netsoc.tcd.ie • NX to cube if you want - all you need is a shell though • No netsoc account? • CS: macneill.scss.tcd.ie • Maths servers? • Talk to an admin before you leave so you have an account for next time

UNIX

• Multi-user, multi-tasking operating system • Origins in the late ’60s - UNICS • The ancestor of many modern operating systems: • BSD • AIX • Solaris • Mac OS X

Kernels

• In most operating systems, the kernel acts as an interface

between the machine’s hardware and the application software running on it
• Linux is a kernel which was developed in the early ’90s to

provide a free alternative to proprietary kernels
• Generally the user doesn’t interact directly with the kernel

Shells

• A shell is a user-friendly, high-level ‘wrapper’ around the

kernel • Some shells:
• • • • •

sh bash ksh tcsh csh

• bash is one of the more popular shells • This talk will be based on
bash

chsh

• Are you using bash? •
echo $SHELL bash,

• If you’re not using
chsh -s /bin/bash

you can switch to it by running

• Log out, then log back in again

Your prompt
• You should see something like
1

stesh@cube :~$

• This is called the prompt • stesh - username • cube - hostname • ~ - current working directory • $ - privilege level • The format of the prompt is maintained in a variable called
PS1:
1 2

stesh@cube :~$ echo $PS1 ${debian_chroot :+( $debian_chroot )}\u@\h:\w\$ $

• We’ll use

as a shorthand for the prompt

Variables
• All variables in bash are strings • This is both a blessing and a curse • Variables are assigned values with • Variables are evaluated with
1 2 3 4 5 6 7

=

$

$ foo=bar $ echo $foo bar $ echo zanzi${foo} zanzibar $ echo "zanzi$foo" zanzibar

• No spaces around the equals - otherwise it’s ambiguous

(how?)

Special variables

• • • • • • • • •

$RANDOM: $$: $?: $!: $@:

random integer

current PID exit status of last process exited PID of last fork argv 0th to 9th argument current shell current user number of arguments

$0…$9: $#:

$SHELL: $USER:

Quotes
• Quotes are very important in shell scripts • Single quotes mean ‘literally’:
1 2 3

$ foo='bar ' $ echo '$foo ' $foo

• Double quotes cause variable names in strings to be replaced

with their values:
1 2 3

$ today="Monday" $ echo "today␣is␣$today" today is Monday

• This opens up interesting security issues

Backticks

• Enclose a string in backticks, and bash will execute it and

return a result:
1 2 3 4

echo $(whoami) stesh echo `uptime ` 05:07:59 up 120 days , 16:17 , 115 users , 0.46, 0.47, 0.42 $()

load average: →

can be easier to read

• But many older versions of many shells don’t support it

stdin, stdout, stderr

• Three standard data streams • • •
stdin: stdout: stderr:

Data going in (buffered) Data coming out (buffered) Warnings coming out (not buffered)

cat
• ‘concatenate’ • Copy
stdin

to

stdout

• Specify filenames as arguments, and cat will copy them to

stdout one by one
• Use it to concatenate files together • On some systems,
cat -n

adds line numbers to each line

printed on

1 2 3

stdout

tac

is like

cat,

but it prints in reverse order:

$ echo "Stephen\nShaw" | tac Shaw Stephen

Pipes

• Pipes make shell scripts really powerful • connect
1 2 3 4 5 6 7 8 9

stdout

of one process to

stdin

of another

$ ls /home | sort | head -n 5 # First five home folders by→ alphabetical order alxsoky andyrew arboroia at_god baran $ ps -ef | grep emacs | grep -v grep | wc -l # How many → emacs users are there ? 4

Redirects

• • • •
1 2

< foo > foo

feeds

stdin

from

foo foo foo

redirects

stdout

to

2> foo 2>&1

redirects

stderr

to

redirects

stderr

to

stdout

$ mysql < my_database_backup .mysql $ top > running_processes .mysql

Fun with redirects

• Silence error messages: • Record error messages:

find . 2> /dev/null find . 2>&1 | less

• Writing our first script quickly:
1 2 3 4 5 6

$ cat <<EOF > myscript.sh echo 'Hello netsoc!' EOF $ chmod +x myscript .sh $ ./ myscript.sh Hello netsoc!

Conditional execution

&&

for conjunction and

||

for disjunction

• shells are like most programming languages in that they

‘shortcut’ boolean expressions ∧ • F ∧ n pi ≡ F no matter what each pi is i=0
• T ∨ ∨n pi ≡ T no matter what each pi is i=0 • Abuse this to do conditional execution:
1 2 3 4 5 6 7

$ t r u e && echo "Hi␣$USER" Hi stesh $ f a l s e && echo "Hi␣$USER" $ t r u e || echo "Hi␣$USER" $ f a l s e || echo "Hi␣$USER" Hi stesh $ ./ configure && make

if

and exit codes

• Processes have exit codes • They tell you something about the status of the process when

it ended
• Success? Failure? • You exit a script with • •
1 2 3

exit

exit exit

followed by zero is ‘true’ followed by a non-zero positive integer is ‘false’

$ i f ( e x i t 0); then echo 'yay!'; f i yay! $ i f ( e x i t 1); then echo 'yay!'; f i

Traditional-style conditionals

• Programs have exit codes • So why not write a program which turns condition tests into

exit codes?
• •
1 2 3 4 5

is such a program. It tests conditions on strings, as well as characteristics of files
[ $ i f [ -e /home/stesh ]; then > ls /home/stesh > else > echo "oh␣no!␣my␣home␣directory␣is␣gone!" > fi

[ and logic

condition $p $p -a $q $p -o $q -z $str -n $str $a = $b $a != $b

true if $p is not true $p is true and $q is true $p is true or $q is true length of $str is zero length of $str is greater than zero $a and $b are equal $a and $b differ

[ and files
condition -e file -f file -d file -r file -w file -x file -p file true if file exists file exists file exists file exists file exists file exists file exists

and and and and and and

is is is is is is

a regular file a directory readable by me writable by me executable by me a pipe

• You have to be careful using these file tests • The condition is true as of when it was evaluated • Race conditions

Looping

1 2 3

while: w h i l e [ -e $lock ]; do > sleep 1 > done for

iterates over arguments separated by spaces
$()

• use
1 2 3 4 5 6 7 8

to make things more readable

f o r i in 1 2 3; do > echo $i > done 1 2 f o r file in $(ls); do > du -sh $i > done

Vocabulary

• Now let’s run through some fun programs we can glue

together into scripts

who
• who is logged in, and from where
1 2 3 4 5 6 7 8

$ who jgilbert bunburya stesh stesh scott arboroia ...

pts /225 pts /38 :1010 pts /231 :1006 :1016

2011 -10 -26 2011 -09 -27 2011 -07 -10 2011 -10 -26 2011 -08 -30 2011 -10 -25

21:41 23:58 19:37 00:11 16:56 14:51

(46.7.75.138) (:pts /14:S.0) (spoon:s.0) (:1026.0) (89.126.1.54) (10.6.17.72)

• When did we last boot?
1 2

$ who -b system boot 2011 -06 -27 12:51

• How many people are logged in?
1 2

$ who -q | grep "#" # users =130

w

• who is logged in, and what are they running?
1 2 3 4

$ w stesh pts /199 89.100.25.137 20:12 0.00s 0.06s 0.00s tmux a stesh pts /228 :1026.0 Tue23 24:14m 0.67s 0.61s ssh spoon stesh pts /230 :1026.0 Tue23 24:31m 0.06s 0.06s zsh

• different on BSD Unixes and solaris:
1 2 3 4

$ w USER TTY FROM LOGIN@ IDLE WHAT stesh console - Mer18 6:53 stesh s000 - Mer19 1 ssh cube w -h

removes the header

last

• Login histories
1 2 3 4 5 6 7

$ who mloc pts /129 104.76.534.53 Thu Oct 27 00:01 still logged → in bunburya pts /222 88.151.27.232 Wed Oct 26 23:17 still → logged in mloc pts /193 202.17.56.53 Wed Oct 26 21:35 gone - no → logout scott pts /58 89.116.2.54 Wed Oct 26 21:12 - 00:30 (02:12) t1 pts /129 109.76.162.99 Wed Oct 26 22:16 - 00:06 (01:50) ... /var/log/wtmp

• If

isn’t world-readable, this won’t work without

root

finger
• Look up information about a user
1 2 3 4 5 6 7 8 9 10

$ finger stesh Login: stesh Name: Stephen Shaw Directory: /home/stesh Shell: /usr/bin/→ zsh ... $ finger finger Login: finger Name: Kieran → Manning Directory: /home/finger Shell: /bin/bash ... $ finger stephen # finger everyone called 'Stephen ' $ finger -m stesh # finger stesh in more detail touch ~/.nofinger

• run

to prevent yourself getting fingered1

• Some servers still allow fingers across the network:
1 2

$ finger @maths.tcd.ie User Real Name Console Location
1

What

Idle

TTY

Host

but who doesn’t want to get fingered?

uptime

• How long we’ve been up, and what the load averages are
1 2

$ uptime 00:44:03 up 121 days , 11:53 , 130 users , 0.71, 0.62, 0.56

load average: →

ps
• Get information about the processes that are currently running •
ps

varies widely between operating systems

• GNU ps:
1 2 3

$ ps -e # all processes $ ps -U stesh # all stesh 's processes $ ps -f # full format

• BSD ps:
1 2

$ ps aux # all processes $ ps x # all my processes

• Example: harvest passwords from silly people who place them

on the command line:
1 2

$ w h i l e t r u e ;do ps -ef;done|grep "password"| grep -v grep mysql -u sillyperson --password=RxFLo3YpEd

xargs

• Read command-line arguments from

stdin

and pass them to

the specified program
1 2 3

$ ls ~ | xargs du -h # calculate sizes for my files $ find /srv/webspace/$USER - type d | xargs chmod 755 # fix→ webspace permissions $ find /srv/webspace/$USER - type f | xargs chmod 644 # fix→ webspace permissions

• if you don’t specify a program, prints an argument list on
stdout

cp

• Copy a file
1 2 3 4

$ cp /etc/motd.tail /etc/motd $ cp -r /etc /var/backups/etc # recursively copy a → directory $ cp -a ~/ Docs mnt/spoon # preserve access times and → ownership $ cp -v /home /mnt/backupdrive # notify on stderr when a → copy is made

mv

• Move a file
1 2 3 4

$ $ $ $

mv mv mv mv

/var/log/auth.log /var/log/auth.log.1 -i /etc/profile /etc/passwd # confirm before moving -n new.txt old.txt # don 't move if old.txt exists -v # notify on stderr when a move is made

rm, rmdir

• remove a file or directory
1 2 3 4

$ $ $ $

rm /bin/rm # oops rm -r ~/. Trash # recursively remove a directory rmdir ~/. Trash # remove a directory , fails if non -empty rm -rf --preserve -root / # Refuse to destroy slash

grep,fgrep

• Print lines in a file which match a regular expression
1 2 3 4 5 6 7 8

$ grep root /etc/passwd root:x:0:0: root :/ root :/bin/bash $ ps -e | grep tmux 3279 ? 00:06:15 tmux 4888 pts /183 00:00:00 tmux $ fgrep -i fail /var/log/auth.log # ignore case $ last | grep -v netsoc # reverse the match $ last | grep -e '(\d+) \.(\d+) \.(\d+) \.(\d+)' # use → extended regexes

wc

• Count things in a file
1 2 3 4 5

$ $ $ $ $

wc -l /var/log/sshd.log # count lines wc -m myfile.txt # count characters wc -b myfile.txt # count bytes mv -w myfile.txt # count tokens grep ":0:0" /etc/passwd | wc -l # toor?

Archiving and compressing


1 2 3 4 5 6 7

tar

- tape archive

$ tar -cf homebackup.tar /home/stesh # archive my home → directory $ tar -czf homebackup.tar /home/stesh # same , but with → compression $ tar -xf homebackup.tar # restore from an archive $ gzip access.log # compress a file $ gzip -9 access.log # highest compression level ( between → 1 and 9) $ gunzip access.log.gz # decompress $ zcat access.log.gz # decompress and output to stdout

pv

• ”Pipe viewer” • Just like
cat

except it draws a progress bar on

stderr

• Monitor the flow of data through a pipe:
1 2

$ pv backup.tgz | tar x 0O 0:00:05 [ 0B/s] [<=>

]

sed and tr
• sed - Stream editor • modify input line-by-line • a silly example: replace all the colons in
/etc/passwd

with

hyphens:
1

$ cat /etc/passwd | sed "s/:/-/g"

• tr - Transliterator • modify input character-by-character •
1 2

$ cat ls /home | tr '\n' ' ' # replace newlines with → spaces $ finger stephen | tr -s ' ' # 'squeeze ' multiple spaces → into one

head and tail

• Output the first and last few lines of a file •
1 2 3 4

$ $ $ $

man ssh | head head -n 5 /etc/shadow # first 5 lines last | tail -n 10 # last 10 lines tail -f /var/log/userweb.log # watch for new writes

sort

• Sort lines of input •
1 2 3 4 5

$ $ $ $ $

who | sort sort -g myfile # sort numerically sort -r myfile # reverse order sort -u myfile # don 't print duplicates df -h | sort -h # sort human - readable quantities (1G, 2K→ , etc .)

shuf

• Shuffle lines of input •
1 2 3

$ who | sort $ shuf /etc/passwd | head -n 1 | cut -d ':' -f 1 | # a → random user $ shuf /usr/share/dict/words | head -n 1 # a random word → from the dictionary

cut

• Tokenize lines of data on a given delimiter • modify input character-by-character •
1 2 3

$ cut -d ':' -f 1 /etc/passwd # list the usernames in /etc→ / passwd $ ps -ef | cut -d ' ' -f 2,3,4 # the second , third , and → forth space - delimited tokens $ cut -c 100 ~/. plan # the first 100 characters

comm, diff, uniq

• • • •
1 2 3

comm diff uniq

prints lines common to two files shows the differences between two files shows the unique lines in a file

$ ps -ef | cut -d ' ' -f 1 | sort | uniq $ comm /etc/ssh/ssh_config ~/. ssh/config $ diff myfile.txt myfile.txt.old comm

and

diff

work on adjacent lines only

• You get unexpected results if the input lines are not sorted

perl

• Perl is a general-purpose, interpreted programming language • It is used a lot in text processing and system administration • Very powerful regular expressions

Regular expressions for mathematicians

• Formal language theory • Mathematicians and computational linguists use regular

expressions to define regular sets
• The same expressive power as regular grammmars • All regular expressions have a generatively-equivalent

finite-state automaton
• This is usually irrelevant for the purpose of shell scripting • Use to match patterns in text • Can also perform limited amounts of parsing

Some regular expressions

Expression a . a* a+ a|b ab ab?

Recognizes a single occurrence of a a single occurrence of any character zero or more occurrences of a one or more occurrences of a a single occurrence a or of b (but not both) a single a followed by a single b a single a, optionally followed by a single b

cron
• • • •
1 2 3 4 5 6 7 8

cron lets crontab crontab crontab

you schedule tasks to run at particular times -l to view your cron table -e to edit your cron table -lu user to view user’s cron table (requires root)
command

$ crontab -l # m h dom mon dow

# hourly backups to spoon @hourly /home/stesh/bin/hourly -backups # daily backups from CS 30 4 * * * /home/stesh/bin/daily -backups

• It’s often good to end a cron entry with 2>&1 >/dev/null • Otherwise cron daemon will send emails about your cronjob • It is good manners not to schedule a big cron job during peak

hours
• Notice how my big daily backup job runs at 4:30 in the

morning

nc

• netcat • copy
1 2 3 4

stdin

to

stdout

over a network

$ cat myfile.txt | nc -lp 9999 # serve myfile .txt on port → 9999 $ nc localhost 9999 > myfile.txt.copy $ nc -z spoon.netsoc.tcd.ie 22 # is port 22 open on spoon? $ nc -z spoon 1 -1000 # which ports between 1 and 1000 are → open on spoon? nc

is useful in all sorts of situations

• ‘the TCP/IP swiss army knife’

Example: backup

• I want to upgrade a lot of packages on spoon, so I should take

a backup of

/etc/

in case something goes wrong.

• I need to store the backup on a remote machine • The remote machine isn’t as physically secure as spoon.

Example: backup

• Use

tar and gzip to consolidate compress it.

/etc

into an archive and

• Encrypt the archive using the GNU privacy guard (gpg) • Use
ssh

to transfer the file securely to the remote machine

• we can write a script to automate this

Example: backup

1 2 3 4 5 6 7 8 9 10 11

#!/ bin/bash s e t -e # die if any call exists with an exception ln -s $$ lock || e x i t 1 i f [ ! -e etcbackup.tgz ]; then tar -czf etcbackup.tgz /etc gpg -c etcbackup.tgz scp etcbackup.tgz.gpg prime.netsoc.tcd.ie: fi rm lock

• thoughts?

Example: backup
• thoughts? • locking is important in admin-style scripts, especially cronjobs • make sure at most one instance of the script can run at any

one time
• Be careful when using
[

file tests

• This implementation creates a few unnecessary files • We can condense it down to one line:
1 2

#!/ bin/bash tar -c /etc | gzip --best | gpg -c | ssh prime.netsoc.tcd.→ ie ">etcbackup.tgz.gpg" stdin

• We can have SSH accept

and pass it to stdout on the

remote end

Sign up to vote on this title
UsefulNot useful