BeginnersShellScripting

for
BatchJobs
EvanBollig
and

GeoffreyWomeldorff

Beforewebegin...
Everyonepleasevisitthispageforexample
scriptsandgrabacribsheetfromthefront
http://www.scs.fsu.edu/~bollig/TechSeries

Alsopleasesigninontheclipboardbeing
passedaround

Outline
IntroductiontoShells
History
Whatisthe*nixENV?
WhatisShellScripting?

NoviceScripting
HelloWorld(HelloWorld.sh)
Variables(HelloWorld2.sh)
`...`,'...'and...(Example3.sh)

Outline(2)

IntermediateScripting

FileandOutputControl(Example4.sh)
Loops,ConditionalsandUtilizingAvailablePrograms
(Example5.sh)

AdvancedScripting
SubstringsandCaseStatementsforMachineandArchitecture
SpecificTasks(Example6.sh)
RestartFiles,IntegersandAvoidingRedundantTasks
(Example7.sh)
CallingMatlabforBatchProcessing(Example8.sh)

IntroductiontoShells

Whatare*nixShells?
UnixwasthefirstOStoseparatethecommand
interpreter(shell)fromtheoperatingsystem(kernel)
Shellsabstractthecomplexitiesofsystemsand
providestandardcommandsforallmachines
Ashellhasbuiltincommands,plusallowsusaccess
tootherUnixcommands
history,export,cd>builtin
cp,ls,grep>Unixcommands(/bin,/usr/bin,etc.)

http://www.phys.ualberta.ca/~gingrich/research/shells/shells.html

AGeneralHistory
BourneShell(/bin/sh):
BellLaboratories,byStephenBourne
Basisforallshells,butprovidedlimitedfunctionality

CShell(/bin/csh):
UCBerkley,byBillJoy(creatorofvi)
ExpandedBourneShelltomimicClanguageandprovideinteractivity(in
reality,itwastoobuggytouseforscripts)
TENEXcsh(/bin/tcsh)waslaterdevelopedtogetridofbugsbutUnix
manufacturersstuckwithcsh

http://www.softlab.ntua.gr/facilities/documentation/unix/shelldiff.html#3

History(2)
KornShell(/bin/ksh):
AT&T,byDavidKorn
Hybridofshandcshusingshlanguagewithcshinteractivity
Althoughitwasthebestofbothworlds,itwasstillfromAT&T=>ksh
wasnotfree.

BourneAgainShell(/bin/bash):
GNU'sPOSIXcompatibleBourneShell
EquivalenttoKornbutfreeimplementation
PerfectfitinLinuxwhichresultedinmasspopularity

rc,zshandothersarestillemerging...

UnderstandingtheENV
TheENVistheUnixenvironment
Key=valuepairsdescribecurrentconfiguration
Everyrunningapplication,user,etcthatinteractswiththe
operatingsystemhasanenvironment
Aprocessinheritsitsparent'senvironmentunlesstheparent
overrides/specifiesotherproperties.
Users'initialENVisprovidedbytheshellandbasedon
/etc/profile.d/*,/etc/*shrcand~/.*shrcfiles.
Changestochild'senvironmentonlyaffectchild'schildren
andarelostwhenthesubtreeexits.

ControllingtheEnv
Shellsallowustocontroltheenvironmentwithbuiltin
commands;Unixcommandscanalsobeused:
%>env
Listsyourcurrentenvironment

%>export<key>=<value>
Setanewkeyorupdateanexistingone
NOTE:thisisequivalenttosetenv<key><value>intheCshell
family

%>echo$<key>
Printcurrentvalueofvariable

UnderstandingShellScripts
Shellscriptsarefilesthatcontaincommandsasyou
wouldrunthemonthecommandprompt.
Scriptscommonlyusevariables,conditionalsandloops.
Theseareallavailableonthecommandprompt

Whatmakesascriptsospecial?
Ithastheexecutablebitset(chmod+x<script>)
Itexecutesuntilitencountersanerror(interpreted)
Scriptscanuseanyavailableexecutables;noneedto
worryaboutlibraries,headersorAPIs

WhyScript?
Quickanddirtysolutions
Mostlyyouusepreexistingbinaries
Changestoscripthaveimmediateeffects
noneedtowrite/(re)compilecode

Interprogramcommunicationisperformedthrough
standardfileI/Opipes
Noworriesaboutcommunicationstandards
ShellScriptsuseshellsthatarestandardonall*nixplatforms
NOTE:onlytheshellsarestandard.Thisisnotalwaystrue
forthecommandsascriptcontains.

NoviceShellScripting

HelloWorld
Example1:HelloWorld.sh
Keyfeatures:
Shbang(#!/bin/sh)
Everyshellscriptmuststartwithashbangonthefirst
line.Thistellstheexecutingshellwhattypeofshellshould
interpretit(i.e.sh,bash,csh,perl,python,tcl,etc.)

Comments
Commentsbeginwiththehash(#)andproceedtotheend
oftheline

Variables
Example2:HelloWorld2.sh;Hello.sh
KeyFeatures:
LocalandGlobalVariables
<key>=<value>isalocalvariablethatisnotpassed
tochildren
export<key>=<value>isaglobalvariablethatis
passedtochildrenbutisnotvisibletoparentofthis
script

`...`,'...'and...
Example3:Example3.csh
KeyFeatures:
Useofdifferentshell(seecshinshbang)
`...`:executeprogramperformingsubstitution
onvariables
'...':stringswithoutvariablesubstitution
...:stringswithvariablesubstitution

IntermediateShellScripting

FileandOutputControl
Example4:Example4.sh;HandsFreeCharmm.sh
KeyFeatures:
cat<<EOF...
Printeverythingbetween<<EOFandthenextoccurrenceof
EOFtostdout.
NOTE:the>example4.outtotherightof<<EOFredirects
stdouttoafile

Redirectingbothstdoutandstderr
Use([command]>[stdoutFile])>[stderrFile]tostoreoutput
separately

Use[command]&>[outFile]tostoretogether

Loops,ConditionalsandUtilizingAvailable
Programs
Example5:Example5.sh
KeyFeatures:
Forloopexecutesonlistoffilenames
Conditional(ifthenelse)testsforalockfiletoavoidextra
work
WetakeadvantageofstandardUnixcommandslikemkdir,
pwd,grepandechobutalsousenonstandard
Imagemagicktoperformbatchimageconversion

AdvancedShellScripting

SubstringsandCaseStatementsforMachine
andArchitectureSpecificTasks
Example6:Example6.sh
KeyFeatures:
Casestatementtocomparemachine'sname(hostname)
Casestatementtocomparearchitectureofmachine
Substringsfromvariablesusing${var_name#*/}and
${var_name%/*}

RestartFiles,IntegersandAvoiding
RedundantTasks
Example7:Example7.sh
KeyFeatures:
Restartfileinformsscriptwhattasksithasalready
completed
Integersrequireuniquehandling(wecannotcompare
likestrings)
LocatesALLjpegsinALLsubdirectoriesandrebuilds
subdirsinoutputfolder(importanttokeepbatchjobs
organized)

CallingMatlabasforBatchProcessing
Example8:Example8.sh;testMatlab.m;
mandrill_cvt.m
KeyFeatures:
Permachinetasks
differentcompressionforeachcluster:Class*,Prism*,
Hallway*,andVislabmachines

ExecutesMatlabbehindthesceneswithacomplex
clusteringalgorithmfordatacompression(easilycoded
inMatlabandeasytovisualizeresults)

Conclusion
Younowhaveexampleshellscriptstogetstarted.
UsetheresourcesonthecribsheetandontheSCS
TechSupportTwikitohelpyouwriteyourownscripts.
Uselogging,locksandrestartfilestohelpaccelerate
batchjobs.
Remember:don'treinventthewheel!Ifacommandexists
anditfunctionscorrectly,don'twasteyourtimerewriting
itfromscratch.Ifithasbugsthenitsanotherstory...

SpecialThanks...
toTSGfortheexcusetoplaywithshellscripts
toGordonErlebacherfortheextratimeallotment
awayfromWATT
toSCSCustodialformakinglifeinterestingduringthe
weehoursofthemorning

Andofcourse!Thankyouallforattending...