UNIX Final Study Material Compressed

Filters
Filter is a programm in unix . It takes its input from another program, performs some
operation on that input, and writes the result to the standard output. Thus the common
use of filters is to modify or restructure output.
Some common filters in UNIX are:
• uniq – Removes identical adjacent lines

• head – displays first n lines of a file .
• tail – displays last n lines of a file .
• sort – sorts files by line (lexically or numerically)
• cut – select portions of a line.
• wc – word count (line count, character count)
• tr – translate
• grep, egrep – search files using regular expressions
head
This command list the beginning of a file to standard output. The default is 10 lines,
but a different number can be specified. The command has a number of options.
Syntax:
head [OPTION] [FILE]
Options:
-c Prints the first N bytes of file; with leading -, prints all but the last N bytes of the
file.
-n Prints first N lines; with leading - print all but the last N lines of each file.
Example: To display the first 10 lines of the file myfile.txt. $head myfile.txt
To display the first 100 lines of the file myfile.txt.
$head -n100 myfile.txt

To print the first 5 bytes from the file
$ head -c5 myfile.txt
tail
List the (tail) end of a file to stdout. The default is 10 lines, but this can be changed
with the -n option. Commonly used to keep track of changes to a system log-file,
using the -f option, which outputs lines appended to the file.
Syntax:
tail [OPTION]... [FILE]...
Example:
To display the last 10 lines of the file myfile.txt.
$tail myfile
$ tail -100 myfile.txt
$tail –n 100 myfile.txt
more
more command allows to view text files or other output in a scrollable manner. When
can command is used to view a very long file, all the output scrolls off the top of your
screen and only the last page can be viewed. more command solves this problem by
allowing the output of cat command one screenful of data at a time.
Syntax:
more [option] filename
Options:
-num This option specifies an integer which is the screen size (in lines).
-d more will prompt the user with the message "[Press space to continue, 'q' to quit.]"
and will display "[Press 'h' for instructions.]" instead of ringing the bell when an
illegal key is pressed.
-l more usually treats ^L (form feed) as a special character, and will pause after any
line that contains a form feed. The -l option will prevent this behavior.
-p Do not scroll. Instead, clear the whole screen and then display the text.
tr
tr command automatically translates or substitute characters.
Syntax:
tr [OPTION] set1 [set2]
Translate, squeeze, and/or delete characters from standard input, writing to standard
output.
Options:
c. : complements the set of characters in string.

d. : deletes the characters in set1
s. : replaces repeated characters listed in the set1 with single occurrence

t. : truncates set1
Example: To replace any occurrence of a by x, b by y and c by z in a given string
$echo “about to call “|tr [abc] [xyz]
Output : xyout to zxll
Example: To replace non matching characters
$ echo "Hello"|tr -c e a
Output : aeaaaa
In the above example , except the character “e” other characters are replaced by a
Example: Squeez , we can squeeze more than one occurrence of continuous

characters with single occurrence.
$echo “about to call “|tr – s ‘ ‘
Output : about to call
Above example squeezes two or more blank spaces into one.
sort
sort command reorders the lines of a file in ascending or descending order.
The default order is ascending .
Syntax:
sort -t field_delemeter [OPTION] file1 [file 2]
Options:
-k n sort on the nth field of the line
-t char use char as the field delimiter
-n sort numerically
-r reverse order sort
-u removes repeated lines
-m list merge sorted files in list
Examples:
Below examples will help you to understand sort used with different options:
Example 1:
Consider a file named “list”, which has below data
1, Justin Timberlake, Title 545, Price $7.30
2, Lady Gaga, Title 118, Price $7.30
3, Johnny Cash, Title 482, Price $6.50

4, Elvis Presley, Title 335, Price $7.30
5, John Lennon, Title 271, Price $7.90
To sort on the 2nd field of file named “list” we have to use the below command:
$sort –t’,’ –k 2 list
Note: File list is comma separated file.
Output:
Example 2: Numerically sorting:
To numerically sort data , option to be used is –n
Suppose list is the name of the file having following data:
19
20
49
00
If we sort it as below:
$sort list
Output is :
19
20
200
49
To get the expected output , the command will be
$sort –n list
Output:
19
20
49
200
Sort can sort multiple files also.
$sort file1 file2 file3 …
Example 3: Numerically sort in reverse order
$sort –nr list
Output :
200
49
20
19
Example 4: Sort the file list removing the repeated lines.
Syntax:
$sort –u filename
File list has following content:
Unix
Unix
Linux
Linux
Solaris
Axis
Axis
$sort –u list
Output:
Unix
Linux
Solaris
Axis
uniq
uniq command is used to suppress the duplicate lines from a file. It discards all the
successive identical lines except one from the input and writes the output.
Syntax:
uniq [option] filename
Options:
-u lists only the lines that are unique
lists only the lines that are duplicates
-c counts the frequency of occurrences
Suppress duplicate lines:
The default behavior of the uniq command is to suppress the duplicate line. Note that,
you have to pass sorted input to the uniq, as it compares only successive lines.
If the lines in the file are not in sorted order, then use the sort command and then pipe
the output to the uniq command.
Count of lines:
The -c option is used to find how many times each line occurs in the file. It prefixes
each line with the count.
Display only duplicate lines:
You can print only the lines that occur more than once in a file using the -d option.
The -D option prints all the duplicate lines.
Skip first N fields in comparison:

the -f option is used to skip the first N columns in comparison. Here the fields are
delimited by the space character.
cut
This command is used for text processing. You can use this command to extract
portion of text from a file by selecting columns.
Syntax:
cut –option filename
Select Column of Characters :
To extract only a desired column from a file use -c option.
The following example displays 2nd character from each line of a file test.txt.
$cut –c2 test.txt
Select Column of Characters using Range :
Range of characters can also be extracted from a file by specifying start and end
position delimited with -.
The following example extracts first 3 characters of each line from a file called test.txt
$cut –c 1-3 test.txt
Select Column of Characters using either Start or End Position :
Either start position or end position can be passed to cut command with -c option.
Following example extracts from 3rd character to end of each line from test.txt file.
$cut –c3- test.txt
To extract 8 characters from the beginning from the file test.txt,

$cut –c-8 test.txt
Select a Specific Field from a File :
Instead of selecting x number of characters you can combine option -f and –d to
extract a whole field.
The option -f specifies which field you want to extract,
The option -d specifies what delimiter that is used in the input file.
The following example displays only first field of each lines from /etc/passwd file
using the field delimiter: (colon). In this case, the 1st field is the username.
$ cut -d':' -f1 etc/passwd
paste
This is the command for merging together different files into a single, multi-column
file. In combination with cut, useful for creating system log files.
Syntax:
paste file1 file2
join
This utility allows merging two files in a meaningful fashion, which essentially
creates a simple version of a relational database.
The command join operates on exactly two files, but pastes together only those lines
with a common tagged field (usually a numerical label), and writes the result to
standard output.
The files to be joined should be sorted according to the tagged field for the matchups
to work properly.
Example:
The content of two files file1 and file2 are as below,

$cat file1
100 Shoes
200 Laces
300 Socks
$cat file2
100 $40.0
200 $1.00
300 $2.00
The following command will join these two files.

$ join 1.data 2.data
100 Shoes $40.00
200 Laces $1.00
300 Socks $2.00
Pipe
In unix , you can connect two commands together so that the output from one program
becomes the input of the next program. Two or more commands connected in this
way form a pipe. In shell the symbol '|’ is used to represent pipe.
Purpose of Pipes :
Using pipe you can construct powerful unix command lines by combining basic unix
commands. UNIX commands are powerful; however by using pipe you can combine
them together, to accomplish complex tasks with ease.
Through the standard output of one command (the command to the left of the pipe)
gets sent as standard input to another command (the command to the right of the
pipe). Pipe functions in a similar manner like the output redirection in UNIX (using >
symbol to redirect the standard output of a command to a file. However, the pipe is
different because it is used to pass the output of a command to another command, not
a file.
Example:
$ cat apple.txt | wc
3 4 21
In this example, the contents of the file apple.txt are sent through pipe to wc (word
count) command. The wc command then does its job and counts the lines, words, and
characters in the file.
You can combine many commands with pipes on a single command line. Here's an
example where the characters, words, and lines of the file apple.txt is sent to wc and
then the output of wc mailed to nobody@december.com with the subject line "The
count."
$ cat apple.txt | wc | mail -s "The count" nobody@december.com
awk
awk is a scripting language which is used for processing or analyzing text files.
awk is used for grouping of data based on either a column or field, or on a set of
columns.
It derives its name from the first letter of the last name of its three authors namely
Alfred V. Aho, Peter J.Weinberger and Brian W. Kernighan.
awk can be used for reporting data in a useful manner. It searches one or more files to
see if they contain lines that match specified patterns and then perform associated
actions. awk is an advanced filter.
Simple awk Filtering
Syntax of awk:
~$ awk 'pattern {action}' input-file

Let’s take a input file with the following data
~$cat awk_file
Name,Marks,Max_Marks
Peter,200,1000
Sam,500,1000
Greg,1000
Abharam,800,1000
Henry,600,1000
Peter,400,1000
Example: Default behavior of awk

Print all the lines from a file.
By default, awk prints all lines of a file, so to print every line of above created file ,
use below command:
~$ awk '{print}' awk_file
Peter,200,1000
Sam,500,1000
Greg,1000
Abharam,800,1000
Henry,600,1000
Peter,400,1000
Example 2: Print only specific field

Print 2nd & 3rd fileds
~$ awk -F”,” {print $2,$3;}' awk_file
Example: Pattern Matching

Print the lines which matches the pattern (lines which contains the word “Henry" or
"Peter”)
~$ awk '/Henry|Peter/' awk_file
Peter,200,1000
Henry,600,1000
Peter,400,1000
Initialization and Final Action
BEGIN and END blocks are helpfull in displaying information before and after
executing actual awk script.
BEGIN block is evaluated before awk starts processing the actual awk script; it’s an
excellent place to initialize the FS (field separator) variable, print a heading, or
initialize other global variables.
BEGIN block Usages:
• Declaring variables.
• Initialization variables for doing increment/decrements operations in main
AWK code.
• Printing Headings/info before actual AWK code output.
END block is evaluated after all the lines in the awk script have been processed.
Typically, the END block is used to perform final calculations or print summaries that
should appear at the end of the output stream.
END block Usages:
• Printing final results, after doing operations in main AWK block.

• Printing Completion/info after actual AWK code output.
awk tool is mainly used for reporting some data in useful manner. Without these
BEGIN and END
blocks the output will be meaningless.
Consider db.txt which contains below data:
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
awk BEGIN block
This is a block of code which is executed before executing actual awk script.
BEGIN block Syntax
awk ‘BEGIN{awk initializing code}{actual AWK code}’ filename.txt
Example: Print a meaning full info before actual AWK output.
~$ awk ‘BEGIN{print “########################\nThis is the output of
filtered
data\n########################”}{print $0}’ db.txt
Output:
##########################
This is the output of filtered data
##########################
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47
awk END block
This is the block which is executed after executing all the awk code.
Example:
Print some meaning full info after processing awk code.

~$ awk ‘{print $0} END {print “#########################\n Completed
printing
filtered data\n########################”}’ db.txt
Output:
Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
#########################
Completed printing filtered data
#########################
Combining BEGIN and END block
Example:
~$ awk ‘BEGIN{print “##########################\n This is the output of

filtered
data\n##########################”}{print $0}END{print
“########################\n Completed printing filtered
data\n########################”}’ db.txt
Output:
#########################
#########################
Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
########################
awk inbuilt variables
awk is supplied with good number of built-in variables which comes in handy when
working with data files. We will see usages of awk built-in variables with one or two
examples . These variable are used to format the output of an awk command.
List of built-in variables:
FS field separator character (default blank & tab)

OFS output field separator string (default blank)
RS input record separator character (default newline)
ORS output record separator string (default newline)
NF number of fields in input record
NR number of input record
FNR output number of lines
FILENAME name of current input file
Consider below db.txt as sample file.

~$ cat db.txt
John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married
In order to make it simple we can divide above inbuilt variables in to groups on basis
of their operations.
Group1: FS(input field separator), OFS(Output Field Separator)
Group2: RS(Row separator) and ORS(Output Record Separator)
Group3: NR, NF and FNR
Group4: FILENAME variable
FS (Input Field Separator)
This variable is useful in storing the input field separator. By default AWK can
understand only spaces, tabs as input and output separators. But if your file contains
some other character as separator other than these mention one’s, awk cannot
understand them.
For example UNIX password file which contain ‘:’ as a separator. So in order to
mention the input filed separator we use this inbuilt variable. We will see what issue
we face if we don’t mention the field separator for our db.txt.
Example: without using FS
Print first column data from db.txt file.

~$ awk ‘{print $1}’ db.txt
Output:
OFS (Output Field Separator)
This variable is useful for defining the output field separator for the expected output
data.
Example:
Display only 1st and 4th column and with $ as field separator for the output .
~$ awk ‘BEGIN{FS=”,”;OFS=” $ “}{print $1,$4}’ db.txt
Output:
John $ IBM
Barbi $ JHH
Mitch $ BofA
Tim $ DELL
Lisa $ SmartDrive
Note: Space is give before and after $ in OFS variable to show better output.
RS (Row separator)
Row Separator is helpful in defining separator between rows in a file. By default awk
takes row separator as new line. We can change this by using RS built-in variable.
Example:
Convert a sentence to a word per line. We can use RS variable for doing it.
~$ echo “This is how it works” | awk ‘BEGIN{RS=” ”}{print $0}’
Output:
This
is
how
it
Works
ORS (Output Record Separator)

This variable is useful for defining the record separator for the awk command output.
By default ORS is set to new line.
Example:
Print all the company names in single line which are in 4th column.
~$ awk -F’,’ ‘BEGIN{ORS=” “}{print $4}’ db.txt
Output:
IBM JHH BofA DELL SmartDrive
NF
This variable keeps information about total fields in a given row. The final
value of a row can be represented with $NF.
Example: Consider abc.txt which contains below data:

Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47
Print number of fields in each row in abc.txt.
~$ awk ‘{print NF}’ abc.txt
Output:
5
5
4
5
4
NR
This variable keeps the value of present line number. This will come handy when you
want to print line numbers in a file.
Example:
Print line number for each line in a given file.
~$ awk ‘{print NR, $0}’ abc.txt
Output:
1 Jones 2143 78 84 77
2 Gondrol 2321 56 58 45
3 RinRao 2122234 38 37
4 Edwin 253734 87 97 95
5 Dayan 24155 30 47
This can be treated as cat command -n option for displaying line number for a file
FNR
This variable keeps count of number of lines present in a given file/data. This will
come handy when
you want to print no of line present in a given file. This command is equivalent to wc
-l command.
Example:
Print total number of lines in a given file.
~$ awk ‘END{print FNR}’ abc.txt
Output:
5
FILENAME
This variable contain file awk command is processing.
Example:
Print filename for each line in a given file.
~$ awk ‘{print FILENAME, NR, $0}’ abc.txt
Output:
abc.txt 1 Jones 2143 78 84 77
abc.txt 2 Gondrol 2321 56 58 45
abc.txt 3 RinRao 2122234 38 37
abc.txt 4 Edwin 253734 87 97 95
abc.txt 5 Dayan 24155 30 47
awk Built in Function
A function is a self-contained computation that accepts a number of arguments as

input and returns some value. awk has a number of built-in functions in two groups:
arithmetic and string functions.
Arithmetic Functions
Nine of the built-in functions can be classified as arithmetic functions. Most of them
take a numeric argument and return a numeric value. Below table summarizes these
arithmetic functions with some Examples.
awk Function Description

cos ( x ) Returns cosine of x (x is in radians).
exp ( x ) Returns e to the power x.
index (s1,s2) Position of string s2 in s1; returns 0 if not present
int ( x ) Returns truncated value of x.
log ( x ) Returns natural logarithm (base- e) of x.
sin ( x ) Returns sine of x (x is in radians)
sqrt ( x ) Returns square root of x.

atan2 ( y , x ) Returns arctangent of y / x in the range - to .
rand () Returns pseudo-random number r, where 0 <= r < 1.
sqrt(expr) Returns the square root of the expression or value given
Examples:
~$ awk 'BEGIN{
print sqrt(16);
print sqrt(0);
print sqrt(-12);
}'
Output:
4
0
nan
Here nan stands for not a valid number.
String Functions
The built-in string functions are much more significant and interesting than the
numeric functions. Because awk is essentially designed as a string-processing
language, a lot of its power derives from these functions. Below table lists the string
functions found in awk. awk's Built-In String Functions
Filters and Regular Expression
grep
grep command allows you to search one file or multiple files for lines that contain a
pattern.
Full form of grep is global regular expression print.
It is a powerful file pattern searcher in Linux

grep's exit status is 0 if matches were found, 1 if no matches were found, and 2 if
errors occurred.
grep search the target file(s) for occurrences of pattern, where pattern may be literal
text or a Regular Expression.
Syntax:
grep pattern [file...]
Search for the given string in a single file

The basic usage of grep command is to search for a specific string in the specified file
as shown below.
Checking given string in multiple files

We can use grep command for searching for a given string in multiple files.
For example, let us copy the demo_file to demo_file1and use the grep on both the
files to search the pattern this.
The output will include the file name in front of the line that matched the specific
pattern as shown below.
When the Linux shell sees the meta character, it does the expansion and gives all the
files as input to grep.
Case insensitive search
We can use grep to search for the given string/pattern case insensitively. So it matches
all the words such as “the”, “THE” and “The” case insensitively as shown below.
Match regular expression in files

This is a very powerful feature of grep . In the following example, it searches for all
the pattern that starts with “lines” and ends with “empty” with anything in-between.
i.e To search “lines[anything in-between]empty” in the demo_file.
A regular expression may be followed by one of several repetition operators:
1. ? The preceding item is optional and matched at most once.

2. The preceding item will be matched zero or more times.
3. + The preceding item will be matched one or more times.
4. {n} The preceding item is matched exactly n times.
5. {n,} The preceding item is matched n or more times.
6. {,m} The preceding item is matched at most m times.
7. {n,m} The preceding item is matched at least n times, but not more than m
times.
Checking for full words

To search for a word, and to avoid it to match the substrings -w option is used. The
following example is the regular grep where it is searching for “is”. When you search
for “is”, without any option it will show out “is”, “his”, “this” and everything which
has the substring “is”.
Searching in all files recursively
When you want to search in all the files under the current directory and its sub
directory ‘–r’ option is the one which you need to use. The following example will
look for the string “ramesh” in all the files in the current directory and all its
subdirectory.
$ grep -r "ramesh" *
Invert match
If you want to display the lines which does not matches the given string/pattern, use
the option -v as shown below. This example will display all the lines that did not
match the word “Two”.
Displaying the lines which does not matches the entire given pattern.
Syntax:
grep -v -e pattern -e pattern
For example, the file file1 has the following content
Apple
Banana
Cauliflower
Grapes
Orange
Counting the number of matches

Count the number of lines matched in the given pattern/string, then use the option -c.
Syntax:
grep -c pattern filename
Displaying only the file names which matches the given pattern
The -l option is used to display only the file names which matched the given pattern.
When you give multiple files to the grep as input, it displays the names of file which
contains the text that matches the pattern, will be very handy when you try to find
some notes in your whole directory structure.
Showing line number while displaying the output

To show the line number of file with the line matched, -n option is used.
Syntax:
grep -n pattern filename
Example:
grep -n "this" demo_file
2: this line is the 1st lower case line in this file.
6: Two lines above this line is empty.
sed
sed is a stream editor used to perform basic text transformations on an input stream (a
file, or input from a pipeline).
Working methodology
sed works by making only one pass over the input(s) s called as one execution cycle.
Cycle continues till end of file/input is reached.
• Read entire line from stdin/file.

• Removes any trailing newline.
• Places the line, in its pattern buffer.
• Modify the pattern buffer according to the supplied commands.
• Print the pattern buffer to stdout.
Printing Operation in sed

sed allows you to print only specific lines based on the line number or pattern
matches. “p” is the command for printing the data from the pattern buffer. To suppress
automatic printing of patternspace -n option is used with sed. sed -n option will not
print anything, unless an explicit request to print is found.
Syntax:
sed -n 'ADDRESS'p filename
sed -n '/pattern/p' filename
Examples:
Let us assume the demo_file has the following content
To prints third line of input file
$sed -n '3p' demo_file

3. Hardware
To print every nth line starting from the line m
$sed -n 'm~np' filename
To print only the last line
To print the lines containing the given pattern:

Syntax:
sed -n /PATTERN/p filename
Deletion operation in sed

In sed the d command is used to delete the pattern space buffer and immediately starts
the next cycle.
Syntax:
sed nd filename
'nd’ deletes the nth line and prints the other lines.
sed 'ADDRESS'd filename
sed /PATTERN/d filename
The process is
• It reads the first line and places in its pattern buffer
• checks whether supplied command is true for this line , if true, deletes pattern space
buffer and starts next cycle and reads the next line.
• If supplied command is not true, it prints the content of the pattern space buffer.
To delete the 3rd line and print other lines from the file demo_file
Substitution operation in sed

In sed the s command is used to substitute the pattern. The `s’ command attempts to
match the pat-tern space against the supplied expression/ pattern; if the match is
successful, then that portion of the pattern space which was matched is replaced with
the replacement given.
Syntax:
$sed 'ADDRESSs/REGEXP/REPLACEMENT/FLAGS' filename
$sed 'PATTERNs/REGEXP/REPLACEMENT/FLAGS' filename
1. s is substitute command
2. / is a delimiter
3. REGEXP is regular expression to match
4. REPLACEMENT is a value to replace
FLAGS can be any of the following:
1. g Replace all the instance of REGEXP with REPLACEMENT

2. n Could be any number,replace nth instance of the REGEXP with
REPLACEMENT.
3. p If substitution was made, then prints the new pattern space.
4. i match REGEXP in a case-insensitive manner.
5. w file If substitution was made, write out the result to the given file.
6. We can use different delimiters ( one of @ % ; : ) instead of /
To Write Changes to a File and Print the Changes
To combine multiple sed commands we have to use option -e
Syntax:
$sed -e 'command' e 'command' filename
To Delete the first,last and all the blank lines from input
FILTERS USING REGULAR EXPRESSION
A regular expression is a set of characters that specify a pattern. Regular expressions

are used when you want to search for specific lines of text containing a particular
pattern. Most of the UNIX utilities operate on ASCII files a line at a time. Regular
expressions search for patterns on a single line, and not for patterns that start on one
line and end on another.
The Structure of a Regular Expression
There are three important parts to a regular expression.

• Anchors : These are used to specify the position of the pattern in relation to a line of
text.
• Character Sets : The set of characters that match one or more characters in a single
position.
• Modifiers: They specify how many times the previous character set is repeated.
A simple example that demonstrates all three parts is the regular expression is : "^#*"
Here ,
• The up arrow , “^”, is an anchor that indicates the beginning of the line.
• The character "#" is a simple character set that matches the single
character "#".
• The asterisk “*” is a modifier. In a regular expression it specifies that the
previous character set can appear any number of times, including zero.
There are also two types of regular expressions:

• the "Basic" regular expression,(BRE)
• the "extended" regular expression.(ERE)
A few utilities like awk and egrep use the extended expression. Most use the "regular"
regular expression. From now on, if I talk about a "regular expression," it describes a
feature in both types.
The Anchor Characters: ^ and $ Anchores are used when we want to search for a
pattern that is at one end or the other, of a line. The character "^" is the starting
anchor, and the character "$" is the end anchor.Following list provides a summary:
Pattern Matches
Â "A" at the beginning of a line
A$ "A" at the end of a line
A^ "A^" anywhere on a line
$A "$A" anywhere on a line
^^ "^" at the beginning of a line
$$ "$" at the end of a line
The Character Set

The character set also called “character class” in a regular expression , is used to tell
the regex engine to match only one out of several characters
• A character set matches only a single character. In case of the above example,
gr[ae]y does not match graay, graey or any such thing.
• The order of the characters inside a character set does not matter. The results
are identical.
• Some characters have a special meaning in regular expressions. If we want to
search for such a character, we have to escape it with a backslash.
Exception in the character class

If we want to search for all the characters except those in the square bracket,
then the ^ (Caret) symbol needs to be used as the first character after open
square bracket. The expression "^[âeiou]" is to searc for a line which does
not start with the vowel letter.
Regular Expression Matches
[] The characters "[]"
[0] The character "0"
[0-9] Any number
[^0-9] Any character other than a number
[-0-9] Any number or a "-"
[0-9-] Any number or a "-"
[^-0-9] Any character except a number or a "-"
[]0-9] Any number or a "]"
[0-9]] Any number followed by a "]"
[0-9-z] Any number, or any character between "9" and "z".
[0-9\-a\]] Any number, or a "-", a "a", or a "]"
Match any character

The character "." is one one of thespecial meta-characters. By itself it will match any
character, except the end-of-line character. Thus the pattern that will match a line with
a single characters is ^.$
Repeating character sets

The third part of a regular expression is the modifier. It is used to specify how may
times you expect to see the previous character set. The repetition modifier * find no or
one, one or more, and zero or more
repeats, respectively.
Examples:
Expression Matches
Go*gle Gogle,Google,Gooogle, and so on.
"[0-9]*" zero or more numbers.
Matching a specific number of sets with \{ and \}

We cannot specify a maximum number of sets with the "*" modifier. There is a
special pattern we can use to specify the minimum and maximum number of repeats,
by putting those two numbers between "\{" and "\}".
▪ A modifier can specify amounts such as none, one, or more;
For example , A user name is a string beginning with a letter followed by at least two,
but not more than seven letters or numbers followed by the end of the string. Then the
regular expression is
^[A-z][A-z0-9]{2,7}
▪ A repetition modifier must be combined with other patterns; the modifier has no
meaning by itself.
For example , modifiers like "*" and "\{1,5\}" only act as modifiers if they follow a
character set. If they were at the beginning of a pattern, they would not be a modifier.
grep with Regular expression
Search for 'vivek' in /etc/passswd

grep vivek /etc/passwd
Search vivek in any case (i.e. case insensitive search)
grep -i -w vivek /etc/passwd
Search vivek or raj in any case
grep -E -i -w 'vivek|raj' /etc/passwd
Line and word anchors
Search lines starting with the vivek only

grep ^vivek /etc/passwd
To display only lines starting with the word vivek only i.e. do not display vivekgite,
vivekg
grep -w ^vivek /etc/passwd
To Find lines ending with word foo
grep 'foo$' filename
Character classes
To match Vivek or vivek.

grep '[vV]ivek' filename
OR
grep '[vV][iI][Vv][Ee][kK]' filename
To match digits (i.e match vivek1 or Vivek2 etc)

grep -w '[vV]ivek[0-9]' filename
Wildcards
To match all 3 character word starting with "b" and ending in "t".
grep '\' filename
Where,
•\< Match the empty string at the beginning of word
•\> Match the empty string at the end of word.
Print all lines with exactly two characters
grep '^..$' filename
Display any lines starting with a dot and digit
grep '^\.[0-9]' filename
Escaping the dot
To find an IP address 192.168.1.254

grep '192\.168\.1\.254' /etc/hosts
Search a Pattern Which Has a Leading – Symbol
Searches for all lines matching '--test--' using -e option . Without -e, grep would
attempt to parse '--test--' as a list of options
grep -e '--test--' filename
Test Sequence
To Match a character "v" two times

egrep "v{2}" filename
To match both "col" and "cool"

egrep 'co{1,2}l' filename
grep OR Operator
Suppose the file “employee” has the following data:
To find the records of those who are either from Tech or Sales dept.
We can use the following syntaxes :
1) Syntax : grep 'word1\|word2' filename

grep 'Tech\|Sales' employee
2) Syntax : grep -E 'pattern1|pattern2' fileName

grep -E 'Tech|Sales' employee
grep AND Operator

There is no AND operator in grep. But, we can simulate AND using
• grep -E option.
Syntax : grep -E 'word1.*word2 ' filename
grep -E 'word1.*word2|'word2.*word1' filename
• multiple grep command separated by pipe

Syntax : grep 'word1' filename | grep 'word2'
Environment Variables:
An important concept in Unix is the environment, which is defined by values stored

in a set of predefined variables called environment variables.
Values of some of these variables are set by the system, some can be set by the user,
some of them by the shell, or any program that loads another program.
When a user logs in to the system, the shell undergoes a twostep process
called initialization to set up the environment. This process involves the shell reading
two files −/etc/profile and .profile in the home directory of the user.
The file /etc/profile is maintained by the system administrator of the Unix machine
and contains shell initialization information required by all users on a system.
The file .profile is under the control of the user. User can add as much shell
customization information as he/she wants to this file. The minimum set of
information that a user need to configure includes −
• The type of terminal user is using.

• A list of directories in which to locate the commands.
• A list of variables affecting the look and feel of the terminal.
Environment around the UNIX Process:
When a process is created, UNIX opens 3 streams stdin/stdout/stderr for basic

communication with respect to the process control terminal. In addition, it knows the
current working directory for performing file I/O.
Each login shell maintains a description of the environment as a table of pointers to

strings.
A global shell environment pointer called environ is maintained by UNIX kernel and
it can be used by a process to access its own table
The shell does not directly use this table, but it creates a child process and calls exec()
system call to execute a command program that uses the table inherited from the shell
parent. Children inherit the entire execution environment from the parent.
Some examples of the environment variables are the USER, LOGNAME, HOME,
PATH, PS1, PS2, TERM MAIL, etc.
The HOME Variable

It specifies an associated directory with every user in a UNIX system. If the HOME
variable for the user Sita contains /usr/sita/stores, every time Sita logs in, she is taken
to the directory stores.
The variable HOME is referenced the same way:
$ echo ${HOME}
The PATH Variable

contains a list of all full path-names (separated by a colon) of directories that are to be
searched for an executable program. For example, the command
$PATH=.:/usr/bin:/bin specifies directories to be searched for any executable file or a
command file (current directory, /usr/bin and /bin, in that order).
The PS1 Variable

The system prompt may be changed by setting the value of this variable to the desired
prompt:
$ PS1=“Hello>”
Hello> #can be changed only at the UNIX command line, not within a shell script.
The PS2 Variable: prompt string for continued command line (default ‘> ‘).
The LOGNAME Variable

contains user’s login name. Its contents cannot be changed by the user, but can be
displayed:
echo “${LOGNAME}”
The TERM Variable

Names the kind of terminal you are using; setting it helps to manage your screen more
effectively, say,
$ TERM=vt100
The PWD Variable

The current working directory can be displayed:
echo “${PWD}”
In fact the whole environment table can be displayed.
$IFS: String of characters which are used as word separators in command line ( space,
tab, newline chars).
The MAIL Variable

Names the standard file where your mail is kept
The .profile File
Some of above variables like HOME and LOGNAME are set automatically each time
the user logs in. The others, however, have to be set. The .profile is used for this
purpose as it is executed as soon the user logs in. A sample .profile file would look
like:
PATH=.:/bin:/usr/bin
export HOME PATH PS1 MAIL
Filters
Filter is a programm in unix . It takes its input from another program, performs some
operation on that input, and writes the result to the standard output. Thus the common
use of filters is to modify or restructure output.
Some common filters in UNIX are:
• uniq – Removes identical adjacent lines

• head – displays first n lines of a file .
• tail – displays last n lines of a file .
• sort – sorts files by line (lexically or numerically)
• cut – select portions of a line.
• wc – word count (line count, character count)
• tr – translate
• grep, egrep – search files using regular expressions
head
This command list the beginning of a file to standard output. The default is 10 lines,
but a different number can be specified. The command has a number of options.
Syntax:
head [OPTION] [FILE]
Options:
-c Prints the first N bytes of file; with leading -, prints all but the last N bytes of the
file.
-n Prints first N lines; with leading - print all but the last N lines of each file.
Example: To display the first 10 lines of the file myfile.txt. $head myfile.txt
To display the first 100 lines of the file myfile.txt.
$head -n100 myfile.txt
To print the first 5 bytes from the file
$ head -c5 myfile.txt
tail
List the (tail) end of a file to stdout. The default is 10 lines, but this can be changed
with the -n option. Commonly used to keep track of changes to a system log-file,
using the -f option, which outputs lines appended to the file.
Syntax:
tail [OPTION]... [FILE]...
Example:
$tail myfile
$ tail -100 myfile.txt
$tail –n 100 myfile.txt
more
more command allows to view text files or other output in a scrollable manner. When
can command is used to view a very long file, all the output scrolls off the top of your
screen and only the last page can be viewed. more command solves this problem by
allowing the output of cat command one screenful of data at a time.
Syntax:
more [option] filename
Options:
-num This option specifies an integer which is the screen size (in lines).
-d more will prompt the user with the message "[Press space to continue, 'q' to quit.]"
and will display "[Press 'h' for instructions.]" instead of ringing the bell when an
illegal key is pressed.
-l more usually treats ^L (form feed) as a special character, and will pause after any
line that contains a form feed. The -l option will prevent this behavior.
-p Do not scroll. Instead, clear the whole screen and then display the text.
tr
tr command automatically translates or substitute characters.
Syntax:
tr [OPTION] set1 [set2]
Translate, squeeze, and/or delete characters from standard input, writing to standard
output.
Options:
e. : complements the set of characters in string.

f. : deletes the characters in set1
u. : replaces repeated characters listed in the set1 with single occurrence

v. : truncates set1
Example: To replace any occurrence of a by x, b by y and c by z in a given string

$echo “about to call “|tr [abc] [xyz]
Output : xyout to zxll
Example: To replace non matching characters
$ echo "Hello"|tr -c e a
Output : aeaaaa
In the above example , except the character “e” other characters are replaced by a
Example: Squeez , we can squeeze more than one occurrence of continuous

characters with single occurrence.
$echo “about to call “|tr – s ‘ ‘
Output : about to call
Above example squeezes two or more blank spaces into one.
sort
sort command reorders the lines of a file in ascending or descending order.
The default order is ascending .
Syntax:
sort -t field_delemeter [OPTION] file1 [file 2]
Options:
-k n sort on the nth field of the line
-t char use char as the field delimiter
-n sort numerically
-r reverse order sort
-u removes repeated lines

-m list merge sorted files in list
Examples:
Below examples will help you to understand sort used with different options:
Example 1:
Consider a file named “list”, which has below data
To sort on the 2nd field of file named “list” we have to use the below command:
$sort –t’,’ –k 2 list
Note: File list is comma separated file.
Output:
Example 2: Numerically sorting:
To numerically sort data , option to be used is –n

Suppose list is the name of the file having following data:
19
20
49
00
If we sort it as below:
$sort list
Output is :
19
20
200
49
To get the expected output , the command will be
$sort –n list
Output:
19
20
49
200
Sort can sort multiple files also.
$sort file1 file2 file3 …
Example 3: Numerically sort in reverse order
$sort –nr list
Output :
200
49
20
19
Example 4: Sort the file list removing the repeated lines.
Syntax:
$sort –u filename
File list has following content:
Unix
Unix
Linux
Linux
Solaris
Axis
Axis
$sort –u list
Output:
Unix
Linux
Solaris
Axis
uniq
uniq command is used to suppress the duplicate lines from a file. It discards all the
successive identical lines except one from the input and writes the output.
Syntax:
uniq [option] filename
Options:
-u lists only the lines that are unique
lists only the lines that are duplicates
-c counts the frequency of occurrences
Suppress duplicate lines:
The default behavior of the uniq command is to suppress the duplicate line. Note that,
you have to pass sorted input to the uniq, as it compares only successive lines.
If the lines in the file are not in sorted order, then use the sort command and then pipe
the output to the uniq command.
Count of lines:
The -c option is used to find how many times each line occurs in the file. It prefixes
each line with the count.
Display only duplicate lines:
You can print only the lines that occur more than once in a file using the -d option.
The -D option prints all the duplicate lines.
Skip first N fields in comparison:
the -f option is used to skip the first N columns in comparison. Here the fields are
delimited by the space character.
cut
This command is used for text processing. You can use this command to extract
portion of text from a file by selecting columns.
Syntax:
cut –option filename
Select Column of Characters :
To extract only a desired column from a file use -c option.
The following example displays 2nd character from each line of a file test.txt.
$cut –c2 test.txt
Select Column of Characters using Range :
Range of characters can also be extracted from a file by specifying start and end
position delimited with -.
The following example extracts first 3 characters of each line from a file called test.txt
$cut –c 1-3 test.txt

Select Column of Characters using either Start or End Position :
Either start position or end position can be passed to cut command with -c option.
Following example extracts from 3rd character to end of each line from test.txt file.
$cut –c3- test.txt
To extract 8 characters from the beginning from the file test.txt,
$cut –c-8 test.txt

Select a Specific Field from a File :
Instead of selecting x number of characters you can combine option -f and –d to
extract a whole field.
The option -f specifies which field you want to extract,
The option -d specifies what delimiter that is used in the input file.
The following example displays only first field of each lines from /etc/passwd file
using the field delimiter: (colon). In this case, the 1st field is the username.
$ cut -d':' -f1 etc/passwd
paste
This is the command for merging together different files into a single, multi-column
file. In combination with cut, useful for creating system log files.
Syntax:
paste file1 file2
join
This utility allows merging two files in a meaningful fashion, which essentially
creates a simple version of a relational database.
The command join operates on exactly two files, but pastes together only those lines
with a common tagged field (usually a numerical label), and writes the result to
standard output.
The files to be joined should be sorted according to the tagged field for the matchups
to work properly.
Example:
The content of two files file1 and file2 are as below,

$cat file1
100 Shoes
200 Laces
300 Socks
$cat file2
100 $40.0
200 $1.00
300 $2.00
The following command will join these two files.

$ join 1.data 2.data
100 Shoes $40.00
200 Laces $1.00
300 Socks $2.00
Pipe
In unix , you can connect two commands together so that the output from one program
becomes the input of the next program. Two or more commands connected in this
way form a pipe. In shell the symbol '|’ is used to represent pipe.
Purpose of Pipes :
Using pipe you can construct powerful unix command lines by combining basic unix
commands. UNIX commands are powerful; however by using pipe you can combine
them together, to accomplish complex tasks with ease.
Through the standard output of one command (the command to the left of the pipe)
gets sent as standard input to another command (the command to the right of the
pipe). Pipe functions in a similar manner like the output redirection in UNIX (using >
symbol to redirect the standard output of a command to a file. However, the pipe is
different because it is used to pass the output of a command to another command, not
a file.
Example:
$ cat apple.txt | wc
3 4 21
In this example, the contents of the file apple.txt are sent through pipe to wc (word
count) command. The wc command then does its job and counts the lines, words, and
characters in the file.
You can combine many commands with pipes on a single command line. Here's an
example where the characters, words, and lines of the file apple.txt is sent to wc and
then the output of wc mailed to nobody@december.com with the subject line "The
count."
$ cat apple.txt | wc | mail -s "The count" nobody@december.com

awk
awk is a scripting language which is used for processing or analyzing text files.
awk is used for grouping of data based on either a column or field, or on a set of
columns.
It derives its name from the first letter of the last name of its three authors namely
Alfred V. Aho, Peter J.Weinberger and Brian W. Kernighan.
awk can be used for reporting data in a useful manner. It searches one or more files to
see if they contain lines that match specified patterns and then perform associated
actions. awk is an advanced filter.
Simple awk Filtering
Syntax of awk:
~$ awk 'pattern {action}' input-file

Let’s take a input file with the following data
~$cat awk_file
Peter,200,1000
Sam,500,1000
Greg,1000
Abharam,800,1000
Henry,600,1000
Peter,400,1000
Example: Default behavior of awk

Print all the lines from a file.
By default, awk prints all lines of a file, so to print every line of above created file ,
use below command:
~$ awk '{print}' awk_file
Peter,200,1000
Sam,500,1000
Greg,1000
Abharam,800,1000
Henry,600,1000
Peter,400,1000
Example 2: Print only specific field

Print 2nd & 3rd fileds
~$ awk -F”,” {print $2,$3;}' awk_file
Example: Pattern Matching

Print the lines which matches the pattern (lines which contains the word “Henry" or
"Peter”)
~$ awk '/Henry|Peter/' awk_file
Peter,200,1000
Henry,600,1000
Peter,400,1000
Initialization and Final Action
BEGIN and END blocks are helpfull in displaying information before and after
executing actual awk script.
BEGIN block is evaluated before awk starts processing the actual awk script; it’s an
excellent place to initialize the FS (field separator) variable, print a heading, or
initialize other global variables.
BEGIN block Usages:
• Declaring variables.
• Initialization variables for doing increment/decrements operations in main
AWK code.
• Printing Headings/info before actual AWK code output.
END block is evaluated after all the lines in the awk script have been processed.
Typically, the END block is used to perform final calculations or print summaries that
should appear at the end of the output stream.
END block Usages:
• Printing final results, after doing operations in main AWK block.

• Printing Completion/info after actual AWK code output.
awk tool is mainly used for reporting some data in useful manner. Without these
BEGIN and END
blocks the output will be meaningless.
Consider db.txt which contains below data:
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
awk BEGIN block
This is a block of code which is executed before executing actual awk script.
BEGIN block Syntax
awk ‘BEGIN{awk initializing code}{actual AWK code}’ filename.txt
Example: Print a meaning full info before actual AWK output.
~$ awk ‘BEGIN{print “########################\nThis is the output of
filtered
data\n########################”}{print $0}’ db.txt
Output:
##########################
##########################
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47
awk END block
This is the block which is executed after executing all the awk code.
Example:
Print some meaning full info after processing awk code.
~$ awk ‘{print $0} END {print “#########################\n Completed
printing
filtered data\n########################”}’ db.txt
Output:
Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
#########################
#########################
Combining BEGIN and END block
Example:
~$ awk ‘BEGIN{print “##########################\n This is the output of

filtered
data\n##########################”}{print $0}END{print
“########################\n Completed printing filtered
data\n########################”}’ db.txt
Output:
#########################
#########################
Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
########################
awk inbuilt variables

awk is supplied with good number of built-in variables which comes in handy when
working with data files. We will see usages of awk built-in variables with one or two
examples . These variable are used to format the output of an awk command.
List of built-in variables:
FS field separator character (default blank & tab)

OFS output field separator string (default blank)
RS input record separator character (default newline)
ORS output record separator string (default newline)
NF number of fields in input record
NR number of input record
FNR output number of lines
FILENAME name of current input file
Consider below db.txt as sample file.

~$ cat db.txt
In order to make it simple we can divide above inbuilt variables in to groups on basis
of their operations.
Group1: FS(input field separator), OFS(Output Field Separator)
Group2: RS(Row separator) and ORS(Output Record Separator)
Group3: NR, NF and FNR
Group4: FILENAME variable
FS (Input Field Separator)
This variable is useful in storing the input field separator. By default AWK can
understand only spaces, tabs as input and output separators. But if your file contains
some other character as separator other than these mention one’s, awk cannot
understand them.
For example UNIX password file which contain ‘:’ as a separator. So in order to
mention the input filed separator we use this inbuilt variable. We will see what issue
we face if we don’t mention the field separator for our db.txt.
Example: without using FS
Print first column data from db.txt file.

~$ awk ‘{print $1}’ db.txt
Output:
OFS (Output Field Separator)

This variable is useful for defining the output field separator for the expected output
data.
Example:
Display only 1st and 4th column and with $ as field separator for the output .
~$ awk ‘BEGIN{FS=”,”;OFS=” $ “}{print $1,$4}’ db.txt
Output:
John $ IBM
Barbi $ JHH
Mitch $ BofA
Tim $ DELL
Lisa $ SmartDrive
Note: Space is give before and after $ in OFS variable to show better output.
RS (Row separator)
Row Separator is helpful in defining separator between rows in a file. By default awk
takes row separator as new line. We can change this by using RS built-in variable.
Example:
Convert a sentence to a word per line. We can use RS variable for doing it.
~$ echo “This is how it works” | awk ‘BEGIN{RS=” ”}{print $0}’
Output:
This
is
how
it
Works
ORS (Output Record Separator)

This variable is useful for defining the record separator for the awk command output.
By default ORS is set to new line.
Example:
Print all the company names in single line which are in 4th column.
~$ awk -F’,’ ‘BEGIN{ORS=” “}{print $4}’ db.txt
Output:
IBM JHH BofA DELL SmartDrive
NF
This variable keeps information about total fields in a given row. The final
value of a row can be represented with $NF.
Example: Consider abc.txt which contains below data:

Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47
Print number of fields in each row in abc.txt.
~$ awk ‘{print NF}’ abc.txt
Output:
5
5
4
5
4
NR
This variable keeps the value of present line number. This will come handy when you
want to print line numbers in a file.
Example:
Print line number for each line in a given file.
~$ awk ‘{print NR, $0}’ abc.txt
Output:
1 Jones 2143 78 84 77
2 Gondrol 2321 56 58 45
3 RinRao 2122234 38 37
4 Edwin 253734 87 97 95
5 Dayan 24155 30 47
This can be treated as cat command -n option for displaying line number for a file
FNR
This variable keeps count of number of lines present in a given file/data. This will
come handy when
you want to print no of line present in a given file. This command is equivalent to wc
-l command.
Example:
Print total number of lines in a given file.
~$ awk ‘END{print FNR}’ abc.txt
Output:
5
FILENAME
This variable contain file awk command is processing.
Example:
Print filename for each line in a given file.
~$ awk ‘{print FILENAME, NR, $0}’ abc.txt
Output:
abc.txt 1 Jones 2143 78 84 77
abc.txt 2 Gondrol 2321 56 58 45
abc.txt 3 RinRao 2122234 38 37
abc.txt 4 Edwin 253734 87 97 95
abc.txt 5 Dayan 24155 30 47
awk Built in Function
A function is a self-contained computation that accepts a number of arguments as

input and returns some value. awk has a number of built-in functions in two groups:
arithmetic and string functions.
Arithmetic Functions
Nine of the built-in functions can be classified as arithmetic functions. Most of them
take a numeric argument and return a numeric value. Below table summarizes these
arithmetic functions with some Examples.
awk Function Description

cos ( x ) Returns cosine of x (x is in radians).
exp ( x ) Returns e to the power x.
index (s1,s2) Position of string s2 in s1; returns 0 if not present
int ( x ) Returns truncated value of x.
log ( x ) Returns natural logarithm (base- e) of x.
sin ( x ) Returns sine of x (x is in radians)
sqrt ( x ) Returns square root of x.

atan2 ( y , x ) Returns arctangent of y / x in the range - to .
rand () Returns pseudo-random number r, where 0 <= r < 1.
sqrt(expr) Returns the square root of the expression or value given
Examples:
~$ awk 'BEGIN{
print sqrt(16);
print sqrt(0);
print sqrt(-12);
}'
Output:
4
0
nan
Here nan stands for not a valid number.
String Functions
The built-in string functions are much more significant and interesting than the
numeric functions. Because awk is essentially designed as a string-processing
language, a lot of its power derives from these functions. Below table lists the string
functions found in awk. awk's Built-In String Functions
Filters and Regular Expression

grep
grep command allows you to search one file or multiple files for lines that contain a
pattern.
Full form of grep is global regular expression print.
It is a powerful file pattern searcher in Linux
grep's exit status is 0 if matches were found, 1 if no matches were found, and 2 if
errors occurred.
grep search the target file(s) for occurrences of pattern, where pattern may be literal
text or a Regular Expression.
Syntax:
grep pattern [file...]
Search for the given string in a single file

The basic usage of grep command is to search for a specific string in the specified file
as shown below.
Checking given string in multiple files

We can use grep command for searching for a given string in multiple files.
For example, let us copy the demo_file to demo_file1and use the grep on both the
files to search the pattern this.
The output will include the file name in front of the line that matched the specific
pattern as shown below.
When the Linux shell sees the meta character, it does the expansion and gives all the
files as input to grep.
Case insensitive search

We can use grep to search for the given string/pattern case insensitively. So it matches
all the words such as “the”, “THE” and “The” case insensitively as shown below.
Match regular expression in files

This is a very powerful feature of grep . In the following example, it searches for all
the pattern that starts with “lines” and ends with “empty” with anything in-between.
i.e To search “lines[anything in-between]empty” in the demo_file.
A regular expression may be followed by one of several repetition operators:
8. ? The preceding item is optional and matched at most once.

9. The preceding item will be matched zero or more times.
10. + The preceding item will be matched one or more times.
11. {n} The preceding item is matched exactly n times.
12. {n,} The preceding item is matched n or more times.
13. {,m} The preceding item is matched at most m times.
14. {n,m} The preceding item is matched at least n times, but not more than m
times.
Checking for full words

To search for a word, and to avoid it to match the substrings -w option is used. The
following example is the regular grep where it is searching for “is”. When you search
for “is”, without any option it will show out “is”, “his”, “this” and everything which
has the substring “is”.
Searching in all files recursively

When you want to search in all the files under the current directory and its sub
directory ‘–r’ option is the one which you need to use. The following example will
look for the string “ramesh” in all the files in the current directory and all its
subdirectory.
$ grep -r "ramesh" *
Invert match
If you want to display the lines which does not matches the given string/pattern, use
the option -v as shown below. This example will display all the lines that did not
match the word “Two”.
Displaying the lines which does not matches the entire given pattern.
Syntax:
grep -v -e pattern -e pattern
For example, the file file1 has the following content

Apple
Banana
Cauliflower
Grapes
Orange
Counting the number of matches

Count the number of lines matched in the given pattern/string, then use the option -c.
Syntax:
grep -c pattern filename
Displaying only the file names which matches the given pattern
The -l option is used to display only the file names which matched the given pattern.
When you give multiple files to the grep as input, it displays the names of file which
contains the text that matches the pattern, will be very handy when you try to find
some notes in your whole directory structure.
Showing line number while displaying the output

To show the line number of file with the line matched, -n option is used.
Syntax:
grep -n pattern filename
Example:
grep -n "this" demo_file
2: this line is the 1st lower case line in this file.
6: Two lines above this line is empty.
sed
sed is a stream editor used to perform basic text transformations on an input stream (a
file, or input from a pipeline).
Working methodology
sed works by making only one pass over the input(s) s called as one execution cycle.
Cycle continues till end of file/input is reached.
• Read entire line from stdin/file.

• Removes any trailing newline.
• Places the line, in its pattern buffer.
• Modify the pattern buffer according to the supplied commands.
• Print the pattern buffer to stdout.
Printing Operation in sed

sed allows you to print only specific lines based on the line number or pattern
matches. “p” is the command for printing the data from the pattern buffer. To suppress
automatic printing of patternspace -n option is used with sed. sed -n option will not
print anything, unless an explicit request to print is found.
Syntax:
sed -n 'ADDRESS'p filename
sed -n '/pattern/p' filename
Examples:
Let us assume the demo_file has the following content
To prints third line of input file
$sed -n '3p' demo_file

3. Hardware
To print every nth line starting from the line m
$sed -n 'm~np' filename
To print only the last line
To print the lines containing the given pattern:

Syntax:
sed -n /PATTERN/p filename
Deletion operation in sed

In sed the d command is used to delete the pattern space buffer and immediately starts
the next cycle.
Syntax:
sed nd filename
'nd’ deletes the nth line and prints the other lines.
sed 'ADDRESS'd filename
sed /PATTERN/d filename
The process is
• It reads the first line and places in its pattern buffer
• checks whether supplied command is true for this line , if true, deletes pattern space
buffer and starts next cycle and reads the next line.
• If supplied command is not true, it prints the content of the pattern space buffer.
To delete the 3rd line and print other lines from the file demo_file
Substitution operation in sed

In sed the s command is used to substitute the pattern. The `s’ command attempts to
match the pat-tern space against the supplied expression/ pattern; if the match is
successful, then that portion of the pattern space which was matched is replaced with
the replacement given.
Syntax:
$sed 'ADDRESSs/REGEXP/REPLACEMENT/FLAGS' filename
$sed 'PATTERNs/REGEXP/REPLACEMENT/FLAGS' filename
5. s is substitute command
6. / is a delimiter
7. REGEXP is regular expression to match
8. REPLACEMENT is a value to replace
FLAGS can be any of the following:
7. g Replace all the instance of REGEXP with REPLACEMENT

8. n Could be any number,replace nth instance of the REGEXP with
REPLACEMENT.
9. p If substitution was made, then prints the new pattern space.
10. i match REGEXP in a case-insensitive manner.
11. w file If substitution was made, write out the result to the given file.
12. We can use different delimiters ( one of @ % ; : ) instead of /
To Write Changes to a File and Print the Changes
To combine multiple sed commands we have to use option -e
Syntax:
$sed -e 'command' e 'command' filename
To Delete the first,last and all the blank lines from input
FILTERS USING REGULAR EXPRESSION
A regular expression is a set of characters that specify a pattern. Regular expressions

are used when you want to search for specific lines of text containing a particular
pattern. Most of the UNIX utilities operate on ASCII files a line at a time. Regular
expressions search for patterns on a single line, and not for patterns that start on one
line and end on another.
The Structure of a Regular Expression
There are three important parts to a regular expression.

• Anchors : These are used to specify the position of the pattern in relation to a line of
text.
• Character Sets : The set of characters that match one or more characters in a single
position.
• Modifiers: They specify how many times the previous character set is repeated.
A simple example that demonstrates all three parts is the regular expression is : "^#*"
Here ,
• The up arrow , “^”, is an anchor that indicates the beginning of the line.
• The character "#" is a simple character set that matches the single
character "#".
• The asterisk “*” is a modifier. In a regular expression it specifies that the
previous character set can appear any number of times, including zero.
There are also two types of regular expressions:

• the "Basic" regular expression,(BRE)
• the "extended" regular expression.(ERE)
A few utilities like awk and egrep use the extended expression. Most use the "regular"
regular expression. From now on, if I talk about a "regular expression," it describes a
feature in both types.
The Anchor Characters: ^ and $ Anchores are used when we want to search for a
pattern that is at one end or the other, of a line. The character "^" is the starting
anchor, and the character "$" is the end anchor.Following list provides a summary:
Pattern Matches
Â "A" at the beginning of a line
A$ "A" at the end of a line
A^ "A^" anywhere on a line
$A "$A" anywhere on a line
^^ "^" at the beginning of a line
$$ "$" at the end of a line
The Character Set

The character set also called “character class” in a regular expression , is used to tell
the regex engine to match only one out of several characters
• A character set matches only a single character. In case of the above example,
gr[ae]y does not match graay, graey or any such thing.
• The order of the characters inside a character set does not matter. The results
are identical.
• Some characters have a special meaning in regular expressions. If we want to
search for such a character, we have to escape it with a backslash.
Exception in the character class

If we want to search for all the characters except those in the square bracket,
then the ^ (Caret) symbol needs to be used as the first character after open
square bracket. The expression "^[âeiou]" is to searc for a line which does
not start with the vowel letter.
Regular Expression Matches
[] The characters "[]"
[0] The character "0"
[0-9] Any number
[^0-9] Any character other than a number
[-0-9] Any number or a "-"
[0-9-] Any number or a "-"
[^-0-9] Any character except a number or a "-"
[]0-9] Any number or a "]"
[0-9]] Any number followed by a "]"
[0-9-z] Any number, or any character between "9" and "z".
[0-9\-a\]] Any number, or a "-", a "a", or a "]"
Match any character

The character "." is one one of thespecial meta-characters. By itself it will match any
character, except the end-of-line character. Thus the pattern that will match a line with
a single characters is ^.$
Repeating character sets

The third part of a regular expression is the modifier. It is used to specify how may
times you expect to see the previous character set. The repetition modifier * find no or
one, one or more, and zero or more
repeats, respectively.
Examples:
Expression Matches
Go*gle Gogle,Google,Gooogle, and so on.
"[0-9]*" zero or more numbers.
Matching a specific number of sets with \{ and \}

We cannot specify a maximum number of sets with the "*" modifier. There is a
special pattern we can use to specify the minimum and maximum number of repeats,
by putting those two numbers between "\{" and "\}".
▪ A modifier can specify amounts such as none, one, or more;
For example , A user name is a string beginning with a letter followed by at least two,
but not more than seven letters or numbers followed by the end of the string. Then the
regular expression is
^[A-z][A-z0-9]{2,7}
▪ A repetition modifier must be combined with other patterns; the modifier has no
meaning by itself.
For example , modifiers like "*" and "\{1,5\}" only act as modifiers if they follow a
character set. If they were at the beginning of a pattern, they would not be a modifier.
grep with Regular expression
Search for 'vivek' in /etc/passswd

grep vivek /etc/passwd
Search vivek in any case (i.e. case insensitive search)
grep -i -w vivek /etc/passwd
Search vivek or raj in any case
grep -E -i -w 'vivek|raj' /etc/passwd
Line and word anchors
Search lines starting with the vivek only

grep ^vivek /etc/passwd
To display only lines starting with the word vivek only i.e. do not display vivekgite,
vivekg
grep -w ^vivek /etc/passwd
To Find lines ending with word foo
grep 'foo$' filename
Character classes
To match Vivek or vivek.

grep '[vV]ivek' filename
OR
grep '[vV][iI][Vv][Ee][kK]' filename
To match digits (i.e match vivek1 or Vivek2 etc)

grep -w '[vV]ivek[0-9]' filename
Wildcards
To match all 3 character word starting with "b" and ending in "t".
grep '\' filename
Where,
•\< Match the empty string at the beginning of word
•\> Match the empty string at the end of word.
Print all lines with exactly two characters
grep '^..$' filename
Display any lines starting with a dot and digit
grep '^\.[0-9]' filename
Escaping the dot
To find an IP address 192.168.1.254

grep '192\.168\.1\.254' /etc/hosts
Search a Pattern Which Has a Leading – Symbol
Searches for all lines matching '--test--' using -e option . Without -e, grep would
attempt to parse '--test--' as a list of options
grep -e '--test--' filename
Test Sequence
To Match a character "v" two times

egrep "v{2}" filename
To match both "col" and "cool"

egrep 'co{1,2}l' filename
grep OR Operator
Suppose the file “employee” has the following data:
To find the records of those who are either from Tech or Sales dept.
We can use the following syntaxes :
1) Syntax : grep 'word1\|word2' filename

grep 'Tech\|Sales' employee
2) Syntax : grep -E 'pattern1|pattern2' fileName

grep -E 'Tech|Sales' employee
grep AND Operator

There is no AND operator in grep. But, we can simulate AND using
• grep -E option.
Syntax : grep -E 'word1.*word2 ' filename
grep -E 'word1.*word2|'word2.*word1' filename
• multiple grep command separated by pipe

Syntax : grep 'word1' filename | grep 'word2'
Environment Variables:
An important concept in Unix is the environment, which is defined by values stored

in a set of predefined variables called environment variables.
Values of some of these variables are set by the system, some can be set by the user,
some of them by the shell, or any program that loads another program.
When a user logs in to the system, the shell undergoes a twostep process
called initialization to set up the environment. This process involves the shell reading
two files −/etc/profile and .profile in the home directory of the user.
The file /etc/profile is maintained by the system administrator of the Unix machine
and contains shell initialization information required by all users on a system.
The file .profile is under the control of the user. User can add as much shell
customization information as he/she wants to this file. The minimum set of
information that a user need to configure includes −
• The type of terminal user is using.

• A list of directories in which to locate the commands.
• A list of variables affecting the look and feel of the terminal.
Environment around the UNIX Process:
When a process is created, UNIX opens 3 streams stdin/stdout/stderr for basic

communication with respect to the process control terminal. In addition, it knows the
current working directory for performing file I/O.
Each login shell maintains a description of the environment as a table of pointers to

strings.
A global shell environment pointer called environ is maintained by UNIX kernel and
it can be used by a process to access its own table
The shell does not directly use this table, but it creates a child process and calls exec()
system call to execute a command program that uses the table inherited from the shell
parent. Children inherit the entire execution environment from the parent.
Some examples of the environment variables are the USER, LOGNAME, HOME,
PATH, PS1, PS2, TERM MAIL, etc.
The HOME Variable

It specifies an associated directory with every user in a UNIX system. If the HOME
variable for the user Sita contains /usr/sita/stores, every time Sita logs in, she is taken
to the directory stores.
The variable HOME is referenced the same way:
$ echo ${HOME}
The PATH Variable

contains a list of all full path-names (separated by a colon) of directories that are to be
searched for an executable program. For example, the command
$PATH=.:/usr/bin:/bin specifies directories to be searched for any executable file or a
command file (current directory, /usr/bin and /bin, in that order).
The PS1 Variable

The system prompt may be changed by setting the value of this variable to the desired
prompt:
$ PS1=“Hello>”
Hello> #can be changed only at the UNIX command line, not within a shell script.
The PS2 Variable: prompt string for continued command line (default ‘> ‘).
The LOGNAME Variable

contains user’s login name. Its contents cannot be changed by the user, but can be
displayed:
echo “${LOGNAME}”
The TERM Variable

Names the kind of terminal you are using; setting it helps to manage your screen more
effectively, say,
$ TERM=vt100
The PWD Variable

The current working directory can be displayed:
echo “${PWD}”
In fact the whole environment table can be displayed.
$IFS: String of characters which are used as word separators in command line ( space,
tab, newline chars).
The MAIL Variable

Names the standard file where your mail is kept
The .profile File
Some of above variables like HOME and LOGNAME are set automatically each time
the user logs in. The others, however, have to be set. The .profile is used for this
purpose as it is executed as soon the user logs in. A sample .profile file would look
like:
PATH=.:/bin:/usr/bin
export HOME PATH PS1 MAIL
PRACTICE ASSIGNMENTS FOR UNIX

Basic Unix Commands :
1. Create a directory Unix in your home directory

2. Create two sub directories dir1 and dir2 under Unix directory.
3. Go to dir1.
4. Create a file myFile in the dir1 and insert few lines.
5. Make a duplicate copy of the file myFile in dir2
6. Append few lines of statements in file myFile in dir2.
7. Change the name of the file myFile in dir2 to newFile.
Filters and Pipes:
Consider this file as the sample myFile.
Hello!
Welcome to this course on Unix.
You are expected to complete the assignments to complete the course.
You can save this file in your system to use it. This file can be used for practice.
I have created this file to complete this assignment.

Happy Learning!
8. Display how many number of lines are there in myFile.
Expected Output : 6
9.Count how many users are currently logged into the system
10.Display the lines having the word "this"/”This” in myFile
Expected Output :
11.Count number of lines having the word "this"/”This” in myFile.
Expected Output : 3
12.Replace the word this by that for all occurrences in myFile.
Expected Output :
Hello!
Welcome to that course on Unix.

You can save that file in your system to use it. This file can be used for practice.
I have created that file to complete that assignment.
Happy Learning!
13.Replace the word this by that only for the first occurrence of it in a line.
Expected Output :
Hello!
I have created that file to complete this assignment.
Happy Learning!
14. Print the first 3 lines from the file myFile
Expected Output :
Hello!
15. Print the last 3 lines from the file myFile
Expected Output :
Happy Learning!
16. Append a new line in the beginning of the file myFile
awk Scripting :
17. Create a file with name student.txt with the following contents in it
RollNo|Name|Marks1|Marks2|Marks3
123|Raghu|80|90|60
342|Maya|45|78|82
561|Gita|56|71|89
480|Mohan|71|90|89
a. Write an awk command to print the name and roll number of the students.
Expected Output:
Name RollNo
Raghu 123
Maya 342
Gita 561
Mohan 480
b . Write the awk command to calculate the total marks of each student and display
the total marks along with the name of the student.
Expected Output:
Raghu-230
Maya-205
Gita-216
Mohan-250
Shell Scripting
18. Write a shell script to print the sum of the 1st 10 natural numbers.
Expected Output: 55
19. Write a shell script to print the odd numbers from a given list of numbers. The list
of numbers will be given as command line argument.
Example :
If input numbers are 10,3,45,6,1,100, then
Expected Output :
45
20. Create a file studentScores.txt and append the following content to it.
RollNo|Name|Marks
123|Raghu|80
342|Maya|45
561|Gita|56
480|Mohan|71
write a shell script named scores.sh to find the name of the student who scored highest
marks. Incorporate validations for the following :
a. Whether correct number of command line arguments are passed or not.
b. whether the given files exists or not.
c. whether the given file is readable or not.
Expected Output :
a. When executed as sh scores.sh, output should be
“Pass correct number of arguments.”
b. When executed as sh scores.sh studentScores.txt, but studentScores.txt does not

exist, output should be
“ File does not exist”
c.When executed as sh scores.sh studentScores.txt, but studentScores.txt does not

have read permission, output should be
“ File is not readable”.
d. When executed as sh scores.sh studentScores.txt, output should be
“Name of the student with highest marks : Raghu”

Practice the questions in any of the below given link :
1. https://www.tutorialspoint.com/unix_terminal_online.php
2. http://www.compileonline.com/execute_bash_online.php
3. https://bellard.org/jslinux/vm.html?url=https://bellard.org/jslinux/buildroot-
x86.cfg
The UNIX Process:
A process in UNIX is a program in execution with definite life-time and well-

defined hierarchy.
• The context of a process is a snapshot of its current run-time environment

that consists of the current state of the processor registers and
• User program image - Process execution on user data structure that
represents an application and is manipulated by its own functions (user
mode execution).
• System image - Process execution on system’s data structures that
represent the resources (memory, files, devices) and supported by the
kernel routines. Depending upon resource requirement and availability,
process’s states are controlled by executing the kernel routines accessed
through system calls (system mode execution).
• The kernel maintains a process table to manage all processes. The two
data structures per process are the user structure and the process structure.
• The kernel process is the first (root) process that comes into existence
when the system is booted. Its process_id and group_id are both 0.
• In establishing a multi-programming environment, the kernel
process creates the init process with its process_id of 1 and group_id, 0,
showing that process 1 is part of process 0. The init process creates and
manages terminal processes for active terminals in the system.
• At the time of creation, each terminal process belongs to process group 0
and is made to execute a program called getty. Once each terminal
process (now called a getty process) establishes communication with the
associated terminal, it displays a login message and waits for a user to
input a user name (Process group of the init process).
• When a user starts a dialogue, the getty process receives the user name
and leaves the task of validating the user’s password to another program
called login. The same terminal process is now called a login process.
• The login process, after validating the user’s password, calls a command
line program, the login shell to run in the same process. The same
terminal process is now called a shell process.
• Each shell process now establishes a new process group and becomes
ready to process the user commands. A shell process is the initiating
process by which each terminal maintains the user session.
• While interpreting a command, the shell creates an execution thread and
then assigns the requested command program to this new process.
• Both the shell and the new process proceed independently in separate
execution threads. The parent shell process normally waits until child
process completes its execution.
A process require certain resources to accomplish its intended task, they are
>> CPU time
>> System memory
>> Disk Files
>> I/O devices.
Process control blocks (PCB)

The OS must know all the information about a specific process in order to manage
and control it. The OS maintains a table (an array of structures), called the process
table, with one entry per process. These entries are called process control blocks
(PCB) - also known as task control block.This entry contains information about the
process' state, its program counter, stack pointer,memory allocation, the status of its
open files, its accounting and scheduling information, and everything else about the
process that must be saved when the process is switched from ready to running or
blocked state so that it can be restarted later with the status it left off.
Process state:- The state may be new, ready, running, waiting, halted, and so on.
Program counter:- The counter indicates the address of the next instruction to be
executed for this process.
CPU registers:- The registers vary in number and type, depending on the computer
architecture. They include accumulators, index registers, stack pointers, and general-
purpose registers, plus any condition-code information.
CPU-scheduling information:- This information includes a process priority, pointers
to scheduling queues, and any other scheduling parameters.
Memory-management information:- This information may include such
information as the value of the base and limit registers, the page tables, or the segment
tables, depending on the memory system used by the OS.
Accounting information:- This information includes the amount of CPU and real
time used, time limits, account numbers, job or process numbers, and so on.
I/O status information:- This information includes the list of I/O devices allocated to
the process, a list of open files, and so on.
Process state:
A process may present in one of the following state.
New:- The process is being created or just created.
Ready:- The process is waiting to be assigned/allocated to a processor time.
Running:- Instructions of the process are being executed by processor.
Waiting:- The process is waiting for some other event/process to execute.
Terminated:- The process has finished its execution.
ps command:
The ps (i.e., process status) command is used to provide information about the
currently running processes in the system. When ps is used without any option four
columns of information labeled PID,TTY, TIME and CMD for at least two processes,
the shell and ps will be visible.
PID :- The process are identified by a 5 digit number known as PID (Process
Identification Number).
TTY :- Is the name of the console or terminal that the user logged into (Stands for
terminal type now
but originally stood for teletype).
TIME :- Is the amount of CPU time in minutes and seconds that the process has been
running.
CMD :- is the name of the command that launched the process.
The information that ps -aux provides about each process is :

>> The user of the process,
>> PID of the process,
>> Percentage of CPU used by the process,
>> Percentage of memory used by the process,
>> VSZ (virtual size in kilobytes),
>> RSS (real memory size or resident set size in 1024 byte units),
>> STAT (the process state code, explained later),
>> Starting time of the process,
>> Length of time the process has been active
>> The command that initiated the process.
Types of process
Parent and Child Process:
A process can initiate a sub process, which is a called a child process, the initiating
process is referred to as its parent.The child processes, in turn create other child
processes forming a tree of processes
( which can be displayed using ps command with –forest option)
Orphan Process:
When a child process is killed, parent process gets the notification via a signal. Parent
then, can continue other task. However if the parent process is killed before, its child,
is called an orphan process.
Zombie Process:
When a process finished its execution and exit status not received by the parent ( or
parent did not read the exit status till now), the process state becomes zombie.
The process is dead (not to be scheduled for further execution) but cannot
becompletely removed from process table, until it has been determined that exit status
is no longer needed.
Daemon Process
Some programs are not designed to be run with continuous user input and disconnect
from the terminal when task completed. For example, a web server responds to web
requests, rather than user input. Mail servers are another example of this type of
application. These types of programs are known as daemons.
Starting a Process:
Process can be started in two ways:
In Foreground: By default every process starts in foreground, ie. Gets the input from
keyboard and sends the output in monitor. But in this case till the process completes
its execution no other process can be started in foreground.
In Background: To take the advantage multiprocessing environment, a process can
be started in background, so that other process can be started in the foreground
without waiting for the previous process to complete execution.
A process can be started in background by adding ampersand(&) after it.
[390119 @ INGNRILPORCL] $ $ ls | wc -l > file1 &
Switching process from foreground to background:

A process running in foreground can be send to background using the following steps:
>> Press + Z to suspend the job
>> bg command puts the job in background
>> nohup unattaches the job from the terminal*
$ cat > file

File content...
+z
[1] + Stopped
$ bg
[1] + cat > file &
Switching process from background to foreground

Process running in the background can be taken into foreground using the following
steps:
>> Find the job id of the process by the command jobs
>> Use fg %to get the job to foreground
Stopping/Killing a Process
A process dies(terminates ) automatically when it completes the job it intended to.
A process can be killed abnormally with the command kill.
>> Use the ps command to find out the process-id for the process
>> Use the command kill to terminate it
>> Use the command kill -9 to kill a process forcefully.
Option settings for background process

While a process is executing/running , if the owner tries to log off the process ,process
will get killed. Sometimes a job or command takes a long time to complete and it is
required the job to be completed without interruption. This situation can be handled in
two different ways:
>> Does not allow the use to log off
>> Continue execution in background even after the user logged off
This can be achieved using command nohup
PERIODICAL EXECUTION OF JOBS
Sometimes it may be required to execute certain Job or task on specific time. It may
be not possible for any user to start or execute the job physically on that particular
time due to other business requirements. Again if similar task need to be executed
periodically on a regular interval, then the task becomes very hectic. Unix internally
has solutions to handle this type of situations using utilities such as cron and at.
Crontab
The word 'crontab' in UNIX stand for chron(chronograph) table, or time table for
Unix system .
The crontab is a special table where it is possible to specify commands and time or
interval to execute the command once or repeatedly. Commands can be any
executable programs, for example, a script can be written to take regular backup of all
the required tables in database. A crontab instruction can be created to execute the
script on the specified time. The cron daemon reads the table and executes the
commands at the times specified.
Crontab Command Syntax

Syntax
Components of Crontab
The basic format of a crontab schedule consists of 6 fields, placed on a single line and
separated by spaces, formatted as follows:
The various fields and their utility are explained in the image below:
Example:
To executes an incremental backup shell script "takeBackup.sh" available

in your home directory at 11:00 on every day the cron entry will be as
below:
00 11 * * * ~/takeBackup.sh
To schedule the same job for every minute, the cron entry will be as
follows:
* * * * * ~/takeBackup.sh
Shell Programming
The UNIX shell program interprets user commands, which are either directly entered
by the user, or which can be read from a file called the shell script or shell program.
Shell scripts are interpreted, not compiled. The shell reads commands from the script
line by line and searches for those commands on the system.
Shell script
Shell script is a file-containing list of commands to be executed in a particular order.
A good shell script will have comments, (a statement preceded by a pound sign, #) ,
describing the purpose of the statement.
In a script we can use conditional tests, such as value A is greater than value B, loops
or iterative statements to execute some steps repetitively or to navigate through a list
of data or records and do processing. We can use files to read and store data. Can
use variables to read and store data. A script may include functions also.
When a script is executed, the shell reads the commands one by one and executes
them .
We can create the simple shell script file by using vi editor or cat command like,
$ vi test.sh
$ cat > test.sh
Below mentioned shebang statement should be the first statement in the shell script as
it tells the system that the commands mentioned in the shell script are to be executed
by the shell /bin/sh
#!/bin/sh
Consider the shell script with just two commands pwd & ls.
$cat test.sh
#!/bin/bash
pwd
ls
Importance of shell script

Shell scripts are basically used for automating processes that we repeat at the prompt .
Following are some activities we can do using shell scripts:
• Automation of repetitive task

• Creating our own power tools/utilities.
• Automating command input or entry.
• Customizing administrative tasks.
• Creating simple applications.
• Automating administration tasks such as adding new users, removing obsolete
users etc
Some Practical examples where shell scripting can be actively used:
• Monitoring your Linux system.

• Data backup and creating snapshots.
• Dumping Oracle or MySQL database for backup.
• Creating email based alert system.
• Find out what processes are eating up your system resources.
• Find out available and free memory.
• Find out all logged in users and what they are doing.
• Find out if all necessary network services are running or not. For example if
web server failed then send an alert to system administrator via a pager or an
email.
• Find out all failed login attemp. If login attempt are continued repeatedly from
same network IP, automatically block all those IPs accessing your
network/service via firewall.
• User administration as per your own security policies.
• Find out information about local or remote servers.
• Configure server such as BIND (DNS server) to add zone entries.
Executing a shell script

Below two methods can be used to execute the shell script.
$ sh filename
Or
$ ./filename
In this case we have to modify the file access permissions of the shell script before
execution.
To provide execute permission , following command is used.
$ chmod u+x filename
Basic Operators in shell scripting

Below operators are supported by shell.
• Arithmetic Operators.
• Relational Operators.
• Boolean Operators.
• String Operators.
• File Test Operators.
Here is simple example to add two numbers:
Example:
#!/bin/sh
val=èxpr 2 + 2`
echo "Total value : $val"
Output:
$ Total value : 4
There are following points to note down:
• There must be spaces between operators and expressions for example 2+2 is
not correct, where as it should be written as 2 + 2.
• Complete expression should be enclosed between ``, called inverted commas to

execute expr command correctly.
Arithmetic Operators
Assume variable a holds 10 and variable b holds 20 then:
Operator Description Example

+ Addition èxpr $a + $b ` will give 30
- Substraction èxpr $a - $b ` will give -10
* Multiplication èxpr $a \* $b` will give 200
/ Division èxpr $b / $a` will give 2
% Modulus èxpr $a % $b` will give 0
!= Not equal [ $a != $b ] will give true
= assignment a=$b will assign value of b to a.
== Equality [ $a == $b ] will return false.
It is very important to note here that all the conditional expressions would be put
inside square braces with one spaces around them, for example [ $a == $b ] is correct
where as [$a==$b] is incorrect.
Relational Operators
Below are relational operators which are specific to numeric values. These operators
would not work for string values unless their value is numeric.
For example, following operators would work to check a relation between 10 and 20
as well as in between "10" and "20" but not in between "ten" and "twenty".
-eq Check if the values of 2 operands are [ $a -eq $b ] is

equal or not, if yes then condition false
becomes true.
-ne Check if the values of 2 operands are [ $a -eq $b ] is
equal or not, if values are not equal true
then condition becomes true.
-gt Check if the value of left operand is [ $a -gt $b ] is
greater than the value of right operand, false
if yes then condition becomes true.
-lt Check if the value of left operand is [ $a -lt $b ] is
less than the value of right operand, if true
yes then condition becomes true.
-ge Check if the value of left operand is [ $a -ge $b ] is
greater than or equal to the value of false
right operand, if yes then condition
becomes true.
-le Check if the value of left operand is [ $a -le $b ] is
less than or equal to the value of right true
operand, if yes then condition becomes
true.
inside square braces with one spaces around them, for example [ $a <= $b ] is correct
where as [$a <= $b] is incorrect.
Boolean Operators
Assume variable a holds 10 and variable b holds 20 then

! This is logical negation. This [ !false ] is true
inverts a true condition into false
and vice versa.
-o This is logical OR. If one of the [ $a -lt 20 -o $b -gt
operands is true then condition 100 ] is true
would be true.
-a This is logical AND. If both the [ $a -lt 20 -a $b -gt
operands are true then condition 100 ] is false.
would be true otherwise it would
be false.
String Operators
These are string operators. Assume variable a holds "abc" and variable b holds "efg"
then:

-z Check if the given string operand [ -z $a ] will return
size is zero. If it is zero length then false.
it returns true.
-n Check if the given string operand [ -z $a ] will return
size is non- zero. If it is non-zero true.
length then it returns true.
= Check if the value of two operands [ $a = $b ] will return
is equal or not, if yes then false
condition becomes true.
!= Check if the value of two operands [ $a != $b ] will
is equal or not, if the values are not return true
equal then condition becomes true.
str Check if the str is not the empty [ $a ] will return true
string. If it is empty then it returns
false.
File Test Operators:
Assume a variable file holds an existing file name "test" whose size is 100 bytes and
has read, write and execute permission on:

-b file Returns true, if file is a block [ -b $file ] is false.
special file
-c file Returns true, if file is a character [ -b $file ] is false.
special file
-d file Returns true, Check if file is a [ -d $file ] is not true.
directory
-f file Returns true, Check if file is an [ -f $file ] is true.
ordinary file or special file
-r file Returns true, Checks if file is [ -r $file ] is true.
readable
-w file Returns true, Check if file is [ -w $file ] is true.
writable
-x file Returns true, Check if file is [ -x $file ] is true.
execute
-s file Returns true, Check if file has size [ -s $file ] is true.
greater than 0
-e file Returns true, Check if file exists [ -e $file ] is true.
Wild Card Characters
Symbol used to replace or represent one or more characters. Wildcards or wild

characters are either asterisk (*), which represent one or more characters or question
mark (?), which represent a single character.
Wild card
Meaning Examples
/Shorthand
Matches any $ ls * will show all files
string or group of $ ls a* will show all files whose first
characters. name is starting with letter 'a'
* $ ls *.c will show all files having
extension .c
$ ls Will show all files having
ut*.c extension .c but file name
must begin with 'ut'.
? Matches any $ ls ? will show all files whose
single character. names are 1 character long
$ ls will show all files whose
fo? names are 3 character long
and file name begin with fo
[...] Matches any one $ ls Will show all files beginning
of the enclosed [abc]* with letters a,b,c
characters
Note: [..-..] A pair of characters separated by a minus sign denotes a range.
Example:
$ ls /bin/[a-c]*
Will show all File name beginning with letter a,b or c.

Output:
/bin/arch /bin/awk /bin/bsh /bin/chmod /bin/cp /bin/as /bin/basename /bin/cat
Shell Quoting Mechanism
The Metacharacters
Unix Shell provides various metacharacters which have special meaning while using
them in any Shell Script and causes termination of a word unless quoted.
Example:
? Matches with a single character while listing files in a directory and an * would
match more than one characters.
Here is a list of most of the shell special characters (also called metacharacters):
* ? [ ] ' " \ $ ; & ( ) | ^ < > new-line space tab
A character may be quoted (i.e., made to stand for itself) by preceding it with a \.
Example:
#!/bin/sh echo Hello; Word
This would produce following result.
Hello ./test.sh: line 2: Word: command not found shell returned 127
Now let us try using a quoted character:
#!/bin/sh echo Hello\; Word
This would produce following result:
Hello; Word
The $ sign is one of the metacharacters, so it must be quoted to avoid special handling
by the shell:
#!/bin/sh echo "I have \$1200"

I have $1200
Quote Description
Single quote All special characters between these quotes lose their
special meaning.
Double Most special characters between these quotes lose their
quote special meaning with these exceptions:
\$
\'
\"
\\
Backslash Any character immediately following the backslash loses
its special meaning.
Back Quote Anything in between back quotes would be treated as a
command and would be executed.
The Single Quotes

Consider an echo command that contains many special shell characters:
echo <-$1500.**>; (update?) [y|n]
Putting a backslash in front of each special character is tedious and makes the line
difficult to read:
echo \<-\$1500.\*\*\>\; update\?update\?
y∥ny‖n
There is an easy way to quote a large group of characters. Put a single quote ( ') at the
beginning and at the end of the string:
echo '<-$1500.**>; (update?) [y|n]'
Any characters within single quotes are quoted just as if a backslash is in front of each
character. So now this echo command displays properly.
If a single quote appears within a string to be output, you should not put the whole
string within single quotes instead you would precede that using a backslash (\) as
follows:
echo 'It\'s Shell Programming
The Double Quotes:

Try to execute the following shell script. This shell script makes use of single quote:
VAR=ZARA
echo '$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]'
$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]
So this is not what you wanted to display. It is obvious that single quotes prevent
variable substitution. If you want to substitute variable values and to make invert
commas work as expected then you would need to put your commands in double
quotes as follows:
VAR=ZARA
echo "$VAR owes <-\$1500.**>; [ as of (`date +%m/%d`) ]"
Now this would produce following result:
ZARA owes <-$1500.**>; [ as of (07/02) ]
Double quotes take away the special meaning of all characters except the following:
• $ for parameter substitution.

• Backquotes for command substitution.
• \$ to enable literal dollar signs.
• \` to enable literal backquotes.
• \" to enable embedded double quotes.
• \\ to enable embedded backslashes.
• All other \ characters are literal (not special).
Back Quotes: Command Substitution

Putting any Shell command in between back quotes would execute the command
Syntax: var=`command`
Example:
Following would execute date command and produced result would be stored in
DATA variable.
DATE=`date` echo "Current Date: $DATE"
Current Date: Thu Jul
Shell Decision Statement
While writing a shell script, there may be situations when you need to adopt one path
out of many available paths. In such cases you need to make use of conditional
statements that allow your program to make correct decisions and perform right
actions.
Unix Shell supports conditional statements, which are used to perform different
actions based on different conditions.
Here we will explain following two decision-making statements
• The if...else statement

• The case...esac statement
If..else statements
We can use “if..else” statement which can be used as decision making statement to
select an option from a given set of options.
Unix Shell supports following forms of if..else statement:
• if...fi statement
• if...else...fi statement
• if...elif...else...fi statement
Syntax:
if [condition]
then
command(s)
fi
if [ condition(s) ] then
command(s)
else
command(s)
fi
if [ condition(s) ]
then
command(s)
elif [ condition(s) ] # Many elif-then allowed

then
command(s)
else
command(s)
fi
test command in if condition
We can use test command as condition of if condition as used in the below script.
$cat if_test.sh
#!/bin/sh
echo “Do you want to quit (Y/y)? ”
read ans
if test $ans ==’y’ –o $ans==’Y’
then
exit
else
echo “ to exit enter N or n”
fi
case...esac Statement
We can use multiple if...elif statements to perform a multiway branch. However, this
is not always the best solution, especially when all of the branches depend on the
value of a single variable.
Unix Shell supports case...esac statement which handles exactly this situation, and it
does so more efficiently than repeated if...elif statements.
The interpreter checks each case against the value of the expression until it finds a
match. If nothing matches, goes with the default condition.
Syntax:
case word in pattern1)
Statement(s) to be executed if pattern1 matches
;;
pattern2)
;;
pattern3)
;;
esac
Here string word is compared against every pattern until a match is found and the
statement(s) following the match pattern executes. If shell cannot find any match, the
case statement exits without performing any action. When statement(s) part is
executed. The command ;; indicates program flow should jump to the end of the
entire case statement.
There is no maximum number of patterns, but the minimum is one.
Example:
#!/bin/sh
COURSE=”DB”
case “$COURSE” in
“Java”) echo “Java is a programming language”
;;
“Perl”)echo “Perl is scripting language”
;;
“DB”)echo “Oracle is a DB”
;;
esac
Output:
Oracle is a DB
Iterative Statements/Loop :
Loops are a powerful programming tool that enables you to execute a set of
commands repeatedly.
• while loop
• for loop
• until loop
while loop
Here condition is evaluated by shell at the beginning of each iteration. If the resulting
value is true, given command(s) are executed. If condition is false then no command
would be executed and program would jump to the next line after done statement.
Syntax:
while condition
do # executed as long as the condition is true

command(s)
done
Example:
a=0
while [ $a -lt 3 ]
do
echo $a
a=èxpr $a + 1`
done
Output:
until loop
Here condition is evaluated by shell. If the resulting value is false, given command(s)
are executed. If condition is true then no command would be executed and program
would jump to the next line after done statement.
Syntax:
until condition # complement of while
do
# executes the body as long as the condition is false
command(s)
done
Example:
a=0
until [ ! $a -lt 3 ] do
echo $a a=èxpr $a + 1`
done
Output:
for loop
For loop operate on lists of items. It repeats a set of commands for every item in a list.
Syntax:
for var in w1 w2 w3.. wn
do
command #Statement(s) to be executed for every w in the list
done
Example:
for var in 0 1 2 3 4 5
do
echo $var
done
Output:
Example:
for filename in `ls tempdir`
do
echo “Displaying contents of ${filename}”
cat ${filename}
done
We can use "break" and "continue" commands to control the loop.
Command "break" causes the termination of a loop and “continue” resumes execution
at its top.
String Handling
• String handling with test command:
test str Returns true if str is not null

test –n str Returns true if length of str is
greater than zero
test –z str Returns true if length of str is
equal to zero
• String handling with expr command
The expr is quite handy for finding the length of a string and extracting a sub-string:
Length of the string:
$ str=”abcdefghijk” ;
$ n=èxpr "$str" : ‘.*’` ;
$ echo $n
11
expr gave how many times any character (.*) occurs. This feature is very useful in
validating data entry.
Extracting a sub-string:
$ expr “$str” : ‘……....’
gh
Note that there are 6 dots preceding the sequence ..... This advanced regular
expression signifies that the first six characters of the string are to be ignored and
extraction should start from the 7th character. Two dots inside .... suggests that this
extraction is limited to two characters only (backslashes override the usual
interpretation of ‘()’).
Extracting string from 3rd character to end of the string:
$ str="abcdefghijk"
$ expr "$str" : '...∗.∗'
cdefghijk
Location of first occurrence of a character “d” inside string:
$ expr "$str" : '[^d]*d‘
Location of last occurrence if a character inside string:
Below will give the last occurrence of character 'a' from string str.
$str=”abc def abc”
$expr "$str" : '[û]*a'
9.
6. Command Line Arguments

To make a shell script a generalized script, we should avoid hard coding. User should
be able to provide the values required for processing as input while running the shell
script. To facilitate this we have to use command line arguments.
The statement we write in the command prompt starting with the command followed
by list of arguments is the command line argument.
Example:
$ ls dir1 dir2 dir3

A set of shell variables are there to represent the command line arguments.
$0 – represents the command name (first word on command line). For above example
“ls”
$1 - the first argument of command (second word/symbol ).For the above example
dir1
$2 – the second argument to the command. For the above example, dir2
In this way we can use up to $9
The command-line arguments $1, $2, $3...$9 are also called positional parameters,
with $0 pointing to the actual command/program/shell script and $1, $2, $3, ...$9 as
the arguments to the command.
Consider below sample shell script “test.sh”
$cat test.sh
#!/bin/sh
echo “Program name is $0”

echo “First argument is $1”
echo “Number of arguments passed =$#”
echo “The arguments are $@”
Let’s execute the above shell script “test.sh”. Remember to change File access
permission of script to execute
$ sh test.sh arg1 arg2 arg3
Program name is test
First argument is arg1
Second argument is arg2
Number of arguments passed = 3

The arguments are arg1 arg2 arg3
There’s another way to execute the script using ./
$ ./test.sh arg1 arg2 arg3
User Input: read
To get input from the keyboard, you use the read command. The read command takes
input from the keyboard and assigns it to a variable.
Example: read.sh
#!/bin/sh
echo –n “Enter some text > “
read text
echo “You entered: $text”
Note that “-n” given to the echo command causes it to keep the cursor on the same
line; i.e., it does not output a carriage return at the end of the prompt.
Next, we invoke the read command with “text” as its argument. What this does is wait
for the user ro type something followed by a carriage return (the Enter key) and then
assign whatever was typed to the variable text.
Execution:
$sh read.sh
Enter some text > This is some text
You entered: This is some text
Validation of Command Line Arguments
Let us assume, we have one shell script which requires exactly 2 arguments to
execute. Shell script should throw proper error message in case user has not given
exactly two arguments.
Let us have a small shell script "test.sh" to show how to implement validations.
#!/bin/sh
if [ $# -ne 2 ]
then
echo " Not sufficient arguments, please give exact 2 arguments"
else
echo "Continue with our task" fi
$ ./test.sh
Not sufficient arguments, please give exact 2 arguments
$ sh test.sh abc
$ ./test.sh abc def ghi
$ ./test.sh abc def 2
$ ./test.sh abc def
Continue with our task
For scripts where input expected is a file , validations can be incorporated for file tests
like whether the file given as input exists or not, whether it is empty or not, whether it
is readable or not and so on as per requirements.
Basic UNIX commands :
The basic Unix commands are:
echo
It displays a line text.
Echo the given string or strings to standard output.
Syntax:
echo [OPTION]... [STRING]...
Example:
echo "Good Morning"
This will display the following in the standard output
Good Morning
passwd command
The passwd command is used to change passwords for user accounts. A normal user
may only change the password for his or her own account, while the superuser may
change the password for any account.
Syntax : passwd [options] [LOGIN]
Options:
uname command
Print certain system information.
Syntax : uname [OPTION]...
Options:
Examples: To show the name of the OS
uname command:
Used to get the information about the current system (the name, version and other
details)
$uname
To get the kernel name, you can use -s parameter.
$ uname -s
To get the kernel release, -r option is used
$ uname -r
who command:
who command prints information about all users who are currently logged in.
Syntax:
who [OPTION]... [ FILE ] [ am i ]
find command:
find is one of the powerful utility of Unix (or Linux). It can search
the entire file-system to locate files and directories according to the specific
search criteria. Other than searching files in can also perform actions(executing
commands) on searched files.
Syntax:
$ find [option] [action]
Examples :
1 . File Type based Search:
>> The following command finds the file named Abc in current(.) directory and
all its sub-directories
[390119@InGhyUnix ~]$ find . -name Abc

./Abc
./Dir1/Abc
2. Wild-Card based Search : Wild-card characters like * and ? can be used:

>> To find all files where filename starts with Abc the command will be:
[390119@InGhyUnix ~]$ find . -name "Abc*"

./Abc
./Dir1/Abc
./Dir1/Abc.php
./Abc.txt
./Abc.dat
3 . Source Location based Search:

>>The command find can search in multiple source location :
[390119@InGhyUnix ~]$ find Dir1 Dir2 -name "Abc.*"

Dir1/Abc.php
Dir2/Abc.htm
Searches for the file Abc with any extension only in directories Dir1 and Dir2
4 . Size Based Search : Using find files can be searched based on its size .
>> To find the files with size more than 10Mb the command will be
[390119@InGhyUnix ~]$ find . -size +10M

./Abc.dat
5 . Access/Modification-Time Based Search : Using find files can be searched

based on the time of last access/modification :
>> To find the files which are accessed within last 3 days the command will be
[390119@InGhyUnix ~]$ find . -atime -3
./Abc
6 . Permission Based Search : Using find command files can be searched

based on the access permission they have
>> To find the files having read write and execute permission only for the owner
the command will be :
[390119@InGhyUnix ~]$ find . -perm 700

./Abc.txt
7 . Perform Action on the search result : Using exec option any action (command)
can be executed on the search result.
>> To remove all the files having extension .swp from the current directory the
command will be :
[390119@InGhyUnix ~]$ find . -name “*.swp” -exec rm {} \;
bc command:
bc is the command used for basic mathematical calculations.
It can be used either a mathematical scripting language or as an interactive

mathematical shell .
In interactive mode it accepts input from the terminal and providing calculations on
requested arithmatic expression.
That means typing the command bc on a Unix command prompt and entering a
mathematical expression, such as (1 + 4) * 2 , whereupon 10 will be the output
While used with files , bc starts by processing code from all the files listed on the
command line in the order listed. After all files have been processed, bc reads from
the standard input. All code is executed as it is read.
Syntax: bc [ -hlwsqv ] [long-options] [ file ... ]
Example:
$ echo "2+5" | bc
Output will be:
tty command:
Prints the file name of the terminal connected in standard input.
Syntax : tty [OPTION]...
UNIX Text Editors

To write a program or a paper even to edit a mail message, you need to create,
write,store, retrieve, view, and edit files. In unix several text editors available for this .
Editors available unix are ed, node,NEdit, emacs and vi .
One of the first end-user programs hosted on the system and standardized in UNIX-
based systems ever since is the vi editor. ed is a line editor for the UNIX operating
system.
vi Editor:
vi (pronounced "vee-eye") is short for "vi"sual editor. It is a visual editor used to

create and edit text files.
It displays a window into the file being edited that shows 24 lines of text. vi lets you
add, change and delete text, but does not provide formatting capabilities
like centering lines or indenting paragraphs.
vi works in three different modes:
1. Edit Mode -where any key is entered as text

2. Command Mode -where keys are used as commands
3. Ex Mode -ex commands can be entered in last line to act on text
Invoking vi with/without filename puts it in command mode:
vi []
Commands to be used in vi:
For cursor movement within the vi editor, we can not use the mouse . We have to use
key commands. To use a key command, user has to be in the command mode. To go
to command mode, press the Esc key.
Commands to Enter and Exit vi:
command Purpose of the command

vi edit filename starting at line 1
filename
vi –r recover filename that was being edited when system
filename crashed
The new or modified file is saved when user exits from vi. But it is also possible to
exit without saving the file.
To use a command to exit from vi , after pressing the Esc key ,colon : is to be pressed.
The cursor moves to bottom of screen whenever a colon (:) is typed. This type of
command is completed by hitting the (or ) key.

:wq quit vi, writing out modified file to file named in
original invocation
:q quit (or exit) vi
:q! quit vi even though latest changes have not been saved
for this vi call
:w write current contents to file named in original vi call
Commands to insert or add text:
Following are the commands to be used to insert or add text.
key Purpose of the command

u Undo the last command’s action
i
insert text before the cursor,until is hit
I
insert text at beginning of current
line, until hit
a append text after cursor, until hit

A append text to end of current line,
until hit
o open and put text in a new line below
current line, until hit
O open and put text in a new line above
Commands to modify texts:

r replace single character under cursor (no needed)
R replace characters, starting with current cursor position,
until hit
cw change the current word with new text,
starting with the character under cursor, until hit
cNw change N words beginning with character under cursor,
until hit;
e.g., c5w changes 5 words
C change (replace) the characters in the current line, until hit
cc change (replace) the entire current line, stopping when is
hit
Ncc change (replace) the next N lines, starting with the current
or line,
cNc stopping when is hit
Commands to Delete texts:

x delete single character under cursor
Nx delete N characters, starting with character under cursor
cw delete the single word beginning with character under
cursor
dw change N words beginning with character under cursor,
until hit;
dNw delete N words beginning with character under cursor;
e.g., d5w deletes 5 words
D delete the remainder of the line, starting with current cursor
position
dd delete entire current line
Ndd delete N lines, beginning with the current line;
or e.g., 5dd deletes 5 lines
dNd
Commands to copy and paste text:
yy copy (yank, cut) the current line into the buffer

Nyy copy (yank, cut) the next N lines, including the current line,
into the buffer
p put (paste) the line(s) in the buffer into the text after the
current line
Commands for searching text
/string search forward for occurrence of string in text

?string search backward for occurrence of string in text
UNIX FILE COMMANDS

In this module following important file handling commands will be discussed:
• touch
• cat
• cat > (single redirection)
• cat >>(double redirection)
• cp
• mv
• cmp
• comm
• diff
touch command
Touch command is used in Unix to create an empty file.
$ touch file1 file2

$ ls -l
-rw-r--r-- 1 user group 0 2014-02-08 7:14 file1
$
Touch command is used to change the timestamps (access time and modification time
of a file).
cat command
Use of cat command:
'cat' command is used to display a file content.
Syntax:
$ cat filename [ filename ]
$ cat data
This is an unix file
This is line two
$
cat > and cat >> command

Redirection operator ( > and >>) can be used with cat command to take input from
standard input device(keyboard) and store them in a file.
Syntax: $ cat > [filename]
$ cat > data

The file created using
cat > filename
^d
$
cat with single redirection(cat >filename) , creates a new file if no file with the given
file name already exists and overrites the file if a file with given file name already
exists.
cat with double redirection (cat >>filename) appends the content given through
standard input to the file.
cp command
cp command is used to copy one file's content to another file.
Syntax: $ cp [source filename] [destination filename]
Switches:
1. cp -i [sourcefilename] [destination filename] This command copies the content of a
file interactively.
2.cp -r [source_directory] [destination_directory] This command copies the whole file

hierarchy.
3. cp -v [sourcefilename] [destination filename] Copies in verbose mode, explains

what is being done
mv command
mv command is used for:
1. for renaming a file in same directory.
2. for moving a file to a different directory.
Syntax:
$ mv [sourcefilename] [destinationfilename]
$ mv [source_directory] [destination_directory]
Directory Commands:
Every item in the UNIX filesystem tree is either a file, or a directory. A directory is
like a file folder. A directory can contain files, and other directories. A directory
contained within another is called the child of the other. A directory in the filesystem
tree may have many children, but it can only have one parent. A file can hold
information, but cannot contain other files, or directories.
The place in the file system tree where an user is located is called the current
working directory (which can be listed using pwd command).
As a user logs in, user is put in his/her home directory.

mkdir(make directory) command:
The mkdir command creates a directory with specified name in the present working
directory or specified path.
Syntax: $ mkdir [ ]
ls Command
Lists the contents of any directory (current directory if no directory specified).
Sorts entries without option
$ ls [option(s)] [filename]
File comparison commands:

cmp – This command is used to compare two files, and if they differ, tells the first
byte and line number where they differ. If input files are same , returns nothing.
Syntax :
cmp options file1 file2
To use cmp commands both the files need to be sorted.
comm- This command is used to compare two sorted files.
Syntax:
comm [options] file1 file2
• One set of options allows selection of ‘columns’ to suppress.

o -1: suppress lines unique to file1 (column 1)
o -3: suppress lines common to file1 and file2 (column3)
Example: To show only lines common between file1 and file2, command is
$ comm -12 file1 file2

diff - This command is used to compare two files line by line. The output indicates
how the lines in each file are different, and the steps that will be involved in changing
file1 to file2.
The change commands are in the format [range][acd][range]. The range on the left
may be a line number or a comma-separated range of line numbers referring to file1,
and the range on the right similarly refers to file2. The character in the middle
indicates the action i.e. add, change or delete.
• ‘LaR’ – Add lines in range ‘R’ from file2 after line ‘L’ in file1.
• ‘FcT’ – Change lines in range ‘F’ of file1 to lines in range ‘T’ of file2.
• ‘RdL’ – Delete lines in range ‘R’ from file1 that would have appeared at line
‘L’ in file2
Syntax: diff [options] file1 file2

USAGE OF WinSCP
Introduction to WinSCP
WinSCP (Windows Secure CoPy) is a popular , free and open-source Secure

File Transfer application (SFTP) for Windows computers .It is used to transfer
files securely between remote server and local computer and vice verse. It
includes support for SCP file transfers, and has the ability to create secure
tunnels via SSH. Secure FTP (SFTP) is similar to FTP, but with SFTP the entire
session is encrypted, so that passwords are never sent in the clear text , and are
therefore much less vulnerable to interception. That is ,for
transferring files securely, we can use Secure copy or SCP protocol, it uses
Secure Shell (SSH), which encrypts the content before transferring data over
network. It runs over TCP port 22 by default.
How to use WinSCP
Step I : Start WinSCP

To start with double click on WinSCP icon in the desktop or select it from
Programs Group . The
first screen will look as below
Step II : Logging in to Server
Enter IP address of the server in Host Name field.
Unix login userid in the User name field
Unix login Password in Password field
Choose the protocol ( SFTP/ SCP / FTP)
For example
1) Host name used : 172.26.132.40 (For example)
2) User name used : 390119
3) Password used : ********
3) Protocol selected : SCP
Step III : Select/Navigate to working directory
1. We can select the working directory in both Local as well as remote
machine from the drop
down menu.
2. Tool bar can be used to navigate to desired folder
Step IV : Transferring content Content can be transferred in two way
A . Downloading : Transferring content from Remote server to Local
machine
B . Uploading : Transferring content from Local machine to Remote server
Following Steps to perform for Downloading file/Directory from Server
Step - A1 : Select the file to transfer in the Remote pane,
Step - A2 : Right click → Select Copy or move option
Step - A3 : Select the destination folder in the local machine(can be

changed by selecting
Browse button )
Step - A4 : The file Unix_Recap.txt copied to desktop.
Uploading files to Remote Server

Step - B1 : Select the file to copy in Local machine(Left) pane.
Step - B2 : Drag it and drop it to the remote pane desired location.
Step - B3 : Select the desired location from drop down list and select copy
Saving sessions for future use

All the session information can be saved for future use and faster access.
The information can be saved are :
>> Server IP
>> User name,
>> Password(Not recommended)
>> Communication protocol
Operating Systems
The structure of a Computer system can be visualized as having four basic
components:
1. Hardware – Provides basic computing resources - CPU, memory, I/O devices
2. Operating system – Controls and coordinates use of hardware among various
applications and users
3. Application programs – Define the ways in which the system resources are used to
solve the computing problems of the users
4. Users – Anybody who desires work to be done by a computer system. People,
machines, other computers
Types of OS
OS are classified into different types depending on their capability of processing.
Single User and Single Tasking OS:
These are simple operating system designed to manage one task at a time ,for use by a
single user for a standalone single computer for performing a single task .
Single User and Multitasking OS:
These OS allow execution of more than one task or process concurrently by dividing
the processor time amongst different tasks.
Multi-programming OS:
These OS allow more than one programs to run at the same time .
Real Time OS:
These are designed to respond to an event within a predetermined time. These

operating systems are used to control processes
Embedded OS:
Embedded in a device in the ROM. They are specific to a device and are less resource
intensive.
Function Of OS
The OS performs basic tasks such as controlling and allocating memory, prioritizing
system requests, controlling input and output devices, facilitating networking, and
managing files
UNIX Introduction
• Unix is an OS for Programmers as shell(the command interpreter)provides the

programming facility.
• It provides an in-built security mechanism through the user name and
password, combined with the access rights associated with files
• Developed by Ken Thompson and Ritchie originally in assembly, and later in
C, thus making it portable to other machines
Supports C, Fortran, Basic, Pascal, COBOL, Lisp, Prolog, Java, Ada compilers
Features of Unix Operating System
• Multi-user and Multitasking

• Everything is a file
• Configuration data stored in text
• Small, single purpose programs
• Ability to chain programs together to perform complex task
• Facility of background processing
Architecture of the UNIX System
The UNIX operating system has a layered architecture having three main components
• Kernel
• Shell
• Utilities
The Unix Kernel
Kernel is a collection of programs mostly written in C which allocate the system

resources and coordinate all the details of the computer's internals.
Functions of Kernel:
• It allocates time and memory to programs and handles the file store and
communications
• Interacts directly with the hardware through device drivers
• Provides sets of services to programs
• Manages memory, controls access, maintains file system, handles interrupts,
allocates resources of the computer
System calls
The system calls are functions used in the kernel itself. UNIX system calls are used to
manage the file system, control processes, and to provide interpocess communication.
System calls can be categorized as:
• File structure related calls -For example create, open,read, write,lseek,dup etc.
• Process Related calls -For example fork,exec,wait,exit etc
• Inter process related calls - For example pipe,msgget,msgsnd etc.
The Unix File System:

It is a hierarchical collection of 3 types of files:
• ordinary
• directory
• special files (device, pipe, fifo, socket).
A UNIX file is featureless because it is simply an array of bytes.

Dominant file type in UNIX is the text file.
System related files are also stored in text form.
Separate device can be added by creating a file for it.
Root is the supremo and is represented by the ‘/’. Every subdirectory must have a
parent.
File names can be up to 14 characters long; can contain both upper and lower case
alphabets, digits, a dot, hyphen (-), underscore (_) anywhere; should not have a blank
or tab; are case-sensitive.
Path names are a sequence of directory names separated by ‘/’. They are used to
access files.
Absolute pathname - file location is determined with respect to the root.
Relative pathname - file location is determined with respect to the current directory.
Though the UFS looks hierarchical, it is actually a directed acyclic graph because files
can be shared.
/home – It holds user’s home directories. In other UNIX systems, this can be /usr
directory.
/bin – It holds many of the basic Linux programs; bin stands for binaries, files that are
executable.
/usr – It holds many user-oriented directories:
/sbin – It holds system files that are usually run automatically.
/etc – It and its subdirectories hold many of Linux config files.
/dev – It holds device files. All info sent to /dev/null is thrown into trash. Your
terminal is one of the /dev/tty files.
The UFS resides on a single logical disk. A logical disk is a disk partition comprising
of a set of consecutive cylinders.
UFS further subdivides a partition into one or more cylinder groups and attempts to
allocate inodes and related data blocks from the same cylinder group, thus minimizing
the disk head movements.
At the beginning of the logical disk lies the boot block of UNIX operating system
containing the bootstrap program.
It is followed by repetitive cylinder groups each one containing a super block,

cylinder group block, inode list and the data area.
Each cylinder group contains a duplicate copy of the super block. The super block
contains the size of file system, number of free blocks, index of next free block in free
block list, size of inode list, number of free inodes, index of next free inode in free
inode list.
The cylinder group block contains a number of inodes and corresponding data blocks
for that cylinder group. The block size is a power of 2 (>=4096).
Internal File Maintenance
For each file created in the system, an inode is also created. Inode is a disk file record
of 64 bytes that maintains the permanent attributes of a file.
An inode is permanent and it exists until the corresponding file is removed from the
system.
Sample details of an inode –
• Owner and group identifiers

• File type and file size
• Number of links for this file
• Times of file creation, last file access and modification, and last inode
modification
• List of access rights – read/write/execute permissions
• Reference count showing number of times file is opened
• Physical address of file on the disk: array of 13 pointers for data storage
Whenever a file is opened, its inode is brought into main memory. The active inode is
kept there until the file is closed and is used to locate the beginning of an open file on
disk and to verify that every I/O request comes from a valid user as per specified
access permissions.
Users and Access Rights
Objective
This module is to understand about different user types in UNIX and their access
rights.
Course Content
• User types
• File access permissions
• Changing file access permissions using symbolic mode
• Changing file access permissions using octal mode
User Types in Unix
There are three types of accounts on a Unix system:

- Root Account
- System Account
- User Account
Managing users and groups

Unix supports a concept of Group which logically groups a number of accounts.
➢ Every account would be a part of any group.
➢ Unix groups plays important role in handling file permissions and process
management.
➢ Grouping of users allows to grant and revoke file permissions collectively.
File Permission: Ownerships level

File ownership is an important component of Unix that provides a secure method for
storing files. Every file in Unix has the following attributes:
File Permission Types (Mode)
Three type of permissions can be set for any ownership level:
Read, Write and Execute Three modes or permissions have different meaning for file
and directory;
Changing Permissions (chmod)

To change file or directory permissions, the chmod (change mode) command is used.
There are two ways to use chmod:
• Symbolic mode
• Absolute mode
chmod command options:
Using chmod in Symbolic Mode:
With symbolic representation, permission set can be added, deleted, or specified using
the operators listed in the following table:
Example:
Let us assume there is a file name "abc" , whose initial permissions are only read
permission to user and group and others. We will provide full permissions to all three
types of users.
Let us verify the existing file permissions using ls -l command as below and observe
the next steps, like changing the permissions and verifying the changes.
In the above example,
chmod u+wx,g+wx abc , adds write and execute permission to user and group to the
file abc
chmod g-x abc, removes execute permission from group from the file abc.
chmod o=r-- abc, provides read permission and removes write and execute
permission from others for file abc
Using chmod in Absolute(octal) Mode
The second way to modify permissions with the chmod command is to use a number
to specify each set of permissions for the file. Each permission is assigned a value, as
the following table shows, and the total of each set of permissions provides a number
for that set.
Examples: Observe the following sequence of command execution statements and
analyze the output
1. Let us view the initial permissions of the file abc

ls -l abc
-rw-r--r-- 1 735873 oinstall 0 Feb 7 12:37 abc
If we observe the output, initially owner have read and write permissions and group
and others have only read permission
2. Let us remove all the permission from all users for abc file, as below
chmod 0 abc
Let us verify the changed permissions
ls -l abc
---------- 1 735873 oinstall 0 Feb 7 12:37 abc
Yes all permissions are taken away from all users on the file abc
3. To provide all permissions to the user, read permission to group and write and
execute permissions for Others,
use the below command

chmod 743 abc
Let us verify the changed permissions

ls -l abc
-rwxr---wx 1 735873 oinstall 0 Feb 7 12:37 abc
From the output all permissions to user, read permission to group and write and
execute
permissions for Others, Use the below command
4. chmod 755 abc

The above command provides all permissions to the user, read and execute
permissions to group and others
Let us verify ls -l abc
-rwxr-xr-x 1 735873 oinstall 0 Feb 7 12:37 abc
5. chmod 777 abc

The above command provides all permissions to all users . We can verify using
below command
ls -l abc
-rwxrwxrwx 1 735873 oinstall 0 Feb 7 12:37 abc
PRACTICE ASSIGNMENTS FOR UNIX

Basic Unix Commands :
8. Create a directory Unix in your home directory

9. Create two sub directories dir1 and dir2 under Unix directory.
10. Go to dir1.
11. Create a file myFile in the dir1 and insert few lines.
12. Make a duplicate copy of the file myFile in dir2
13. Append few lines of statements in file myFile in dir2.
14. Change the name of the file myFile in dir2 to newFile.
Filters and Pipes:
Consider this file as the sample myFile.
Hello!
Happy Learning!
9. Display how many number of lines are there in myFile.
Expected Output : 6
9.Count how many users are currently logged into the system
10.Display the lines having the word "this"/”This” in myFile
Expected Output :
11.Count number of lines having the word "this"/”This” in myFile.
Expected Output : 3
12.Replace the word this by that for all occurrences in myFile.
Expected Output :
Hello!
I have created that file to complete that assignment.

Happy Learning!
13.Replace the word this by that only for the first occurrence of it in a line.
Expected Output :
Hello!
I have created that file to complete this assignment.
Happy Learning!
14. Print the first 3 lines from the file myFile
Expected Output :
Hello!
15. Print the last 3 lines from the file myFile
Expected Output :

Happy Learning!
16. Append a new line in the beginning of the file myFile
awk Scripting :
18. Create a file with name student.txt with the following contents in it
RollNo|Name|Marks1|Marks2|Marks3
123|Raghu|80|90|60
342|Maya|45|78|82
561|Gita|56|71|89
480|Mohan|71|90|89
a. Write an awk command to print the name and roll number of the students.
Expected Output:
Name RollNo
Raghu 123
Maya 342
Gita 561
Mohan 480
b . Write the awk command to calculate the total marks of each student and display
the total marks along with the name of the student.
Expected Output:
Raghu-230
Maya-205
Gita-216
Mohan-250
Shell Scripting
19. Write a shell script to print the sum of the 1st 10 natural numbers.
Expected Output: 55
19. Write a shell script to print the odd numbers from a given list of numbers. The list
of numbers will be given as command line argument.
Example :
If input numbers are 10,3,45,6,1,100, then
Expected Output :
45
20. Create a file studentScores.txt and append the following content to it.
RollNo|Name|Marks
123|Raghu|80
342|Maya|45
561|Gita|56
480|Mohan|71
write a shell script named scores.sh to find the name of the student who scored highest
marks. Incorporate validations for the following :
a. Whether correct number of command line arguments are passed or not.
b. whether the given files exists or not.

c. whether the given file is readable or not.
Expected Output :
a. When executed as sh scores.sh, output should be
“Pass correct number of arguments.”
b. When executed as sh scores.sh studentScores.txt, but studentScores.txt does not

exist, output should be
“ File does not exist”
c.When executed as sh scores.sh studentScores.txt, but studentScores.txt does not

have read permission, output should be
“ File is not readable”.
d. When executed as sh scores.sh studentScores.txt, output should be
“Name of the student with highest marks : Raghu”

Practice the questions in any of the below given link :
4. https://www.tutorialspoint.com/unix_terminal_online.php
5. http://www.compileonline.com/execute_bash_online.php
6. https://bellard.org/jslinux/vm.html?url=https://bellard.org/jslinux/buildroot-
x86.cfg
The UNIX Process:
A process in UNIX is a program in execution with definite life-time and well-

defined hierarchy.
• The context of a process is a snapshot of its current run-time environment

that consists of the current state of the processor registers and
• User program image - Process execution on user data structure that
represents an application and is manipulated by its own functions (user
mode execution).
• System image - Process execution on system’s data structures that
represent the resources (memory, files, devices) and supported by the
kernel routines. Depending upon resource requirement and availability,
process’s states are controlled by executing the kernel routines accessed
through system calls (system mode execution).
• The kernel maintains a process table to manage all processes. The two
data structures per process are the user structure and the process structure.
• The kernel process is the first (root) process that comes into existence
when the system is booted. Its process_id and group_id are both 0.
• In establishing a multi-programming environment, the kernel
process creates the init process with its process_id of 1 and group_id, 0,
showing that process 1 is part of process 0. The init process creates and
manages terminal processes for active terminals in the system.
• At the time of creation, each terminal process belongs to process group 0
and is made to execute a program called getty. Once each terminal
process (now called a getty process) establishes communication with the
associated terminal, it displays a login message and waits for a user to
input a user name (Process group of the init process).
• When a user starts a dialogue, the getty process receives the user name
and leaves the task of validating the user’s password to another program
called login. The same terminal process is now called a login process.
• The login process, after validating the user’s password, calls a command
line program, the login shell to run in the same process. The same
terminal process is now called a shell process.
• Each shell process now establishes a new process group and becomes
ready to process the user commands. A shell process is the initiating
process by which each terminal maintains the user session.
• While interpreting a command, the shell creates an execution thread and
then assigns the requested command program to this new process.
• Both the shell and the new process proceed independently in separate
execution threads. The parent shell process normally waits until child
process completes its execution.
A process require certain resources to accomplish its intended task, they are
>> CPU time
>> System memory
>> Disk Files
>> I/O devices.
Process control blocks (PCB)

The OS must know all the information about a specific process in order to manage
and control it. The OS maintains a table (an array of structures), called the process
table, with one entry per process. These entries are called process control blocks
(PCB) - also known as task control block.This entry contains information about the
process' state, its program counter, stack pointer,memory allocation, the status of its
open files, its accounting and scheduling information, and everything else about the
process that must be saved when the process is switched from ready to running or
blocked state so that it can be restarted later with the status it left off.
Process state:- The state may be new, ready, running, waiting, halted, and so on.
Program counter:- The counter indicates the address of the next instruction to be
executed for this process.
CPU registers:- The registers vary in number and type, depending on the computer
architecture. They include accumulators, index registers, stack pointers, and general-
purpose registers, plus any condition-code information.
CPU-scheduling information:- This information includes a process priority, pointers
to scheduling queues, and any other scheduling parameters.
Memory-management information:- This information may include such
information as the value of the base and limit registers, the page tables, or the segment
tables, depending on the memory system used by the OS.
Accounting information:- This information includes the amount of CPU and real
time used, time limits, account numbers, job or process numbers, and so on.
I/O status information:- This information includes the list of I/O devices allocated to
the process, a list of open files, and so on.
Process state:
A process may present in one of the following state.
New:- The process is being created or just created.
Ready:- The process is waiting to be assigned/allocated to a processor time.
Running:- Instructions of the process are being executed by processor.
Waiting:- The process is waiting for some other event/process to execute.
Terminated:- The process has finished its execution.
ps command:
The ps (i.e., process status) command is used to provide information about the
currently running processes in the system. When ps is used without any option four
columns of information labeled PID,TTY, TIME and CMD for at least two processes,
the shell and ps will be visible.
PID :- The process are identified by a 5 digit number known as PID (Process
Identification Number).
TTY :- Is the name of the console or terminal that the user logged into (Stands for
terminal type now
but originally stood for teletype).
TIME :- Is the amount of CPU time in minutes and seconds that the process has been
running.
CMD :- is the name of the command that launched the process.
The information that ps -aux provides about each process is :

>> The user of the process,
>> PID of the process,
>> Percentage of CPU used by the process,
>> Percentage of memory used by the process,
>> VSZ (virtual size in kilobytes),
>> RSS (real memory size or resident set size in 1024 byte units),
>> STAT (the process state code, explained later),
>> Starting time of the process,
>> Length of time the process has been active
>> The command that initiated the process.
Types of process
Parent and Child Process:
A process can initiate a sub process, which is a called a child process, the initiating
process is referred to as its parent.The child processes, in turn create other child
processes forming a tree of processes
( which can be displayed using ps command with –forest option)
Orphan Process:
When a child process is killed, parent process gets the notification via a signal. Parent
then, can continue other task. However if the parent process is killed before, its child,
is called an orphan process.
Zombie Process:
When a process finished its execution and exit status not received by the parent ( or
parent did not read the exit status till now), the process state becomes zombie.
The process is dead (not to be scheduled for further execution) but cannot
becompletely removed from process table, until it has been determined that exit status
is no longer needed.
Daemon Process
Some programs are not designed to be run with continuous user input and disconnect
from the terminal when task completed. For example, a web server responds to web
requests, rather than user input. Mail servers are another example of this type of
application. These types of programs are known as daemons.
Starting a Process:
Process can be started in two ways:
In Foreground: By default every process starts in foreground, ie. Gets the input from
keyboard and sends the output in monitor. But in this case till the process completes
its execution no other process can be started in foreground.
In Background: To take the advantage multiprocessing environment, a process can
be started in background, so that other process can be started in the foreground
without waiting for the previous process to complete execution.
A process can be started in background by adding ampersand(&) after it.
[390119 @ INGNRILPORCL] $ $ ls | wc -l > file1 &
Switching process from foreground to background:

A process running in foreground can be send to background using the following steps:
>> Press + Z to suspend the job
>> bg command puts the job in background
>> nohup unattaches the job from the terminal*
$ cat > file

File content...
+z
[1] + Stopped
$ bg
[1] + cat > file &
Switching process from background to foreground

Process running in the background can be taken into foreground using the following
steps:
>> Find the job id of the process by the command jobs
>> Use fg %to get the job to foreground
Stopping/Killing a Process
A process dies(terminates ) automatically when it completes the job it intended to.
A process can be killed abnormally with the command kill.
>> Use the ps command to find out the process-id for the process
>> Use the command kill to terminate it
>> Use the command kill -9 to kill a process forcefully.
Option settings for background process

While a process is executing/running , if the owner tries to log off the process ,process
will get killed. Sometimes a job or command takes a long time to complete and it is
required the job to be completed without interruption. This situation can be handled in
two different ways:
>> Does not allow the use to log off
>> Continue execution in background even after the user logged off
This can be achieved using command nohup
PERIODICAL EXECUTION OF JOBS
Sometimes it may be required to execute certain Job or task on specific time. It may
be not possible for any user to start or execute the job physically on that particular
time due to other business requirements. Again if similar task need to be executed
periodically on a regular interval, then the task becomes very hectic. Unix internally
has solutions to handle this type of situations using utilities such as cron and at.
Crontab
The word 'crontab' in UNIX stand for chron(chronograph) table, or time table for
Unix system .
The crontab is a special table where it is possible to specify commands and time or
interval to execute the command once or repeatedly. Commands can be any
executable programs, for example, a script can be written to take regular backup of all
the required tables in database. A crontab instruction can be created to execute the
script on the specified time. The cron daemon reads the table and executes the
commands at the times specified.
Crontab Command Syntax

Syntax
Components of Crontab
The basic format of a crontab schedule consists of 6 fields, placed on a single line and
separated by spaces, formatted as follows:
The various fields and their utility are explained in the image below:
Example:
To executes an incremental backup shell script "takeBackup.sh" available

in your home directory at 11:00 on every day the cron entry will be as
below:
00 11 * * * ~/takeBackup.sh
To schedule the same job for every minute, the cron entry will be as
follows:
* * * * * ~/takeBackup.sh
Shell Programming
The UNIX shell program interprets user commands, which are either directly entered
by the user, or which can be read from a file called the shell script or shell program.
Shell scripts are interpreted, not compiled. The shell reads commands from the script
line by line and searches for those commands on the system.
Shell script
Shell script is a file-containing list of commands to be executed in a particular order.
A good shell script will have comments, (a statement preceded by a pound sign, #) ,
describing the purpose of the statement.
In a script we can use conditional tests, such as value A is greater than value B, loops
or iterative statements to execute some steps repetitively or to navigate through a list
of data or records and do processing. We can use files to read and store data. Can
use variables to read and store data. A script may include functions also.
When a script is executed, the shell reads the commands one by one and executes
them .
We can create the simple shell script file by using vi editor or cat command like,
$ vi test.sh
$ cat > test.sh
Below mentioned shebang statement should be the first statement in the shell script as
it tells the system that the commands mentioned in the shell script are to be executed
by the shell /bin/sh
#!/bin/sh
Consider the shell script with just two commands pwd & ls.
$cat test.sh
#!/bin/bash
pwd
ls
Importance of shell script

Shell scripts are basically used for automating processes that we repeat at the prompt .
Following are some activities we can do using shell scripts:
• Automation of repetitive task

• Creating our own power tools/utilities.
• Automating command input or entry.
• Customizing administrative tasks.
• Creating simple applications.
• Automating administration tasks such as adding new users, removing obsolete
users etc
Some Practical examples where shell scripting can be actively used:
• Monitoring your Linux system.

• Data backup and creating snapshots.
• Dumping Oracle or MySQL database for backup.
• Creating email based alert system.
• Find out what processes are eating up your system resources.
• Find out available and free memory.
• Find out all logged in users and what they are doing.
• Find out if all necessary network services are running or not. For example if
web server failed then send an alert to system administrator via a pager or an
email.
• Find out all failed login attemp. If login attempt are continued repeatedly from
same network IP, automatically block all those IPs accessing your
network/service via firewall.
• User administration as per your own security policies.
• Find out information about local or remote servers.
• Configure server such as BIND (DNS server) to add zone entries.
Executing a shell script

Below two methods can be used to execute the shell script.
$ sh filename
Or
$ ./filename
In this case we have to modify the file access permissions of the shell script before
execution.
To provide execute permission , following command is used.
$ chmod u+x filename
Basic Operators in shell scripting

Below operators are supported by shell.
• Arithmetic Operators.
• Relational Operators.
• Boolean Operators.
• String Operators.
• File Test Operators.
Here is simple example to add two numbers:
Example:
#!/bin/sh
val=èxpr 2 + 2`
echo "Total value : $val"
Output:
$ Total value : 4
There are following points to note down:
• There must be spaces between operators and expressions for example 2+2 is
not correct, where as it should be written as 2 + 2.
• Complete expression should be enclosed between ``, called inverted commas to

execute expr command correctly.
Arithmetic Operators

+ Addition èxpr $a + $b ` will give 30
- Substraction èxpr $a - $b ` will give -10
* Multiplication èxpr $a \* $b` will give 200
/ Division èxpr $b / $a` will give 2
% Modulus èxpr $a % $b` will give 0
!= Not equal [ $a != $b ] will give true
= assignment a=$b will assign value of b to a.
== Equality [ $a == $b ] will return false.
inside square braces with one spaces around them, for example [ $a == $b ] is correct
where as [$a==$b] is incorrect.
Relational Operators
Below are relational operators which are specific to numeric values. These operators
would not work for string values unless their value is numeric.
For example, following operators would work to check a relation between 10 and 20
as well as in between "10" and "20" but not in between "ten" and "twenty".
-eq Check if the values of 2 operands are [ $a -eq $b ] is

equal or not, if yes then condition false
becomes true.
-ne Check if the values of 2 operands are [ $a -eq $b ] is
equal or not, if values are not equal true
then condition becomes true.
-gt Check if the value of left operand is [ $a -gt $b ] is
greater than the value of right operand, false
if yes then condition becomes true.
-lt Check if the value of left operand is [ $a -lt $b ] is
less than the value of right operand, if true
yes then condition becomes true.
-ge Check if the value of left operand is [ $a -ge $b ] is
greater than or equal to the value of false
right operand, if yes then condition
becomes true.
-le Check if the value of left operand is [ $a -le $b ] is
less than or equal to the value of right true
operand, if yes then condition becomes
true.
inside square braces with one spaces around them, for example [ $a <= $b ] is correct
where as [$a <= $b] is incorrect.
Boolean Operators
Assume variable a holds 10 and variable b holds 20 then

! This is logical negation. This [ !false ] is true
inverts a true condition into false
and vice versa.
-o This is logical OR. If one of the [ $a -lt 20 -o $b -gt
operands is true then condition 100 ] is true
would be true.
-a This is logical AND. If both the [ $a -lt 20 -a $b -gt
operands are true then condition 100 ] is false.
would be true otherwise it would
be false.
String Operators
These are string operators. Assume variable a holds "abc" and variable b holds "efg"
then:

-z Check if the given string operand [ -z $a ] will return
size is zero. If it is zero length then false.
it returns true.
-n Check if the given string operand [ -z $a ] will return
size is non- zero. If it is non-zero true.
length then it returns true.
= Check if the value of two operands [ $a = $b ] will return
is equal or not, if yes then false
condition becomes true.
!= Check if the value of two operands [ $a != $b ] will
is equal or not, if the values are not return true
equal then condition becomes true.
str Check if the str is not the empty [ $a ] will return true
string. If it is empty then it returns
false.
File Test Operators:
Assume a variable file holds an existing file name "test" whose size is 100 bytes and
has read, write and execute permission on:

-b file Returns true, if file is a block [ -b $file ] is false.
special file
-c file Returns true, if file is a character [ -b $file ] is false.
special file
-d file Returns true, Check if file is a [ -d $file ] is not true.
directory
-f file Returns true, Check if file is an [ -f $file ] is true.
ordinary file or special file
-r file Returns true, Checks if file is [ -r $file ] is true.
readable
-w file Returns true, Check if file is [ -w $file ] is true.
writable
-x file Returns true, Check if file is [ -x $file ] is true.
execute
-s file Returns true, Check if file has size [ -s $file ] is true.
greater than 0
-e file Returns true, Check if file exists [ -e $file ] is true.
Wild Card Characters
Symbol used to replace or represent one or more characters. Wildcards or wild

characters are either asterisk (*), which represent one or more characters or question
mark (?), which represent a single character.
Wild card
Meaning Examples
/Shorthand
Matches any $ ls * will show all files
string or group of $ ls a* will show all files whose first
characters. name is starting with letter 'a'
* $ ls *.c will show all files having
extension .c
$ ls Will show all files having
ut*.c extension .c but file name
must begin with 'ut'.
? Matches any $ ls ? will show all files whose
single character. names are 1 character long
$ ls will show all files whose
fo? names are 3 character long
and file name begin with fo
[...] Matches any one $ ls Will show all files beginning
of the enclosed [abc]* with letters a,b,c
characters
Note: [..-..] A pair of characters separated by a minus sign denotes a range.
Example:
$ ls /bin/[a-c]*
Will show all File name beginning with letter a,b or c.

Output:
/bin/arch /bin/awk /bin/bsh /bin/chmod /bin/cp /bin/as /bin/basename /bin/cat
Shell Quoting Mechanism
The Metacharacters
Unix Shell provides various metacharacters which have special meaning while using
them in any Shell Script and causes termination of a word unless quoted.
Example:
? Matches with a single character while listing files in a directory and an * would
match more than one characters.
Here is a list of most of the shell special characters (also called metacharacters):
* ? [ ] ' " \ $ ; & ( ) | ^ < > new-line space tab
A character may be quoted (i.e., made to stand for itself) by preceding it with a \.
Example:
#!/bin/sh echo Hello; Word
This would produce following result.
Hello ./test.sh: line 2: Word: command not found shell returned 127
Now let us try using a quoted character:
#!/bin/sh echo Hello\; Word
Hello; Word
The $ sign is one of the metacharacters, so it must be quoted to avoid special handling
by the shell:
#!/bin/sh echo "I have \$1200"

I have $1200
Quote Description
Single quote All special characters between these quotes lose their
special meaning.
Double Most special characters between these quotes lose their
quote special meaning with these exceptions:
\$
\'
\"
\\
Backslash Any character immediately following the backslash loses
its special meaning.
Back Quote Anything in between back quotes would be treated as a
command and would be executed.
The Single Quotes

Consider an echo command that contains many special shell characters:
echo <-$1500.**>; (update?) [y|n]
Putting a backslash in front of each special character is tedious and makes the line
difficult to read:
echo \<-\$1500.\*\*\>\; update\?update\?
y∥ny‖n
There is an easy way to quote a large group of characters. Put a single quote ( ') at the
beginning and at the end of the string:
echo '<-$1500.**>; (update?) [y|n]'
Any characters within single quotes are quoted just as if a backslash is in front of each
character. So now this echo command displays properly.
If a single quote appears within a string to be output, you should not put the whole
string within single quotes instead you would precede that using a backslash (\) as
follows:
echo 'It\'s Shell Programming
The Double Quotes:

Try to execute the following shell script. This shell script makes use of single quote:
VAR=ZARA
echo '$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]'
$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]
So this is not what you wanted to display. It is obvious that single quotes prevent
variable substitution. If you want to substitute variable values and to make invert
commas work as expected then you would need to put your commands in double
quotes as follows:
VAR=ZARA
echo "$VAR owes <-\$1500.**>; [ as of (`date +%m/%d`) ]"
Now this would produce following result:
ZARA owes <-$1500.**>; [ as of (07/02) ]
Double quotes take away the special meaning of all characters except the following:
• $ for parameter substitution.

• Backquotes for command substitution.
• \$ to enable literal dollar signs.
• \` to enable literal backquotes.
• \" to enable embedded double quotes.
• \\ to enable embedded backslashes.
• All other \ characters are literal (not special).
Back Quotes: Command Substitution

Putting any Shell command in between back quotes would execute the command
Syntax: var=`command`
Example:
Following would execute date command and produced result would be stored in
DATA variable.
DATE=`date` echo "Current Date: $DATE"
Current Date: Thu Jul
Shell Decision Statement
While writing a shell script, there may be situations when you need to adopt one path
out of many available paths. In such cases you need to make use of conditional
statements that allow your program to make correct decisions and perform right
actions.
Unix Shell supports conditional statements, which are used to perform different
actions based on different conditions.
Here we will explain following two decision-making statements
• The if...else statement

• The case...esac statement
If..else statements
We can use “if..else” statement which can be used as decision making statement to
select an option from a given set of options.
Unix Shell supports following forms of if..else statement:
Syntax:
if [condition]
then
command(s)
fi
if [ condition(s) ] then
command(s)
else
command(s)
fi
if [ condition(s) ]
then
command(s)
elif [ condition(s) ] # Many elif-then allowed

then
command(s)
else
command(s)
fi
test command in if condition
We can use test command as condition of if condition as used in the below script.
$cat if_test.sh
#!/bin/sh
echo “Do you want to quit (Y/y)? ”
read ans
if test $ans ==’y’ –o $ans==’Y’
then
exit
else
echo “ to exit enter N or n”
fi
case...esac Statement
We can use multiple if...elif statements to perform a multiway branch. However, this
is not always the best solution, especially when all of the branches depend on the
value of a single variable.
Unix Shell supports case...esac statement which handles exactly this situation, and it
does so more efficiently than repeated if...elif statements.
The interpreter checks each case against the value of the expression until it finds a
match. If nothing matches, goes with the default condition.
Syntax:
case word in pattern1)
;;
pattern2)
;;
pattern3)
;;
esac
Here string word is compared against every pattern until a match is found and the
statement(s) following the match pattern executes. If shell cannot find any match, the
case statement exits without performing any action. When statement(s) part is
executed. The command ;; indicates program flow should jump to the end of the
entire case statement.
There is no maximum number of patterns, but the minimum is one.
Example:
#!/bin/sh
COURSE=”DB”
case “$COURSE” in
“Java”) echo “Java is a programming language”
;;
“Perl”)echo “Perl is scripting language”
;;
“DB”)echo “Oracle is a DB”
;;
esac
Output:
Oracle is a DB
Iterative Statements/Loop :
Loops are a powerful programming tool that enables you to execute a set of
commands repeatedly.
• while loop
• for loop
• until loop
while loop
Here condition is evaluated by shell at the beginning of each iteration. If the resulting
value is true, given command(s) are executed. If condition is false then no command
would be executed and program would jump to the next line after done statement.
Syntax:
while condition
do # executed as long as the condition is true

command(s)
done
Example:
a=0
while [ $a -lt 3 ]
do
echo $a
a=èxpr $a + 1`
done
Output:
until loop
Here condition is evaluated by shell. If the resulting value is false, given command(s)
are executed. If condition is true then no command would be executed and program
would jump to the next line after done statement.
Syntax:
until condition # complement of while
do
# executes the body as long as the condition is false
command(s)
done
Example:
a=0
until [ ! $a -lt 3 ] do
echo $a a=èxpr $a + 1`
done
Output:
for loop
For loop operate on lists of items. It repeats a set of commands for every item in a list.
Syntax:
for var in w1 w2 w3.. wn
do
command #Statement(s) to be executed for every w in the list
done
Example:
for var in 0 1 2 3 4 5
do
echo $var
done
Output:
Example:
for filename in `ls tempdir`
do
echo “Displaying contents of ${filename}”
cat ${filename}
done
We can use "break" and "continue" commands to control the loop.
Command "break" causes the termination of a loop and “continue” resumes execution
at its top.
String Handling
• String handling with test command:
test str Returns true if str is not null

test –n str Returns true if length of str is
greater than zero
test –z str Returns true if length of str is
equal to zero
• String handling with expr command
The expr is quite handy for finding the length of a string and extracting a sub-string:
Length of the string:
$ n=èxpr "$str" : ‘.*’` ;
$ echo $n
11
expr gave how many times any character (.*) occurs. This feature is very useful in
validating data entry.
Extracting a sub-string:
$ expr “$str” : ‘……....’
gh
Note that there are 6 dots preceding the sequence ..... This advanced regular
expression signifies that the first six characters of the string are to be ignored and
extraction should start from the 7th character. Two dots inside .... suggests that this
extraction is limited to two characters only (backslashes override the usual
interpretation of ‘()’).
Extracting string from 3rd character to end of the string:
$ str="abcdefghijk"
$ expr "$str" : '...∗.∗'
cdefghijk
Location of first occurrence of a character “d” inside string:
$ expr "$str" : '[^d]*d‘
Location of last occurrence if a character inside string:
Below will give the last occurrence of character 'a' from string str.
$str=”abc def abc”
$expr "$str" : '[û]*a'
10.
7. Command Line Arguments

To make a shell script a generalized script, we should avoid hard coding. User should
be able to provide the values required for processing as input while running the shell
script. To facilitate this we have to use command line arguments.
The statement we write in the command prompt starting with the command followed
by list of arguments is the command line argument.
Example:
$ ls dir1 dir2 dir3

A set of shell variables are there to represent the command line arguments.
$0 – represents the command name (first word on command line). For above example
“ls”
$1 - the first argument of command (second word/symbol ).For the above example
dir1
$2 – the second argument to the command. For the above example, dir2
In this way we can use up to $9
The command-line arguments $1, $2, $3...$9 are also called positional parameters,
with $0 pointing to the actual command/program/shell script and $1, $2, $3, ...$9 as
the arguments to the command.
Consider below sample shell script “test.sh”
$cat test.sh
#!/bin/sh
echo “Program name is $0”

echo “First argument is $1”
echo “Number of arguments passed =$#”
echo “The arguments are $@”
Let’s execute the above shell script “test.sh”. Remember to change File access
permission of script to execute
$ sh test.sh arg1 arg2 arg3

There’s another way to execute the script using ./
$ ./test.sh arg1 arg2 arg3
User Input: read
To get input from the keyboard, you use the read command. The read command takes
input from the keyboard and assigns it to a variable.
Example: read.sh
#!/bin/sh
echo –n “Enter some text > “
read text
echo “You entered: $text”
Note that “-n” given to the echo command causes it to keep the cursor on the same
line; i.e., it does not output a carriage return at the end of the prompt.
Next, we invoke the read command with “text” as its argument. What this does is wait
for the user ro type something followed by a carriage return (the Enter key) and then
assign whatever was typed to the variable text.
Execution:
$sh read.sh
Enter some text > This is some text
You entered: This is some text
Validation of Command Line Arguments
Let us assume, we have one shell script which requires exactly 2 arguments to
execute. Shell script should throw proper error message in case user has not given
exactly two arguments.
Let us have a small shell script "test.sh" to show how to implement validations.
#!/bin/sh
if [ $# -ne 2 ]
then
echo " Not sufficient arguments, please give exact 2 arguments"
else
echo "Continue with our task" fi
$ ./test.sh
$ sh test.sh abc
$ ./test.sh abc def ghi
$ ./test.sh abc def 2
$ ./test.sh abc def
Continue with our task
For scripts where input expected is a file , validations can be incorporated for file tests
like whether the file given as input exists or not, whether it is empty or not, whether it
is readable or not and so on as per requirements.
Basic UNIX commands :
The basic Unix commands are:
echo
It displays a line text.
Echo the given string or strings to standard output.
Syntax:
echo [OPTION]... [STRING]...
Example:
echo "Good Morning"
This will display the following in the standard output
Good Morning
passwd command
The passwd command is used to change passwords for user accounts. A normal user
may only change the password for his or her own account, while the superuser may
change the password for any account.
Syntax : passwd [options] [LOGIN]
Options:
uname command
Print certain system information.
Syntax : uname [OPTION]...
Options:
Examples: To show the name of the OS
uname command:
Used to get the information about the current system (the name, version and other
details)
$uname
To get the kernel name, you can use -s parameter.
$ uname -s
To get the kernel release, -r option is used
$ uname -r
who command:
who command prints information about all users who are currently logged in.
Syntax:
who [OPTION]... [ FILE ] [ am i ]
find command:
find is one of the powerful utility of Unix (or Linux). It can search
the entire file-system to locate files and directories according to the specific
search criteria. Other than searching files in can also perform actions(executing
commands) on searched files.
Syntax:
$ find [option] [action]
Examples :
1 . File Type based Search:
>> The following command finds the file named Abc in current(.) directory and
all its sub-directories
[390119@InGhyUnix ~]$ find . -name Abc

./Abc
./Dir1/Abc
2. Wild-Card based Search : Wild-card characters like * and ? can be used:

>> To find all files where filename starts with Abc the command will be:
[390119@InGhyUnix ~]$ find . -name "Abc*"

./Abc
./Dir1/Abc
./Dir1/Abc.php
./Abc.txt
./Abc.dat
3 . Source Location based Search:

>>The command find can search in multiple source location :
[390119@InGhyUnix ~]$ find Dir1 Dir2 -name "Abc.*"

Dir1/Abc.php
Dir2/Abc.htm
Searches for the file Abc with any extension only in directories Dir1 and Dir2
4 . Size Based Search : Using find files can be searched based on its size .
>> To find the files with size more than 10Mb the command will be
[390119@InGhyUnix ~]$ find . -size +10M

./Abc.dat
5 . Access/Modification-Time Based Search : Using find files can be searched

based on the time of last access/modification :
>> To find the files which are accessed within last 3 days the command will be
[390119@InGhyUnix ~]$ find . -atime -3
./Abc
6 . Permission Based Search : Using find command files can be searched

based on the access permission they have
>> To find the files having read write and execute permission only for the owner
the command will be :
[390119@InGhyUnix ~]$ find . -perm 700

./Abc.txt
7 . Perform Action on the search result : Using exec option any action (command)
can be executed on the search result.
>> To remove all the files having extension .swp from the current directory the
command will be :
[390119@InGhyUnix ~]$ find . -name “*.swp” -exec rm {} \;
bc command:
bc is the command used for basic mathematical calculations.
It can be used either a mathematical scripting language or as an interactive

mathematical shell .
In interactive mode it accepts input from the terminal and providing calculations on
requested arithmatic expression.
That means typing the command bc on a Unix command prompt and entering a
mathematical expression, such as (1 + 4) * 2 , whereupon 10 will be the output
While used with files , bc starts by processing code from all the files listed on the
command line in the order listed. After all files have been processed, bc reads from
the standard input. All code is executed as it is read.
Syntax: bc [ -hlwsqv ] [long-options] [ file ... ]
Example:
$ echo "2+5" | bc
Output will be:
tty command:
Prints the file name of the terminal connected in standard input.
Syntax : tty [OPTION]...
UNIX Text Editors

To write a program or a paper even to edit a mail message, you need to create,
write,store, retrieve, view, and edit files. In unix several text editors available for this .
Editors available unix are ed, node,NEdit, emacs and vi .
One of the first end-user programs hosted on the system and standardized in UNIX-
based systems ever since is the vi editor. ed is a line editor for the UNIX operating
system.
vi Editor:
vi (pronounced "vee-eye") is short for "vi"sual editor. It is a visual editor used to

create and edit text files.
It displays a window into the file being edited that shows 24 lines of text. vi lets you
add, change and delete text, but does not provide formatting capabilities
like centering lines or indenting paragraphs.
vi works in three different modes:
4. Edit Mode -where any key is entered as text

5. Command Mode -where keys are used as commands
6. Ex Mode -ex commands can be entered in last line to act on text
Invoking vi with/without filename puts it in command mode:
vi []
Commands to be used in vi:
For cursor movement within the vi editor, we can not use the mouse . We have to use
key commands. To use a key command, user has to be in the command mode. To go
to command mode, press the Esc key.
Commands to Enter and Exit vi:

vi edit filename starting at line 1
filename
vi –r recover filename that was being edited when system
filename crashed
The new or modified file is saved when user exits from vi. But it is also possible to
exit without saving the file.
To use a command to exit from vi , after pressing the Esc key ,colon : is to be pressed.
The cursor moves to bottom of screen whenever a colon (:) is typed. This type of
command is completed by hitting the (or ) key.

:wq quit vi, writing out modified file to file named in
original invocation
:q quit (or exit) vi
:q! quit vi even though latest changes have not been saved
for this vi call
:w write current contents to file named in original vi call
Commands to insert or add text:
Following are the commands to be used to insert or add text.

u Undo the last command’s action
i
insert text before the cursor,until is hit
I
insert text at beginning of current
line, until hit
a append text after cursor, until hit

A append text to end of current line,
until hit
o open and put text in a new line below
O open and put text in a new line above
Commands to modify texts:

r replace single character under cursor (no needed)
R replace characters, starting with current cursor position,
until hit
cw change the current word with new text,
starting with the character under cursor, until hit
cNw change N words beginning with character under cursor,
until hit;
C change (replace) the characters in the current line, until hit
cc change (replace) the entire current line, stopping when is
hit
Ncc change (replace) the next N lines, starting with the current
or line,
cNc stopping when is hit
Commands to Delete texts:

x delete single character under cursor
Nx delete N characters, starting with character under cursor
cw delete the single word beginning with character under
cursor
dw change N words beginning with character under cursor,
until hit;
dNw delete N words beginning with character under cursor;
e.g., d5w deletes 5 words
D delete the remainder of the line, starting with current cursor
position
dd delete entire current line
Ndd delete N lines, beginning with the current line;
or e.g., 5dd deletes 5 lines
dNd
Commands to copy and paste text:
yy copy (yank, cut) the current line into the buffer

Nyy copy (yank, cut) the next N lines, including the current line,
into the buffer
p put (paste) the line(s) in the buffer into the text after the
current line
Commands for searching text
/string search forward for occurrence of string in text

?string search backward for occurrence of string in text
UNIX FILE COMMANDS

In this module following important file handling commands will be discussed:
• touch
• cat
• cat > (single redirection)
• cat >>(double redirection)
• cp
• mv
• cmp
• comm
• diff
touch command
Touch command is used in Unix to create an empty file.
$ touch file1 file2

$ ls -l
$
Touch command is used to change the timestamps (access time and modification time
of a file).
cat command
Use of cat command:
'cat' command is used to display a file content.
Syntax:
$ cat filename [ filename ]
$ cat data
This is an unix file
This is line two
$
cat > and cat >> command

Redirection operator ( > and >>) can be used with cat command to take input from
standard input device(keyboard) and store them in a file.
Syntax: $ cat > [filename]
$ cat > data

The file created using
cat > filename
^d
$
cat with single redirection(cat >filename) , creates a new file if no file with the given
file name already exists and overrites the file if a file with given file name already
exists.
cat with double redirection (cat >>filename) appends the content given through
standard input to the file.
cp command
cp command is used to copy one file's content to another file.
Syntax: $ cp [source filename] [destination filename]
Switches:
1. cp -i [sourcefilename] [destination filename] This command copies the content of a
file interactively.
2.cp -r [source_directory] [destination_directory] This command copies the whole file

hierarchy.
3. cp -v [sourcefilename] [destination filename] Copies in verbose mode, explains

what is being done
mv command
mv command is used for:
1. for renaming a file in same directory.
2. for moving a file to a different directory.
Syntax:
$ mv [sourcefilename] [destinationfilename]
$ mv [source_directory] [destination_directory]
Directory Commands:
Every item in the UNIX filesystem tree is either a file, or a directory. A directory is
like a file folder. A directory can contain files, and other directories. A directory
contained within another is called the child of the other. A directory in the filesystem
tree may have many children, but it can only have one parent. A file can hold
information, but cannot contain other files, or directories.
The place in the file system tree where an user is located is called the current
working directory (which can be listed using pwd command).
As a user logs in, user is put in his/her home directory.

mkdir(make directory) command:
The mkdir command creates a directory with specified name in the present working
directory or specified path.
Syntax: $ mkdir [ ]
ls Command
Lists the contents of any directory (current directory if no directory specified).
Sorts entries without option
$ ls [option(s)] [filename]
File comparison commands:

cmp – This command is used to compare two files, and if they differ, tells the first
byte and line number where they differ. If input files are same , returns nothing.
Syntax :
cmp options file1 file2
To use cmp commands both the files need to be sorted.
comm- This command is used to compare two sorted files.
Syntax:
comm [options] file1 file2
• One set of options allows selection of ‘columns’ to suppress.

o -3: suppress lines common to file1 and file2 (column3)
Example: To show only lines common between file1 and file2, command is
$ comm -12 file1 file2

diff - This command is used to compare two files line by line. The output indicates
how the lines in each file are different, and the steps that will be involved in changing
file1 to file2.
The change commands are in the format [range][acd][range]. The range on the left
may be a line number or a comma-separated range of line numbers referring to file1,
and the range on the right similarly refers to file2. The character in the middle
indicates the action i.e. add, change or delete.
• ‘LaR’ – Add lines in range ‘R’ from file2 after line ‘L’ in file1.
• ‘FcT’ – Change lines in range ‘F’ of file1 to lines in range ‘T’ of file2.
• ‘RdL’ – Delete lines in range ‘R’ from file1 that would have appeared at line
‘L’ in file2
Syntax: diff [options] file1 file2

USAGE OF WinSCP
Introduction to WinSCP
WinSCP (Windows Secure CoPy) is a popular , free and open-source Secure

File Transfer application (SFTP) for Windows computers .It is used to transfer
files securely between remote server and local computer and vice verse. It
includes support for SCP file transfers, and has the ability to create secure
tunnels via SSH. Secure FTP (SFTP) is similar to FTP, but with SFTP the entire
session is encrypted, so that passwords are never sent in the clear text , and are
therefore much less vulnerable to interception. That is ,for
transferring files securely, we can use Secure copy or SCP protocol, it uses
Secure Shell (SSH), which encrypts the content before transferring data over
network. It runs over TCP port 22 by default.
How to use WinSCP
Step I : Start WinSCP

To start with double click on WinSCP icon in the desktop or select it from
Programs Group . The
first screen will look as below
Step II : Logging in to Server
Enter IP address of the server in Host Name field.
Unix login userid in the User name field
Unix login Password in Password field
Choose the protocol ( SFTP/ SCP / FTP)
For example
1) Host name used : 172.26.132.40 (For example)
2) User name used : 390119
3) Password used : ********
3) Protocol selected : SCP
Step III : Select/Navigate to working directory
1. We can select the working directory in both Local as well as remote
machine from the drop
down menu.
2. Tool bar can be used to navigate to desired folder
Step IV : Transferring content Content can be transferred in two way
A . Downloading : Transferring content from Remote server to Local
machine
B . Uploading : Transferring content from Local machine to Remote server
Following Steps to perform for Downloading file/Directory from Server
Step - A1 : Select the file to transfer in the Remote pane,
Step - A2 : Right click → Select Copy or move option
Step - A3 : Select the destination folder in the local machine(can be

changed by selecting
Browse button )
Step - A4 : The file Unix_Recap.txt copied to desktop.
Uploading files to Remote Server

Step - B1 : Select the file to copy in Local machine(Left) pane.
Step - B2 : Drag it and drop it to the remote pane desired location.
Step - B3 : Select the desired location from drop down list and select copy
Saving sessions for future use

All the session information can be saved for future use and faster access.
The information can be saved are :
>> Server IP
>> User name,
>> Password(Not recommended)
>> Communication protocol

UNIX Final Study Material Compressed

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

UNIX Final Study Material Compressed

Uploaded by

Copyright:

Available Formats

Filters

Some common filters in UNIX are:

• uniq – Removes identical adjacent lines

head [OPTION] [FILE]

To display the first 100 lines of the file myfile.txt.

$head -n100 myfile.txt

$ head -c5 myfile.txt

tail [OPTION]... [FILE]...

To display the last 10 lines of the file myfile.txt.

To display the last 100 lines of the file myfile.txt.

$ tail -100 myfile.txt

$tail –n 100 myfile.txt

more [option] filename

tr [OPTION] set1 [set2]

c. : complements the set of characters in string.

s. : replaces repeated characters listed in the set1 with single occurrence

Example: To replace any occurrence of a by x, b by y and c by z in a given string

$echo “about to call “|tr [abc] [xyz]

Output : xyout to zxll

Example: To replace non matching characters

Example: Squeez , we can squeeze more than one occurrence of continuous

Output : about to call

Above example squeezes two or more blank spaces into one.

The default order is ascending .

sort -t field_delemeter [OPTION] file1 [file 2]

-k n sort on the nth field of the line

-t char use char as the field delimiter

-r reverse order sort

-u removes repeated lines

-m list merge sorted files in list

Consider a file named “list”, which has below data

1, Justin Timberlake, Title 545, Price $7.30

2, Lady Gaga, Title 118, Price $7.30

3, Johnny Cash, Title 482, Price $6.50

5, John Lennon, Title 271, Price $7.90

$sort –t’,’ –k 2 list

Note: File list is comma separated file.

4, Elvis Presley, Title 335, Price $7.30

5, John Lennon, Title 271, Price $7.90

3, Johnny Cash, Title 482, Price $6.50

1, Justin Timberlake, Title 545, Price $7.30

2, Lady Gaga, Title 118, Price $7.30

Example 2: Numerically sorting:

To numerically sort data , option to be used is –n

Suppose list is the name of the file having following data:

To get the expected output , the command will be

Sort can sort multiple files also.

$sort file1 file2 file3 …

Example 3: Numerically sort in reverse order

$sort –nr list

Example 4: Sort the file list removing the repeated lines.

File list has following content:

uniq [option] filename

-u lists only the lines that are unique

lists only the lines that are duplicates

-c counts the frequency of occurrences

Suppress duplicate lines:

Display only duplicate lines:

Skip first N fields in comparison:

cut –option filename

Select Column of Characters :

To extract only a desired column from a file use -c option.

$cut –c2 test.txt