Operating Systems

The structure of a Computer system can be visualized as having four basic

components:
1. Hardware - Provides basic computing resources - CPU, memory, I / O devices
2. Operating system - Controls and coordinates use of hardware among various
applications and users
3. Application programs - Define the ways in which the system resources are used
to solve the computing problems of the users
4. Users - Anybody who desires work to be done by a computer system. People,
machines, other computers

Types of OS

            OS are classified into following different types depending on their

capability of processing.
 

Single User and Single Tasking OS: 

These are simple operating system designed to manage one task at a time, for use
by a single user for a standalone single computer for performing a single task.

Single User and Multitasking OS:

These OS allow execution of more than one task or process concurrently by
dividing the processor time amongst different tasks.

Multi-programming OS:

These OS allow more than one programs to run at the same time.

Real Time OS:

These are designed to respond to an event within a predetermined time. These

operating systems are used to control processes

Embedded OS:

Embedded in a device in the ROM. They are specific to a device and are less
resource intensive.

Function Of OS

The OS performs basic tasks such as controlling and allocating memory,

prioritizing system requests, controlling input and output devices, facilitating
networking, and managing files

Introduction to UNIX OS

  Unix is an OS for Programmers as shell (the command interpreter) provides

the programming facility.
  It provides an in-built security mechanism through the user name and
password, combined with the access rights associated with files 
  Developed by Ken Thompson and Ritchie originally in assembly, and later
in C, thus making it portable to other machines 

 Supports C, Fortran, Basic, Pascal, COBOL, Lisp, Prolog, Java, Ada compilers

Features of Unix Operating System

• Multi-user and Multitasking 

• Everything is a file 
• Configuration data stored in text 
• Small, single purpose programs 
• Ability to chain programs together to perform complex task 
• Facility of background processing 

Architecture of the UNIX System

The UNIX operating system has a layered architecture having three main
components 
• Kernel 

• Shell 

• Utilities 

The Unix Kernel

Kernel is a collection of programs mostly written in C which allocate the system

resources and coordinate all the details of the computer's internals.

      Functions of Kernel:

 It allocates time and memory to programs and handles the file store and
communications 
 Interacts directly with the hardware through device drivers
 Provides sets of services to programs
 Manages memory, controls access, maintains file system, handles interrupts,
allocates resources of the computer
System calls

The system calls are functions used in the kernel itself. UNIX system calls are used
to manage the file system, control processes, and to provide interpocess
communication. 

System calls can be categorized as: 

 File structure related calls -For example create, open,read, write,lseek,dup

etc. 
 Process Related calls  -For example fork,exec,wait,exit etc 
 Inter process related calls - For example pipe,msgget,msgsnd etc. 

The Unix File System:

Unix File System is a hierarchical collection of 3 types of files:

  Ordinary Files
  Directory Files
  Special Files (device, pipe, fifo, socket).

Characteristics of Unix Files:

 UNIX file is featureless because it is simply an array of bytes. 

 Dominant file type in UNIX is the text file.
 System related files are also stored in text form.
 Separate device can be added by creating a file for it.
 Root is the supremo and is represented by the '/'. Every sub-directory must
have a parent.
 File names can be up to 14 characters long, can contain both upper and
lower case alphabets, digits, a dot, hyphen (-), underscore (_) anywhere but
should not have a blank or tab. 
 Files are accessed using path names. Path names are a sequence of directory
names separated by '/'. 

There are two types of path names in unix.

  Absolute path name - file location is determined with respect to the root.
  Relative path name - file location is determined with respect to the current
directory.

Structure of Unix File System:

The Unix File System (UFS) looks hierarchical but it is actually a directed a-cyclic
graph because files can be shared.

Following are some directories available under root in UFS.

/ home - It holds user's home directories. In other UNIX systems, this can be / usr
directory.

/ bin - It holds many of the basic Linux programs; bin stands for binaries, files that
are executable.

/ usr - It holds many user-oriented directories:

/ sbin - It holds system files that are usually run automatically.

/ etc - It and its subdirectories hold many of Linux config files.

 
/dev – It holds device files. All info sent to /dev/null is thrown into trash. Your
terminal is one of the /dev/tty files.

Unix file System Organization:

 Unix divides  physical disks into logical disks called partition which is

comprising of a set of consecutive cylinders.
 The UFS resides on a single partition.
 Each file system contains four blocks :

               a) boot block : This block is located in the first few sectors of a file
system. It contains the initial bootstrap program                                          used to
load the operating system.Typically, the first sector contains a bootstrap program
that reads in                                          a larger bootstrap program from the next
few sectors, and so forth.

             b) super block : This block describes the state of the file system such
as  the total size of the partition, the block size,                                        pointers to
a list of free blocks, the inode number of the root directory, magic number or file
signature                                       (the first few bytes of a file which are unique to a
particular file type), etc.

            c) inode list : A linear array  of inodes (index nodes). This is a data

structure which describes the attributes of a                                                file.
There is a one to one mapping of files to inodes and vice versa. An inode is
identified by it's "inode                                      number". Users uses file names to
refer to a file but  Unix represent  files in terms of inodes.

            d) data blocks : blocks containing the actual contents of files

 Internal File Maintenance: 

For each file created in the system, an inode is also created. Inode is a disk file
record of 64 bytes that maintains the permanent attributes of a file.

An inode is permanent and it exists until the corresponding file is removed from
the system.

Sample details of an inode –

  Owner and group identifiers

  File type and file size
  Number of links for this file
  Times of file creation, last file access and modification, and last inode
modification
   List of access rights – read/write/execute permissions
  Reference count showing number of times file is opened
  Physical address of file on the disk: array of 13 pointers for data storage

Whenever a file is opened, its inode is brought into main memory. The active
inode is kept there until the file is closed and is used to locate the beginning of an
open file on disk and to verify that every I/O request comes from a valid user as
per specified access permissions.

Users and Access Rights

Objective

This module is to have an understanding of different user types in UNIX and their
access rights.

Course Content

  User types
  File access permissions
  Changing file access permissions using symbolic mode
   Changing file access permissions using octal mode

User Types in Unix

There are three types of accounts in a Unix system:

- Root Account
- System Account
- User Account

Managing users and groups

Unix supports a concept of Group which logically groups a number of accounts.

➢ Every account would be a part of any group.
➢ Unix groups plays important role in handling file permissions and process
management.
➢ Grouping of users allows to grant and revoke file permissions collectively.

File Permission & Ownership level

File ownership is an important component of Unix that provides a secure method
for storing files. Every file in Unix has the following attributes:
File Permission Types (Mode)
Three type of permissions can be set for any ownership level:

 Read
 Write
 Executed 

These three modes or permissions have different meaning for a file and a directory;

Changing Permissions (chmod)

To change file or directory permissions, the chmod (change mode) command is
used. There are two ways to use chmod:

  Symbolic mode
  Absolute fashion
chmod command options:

Using chmod in Symbolic Mode:

With symbolic representation, permission set can be added, deleted, or specified

using the operators listed in the following table:

Example:

Let us assume there is a file name "abc" , whose initial permissions are only read
permission to user and group and others. We will provide  full permissions to all
three types of users.

Let us verify the existing file permissions using ls -l command as below and
observe the next steps, like changing the permissions and verifying the changes.
In the above example,

chmod u+wx,g+wx abc , adds write and execute permission to user and group to
the file abc

chmod g-x abc, removes execute permission from group from the file abc.

chmod o=r-- abc, provides read permission and removes write and execute
permission from others for file abc

Using chmod in Absolute(octal) Mode

The second way to modify permissions with the chmod command is to use a
number to specify each set of permissions for the file. Each permission is assigned
a value, as the following table shows, and the total of each set of permissions
provides a number for that set.
 

Examples: Observe the following sequence of command execution statements and

analyze the output

1. Let us view the initial permissions of the file abc

    ls -l abc

    -rw-r--r-- 1 735873 oinstall 0 Feb 7 12:37 abc

If we observe the output, initially owner have read and write permissions and
group and others have only read permission

2. Let us remove all the permission from all users for abc file, as below
   chmod 0 abc
   Let us verify the changed permissions

   ls -l abc
   ---------- 1 735873 oinstall 0 Feb 7 12:37 abc
Yes all permissions are taken away from all users on the file abc

3. To provide all permissions to the user, read permission to group and write and
execute permissions for Others,

    use the below command

chmod 743 abc

Let us verify the changed permissions

ls -l abc
-rwxr---wx 1 735873 oinstall 0 Feb 7 12:37 abc
From the output all permissions to user, read permission to group and write and
execute
permissions for Others, Use the below command

4. chmod 755 abc

    The above command provides all permissions to the user, read and execute
permissions to group and others
Let us verify ls -l abc
-rwxr-xr-x 1 735873 oinstall 0 Feb 7 12:37 abc

5. chmod 777 abc

   The above command provides all permissions to all users . We can verify using
below command
   ls -l abc
   -rwxrwxrwx 1 735873 oinstall 0 Feb 7 12:37 abc

UNIX Text Editors

To write a program or a paper even to edit a mail message, you need to create,
write, store, retrieve, view, and edit files. In unix several text editors available for
this.

Editors available unix are   ed, node, NEdit, emacs and vi .

One of the first end-user programs hosted on the system and standardized in
UNIX-based systems ever since is the vi editor. ed is a line editor for the UNIX
operating system.

vi Editor:

 vi (pronounced "vee-eye") is short for "vi" sual editor. It is a visual editor used to
create and edit text files.

It displays a window into the file being edited that shows 24 lines of text. vi lets
you add, change and delete text, but does not provide formatting capabilities like
centering lines or indenting paragraphs.

vi works in three different modes:

1. Edit Mode -where any key is entered as text

2. Command Mode -where keys are used as commands
3. Ex Mode -ex commands can be entered in last line to act on text

Invoking vi with / without filename puts it in command mode:

vi []

Commands to be used in vi:

For cursor movement within the vi editor, we can not use the mouse. We have to
use key commands. To use a key command, user has to be in the command
mode. To go to command mode, press the Esc key.

Commands to Enter and Exit vi:

 command Purpose of the command

vi filename edit filename starting at line 1

vi –r recover filename that was being edited when system crashed

filename

The new or modified file is saved when user exits from vi. But it is also possible to
exit without saving the file.

To use a command to exit from vi, after pressing the Esc key, colon: is to be
pressed.

The cursor moves to bottom of screen whenever a colon (:) is typed. This type of
command is completed by hitting the  (gold ) key.

command Purpose of the command

: wq quit vi, writing out modified file to file named in original invocation

:q quit (or exit) vi

: q! quit vi even though latest changes have not been saved for this vi call

:w write current contents to file named in original vi call

Commands to insert or add text:

Following are the commands to be used to insert or add text.

key Purpose of the command

u Undo the last command's action

i
insert text before the cursor, until is hit

 
 I
insert text at beginning of current line, until  hit

at append text after cursor, until  hit

AT append text to end of current line, until  hit

o open and put text in a new line below current

line, until  hit

O open and put text in a new line above current

line, until  hit

Commands to modify texts:

key Purpose of the command

r replace single character under cursor (no  needed)

R replace characters, starting with current cursor position, until  hit

cw change the current word with new text, 

starting with the character under cursor, until  hit

cNw change N words beginning with character under cursor, until  hit; 

  eg, c5w changes 5 words

VS change (replace) the characters in the current line, until  hit

CC change (replace) the entire current line, stopping when  is hit

Ncc change ( replace) the next N lines, starting with the current line,
or stopping when  is hit
cNc

Commands to Delete texts:

 key Purpose of the command

x delete single character under cursor

Nx delete N characters, starting with character under cursor

cw delete the single word beginning with character under cursor

dw change N words beginning with character under cursor, until  hit; 

  eg, c5w changes 5 words

dNw delete N words beginning with character under cursor; 

  eg, d5w deletes 5 words

D delete the remainder of the line, starting with current cursor position

dd delete entire current line

Ndd delete N lines, beginning with the current line; 

or   e.g., 5dd deletes 5 lines
dNd

Commands to copy and paste text:

yy copy (yank, cut) the current line into the buffer

Nyy copy (yank, cut) the next N lines, including the current line, into the buffer

p put (paste) the line(s) in the buffer into the text after the current line

Commands for searching text

/string search forward for occurrence of string in text

?string    search backward for occurrence of string in text

 
 

UNIX FILE COMMANDS

In this module following important file handling commands will be discussed:

 touch
 cat
 cat> (single redirection)
 cat >> (double redirect)
 cp
 mv
 cmp
 comm
 Diff

touch command
Touch command is used in Unix to create an empty file.

$ touch file1 file2

$ ls -l
-rw-r - r-- 1 user group 0 2014-02-08 7:14 file1
-rw-r - r-- 1 user group 0 2014-02-08 7: 14 file2
$

Touch command is used to change the timestamps (access time and modification
time of a file).

cat command
Use of cat command:
'cat' command is used to display a file content.
Syntax:

$ cat filename [filename]

$ cat data
This is an unix file
This is line two
$

cat> and cat >> command

Redirection operator (> and >>) can be used with cat command to take input from
standard input device (keyboard) and store them in a file.
Syntax: $ cat> [filename]
$ cat> data
The file created using
cat> filename
^d
$

cat with single redirection (cat> filename), creates a new file if no file with the
given file name already exists and overrites the file if a file with given file name
already exists.

cat with double redirection (cat >> filename) appends the content given through
standard input to the file.

cp command
cp command is used to copy one file's content to another file.
Syntax: $ cp [source filename] [destination filename]
Switches:
1. cp -i [sourcefilename] [destination filename] This command copies the content
of a file interactively.

2.cp -r [source_directory] [destination_directory] This command copies the whole

file hierarchy.

3. cp -v [sourcefilename] [destination filename] Copies in verbose mode, explains

what is being done

mv command
mv command is used for:
1. for renaming a file in same directory.
2. for moving a file to a different directory.
Syntax:
$ mv [sourcefilename] [destinationfilename]
$ mv [source_directory] [destination_directory]

Directory Commands:
Every item in the UNIX filesystem tree is either a file, or a directory. A directory is
like a file folder. A directory can contain files, and other directories. A directory
contained within another is called the child of the other. A directory in the
filesystem tree may have many children, but it can only have one parent. A file can
hold information, but cannot contain other files, or directories.

The place in the file system tree where an user is located is called the current
working directory (which can be listed using pwd command).
As a user logs in, user is put in his/her home directory. 

mkdir(make directory) command:

The mkdir command creates a directory with specified name in the present
working directory or specified path.
Syntax: $ mkdir [ ]

 ls Command
 Lists the contents of any directory (current directory if no directory specified).
 Sorts entries without option
$ ls [option(s)] [filename]

File comparison commands:

cmp – This command is used to compare two files, and if they differ, tells the first
byte and line number where they differ. If input files are same , returns nothing.

Syntax :

cmp options file1 file2

To use cmp commands both the files need to be sorted.

comm- This command is used to compare two sorted files.

Syntax:

comm [options] file1 file2

 One set of options allows selection of ‘columns’ to suppress.

o -1: suppress lines unique to file1 (column 1)
o -2: suppress lines unique to file2 (column 2)
o -3: suppress lines common to file1 and file2 (column3)

Example: To show only lines common between file1 and file2, command is

$ comm -12 file1 file2

diff -  This command is used to compare two files line by line. The output indicates
how the lines in each file are different, and the steps that will be involved in
changing file1 to file2.
The change commands are in the format [range][acd][range].  The range on the left
may be a line number or a comma-separated range of line numbers referring to
file1, and the range on the right similarly refers to file2.  The character in the
middle indicates the action i.e. add, change or delete.

 ‘LaR’ – Add lines in range ‘R’ from file2 after line ‘L’ in file1.
 ‘FcT’ – Change lines in range ‘F’ of file1 to lines in range ‘T’ of file2.
 ‘RdL’ – Delete lines in range ‘R’ from file1 that would have appeared at
line ‘L’ in file2

Syntax: diff [options] file1 file2

USAGE OF WinSCP

Introduction to WinSCP

WinSCP (Windows Secure CoPy) is a popular, free and open-source Secure File
Transfer application (SFTP) for Windows computers. It is used to transfer files
securely between remote server and local computer and vice verse. It includes
support for SCP file transfers, and has the ability to create secure tunnels via
SSH. Secure FTP (SFTP) is similar to FTP, but with SFTP the entire session is
encrypted, so that passwords are never sent in the clear text, and are therefore
much less vulnerable to interception. That is, for transferring files securely, we can
use Secure copy or SCP protocol, it uses Secure Shell (SSH), which encrypts the
content before transferring data over network. It runs over TCP port 22 by default.

How to use WinSCP

Step I: Start WinSCP

To start with double click on WinSCP icon in the desktop or select it from
Programs Group. The
first screen will look as below
Step II: Logging in to Server

Enter IP address of the server in Host Name field.

Unix login userid in the User name field
Unix login Password in Password field
Choose the protocol (SFTP / SCP / FTP)
For example
1) Host name used: 172.26.132.40 (For example)
2) User name used: 390119
3) Password used : ********
3) Protocol selected: SCP
    

Step III: Select / Navigate to working directory

1. We can select the working directory in both Local as well as remote machine
from the drop
down menu.
2. Tool bar can be used to navigate to desired folder
  

Step IV: Transferring content

Content can be transferred in two ways

A. Downloading: Transferring content from Remote server to Local machine
B. Uploading: Transferring content from Local machine to Remote server

Following are the Steps to be performed for Downloading file / Directory from
Server
Step - A1: Select the file to transfer in the Remote pane,
Step - A2: Right click → Select Copy or move option

Step - A3: Select the destination folder in the local machine (can be changed by
selecting Browse button)

Step - A4 : The file Unix_Recap.txt copied to desktop.

Following are the steps to be followed for uploading files to Remote Server

Step - B1 : Select the file to copy in Local machine(Left) pane.

Step - B2 : Drag it and drop it to the remote pane desired location.

Step - B3 : Select the desired location from drop down list and select copy
Saving sessions for future use:

All the session information can be saved for future use and faster access.
The information can be saved in following ways :
>> Server IP
>> User name,
>> Password(Not recommended)
>> Communication protocol
Filters
Filter is a programm in unix. It takes its input from another program, performs
some operation on that input, and writes the result to the standard output. Thus the
common use of filters is to modify or restructure output.

Some common filters in UNIX are:

 uniq - Removes identical adjacent lines

 head - displays first n lines of a file.
 tail - displays last n lines of a file.
 sort - sorts files by line (lexically or numerically)
 cut - select portions of a line. 
 wc - word count (line count, character count)
 tr - translate
 grep, egrep - search files using regular expressions

head
This command list the beginning of a file to standard output. The default is 10
lines, but a different number can be specified. The command has a number of
options.

Syntax:

head [OPTION] [FILE]

Options:

-c Prints the first N bytes of file; with leading -, prints all but the last N bytes of the
file.

-n Prints first N lines; with leading - print all but the last N lines of each file.

Example: To display the first 10 lines of the file myfile.txt. $ head myfile.txt

To display the first 100 lines of the file myfile.txt.

$ head -n100 myfile.txt

To print the first 5 bytes from the file

$ head -c5 myfile.txt

tail
List the (tail) end of a file to stdout. The default is 10 lines, but this can be changed
with the -n option. Commonly used to keep track of changes to a system log-file,
using the -f option, which outputs lines appended to the file.

Syntax:

tail [OPTION] ... [FILE] ...

Example:

To display the last 10 lines of the file myfile.txt.

$ tail myfile

To display the last 100 lines of the file myfile.txt.

$ tail -100 myfile.txt

$ tail –n 100 myfile.txt

more
more command allows to view text files or other output in a scrollable
manner. When can command is used to view a very long file, all the output scrolls
off the top of your screen and only the last page can be viewed. more command
solves this problem by allowing the output of cat command one screenful of data at
a time.

Syntax:

more [option] filename

Options:

-num This option specifies an integer which is the screen size (in lines).

-d more will prompt the user with the message "[Press space to continue, 'q' to
quit.]" and will display "[Press 'h' for instructions.]" instead of ringing the bell
when an illegal key is pressed .

-l more usually treats ^ L (form feed) as a special character, and will pause after
any line that contains a form feed. The -l option will prevent this behavior.

-p Do not scroll. Instead, clear the whole screen and then display the text.

tr
tr command automatically translates or substitute characters.
Syntax:

tr [OPTION] set1 [set2]

Translate, squeeze, and / or delete characters from standard input, writing to

standard output.
Options:

c. : complements the set of characters in string.

d. : deletes the characters in set1

s. : replaces repeated characters listed in the set1 with single occurrence

t. : truncates set1

 
Example: To replace any occurrence of a by x, b by y and c by z in a given string

$ echo “about to call“ | tr [abc] [xyz]

Output: xyout to zxll

Example: To replace non matching characters

$ echo "Hello" | tr -cea

Output: aeaaaa

In the above example, except the character “e” other characters are replaced by a

Example: Squeez, we can squeeze more than one occurrence of continuous

characters with single occurrence.

$ echo “about to call“ | tr - s ''

Output: about to call

Above example squeezes two or more blank spaces into one.

fate
sort command reorders the lines of a file in ascending or descending order.

The default order is ascending.

Syntax:

sort -t field_delemeter [OPTION] file1 [file 2]

Options:

-kn sort on the nth field of the line

-t char use char as the field delimiter

-n sort numerically

-r reverse order sort

-u removes repeated lines

-m list merge sorted files in list

Examples:

Below examples will help you to understand sort used with different options:

Example 1:

Consider a file named “ list” , which has below data

1, Justin Timberlake, Title 545, Price $ 7.30

2, Lady Gaga, Title 118, Price $ 7.30

3, Johnny Cash, Title 482, Price $ 6.50

4, Elvis Presley, Title 335, Price $ 7.30

5, John Lennon, Title 271, Price $ 7.90

To sort on the 2 nd field of file named “list” we have to use the below command:

$ sort –t ',' –k 2 list

Note: File list is comma separated file.

Output:

4, Elvis Presley, Title 335, Price $ 7.30

5, John Lennon, Title 271, Price $ 7.90

3, Johnny Cash, Title 482, Price $ 6.50

1, Justin Timberlake, Title 545, Price $ 7.30

2, Lady Gaga, Title 118, Price $ 7.30

Example 2: Numerically sorting:

To numerically sort data, option to be used is –n

Suppose list is the name of the file having following data:

19

20
5

49

00

If we sort it as below:

$ sort list
 

Output is:

19

20

200

49

To get the expected output, the command will be

$ sort –n list

Output:

19

20

49

200

Sort can sort multiple files also.

$ sort file1 file2 file3 ...

Example 3: Numerically sort in reverse order

$ sort - nr list

Output:

200

49

20

19

Example 4: Sort the file list removing the repeated lines.

Syntax:

$ sort –u filename

File list has following content:

Unix

Unix

Linux

Linux

Solaris

Axis

Axis

$ sort –u list

Output:

Unix

Linux

Solaris

Axis

 
uniq
uniq command is used to suppress the duplicate lines from a file. It discards all the
successive identical lines except one from the input and writes the output.
 

Syntax:

uniq [option] filename

Options:

-u lists only the lines that are unique

lists only the lines that are duplicates

-c counts the frequency of occurrences

Suppress duplicate lines:

The default behavior of the uniq command is to suppress the duplicate line. Note
that, you have to pass sorted input to the uniq, as it compares only successive lines.

If the lines in the file are not in sorted order, then use the sort command and then
pipe the output to the uniq command.

Count of lines:

The -c option is used to find how many times each line occurs in the file. It
prefixes each line with the count.

Display only duplicate lines:

You can print only the lines that occur more than once in a file using the -d
option. The -D option prints all the duplicate lines.

Skip first N fields in comparison:

the -f option is used to skip the first N columns in comparison. Here the fields are
delimited by the space character.

Cut
This command is used for text processing. You can use this command to extract
portion of text from a file by selecting columns.

Syntax:

cut –option filename

Select Column of Characters:

To extract only a desired column from a file use -c option.

The following example displays 2nd character from each line of a file test.txt.

$ cut –c2 test.txt

Select Column of Characters using Range:

 

Range of characters can also be extracted from a file by specifying start and end
position delimited with -.

The following example extracts first 3 characters of each line from a file called
test.txt

$ cut –c 1-3 test.txt

Select Column of Characters using either Start or End Position:

 

Either start position or end position can be passed to cut command with -c option.

Following example extracts from 3rd character to end of each line from test.txt file.
 

$ cut –c3- test.txt

To extract 8 characters from the beginning from the file test.txt,

$ cut –c-8 test.txt

Select a Specific Field from a File:
Instead of selecting x number of characters you can combine option -f and –d to
extract a whole field.

The option -f specifies which field you want to extract,

The option -d specifies what delimiter that is used in the input file.
The following example displays only first field of each lines from / etc / passwd
file using the field delimiter : (colon). In this case, the 1st field is the username.

$ cut -d ':' -f1 etc / passwd

paste
This is the command for merging together different files into a single, multi-
column file. In combination with cut, useful for creating system log files.
Syntax:

paste file1 file2

join
This utility allows merging two files in a meaningful fashion, which essentially
creates a simple version of a relational database.
 

The command join operates on exactly two files, but pastes together only those
lines with a common tagged field (usually a numerical label), and writes the result
to standard output.
 

The files to be joined should be sorted according to the tagged field for the
matchups to work properly.

Example:

The content of two files file1 and file2 are as below,

$ cat file1

100 Shoes

200 Laces

300 Socks

$ cat file2

$ 100 40.0

200 $ 1.00
$ 300 2.00
 

The following command will join these two files.

 

$ join 1.data 2.data

100 Shoes $ 40.00

200 Laces $ 1.00

300 Socks $ 2.00

Pipe

In unix, you can connect two commands together so that the output from one
program becomes the input of the next program. Two or more commands
connected in this way form a pipe. In shell the symbol '|' is used to represent pipe.
 

Purpose of Pipes :

Using pipe you can construct powerful unix command lines by combining basic
unix commands. UNIX commands are powerful; however by using pipe you can
combine them together, to accomplish complex tasks with ease.

Through the standard output of one command (the command to the left of the pipe)
gets sent as standard input to another command (the command to the right of the
pipe). Pipe functions in a similar manner like the output redirection in UNIX
(using> symbol to redirect the standard output of a command to a file. However,
the pipe is different because it is used to pass the output of a command to another
command, not a file.

Example:

$ cat apple.txt | wc

3 4 21
In this example, the contents of the file apple.txt are sent through pipe to wc (word
count) command. The wc command then does its job and counts the lines, words,
and characters in the file.

You can combine many commands with pipes on a single command line. Here's an
example where the characters, words, and lines of the file apple.txt is sent to wc
and then the output of wc mailed to nobody@december.com with the subject line
"The count."

$ cat apple.txt | wc | mail -s "The count" nobody@december.com

awk

awk is a scripting language which is used for processing or analyzing text files. 

awk is used for grouping of data based on either a column or field, or on a set of
columns.

It derives its name from the first letter of the last name of its three authors namely
Alfred V. Aho, Peter J. Weinberger and Brian W. Kernighan.

 awk can be used for reporting data in a useful manner. It searches one or more
files to see if they contain lines that match specified patterns and then perform
associated actions. awk is an advanced filter.
Simple awk Filtering

Syntax of awk:

~ $ awk 'pattern {action}' input-file

Let's take a input file with the following data
~ $ cat awk_file
Name, Marks, Max_Marks
Peter, 200,1000
Sam, 500,1000
Greg, 1000
Abharam, 800,1000
Henry, 600,1000
Peter, 400,1000

Example: Default behavior of awk

Print all the lines from a file.
By default, awk prints all lines of a file, so to print every line of above created file,
use below command:
~ $ awk '{print}' awk_file

Name, Marks, Max_Marks

Peter, 200,1000
Sam, 500,1000
Greg, 1000
Abharam, 800,1000
Henry, 600,1000
Peter, 400,1000

Example 2: Print only specific field

Print 2nd & 3rd fileds
~$ awk -F”,” {print $2,$3;}' awk_file

Example: Pattern Matching

Print the lines which matches the pattern (lines which contains the word “Henry"
or "Peter”)
~$ awk '/Henry|Peter/' awk_file
Peter,200,1000
Henry,600,1000
Peter,400,1000

Initialization and Final Action

BEGIN and END blocks are helpfull in displaying information before and after
executing actual awk script.

BEGIN block is evaluated before awk starts processing the actual awk script; it’s
an excellent place to initialize the FS (field separator) variable, print a heading, or
initialize other global variables.

BEGIN block Usages:

 Declaring variables.
 Initialization variables for doing increment/decrements operations in main
AWK code.
 Printing Headings/info before actual AWK code output.

END block is evaluated after all the lines in the awk script have been processed.
Typically, the END block is used to perform final calculations or print summaries
that should appear at the end of the output stream.
END block Usages:

 Printing final results, after doing operations in main AWK block.

 Printing Completion/info after actual AWK code output.

awk tool is mainly used for reporting some data in useful manner. Without these
BEGIN and END
blocks the output will be meaningless.
Consider db.txt which contains below data:
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37

awk BEGIN block

This is a block of code which is executed before executing actual awk script.
BEGIN block Syntax
               awk ‘BEGIN{awk initializing code}{actual AWK code}’ filename.txt
Example: Print a meaning full info before actual AWK output.
~$ awk ‘BEGIN{print “########################\nThis is the output of
filtered
data\n########################”}{print $0}’ db.txt

Output:
##########################
This is the output of filtered data
##########################
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47

awk END block

This is the block which is executed after executing all the awk code.
Example:

Print some meaning full info after processing awk code.

~$ awk ‘{print $0} END {print “#########################\n Completed
printing
filtered data\n########################”}’ db.txt
Output:

Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
#########################
Completed printing filtered data
#########################

Combining BEGIN and END block

Example:

~$ awk ‘BEGIN{print “##########################\n This is the output of

filtered
data\n##########################”}{print $0}END{print
“########################\n Completed printing filtered
data\n########################”}’ db.txt

Output:

#########################
This is the output of filtered data
#########################
Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
########################
Completed printing filtered data

awk inbuilt variables

awk is supplied with good number of built-in variables which comes in handy
when working with data files. We will see usages of awk built-in variables with
one or two examples .  These variable are used to format the output of an awk
command.

List of built-in variables:

FS field separator character (default blank & tab)

OFS output field separator string (default blank)
RS input record separator character (default newline)
ORS output record separator string (default newline)
NF number of fields in input record
NR number of input record
FNR output number of lines
FILENAME name of current input file

Consider below db.txt as sample file.

~$ cat db.txt
John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married

In order to make it simple we can divide above inbuilt variables in to groups on

basis of their operations.
Group1: FS(input field separator), OFS(Output Field Separator)
Group2: RS(Row separator) and ORS(Output Record Separator)
Group3: NR, NF and FNR
Group4: FILENAME variable

FS (Input Field Separator)

This variable is useful in storing the input field separator. By default AWK can
understand only spaces, tabs as input and output separators. But if your file
contains some other character as separator other than these mention one’s, awk
cannot understand them.

For example UNIX password file which contain ‘:’ as a separator. So in order to

mention the input filed separator we use this inbuilt variable. We will see
what issue we face if we don’t mention the field separator for our db.txt.

Example: without using FS

Print first column data from db.txt file.

~$ awk ‘{print $1}’ db.txt

Output:
John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married
OFS (Output Field Separator)
This variable is useful for defining the output field separator for the expected
output data.

Example:

Display only 1st and 4th column and with $ as field  separator for the output .
~$ awk ‘BEGIN{FS=”,”;OFS=” $ “}{print $1,$4}’ db.txt

Output:
John $ IBM
Barbi $ JHH
Mitch $ BofA
Tim $ DELL
Lisa $ SmartDrive
Note: Space is give before and after $ in OFS variable to show better output.

RS (Row separator)
Row Separator is helpful in defining separator between rows in a file. By default
awk takes row separator as new line. We can change this by using RS built-in
variable.

Example:
Convert a sentence to a word per line. We can use RS variable for doing it.
~$ echo “This is how it works” | awk ‘BEGIN{RS=” ”}{print $0}’

Output:
This
is
how
it
Works

ORS (Output Record Separator)

This variable is useful for defining the record separator for the awk command
output. By default ORS is set to new line.

Example:
Print all the company names in single line which are in 4th column.
~$ awk -F’,’ ‘BEGIN{ORS=” “}{print $4}’ db.txt

Output:
IBM JHH BofA DELL SmartDrive
NF
This variable keeps information about total fields in a given row. The final
value of a row can be represented with $NF.

Example: Consider abc.txt which contains below data:

Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47
Print number of fields in each row in abc.txt.
~$ awk ‘{print NF}’ abc.txt

Output:
5
5
4
5
4

NR

This variable keeps the value of present line number. This will come handy when
you want to print line numbers in a file.

Example:
Print line number for each line in a given file.
~$ awk ‘{print NR, $0}’ abc.txt

Output:
1 Jones 2143 78 84 77
2 Gondrol 2321 56 58 45
3 RinRao 2122234 38 37
4 Edwin 253734 87 97 95
5 Dayan 24155 30 47
This can be treated as cat command -n option for displaying line number for a file

FNR
This variable keeps count of number of lines present in a given file/data. This will
come handy when
you want to print no of line present in a given file. This command is equivalent to
wc -l command.

Example:
Print total number of lines in a given file.
~$ awk ‘END{print FNR}’ abc.txt

Output:
5
FILENAME
This variable contain file awk command is processing.

Example:
Print filename for each line in a given file.
~$ awk ‘{print FILENAME, NR, $0}’ abc.txt

Output:
abc.txt 1 Jones 2143 78 84 77
abc.txt 2 Gondrol 2321 56 58 45
abc.txt 3 RinRao 2122234 38 37
abc.txt 4 Edwin 253734 87 97 95
abc.txt 5 Dayan 24155 30 47

awk Built in Function

A function is a self-contained computation that accepts a number of arguments as

input and returns some value. awk has a number of built-in functions in two
groups: arithmetic and string functions.

Arithmetic Functions
Nine of the built-in functions can be classified as arithmetic functions. Most of
them take a numeric argument and return a numeric value. Below table
summarizes these arithmetic functions with some Examples.

awk Function Description

cos ( x ) Returns cosine of x (x is in radians).
exp ( x ) Returns e to the power x.
index (s1,s2) Position of string s2 in s1; returns 0 if not present
int ( x ) Returns truncated value of x.
log ( x ) Returns natural logarithm (base- e) of x.
sin ( x ) Returns sine of x (x is in radians)

sqrt ( x ) Returns square root of x.

atan2 ( y , x ) Returns arctangent of y / x in the range - to .
rand () Returns pseudo-random number r, where 0 <= r < 1.
sqrt(expr) Returns the square root of the expression or value given

Examples:
~$ awk 'BEGIN{
print sqrt(16);
print sqrt(0);
print sqrt(-12);
}'
Output:
4
0
nan
Here nan stands for not a valid number.

String Functions
The built-in string functions are much more significant and interesting than the
numeric functions. Because awk is essentially designed as a string-processing
language, a lot of its power derives from these functions. Below table lists the
string functions found in awk. awk's Built-In String Functions

 
 

Shell Programming

The UNIX shell program interprets user commands, which are either directly
entered by the user, or which can be read from a file called the shell script or shell
program. Shell scripts are interpreted, not compiled. The shell reads commands
from the script line by line and searches for those commands on the system.

Shell script

Shell script is a file-containing list of commands to be executed in a particular

order.

A good shell script will have comments, (a statement preceded by a pound

sign, #)  , describing the purpose of the statement.

In a script we can use conditional tests, such as value A is greater than value B,
loops or iterative statements to execute some steps repetitively or to navigate
through a list of data or records and do processing. We can use files to read and
store data. Can use variables to read and store data. A script may include functions
also.

When a script is executed, the shell reads the commands one by one and executes
them.

We can create the simple shell script file by using vi editor or cat command like,
$ vi test.sh

$ cat> test.sh

Below mentioned shebang statement should be the first statement in the shell

script as it tells the system that the commands mentioned in the shell script are to
be executed by the shell / bin / sh

#! / bin / sh

Consider the shell script with just two commands pwd & ls.

$ cat test.sh

#!/bin/bash

pwd

ls

Importance of shell script

Shell scripts are basically used for automating processes that we repeat at the
prompt .

Following are some activities we can do using shell scripts:

 Automation of repetitive task

 Creating our own power tools/utilities.

 Automating command input or entry.

 Customizing administrative tasks.

 Creating simple applications.

 Automating administration tasks such as adding new users, removing

obsolete users etc

Some Practical examples where shell scripting can be actively used:

 Monitoring your Linux system.

 Data backup and creating snapshots.

  Dumping Oracle or MySQL database for backup.

 Creating email based alert system.

 Find out what processes are eating up your system resources.

 Find out available and free memory.

 Find out all logged in users and what they are doing.

 Find out if all necessary network services are running or not. For example if
web server failed then send an alert to system administrator via a pager or an
email.

 Find out all failed login attemp. If login attempt are continued repeatedly
from same network IP, automatically block all those IPs accessing your
network/service via firewall.

 User administration as per your own security policies.

 Find out information about local or remote servers.

 Configure server such as BIND (DNS server) to add zone entries.

Executing a shell script

Below two methods can be used to execute the shell script.

$ sh filename

Or

$ ./filename

In this case we have to modify the file access permissions of  the shell script before
execution.

To provide execute permission , following command is used.

$ chmod u+x filename

Basic Operators in shell scripting

Below operators are supported by shell.

  Arithmetic Operators.

 Relational Operators.

 Boolean Operators.

 String Operators.

 File Test Operators.

Here is simple example to add two numbers:

Example:

#!/bin/sh

val=`expr 2 + 2`

echo "Total value : $val"

Output:

$ Total value : 4

There are following points to note down:

 There must be spaces between operators and expressions for example 2+2 is
not correct, where as it should be written as 2 + 2.

 Complete expression should be enclosed between ``, called inverted

commas to execute expr command correctly.

 Arithmetic Operators

Assume variable a holds 10 and variable b holds 20 then:

Operator Description Example

+ Addition `expr $a + $b ` will give 30
- Substraction `expr $a - $b ` will give -10
 * Multiplication `expr $a \* $b` will give 200
/ Division `expr $b / $a` will give 2
% Modulus `expr $a % $b` will give 0
!= Not equal [ $a != $b ] will give true
= assignment a=$b will assign value of b to a.
== Equality [ $a == $b ] will return false.

It is very important to note here that all the conditional expressions would be put
inside square braces with one spaces around them, for example [ $a == $b ] is
correct where as [$a==$b] is incorrect.

Relational Operators

Below are relational operators which are specific to numeric values. These
operators would not work for string values unless their value is numeric.

For example, following operators would work to check a relation between 10 and
20 as well as in between "10" and "20" but not in between "ten" and "twenty".

Assume variable a holds 10 and variable b holds 20 then:

Operator Description Example

-eq Check if the values of 2 operands are equal or not, [ $a -eq $b ] is false
if yes then condition becomes true.

-ne Check if the values of 2 operands are equal or not, [ $a -eq $b ] is true
if values are not equal then condition becomes true.
-gt Check if the value of left operand is greater than the [ $a -gt $b ] is false
value of right operand, if yes then condition
becomes true.
-lt Check if the value of left operand is less than the [ $a -lt $b ] is true
value of right operand, if yes then condition
becomes true.
-ge Check if the value of left operand is greater than or [ $a -ge $b ] is false
 equal to the value of right operand, if yes then
condition becomes true.
-le Check if the value of left operand is less than or [ $a -le $b ] is true
equal to the value of right operand, if yes then
condition becomes true.

It is very important to note here that all the conditional expressions would be put
inside square braces with one spaces around them, for example [ $a <= $b ] is
correct where as [$a <= $b] is incorrect.

Boolean Operators

Assume variable a holds 10 and variable b holds 20 then

Operator Description Example

! This is logical negation. This inverts a true [ !false ] is true
condition into false and vice versa.
-o This is logical OR. If one of the operands is [ $a -lt 20 -o $b -gt 100 ] is
true then condition would be true. true
-a This is logical AND. If both the operands [ $a -lt 20 -a $b -gt 100 ] is
are true then condition would be true false.
otherwise it would be false.

String Operators

These are string operators. Assume variable a holds "abc" and variable b holds
"efg" then:

Operator Description Example

-z Check if the given string operand size is [ -z $a ] will return false.
zero. If it is zero length then it returns true.
-n Check if the given string operand size is non- [ -z $a ] will return true.
zero. If it is non-zero length then it returns
true.
= Check if the value of two operands is equal [ $a = $b ] will return false
or not, if yes then condition becomes true.
!= Check if the value of two operands is equal [ $a != $b ] will return true
or not, if the values are not equal then
condition becomes true.
 str Check if the str is not the empty string. If it is [ $a ] will return true
empty then it returns false.
 
File Test Operators:

Assume a variable file holds an existing file name "test" whose size is 100 bytes
and has read, write and execute permission on:

Operator Description Example

-b file Returns true, if file is a block special file [ -b $file ] is false.
-c file Returns true, if file is a character special file [ -b $file ] is false.
-d file Returns true, Check if file is a directory [ -d $file ] is not true.
-f file Returns true, Check if file is an ordinary file [ -f $file ] is true.
or special file
-r file Returns true, Checks if file is readable [ -r $file ] is true.
-w file Returns true, Check if file is writable [ -w $file ] is true.
-x file Returns true, Check if file is execute [ -x $file ] is true.
-s file Returns true, Check if file has size greater [ -s $file ] is true.
than 0
-e file Returns true, Check if file exists [ -e $file ] is true.

Wild Card Characters

Symbol used to replace or represent one or more characters. Wildcards or wild

characters are either asterisk (*), which represent one or more characters or
question mark (?), which represent a single character.

Wild  
card Meaning Examples
/Shorthand
Matches any string or $ ls * will show all files
group of characters. $ ls a* will show all files whose first name is
starting with letter 'a'
*
$ ls *.c will show all files having extension .c
$ ls ut*.c Will show all files having extension .c
but file name must begin with 'ut'.
 ? Matches any single $ ls ? will show all files whose names are 1
character. character long
$ ls fo? will show all files whose names are 3
character long and file name begin with
fo
[...] Matches any one of the $ ls Will show all files beginning with
enclosed characters [abc]* letters a,b,c

Note: [..-..] A pair of characters separated by a minus sign denotes a range.

Example:

$ ls /bin/[a-c]*

Will show all File name beginning with letter a,b or c.

Output:

/bin/arch /bin/awk /bin/bsh /bin/chmod /bin/cp /bin/as /bin/basename /bin/cat

Shell Quoting Mechanism

The Metacharacters

Unix Shell provides various metacharacters which have special meaning while
using them in any Shell Script and causes termination of a word unless quoted.

Example:

? Matches with a single character while listing files in a directory and an * would
match more than one characters.

Here is a list of most of the shell special characters (also called metacharacters):

* ? [ ] ' " \ $ ; & ( ) | ^ < > new-line space tab

A character may be quoted (i.e., made to stand for itself) by preceding it with a \.

Example:

#!/bin/sh echo Hello; Word

This would produce following result.

Hello ./test.sh: line 2: Word: command not found shell returned 127
Now let us try using a quoted character:

#!/bin/sh echo Hello\; Word

This would produce following result:

Hello; Word

The $ sign is one of the metacharacters, so it must be quoted to avoid special

handling by the shell:

#!/bin/sh echo "I have \$1200"

This would produce following result:

I have $1200

Quote Description
Single quote All special characters between these quotes lose their special meaning.

Double quote Most special characters between these quotes lose their special meaning
with these exceptions:

\$

\'

\"

\\
Backslash Any character immediately following the backslash loses its special
meaning.
Back Quote Anything in between back quotes would be treated as a command and
would be executed.

The Single Quotes

Consider an echo command that contains many special shell characters:

echo <-$1500.**>; (update?) [y|n]

Putting a backslash in front of each special character is tedious and makes the line
difficult to read:

echo \<-\$1500.\*\*\>\; \(update\?\) \[y\|n\]

There is an easy way to quote a large group of characters. Put a single quote ( ') at
the beginning and at the end of the string:

echo '<-$1500.**>; (update?) [y|n]'

Any characters within single quotes are quoted just as if a backslash is in front of
each character. So now this echo command displays properly.

If a single quote appears within a string to be output, you should not put the whole
string within single quotes instead you would precede that using a backslash (\) as
follows:

echo 'It\'s Shell Programming

The Double Quotes:

Try to execute the following shell script. This shell script makes use of single
quote:

VAR=ZARA

echo '$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]'

This would produce following result:

$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]

So this is not what you wanted to display. It is obvious that single quotes prevent
variable substitution. If you want to substitute variable values and to make invert
commas work as expected then you would need to put your commands in double
quotes as follows:

VAR=ZARA

echo "$VAR owes <-\$1500.**>; [ as of (`date +%m/%d`) ]"

Now this would produce following result:

ZARA owes <-$1500.**>; [ as of (07/02) ]

Double quotes take away the special meaning of all characters except the
following:

 $ for parameter substitution.

 Backquotes for command substitution.

 \$ to enable literal dollar signs.

 \` to enable literal backquotes.

 \" to enable embedded double quotes.

 \\ to enable embedded backslashes.

 All other \ characters are literal (not special).

Back Quotes: Command Substitution

Putting any Shell command in between back quotes would execute the command

Syntax: var=`command`

Example:

Following would execute date command and produced result would be stored in
DATA variable.

DATE=`date` echo "Current Date: $DATE"

This would produce following result:

Current Date: Thu Jul

Shell Decision Statement

While writing a shell script, there may be situations when you need to adopt one
path out of many available paths. In such cases you need to make use of
conditional statements that allow your program to make correct decisions and
perform right actions.

Unix Shell supports conditional statements, which are used to perform different
actions based on different conditions.

Here we will explain following two decision-making statements

 
  The if...else statement

 The case...esac statement

 If..else statements

We can use “if..else” statement which can be used as decision making statement to
select an option from a given set of options.

Unix Shell supports following forms of if..else statement:

 if...fi statement

 if...else...fi statement

 if...elif...else...fi statement

Syntax:

 if...fi statement

if [condition]

then

     command(s)

fi

 if...else...fi statement

if [ condition(s) ] then

   command(s)

else

   command(s)

fi

 if...elif...else...fi statement

 
if [ condition(s) ]

then

     command(s)

elif [ condition(s) ] # Many elif-then allowed

then

   command(s)

else

   command(s)

fi

test command in if condition

We can use test command as condition of if condition as used in the below script.

$cat if_test.sh

#!/bin/sh

echo “Do you want to quit (Y/y)? ”

read ans

if  test $ans ==’y’ –o $ans==’Y’

then

      exit

else

      echo “ to exit enter N or n”

fi

case...esac Statement
We can use multiple if...elif statements to perform a multiway branch. However,
this is not always the best solution, especially when all of the branches depend on
the value of a single variable.

Unix Shell supports case...esac statement which handles exactly this situation, and
it does so more efficiently than repeated if...elif statements.

The interpreter checks each case against the value of the expression until it finds a
match. If nothing matches, goes with the default condition.

Syntax:

case word in pattern1)

Statement(s) to be executed if pattern1 matches

;;

pattern2)

Statement(s) to be executed if pattern2 matches

;;

pattern3)

Statement(s) to be executed if pattern3 matches

;;

esac

Here string word is compared against every pattern until a match is found and the
statement(s) following the match pattern executes. If shell cannot find any match,
the case statement exits without performing any action. When statement(s) part is
executed. The command ;; indicates program flow should jump to the end of the
entire case statement.

There is no maximum number of patterns, but the minimum is one.

Example:

#!/bin/sh
COURSE=”DB”

case “$COURSE” in

 “Java”) echo “Java is a programming language”

   ;;

 “Perl”)echo “Perl is scripting language”

  ;;

 “DB”)echo “Oracle is a DB”

  ;;

esac

Output:

Oracle is a DB

 Iterative Statements/Loop :

Loops are a powerful programming tool that enables you to execute a set of
commands repeatedly.

 while loop

 for loop

 until loop

while loop

Here condition is evaluated by shell at the beginning of each iteration. If the

resulting value is true, given command(s) are executed. If condition is false then no
command would be executed and program would jump to the next line after done
statement.

Syntax:

while condition
do # executed as long as the condition is true

   command(s)

done

Example:

a=0

while [ $a -lt 3 ]

do

   echo $a

   a=`expr $a + 1`

done

Output:

 until loop

Here condition is evaluated by shell. If the resulting value is false, given

command(s) are executed. If condition is true then no command would be executed
and program would jump to the next line after done statement.

Syntax:

until condition # complement of while

do

# executes the body as long as the condition is false

   command(s)

done

Example:
  a=0

until [ ! $a -lt 3 ] do

echo $a a=`expr $a + 1`

done

Output:

 for loop

For loop operate on lists of items. It repeats a set of commands for every item in a
list.

Syntax:

for var in w1 w2 w3.. wn

do

      command #Statement(s) to be executed for every w in the list

done

Example:

for var in 0 1 2 3 4 5

do

     echo $var

done

Output:

0
1

Example:

  for filename in `ls tempdir`

do

echo “Displaying contents of ${filename}”

cat ${filename}

done

We can use "break" and "continue" commands to control the loop.

Command "break" causes the termination of a loop and “continue” resumes

execution at its top.

String Handling

 String handling with test command:

test str Returns true   if str is not null

test –n str Returns true   if length of str is greater

than zero

test –z str Returns true   if length of str is equal to

zero

 
  String handling with expr command

The expr is quite handy for finding the length of a string and extracting a sub-
string:

Length of the string:

  $ str=”abcdefghijk” ;

$ n=`expr "$str" : ‘.*’` ;

$ echo $n

11

expr gave how many times any character (.*) occurs. This feature is very useful in
validating data entry.

Extracting a sub-string:

  $ str=”abcdefghijk” ;

$ expr “$str” : ‘……\(..\)’

gh

Note that there are 6 dots preceding the sequence \(..\). This advanced regular
expression signifies that the first six characters of the string are to be ignored and
extraction should start from the 7th character. Two dots inside \(..\) suggests that
this extraction is limited to two characters only (backslashes override the usual
interpretation of ‘()’).

Extracting string from 3rd character to end of the string:

  $ str="abcdefghijk"

$ expr "$str" : '..\(.*\)'

cdefghijk

Location of first occurrence of a character “d” inside string:

  $ str=”abcdefghijk” ;

$ expr "$str" : '[^d]*d‘

Location of last occurrence if a character inside string:

Below will give the last occurrence of character 'a' from string str.

  $str=”abc def abc”

$expr "$str" : '[^u]*a'

9.  

 
6. Command Line Arguments
To make a shell script a generalized script, we should avoid hard coding. User
should be able to provide the values required for processing as input while running
the shell script. To facilitate this we have to use command line arguments.

The statement we write in the command prompt starting with the command
followed by list of arguments is the command line argument.

Example:

$ ls dir1 dir2 dir3

 

A set of shell variables are there to represent the command line arguments.

$0 – represents the command name (first word on command line). For above
example “ls”

$1 - the first argument of command (second word/symbol ).For the above example
dir1

$2 – the second argument to the command. For the above example, dir2

In this way we can use up to $9

The command-line arguments $1, $2, $3...$9 are also called positional parameters,
with $0 pointing to the actual command/program/shell script and $1, $2, $3, ...$9
as the arguments to the command.

Consider below sample shell script “test.sh”

$cat test.sh
 

#!/bin/sh

echo “Program name is $0”

echo “First argument is $1”

echo “Number of arguments passed =$#”

echo “The arguments are $@”

Let’s execute the above shell script “test.sh”. Remember to change File access
permission of script to execute

$ sh test.sh arg1 arg2 arg3

Program name is test

First argument is arg1

Second argument is arg2

Number of arguments passed = 3

The arguments are arg1 arg2 arg3

There’s another way to execute the script using ./

$ ./test.sh arg1 arg2 arg3

Program name is test

First argument is arg1

Second argument is arg2

Number of arguments passed = 3

The arguments are arg1 arg2 arg3

User Input: read

To get input from the keyboard, you use the read command. The read command
takes input from the keyboard and assigns it to a variable.

Example: read.sh

#!/bin/sh

echo –n “Enter some text > “

read text

echo “You entered: $text”

Note that “-n” given to the echo command causes it to keep the cursor on the same
line; i.e., it does not output a carriage return at the end of the prompt.

Next, we invoke the read command with “text” as its argument. What this does is
wait for the user ro type something followed by a carriage return (the Enter key)
and then assign whatever was typed to the variable text.

Execution:

$sh read.sh

Enter some text > This is some text

You entered: This is some text

Validation of Command Line Arguments

Let us assume, we have one shell script which requires exactly 2 arguments to
execute. Shell script should throw proper error message in case user has not given
exactly two arguments.

Let us have a small shell script "test.sh" to show how to implement validations.

#!/bin/sh

if [ $# -ne 2 ]

then
echo " Not sufficient arguments, please give exact 2 arguments"

else

echo "Continue with our task" fi

$ ./test.sh

Not sufficient arguments, please give exact 2 arguments

$ sh test.sh abc

Not sufficient arguments, please give exact 2 arguments

$ ./test.sh abc def ghi

Not sufficient arguments, please give exact 2 arguments

$ ./test.sh abc def 2

Not sufficient arguments, please give exact 2 arguments

$ ./test.sh abc def

Continue with our task

For scripts where input expected is a file , validations can be incorporated for file
tests like whether the file given as input exists or not, whether it is empty or not,
whether it is readable or not and so on as per requirements.

Environment Variables:

An important concept in Unix is the  environment , which is defined by values

stored in a set of predefined variables called environment variables.

Values of some of these variables are set by the system, some can be set by the
user, some of them by the shell, or any program that loads another program.

When a user logs in to the system, the shell undergoes a twostep process
called  initialization  to set up the environment. This process involves the shell
reading two files - / etc / profile and .profile in the home directory of the user.
The file  / etc / profile  is maintained by the system administrator of the Unix
machine and contains shell initialization information required by all users on a
system.

The file  .profile  is under the control of the user. User can add as much shell
customization information as he / she wants to this file. The minimum set of
information that a user need to configure includes -

 The type of terminal user is using.

 A list of directories in which to locate the commands.
 A list of variables affecting the look and feel of the terminal.

Environment around the UNIX Process:

When a process is created, UNIX opens 3 streams stdin / stdout / stderr for basic
communication with respect to the process control terminal. In addition, it knows
the current working directory for performing file I / O.

Each login shell maintains a description of the environment as a table of pointers to

strings. 

A global shell environment pointer called environ is maintained by UNIX kernel

and it can be used by a process to access its own table 

The shell does not directly use this table, but it creates a child process and calls
exec() system call to execute a command program that uses the table inherited
from the shell parent. Children inherit the entire execution environment from the
parent.

Some examples of  the environment variables are the USER, LOGNAME, HOME,
PATH, PS1, PS2, TERM MAIL, etc.
 
The HOME Variable
It specifies an associated directory with every user in a UNIX system. If the
HOME variable for the user Sita contains /usr/sita/stores, every time Sita logs in,
she is taken to the directory stores.
The variable HOME is referenced the same way:
$ echo ${HOME}
 
The PATH Variable
contains a list of all full path-names (separated by a colon) of directories that are to
be searched for an executable program. For example, the command
$PATH=.:/usr/bin:/bin specifies directories to be searched for any executable file
or a command file (current directory, /usr/bin and /bin, in that order).
 
The PS1 Variable
The system prompt may be changed by setting the value of this variable to the
desired prompt:
$ PS1=“Hello>”
Hello>   #can be changed only at the UNIX command line, not within a shell
script.
 
The PS2 Variable: prompt string for continued command line (default ‘> ‘).
 
The LOGNAME Variable
contains user’s login name. Its contents cannot be changed by the user, but can be
displayed:
echo “${LOGNAME}”
 
The TERM Variable
Names the kind of terminal you are using; setting it helps to manage your screen
more effectively, say,
$ TERM=vt100
 
The PWD Variable
The current working directory can be displayed:
echo “${PWD}”
In fact the whole environment table can be displayed.
 
$IFS: String of characters which are used as word separators in command line
( space, tab, newline chars).
 
The MAIL Variable
Names the standard file where your mail is kept

The .profile File

Some of above variables like HOME and LOGNAME are set automatically each
time the user logs in. The others, however, have to be set. The .profile is used for
this purpose as it is executed as soon the user logs in. A sample .profile  file would
look like:
PATH=.:/bin:/usr/bin
export HOME PATH PS1 MAIL

The UNIX Process:

A process in UNIX is a program in execution with definite life-time and well-

defined hierarchy.

 The context of a process is a snapshot of its current run-time environment

that consists of the current state of the processor registers and 
 User program image - Process execution on user data structure that
represents an application and is manipulated by its own functions (user
mode execution).
 System image - Process execution on system's data structures that represent
the resources (memory, files, devices) and supported by the kernel
routines. Depending upon resource requirement and availability, process's
states are controlled by executing the kernel routines accessed through
system calls (system mode execution).
 The kernel maintains a process table to manage all processes. The two data
structures per process are the user structure and the process structure.
 The kernel process is the first (root) process that comes into existence when
the system is booted. Its process_id and group_id are both 0. 
 In establishing a multi-programming environment, the kernel process creates
the init process with its process_id of 1 and group_id, 0, showing that
process 1 is part of process 0. The init process creates and manages terminal
processes for active terminals in the system.
 At the time of creation, each terminal process belongs to process group 0
and is made to execute a program called getty. Once each terminal process
(now called a getty process) establishes communication with the associated
terminal, it displays a login message and waits for a user to input a user
name (Process group of the init process).
 When a user starts a dialogue, the getty process receives the user name and
leaves the task of validating the user's password to another program called
login. The same terminal process is now called a login process.
 The login process, after validating the user's password, calls a command line
program, the login shell to run in the same process. The same terminal
process is now called a shell process.
  Each shell process now establishes a new process group and becomes ready
to process the user commands. A shell process is the initiating process by
which each terminal maintains the user session. 
 While interpreting a command, the shell creates an execution thread and
then assigns the requested command program to this new process. 
 Both the shell and the new process proceed independently in separate
execution threads. The parent shell process normally waits until child
process completes its execution. 

A process require certain resources to accomplish its intended tasks. They are

>> CPU time
>> System memory
>> Disk Files
>> I / O devices.

Process control blocks (PCB)

The OS must know all the information about a specific process in order to manage
and control it. The OS maintains a table (an array of structures), called the process
table, with one entry per process. These entries are called process control blocks
(PCB) - also known as task control block.This entry contains information about the
process' state, its program counter, stack pointer,memory allocation, the status of
its open files, its accounting and scheduling information, and everything else about
the process that must be saved when the process is switched from ready to running
or blocked state so that it can be restarted later with the status it left off.

Process state:- The state may be new, ready, running, waiting, halted, and so on.
Program counter:- The counter indicates the address of the next instruction to be
executed for this process.
CPU registers:- The registers vary in number and type, depending on the
computer
architecture. They include accumulators, index registers, stack pointers, and
general-purpose registers, plus any condition-code information.
CPU-scheduling information:- This information includes a process priority,
pointers to scheduling queues, and any other scheduling parameters.
Memory-management information:- This information may include such
information as the value of the base and limit registers, the page tables, or the
segment tables, depending on the memory system used by the OS.
Accounting information:- This information includes the amount of CPU and real
time used, time limits, account numbers, job or process numbers, and so on.
I/O status information:- This information includes the list of I/O devices
allocated to the process, a list of open files, and so on.

Process state: 
A process may present in one of the following state.
New:- The process is being created or just created.
Ready:- The process is waiting to be assigned/allocated to a processor time.
Running:- Instructions of the process are being executed by processor.
Waiting:- The process is waiting for some other event/process to execute.
Terminated:- The process has finished its execution.

ps command:
The ps (i.e., process status) command is used to provide information about the
currently running processes in the system.

When ps is used without any option four columns of information labeled PID,TTY,
TIME and CMD for at least two processes, the shell and ps will be visible.

PID :- The process are identified by a 5 digit number known as PID (Process
Identification Number).
TTY :- Is the name of the console or terminal that the user logged into (Stands for
terminal type now
but originally stood for teletype).
TIME :- Is the amount of CPU time in minutes and seconds that the process has
been running.
CMD :- is the name of the command that launched the process.

The information that ps -aux command provides about each process are :

  The user of the process,

  PID of the process,
  Percentage of CPU used by the process,
  Percentage of memory used by the process,
  VSZ (virtual size in kilobytes),
  RSS (real memory size or resident set size in 1024 byte units),
  STAT (the process state code, explained later),
  Starting time of the process,
  Length of time the process has been active
  The command that initiated the process.

Types of process:

Parent and Child Process

A process can initiate a sub process, which is a called a child process, the initiating
process is referred to as its parent.The child processes, in turn create other child
processes forming a tree of processes
( which can be displayed using ps command with –forest option)

Orphan Process
When a child process is killed, parent process gets the notification via a signal.
Parent then, can continue other task. However if the parent process is killed before,
its child, is called an orphan process.

Zombie Process
When a process finished its execution and exit status not received by the parent ( or
parent did not read the exit status till now), the process state becomes zombie.
The process is dead (not to be scheduled for further execution) but cannot
becompletely removed from process table, until it has been determined that exit
status is no longer needed.

Daemon Process
Some programs are not designed to be run with continuous user input and
disconnect from the terminal when task completed. For example, a web server
responds to web requests, rather than user input. Mail servers are another example
of this type of application. These types of programs are known as daemons.
Starting a Process:

Process can be started in two ways:

 In Foreground: By default every process starts in foreground, ie. Gets the

input from keyboard and sends the output in monitor. But in this case till the
process completes its execution no other process can be started in
foreground.

 In Background: To take the advantage multiprocessing environment, a

process can be started in background, so that other process can be started in
the foreground without waiting for the previous process to complete
execution.

        A process can be started in background by adding ampersand(&) after it.

        [390119 @ INGNRILPORCL] $ $ ls | wc -l > file1 &

Switching process from foreground to background:

A process running in foreground can be send to background using the following

steps:

  Press + Z to suspend the job

  bg command puts the job in background
  nohup unattaches the job from the terminal*

$ cat > file

File content...
+z
[1] + Stopped
$ bg
[1] + cat > file &

Switching process from background to foreground

Process running in the background can be taken into foreground using the
following steps:
>> Find the job id of the process by the command jobs
>> Use fg %to get the job to foreground
Stopping/Killing a Process

A process dies(terminates ) automatically when it completes the job it intended to.

A process can be killed abnormally with the command kill.

 Use the ps command to find out the process-id for the process
 Use the command kill to terminate it
 Use the command kill -9 to kill a process forcefully.

Option settings for background process

While a process is executing/running , if the owner tries to log off the process
,process will get killed. Sometimes a job or command takes a long time to
complete and it is required the job to be completed without interruption. This
situation can be handled in two different ways:
  Does not allow the use to log off
 Continue execution in background even after the user logged off

This can be achieved using command nohup

PERIODICAL EXECUTION OF JOBS

Sometimes it may be required to execute certain Job or task on specific time. It

may be not possible for any user to start or execute the job physically on that
particular time due to other business requirements. Again if similar task need to be
executed periodically on a regular interval, then the task becomes very hectic. Unix
internally has solutions to handle this type of situations using utilities such as cron
and at.

 Crontab
The word 'crontab' in UNIX stand for chron (chronograph) table, or time table for
Unix system.
The crontab is a special table where it is possible to specify commands and time or
interval to execute the command once or repeatedly. Commands can be any
executable programs, for example, a script can be written to take regular backup of
all the required tables in database. A crontab instruction can be created to execute
the script on the specified time. The cron daemon reads the table and executes the
commands at the times specified.

Crontab Command Syntax

Syntax
 

Components of Crontab
The basic format of a crontab schedule consists of 6 fields, placed on a single line
and separated by spaces, formatted as follows:
The various fields and their utility are explained in the image below:
Example:

To executes an incremental backup shell script "takeBackup.sh" available in your

home directory at 11:00 on every day the cron entry will be as below:
00 11 * * * ~ / takeBackup.sh

                      To schedule the same job for every minute, the cron entry will be as
follows:

                  * * * * * ~ / takeBackup.sh