5.SPECIAL TOOLS AND UTILITIES: 5.1,5.2 : Standard files and IO redirection.

y y y y y y y y y y y y y y y y y y y y y

y y y y y y y y y y y y y y

In all OS, there is a std input device and a standard output device. Unix also has keyboard as std input device, display screen as std output device and standard error device, display screen. Unix commands or programs get their input from std input device, send their output to std output device. any error messages are also send to std error device. Unix provide special files to instruct all programs to take input , output from std devices. 3 streams, standard input, standard output and standard error are denoted by no:s0,1,2. (FILE DESCRIPTOR) Stream Device Value Std input Keyboard 0 Std output Terminal Screen 1 Std error Terminal Screen 2 Unix allows you to change the standard input, output and error devices by using what is known as redirection and piping. Eg, redirecting input to a file, output to a printer. Unix provides redirection symbols for the purpose. > redirection of output , < redirection of input. > sends output of a command to file or a printer rather than screen. < takes input from a file rather than keyboard >> adds output of a command to the end of a file without deleting the information already present there. Eg: $ cat file1 Displays the contents on the terminal screen. Eg: $ cat file1 > file2 On executing this, we will come directly to $ prompt. > declares file2 to be the std output device, not the keyboard. If file2 doesn t exist, it is created. If ir does exist, it is wiped clear, filled with new contents. This change in std output is temporary, when the command ends, terminal once again becomes the standard output device. $ cat Hello World Ctrl D Hello World Here, we typed cat and hit the enter key, didn t give the filename.hence what we typed on terminal was the std input, it was displayed on terminal. $ cat < file1 will display contents of file1, file1 being the std input. $ cat file1 also gives the same output. Here, file1 is not the std input, but internal programming of cat caused file1 to be opened instead of std. input. Sometimes we may use both redirection operators at once. Eg : $ cat < currentfile > newfile. Input is to be taken from currentfile, output is to be routed to newfile. Order doesn t matter. $who >> logfile This command appends the current list of users who have logged in to the end of the file logfile. $ cat file1 file2 >> file3

5.3 Piping:
o o o o o o

Sometimes, a single Unix command may not be sufficient to solve a problem. Thats when we join commands together. Chief tools for this are redirection and piping. Redirection helps us to connect commands to files. Piping helps to connect commands to commands. To send the output of one command as input for another, the two commands must be joined by a pipe character (|) Filter is a program or command which can receive a flow of data from standard input, process or filter it, send the result to the standard output. Eg: cat, pg, more, head, tail, grep, sort, wc, uniq, cut etc. o Eg: $ ls | wc l o Here, the output of ls becomes input to wc , which promptly counts the no:of lines. o If we want to store the count value in a file, o $ ls | wc l > countfile. (Piping and redirection used together)
o o

o o

$ who | sort Here, instead of displaying the output of who on the screen , it is piped to sort.Sort displays output on the screen . Because , sort being a filter, unless otherwise mentioned, it sends the output to the screen. We can redirect this sorted output to a file. $ who | sort > sortedlist.

o o o o o o o o o o o o o o o

o Check whether sailesh is there in the list of users logged in. o $ who | grep sailesh o Check whether file1 is there in the list of files. o $ ls | grep file1 o Sort the list of all files in the reverse order. o $ who | sort r o View the first 10 lines of 2011 calender. o $ cal 2010 | head -10 o Find the word love in the file, poetry o $ cat poetry | grep love Above examples show the flow of data through pipe from one command to the next. Redirection routes the output to files, Pipes routes the output to other programs. What if we want to do both? Unix offers the tee command to achieve this purpose. Tee reads the input, send it to another command as well as redirects a copy to another file. $ who | tee logfile | sort Here, output of who becomes standard input of tee. Tee now sends onecopy of input to sort through pipeline , and another copy is stored in a file logfile. If we want to store the output of who in 2 files, and also, pipe one copy to sort, $ who | tee logfile newlogfile | sort If we want to store the output of who in 2 files, display the same on the screen , and pipe to sort, $ who | tee logfile newlogfile /dev/pts1 | sort If we want to store the output of who in 2 files, display the same on the screen , and pipe to sort,and store the sorted output in file1, $ who | tee logfile newlogfile /dev/pts1 | sort > file1 /dev/pts1 is the file associated with the terminal. To append output of tee to a file, -a option is used. $ cat file1 file2 | tee a completefile | more We can also redirect the standard error.

o o

$cat myfile > newfile 2> errorfile Here, if myfile exists, its contents are copied to newfile If it doesnot exist, an error message is produced.Instead of displaying the error message on the screen , it is redirected to errorfile. 2> redirects standard error. (Standard output can be redirected as 1> also, instead of >) $ cat errorfile

The facility of piping and redirection provides a lot of flexibility.It is becoz of the fact that Unix treats all devices as files. This allows same command to work equally well with files as well as devices like terminals, harddisc drives etc. Also , Unix commands are designed as filters, take input from keyboard, send output to screen.This makes connecting Unix commands, a very natural process. EXERCISES: 1. Output of who should be displayed on the screen, with the value of total no:of users who have logged in, displayed at the bottom of the list. 2. Output of ls should be displayed on the screen, from this output,lines containing the word poem should be counted, store count in file file1. 3. Contents of file1 and file2 should be displayed on the screen, this output should be appended to the file file3. 4. From the output of ls, lines containing poem should be displayed on the screen along with its count. 5. All files present in the directory dir should be deleted. Any error if occur while carrying out this operation should be stored in the file errorlog. 6. What are the effects of the following commands? a. $ cat < file1 > file2 b. $ wc l < aaa c. $ who | sort d. $ who | wc l > aaa e. $ sort f. $ date > aaa g. $ ls | grep poem h. $ grep poem file | sort | wc l i. $ ls | tee /dev/pts/0 | grep poem | wc l j. $ cal 2011 | head -15 k. $ cat aaa > bbb 2> ccc 5.10 Compressing and Decompressing Files:
 Unix provides utilities such that we can pack the same information in lesser bytes there by saving precious disc space.  Compressed files can be transmitted over long distances at low costs.  Some compression decompression utilities in Unix are: o Gzip and gunzip (.gz) o Bzip2 and bunzip2 (.bz2) o Zip and unzip (.zip)  Degree of compression depends on type of file and size and compression technique used.  Text files can be compressed to a larger extend.

5.10.1 gzip:


Gzip compresses single file into a single file.

   

Compressed file consists of gzip header and deflated data. If a given file is an argument, gzip compresses it, adds a .gz suffix, deletes the original file. With no arguments, gzip compresses std input and writes compressed file into std output. Options: o c : donot delete the original file, writes compressed file to std output. o d : acts like gunzip. o r : recursive compression (directories + files inside will be compressed) o l : display how much compression is achieved.

5.10.2 gunzip: Uncompresses a file compressed with gzip.(recognizes extension and decompresses all formats.) Options: o c : donot delete original file, Writes uncompressed data to stdout o r : recursive decompression. Older compression utilities are:
 Compress (.Z) -v : verbose (reporting how much space it saved)  Uncompress  Once a file is compressed, we cannot view it using the cat command.Unix provides a utility called zcat for this purpose.It displays an uncompressed version of a compressed file to the terminal, without changing the compressed file or storing the uncompressed version in a file.  Pack (.z)  Unpack : get back the original file  To view the contents of a packed file, we can use pcat command.  Compress offers a higher degree of compression compared to pack.  Zcat can also be used to view files compressed using gzip.