Grep :

- Used to find words in a file.

- You can analyze large sets of log files.
- Grep [options] pattern [list of files]

- Match Selection Options

o –E
o –F
o –G
o –P

- Match Control Options

o –e
o –f
o –i
o –v
o –w
o –x
o –y

- General Output Control

o –c
o --color
o –L
o –l
o –m
o –o
o –q
o –s

- Output line Prefix Control

o –b
o –H
o –h
o –n
o –T
o –u
o –z

- Context Line Control

o –A NUM
o –B NUM
o –C NUM

- File & Directory Selection

o –a
o –binary-files
o –D ACTION
o –d ACTION
o --exclude=GLOB
o --exclude-from=FILE
o --exclude-dir=DIR
o –I
o –include=GLOB
o –r
 o –R

- Repetition
o . : match single character
o ? : zero or one
o * : zero or more
o + : one or more times
o {n} : match exactly n times
o {n,} : match atleast n times
o {n,m} : match atleast n but more than m times

- Meta-characters
o ^ : Match Empty string at the beginning
o $ : Match String at the end of line

1. Run last executed grep command

 !grep

2. Search for specified string in a specified file

 grep “Error” logfile.txt

3. Search for string in multiple files

 grep “Error” file1 file2

4. Search string case insensitively

 grep -i “Unix” file.txt

5. Search exact matching word in a file

 grep -w “The” test.txt

6. Displays matching lines along with two lines before match line
grep –B 2 “Test” file.txt

7. Displays matching lines along with three lines after match line
grep -A 3 “Test” file.txt

8. Displays matching lines along with 5 lines before & after matched lines
grep -C 5 “Test” file.txt

9. Search for a particular string recursively

grep –r “Test” *

10. Display lines that do not have string “Test”

grep -v “Test” file.txt

11. Displaying the non-empty lines

grep -v “^$” file.txt

12. Give the count of number of lines that matches the given string/pattern
grep -c “string” file.txt

13. Display the filename that matches the string/pattern

grep -l “string” *
 14. Display the files which do not contain the matching string/pattern
grep -L “string” *

15. Display entire line which has matched string

grep -o “string” file.txt

16. Display the position of the line which contains the matched string in a file using –n option
grep -n “string” file.txt

17. Display the character position of the matched string in a file

grep -o -b “string” file.txt

18. List all files in the default directory that others can read or write
Ls –l |grep ‘.\{7\}rw’

19. Find lines a file where “sams” occurs twice

grep (sam) .*\1

20. Provide count of how many lines were matched with string “hope”
grep -cw “hope” file.txt

21. Search a string in gzipped file

zgrep -I “Error” /var/log/syslog.2.gz

 Find : Search and locate list of files & directories based on conditions you specify.
1. Find files using Names in current directory
find . –name test.txt

2. Find files under home directory

find /home –name test.txt

3. Find files using Name & Ignore Case

find /home –iname test.txt

4. Find directories using name

find / type d -name Test

5. Find PHP files using name

find . type f -name Test.php

6. Find all PHP files in directory

find . –type f -name “*.php”

7. Find files with 777 permission

find . type f -perm 0777 -print

8. Find files without 777 permission

find . type f ! -perm 777

9. Find SGID files with 777 permission

find / -perm 2644
10. Find sticky bit files with 777 permission
find / -perm 1777

11. Find SUID files

find / -perm /u=s

12. Find SGID files

find / -perm /g=s

13. Find Read Only files

find / -perm /u=r

14. Find Executable files

find / -perm a=x

15. Find files with 777 permission and chmod to 644

find / -type d –perm 0777 -print -exec chmod 644 {} \

16. Find and remove single file

find . -type f -name “*.txt” -exec rm -f {} \

17. Find all Empty files

find /tmp -type f -empty

18. Find all Hidden files

find /tmp -type f –name “.*”

19. Find single file based on user

find / -user root –name test.txt

20. Find all files based on user

find /home -user test

21. Find all files based on group

find /home -group developer

22. Find particular files of user

Find /home -user test -iname “*.txt”

23. Find the files modified N days back

find / -mtime 50

24. Find files accessed in last 50 days

find / -atime 50

25. Find files modified in range of days between 50 to 100 days

find / -mtime +50 -mtime -100

26. Find files changed in last N minutes

find /home/bob -cmin -60
 27. Files modified in last hour
find / -mmin -60

28. Find accessed files in last 1 hour

find / -amin -60

29. Find files of given size

find / -size 50M

30. Find files in a size range

find / -size +50M -size -100M

31. Find largest and smallest files

find . type f -exec ls -s {} \; | sort –n –r | head -5
find . type f -exec ls -s {} \; | sort –n | head -5

32. Find empty files & directories

find /tmp -type f -empty -- for files
find /tmp -type d -empty -- for directories

33. Delete matching files and directories

find /tmp -type f -name “*.txt” -exec rm -f {} \;
find /home/bob/dir -type f -name *.log -size +10M -exec rm -f {} \;

 Sed
o Print only 2nd line
sed -n ‘2 p’ employee.txt

o Print from line 1 through line 4

sed -n ‘1,4 p’ employee.txt

o Print from line 2 through the last line ($ represents the last line)
sed -n ‘2, $ p’ employee.txt

o Print only odd number lines

sed -n ‘1~2 p’ employee.txt

o Print lines matching the pattern “Jane”

sed -n ‘/Jane/p’ employee.txt

o Print lines starting from the 1st match of “jason” until the 4th line
o Print lines starting from 1st match of “Raj” until the last line
o Print lines starting from line matching “Raj” until the line matching “Jane”
o Print the lines matching “Jason” and 2 lines immediately after that
o Delete only the 2nd line
o Delete from line 1 through 4
o Delete from line 2 through the last line
o Delete only odd number of lines
o Delete lines matching pattern “Manager”
o Delete lines starting from the 1st match of “Jason” until the 4th line
o Delete lines starting from the 1st match of “Raj” until the last line
o Delete lines starting from the 1st match of “Raj” until the line matching “Jane”
 o Delete lines starting from the line matching Jason and 2 lines immediately after that.
o Delete all empty lines from a file
o Delete all comment lines (assuming that comment starts with #)
o Write only 2nd line
o Write lines 1 through 4
o Write from line 2 through the last line
o Write lines matching pattern “Manager”
o Write lines starting from the 1st match of “Jason” until the 4th line
o Write lines starting from the 1st match of “Raj” until the last line
o Write lines starting from the 1st match of “Raj” until the line matching “Jane”
o Write lines starting from the line matching Jason and 2 lines immediately after that.
o Write all empty lines from a file
o Write all comment lines (assuming that comment starts with #)

 Sort
o Sort file in alphabetical order : sort file
o Sort months in a file : sort –M file
o Sort 1 or 2 string a file : sort –o ….incomplete
o Sort in human readable format : sort –h file
o Sort remove duplicates : sort –u file
o Sort a file numerically : sort –n file
o Sort file numerically in reverse order : sort –nr file
o Sort multiple files : sort –n file1 file2
o Sort, merge and remove duplicates : sort –nu file1 file2
o Sort file basis of 1st field : sort –t “,” –k1,1 file
o Sort file basis of 2nd field : sort –t “,” –k2,2 file
o Sort file alphabetically 1st field & numerically
2nd field in reverse order : sort –t “,” –k1,1 -k2nr,2 file

 tr
o Convert lowercase to uppercase : tr [:lower:] [:upper:]
o Translate braces and output to new file : tr ‘{}’ ‘()’ input-file output-file
o Translate whitespaces to tab : tr [:space:] ‘\t’
o Replace non-matching characters : tr –c “u” “a” (Replace all characters with “a” except “u”)
o Delete non-printable characters : tr –cd “[:print:]” < filename
o Squeeze one or more occurrence of continuous
Characters : tr –s “”
o Replace space character with any other character: tr –s”” “,” (replace space with comma)

 Cut [Note: cut does not count tab space as column ‘f’ but character ‘c’. Hence, it gets calculated in character.]
o Display 3rd field of each line : cut –f 3 filename
o Display second to fourth field of line : cut –f 2-4 filename
o Display specific fields of file : cut –f 1-2, 4-5 filename
o Display fields after 3 column
rd
: cut –f 3- filename
o Display characters from 3rd character to 12th : cut –c 3-12 filename
o Display first field of file that has “:” delimiter : cut –f 1 –d ‘:’ /etc/passwd
o Display 1 to3 field of a delimiter file : cut –f 1,3 –d ‘:’ /etc/passwd
o Change delimiter from “;” to space : cut –f 1,3, -d ‘:’ –output-delimiter=’ ’ /etc/passwd
o Do not display column 6 of file : cut –d ‘,’ –complement –f 6 filename
 o Select fields that contains “|” delimiter only : cut –d ‘|’ –f1-6

 Diff
o Compare two files : diff file1 file2
o How to show 2 files are different with 1 message: diff –q file1 file2
o How to show message two files are same : diff –s file1 file2
o How to produce difference side by side : diff –y file1 file2
o Restricting column widths : diff –width=5 file1 file2
o Ignore case difference while comparing two files: diff –i file1 file2
o Ignore trailing white space at end of line : diff –Z file1 file2
o Ignore all white space differences : diff –w file1 file2
o Ignore blank lines comparing two files : diff –B file1 file2

 Unique
o Print only unique lines : uniq filename
o Count number of duplicates in a line : uniq -c filename
o Print only duplicate lines : uniq -d filename
o Print only unique lines : uniq -u filename
o Compare to ‘N’ characters & print all duplicates : uniq -c -w 8 filename (compare first 8 character)
o Avoid comparing ‘N’ characters and print all duplicates: uniq -D -s 2 filename (skip 2 characters)
o Avoid comparing first ‘N’ fields : uniq -D -f 2 filename (first 2 fields)

Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Explicit print command, which is specified by using 'p' character : sed 'p' BSD
Sed suppresses the automatic printing by using -n option else all lines are printed twice : sed -n 'p' BSD
To print first line only : sed -n '1p' BSD
To print 1 through 5 lines only : sed -n '1, 5p' BSD
It prints every other line starting after line 1 : sed -n '1~2 p' BSD
Printing lines which does not contain 'X" : sed -n '/X/!p' file
Print lines which end with x or X : sed -n '/[xX]$/p'
Print lines beginning with A or L : sed -n '/^A\|^L/p' file
Print every alternate line : sed 'n;d' file
Print every 2 lines : sed 'n;n;N;d' file
Delete the first line : sed -i '1d' file
Delete a particular line i.e. 3rd line : sed '3d' file
Delete the last line : sed '$d' file
Delete a range of lines: from 2nd line till 4th line : sed '2,4' file
Delete lines other than specified range: other than 2nd till 4th : sed '2,4!d' file
Delete the first & last line of a file : sed '1d,$d'file
Delete all lines beginning with a particular character: L in this case : sed '/^L/d' file
Delete all lines ending with a particular character : sed '/x$/d' file
Delete all lines ending with either x or X : sed '/[xX]$/d' file
Delete all blank lines in the file : sed '/^$/d' file
Remove 1st field or column : sed 's/[^,]*,//' file
Print only last field : sed 's/.*,//' file
Print only 1st field : sed 's/,.*//' file
Print only the 2nd field : sed 's/[^,]*,\([^,]*\).*/\1/' file
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
awk -Fs 'pattern {action}' input-file > output-file
-F : is the field separator
/pattern/ : will process only those records from input-file that match the given pattern
{action} : awk programming commands

<<>> Awk internal variables & expanded format

- FS : Field Separator that denotes each field in record.
- NF : Number of Fields in current record
- NR : Number of current record
- OFS : Field separator for outputted data
- ORS : Record separator for outputted data
- RS : Record separator
- FNR : References the number of current record.
- BEGIN, body, END

 Print complete file : awk '{print}' employee.txt

: awk '{print $2}' employee.txt (is same as above).
 Print only field (column) 2 of every records in the file: awk '{print $2}' employee.txt
 Print employee name whose employee id is '102' : awk -F ',' '/^102/ {print "emp id 102 is", $2}' \ employee.txt
 Print records with , after 1st column, : after 2nd column
, % after 3rd field (Input Field) : awk 'BEING {FS="[,:%]"}' {print $2, $3} \empolyee-multiple.txt
awk - F ',' '(print $2, $3)'employee.txt
 Print : at outfield separator : awk -F ',' '{print $2, ":", $3}' employee.txt
awk -F ',' 'BEGIN {OFS=":"} \ {print $2, $3}' employee.txt
 When we don't separate values with a comma in print
statement awk will not use the OFS : awk 'BEGIN {print "test1" "test2"}' OUTPUT: test1test2
 Separate records with a delimiter '-' : awk -F, 'BEGIN {RS="-"} \ {print $2}' employee.txt
 Use Field separator new line, record separator - and
OFS as ":" : awk 'BEGIN {FS="\n"; RS="\n"; OFS =":"} \ {print $2, $3}'
employee.txt
 Print number of records : awk 'BEGIN {FS=","} \ {print "Record Number is:,NR"}'
 Filename : awk '{print filename}'
FNR file number record will give you number within its current file. New file will count record based on row.
- AWK operators:
- Unary operators
- Arithmetic Operators
- String operator
- Assignment Operators: =, +=, -=, *=, /=, %=
- Comparison Operators: >, <, >=, <=, ==, !=, &&, ||
- Regular Expression Op: ~ (Match), !~ (No match)
Print lines where field two is "Tennis" : awk -F "," '$2 == "Tennis"' items.txt
Print lines where field two contains "Tennis" : awk -F "," '$2 ~ "Tennis"' items.txt
- Conditions in awk
- If Else : if (conditional-exp)
action1
else
action2
- While : While (condition)
actions
- Do-While : do
actions
While (condition)
- For loop : for (Initialization; condition; Increment/decrement)
actions
- break & continue, Exit

- Assigning array elements : arrayname [string] = value e.g. item[100] = "TV" , item[101] = "Fridge"
- Browse array using "for" loop : for (var in arrayname)
- Sort array values using sort : asort(item)
- Print format specifiers :
-s : string
-c : single character
-d : decimal
-e : exponential floating point
-f : fixed floating point
-g : use either e or f depending on which is smaller for given input
-o : octal
-x : hexadecimal
-% : prints the percentage symbol
- Built-in Function
- int(n)
- index
- length
- split (input-string, output-array, separator)
- substr (input-string, location, length)
- match (input-string, search-string)
- ignorecase =1/0
- Getline to a variable : awk -F "," '{getline tmp; print $0}' items.txt