Introduction To LINUX UNIX

Introduction to Unix
Elana J. Fertig
Johns Hopkins University
Department of Oncology, Division of Biostatistics and
Bioinformatics
Computational Genomics Workshop

18 November 2011
Unix filesystem
File processing
Redirecting commands and scripts
Additional information
Lesson Overview
1 Unix filesystem
2 File processing
3 Redirecting commands and scripts
4 Additional information
E. J. Fertig Intro Unix

Unix filesystem
File processing
ssh
The “ssh” command enables secure login to a machine to which

you have access.
$ ssh [user]@[host]
You will then be prompted for your password.

Unix filesystem
File processing
The most useful command: man
For help with the structure of a command, use the following:
$ man command
Type the enter key to get a the next line, spacebar to get the
next page, and q to exit.
For example,:
$ man ssh
$ man man
The “-k” option lets you search within the help pages:
$ man -k bash

Unix filesystem
File processing
Less typing through tab, copy, and paste
If you start typing a command, the tab key will autocomplete

it. E.g., $ ma tab
In the Terminal program on Mac, copy and paste from the
edit menu and shortcuts command-c and command-v will
work.
In putty program on Windows and most unix machines, copy
and paste work by simply highlighting the text that you want
to copy and then clicking the middle mouse button.

Unix filesystem
File processing
Re-using past commands
The history command will display a list of the most recent

commands that were executed.
The up and down arrows will allow you to scroll through
commands that were previously executed.
The !! command will repeat the previous command.
!command will repeat the last command that contains the text
command. E.g., $ !his will repeat the previous history
command. The ! can also be used to repeat a command from
the history when referenced by number.
^text1^text2 will repeat the last command, replacing the
first instance of “text1” with “text2”.

Unix filesystem
File processing
Other special characters
* a wildcard for searching

$ the start of a variable name
# starts a comment (a line that will be ignored)

Unix filesystem
File processing
Filesystem exercises
Where am I?
The “pwd” (print working directory) command will get the current
directory in which you are located.
$ pwd

Unix filesystem
File processing
The home directory ($HOME, ∼)
The “pwd” (print working directory) command will get the current
directory in which you are located.
$ pwd
By default, will be your home directory
$ echo $HOME
(where “echo” prints the command to the screen and $ represents

the start of a variable name).
The ∼ symbol is also used to represent the home directory
$ echo ∼

Unix filesystem
File processing
Finding contents of directories (ls)
To see the contents your current path:
$ ls
To see the contents of another directory (e.g., the Data directory):
$ ls Data
The * command serves as a placeholder for any characters. E.g.,
$ ls myFile*
The -l option will display details of all the files listed and -a
option will also display hidden files.

Unix filesystem
File processing
Changing directories (cd)
The following command will move you into the directory Data
$ cd Data
$ pwd
$ cd ∼ # moves you back to your home directory

Unix filesystem
File processing
Direct vs relative referencing
Relative referencing gives the Direct referencing gives the full

path relative to your current name of the path (always will
working directory. start with /).
e.g., Data e.g., /home/user/Data
What would be reasons for using relative referencing? Direct

referencing?

Unix filesystem
File processing
Structure of the filesystem
Folders are separated by the “/” symbol (e.g., )

./ represents the current directory and ∼/ the home
directory. Usually /home/user.
../ represents the “parent” directory (up one in the tree
structure).
The “/” symbol alone represents the base directory of the file
system like the Machintosh HD on Mac or the C:\\ folder on
Windows.
The “executable” program files are usually in folders labeled
with bin (e.g., /usr/bin; /usr/local/bin). The system
will look in the directories in $PATH for any program.

Unix filesystem
File processing
Exercises
1 List the folders contained in the base directory “/”.

2 Move from the home directory to the Data directory. What
files are contained in this directory?
3 Move from the base directory to the Data directory.
4 Determine all possible programs that this system can run.
5 List the files in another user’s home directory.

Unix filesystem
File processing
Exercises solutions
1 List the folders contained in the base directory “/”.

$ ls /
2 Move from the home directory to the Data directory. What
files are contained in this directory?
$ cd Data/
$ ls
3 Move from the base directory to the Data directory.
$ cd /
$ cd ∼/Data
$ pwd

Unix filesystem
File processing
Exercises solutions, cont.
4 Determine all possible programs that this system can run.

$ echo $PATH
$ ls /usr/local/sbin
$ ˆsbinˆbin # a lazy way of typing ls /usr/local/bin
$ ls /usr/sbin
$ ls /usr/bin
etc
5 List the files in another user’s home directory.
$ ls /home
$ ls /home/secret0

Unix filesystem Reading / writing
File processing Interacting with the server
Redirecting commands and scripts Managing file locations
Additional information Security
Viewing text files
head: View the first 10 lines of a file.

tail: View the last 10 lines of a file.
cat: Display the contents of the entire file.
more or less: Display contents of the entire file in a
controlled format.

Examples of viewing files
Text files
$ cd ∼/Data
$ head HG-U133 Plus 2.na32.annot.csv
$ tail HG-U133 Plus 2.na32.annot.csv
$ cat HG-U133 Plus 2.na32.annot.csv
$ more HG-U133 Plus 2.na32.annot.csv
Non-text files
$ more /bin/more

Editing files
Standard unix file editors:

vi: Gives users a lot of control over file editing and searching
abilities, but has a steep learning curve.
emacs: Easier to use, but presence of mouse-clickable menus
depends on graphical link and version of emacs on the server.
pico: One of the easiest editors to use.
Use can also use your favorite text editor on Windows (e.g.,
Notepad) or Mac (e.g., TextEdit) and copy files over to the server.
Be careful that the text editor saves everything as a true
text file. NEVER USE MICROSOFT WORD FOR THIS!!!!

Demo of file editing with pico
$ pico

Copying files onto server
The “scp” (secure copy) is the most standard command to

copying files from one location to another.
Some servers also allow ftp or sftp access.
The easiest way to copy files between a local Windows
machine and a server is using a program like FileZilla
http://filezilla-project.org/.

Copying files onto server (scp)
Generic command:
$ scp [user1]@[host1]:file1 [user2]@[host2]:file2
To copy to the server:
$ scp file [user]@[host]:∼/

What directory will file be copied into on the server?
To copy from the server:
$ scp [user]@[host]:∼/file .
What directory will file be moved to on your machine?

Copying files to server: exercises
1 Create a text file on your machine using your favorite text file.
Save it in a directory that you can find on your machine.
2 Copy that file into your home directory.
3 Copy that file into your Data directory.
4 Copy that file from your home directory back into the
directory on your machine with a new filename.
5 Try to copy that file into the path /home/secret0.
6 Copy the file /home/secret0/mySecretFile into a directory
on your machine.

Copying files locally (cp)
Generic command:
$ cp file1 file2
To create a new copy of a file in the home directory:
$ cp myCopyFile.txt myCopyFile2.txt
To create a copy of a file in the Data directory:
$ cp myCopyFile.txt ∼/Data
BE CAREFUL: the cp command will overwrite file2 with the

contents of file1 if already file2 exists.

Copying files locally (cp)
Generic command:
$ cp file1 file2
BE CAREFUL: the cp command will overwrite file2 with the

Safer to use the “interactive” command:
$ cp -i file1 file2
Try the following two commands.
$ cp myCopyFile.txt myCopyFile2.txt
$ cp -i myCopyFile.txt myCopyFile2.txt

Moving files (mv)
Generic command:
$ mv file1 file2
BE CAREFUL: the mv command will overwrite file2 with the

$ mv -i file1 file2

Deleting files (rm)
Generic command:
$ rm file
BE CAREFUL: rm will not move files to a trash can, so there is

often no recovering files after using the rm command.
$ rm -i file

Creating directories
Creating a new folder:
$ mkdir folder

Managing directories
Creating a new folder:
$ mkdir folder
Copying contents of one folder to another:
$ cp -r folder1 folder2
Moving contents of one folder to another:
$ mv folder1 folder2
Be careful to check when folder2 already exists. Often will create a

copy of folder1 within folder2.

Deleting directories
The following command will irretrievably remove a folder and all of

its contents:
$ rm -r folder
Some versions of unix will not allow you to remove a folder

containing any files. In these cases, use the following command
with great care:
$ rm -rf folder

Exercises
1 Create a new directory within the Data directory from within

that directory.
2 Create another new directory within the Data directory from
within the home directory.
3 Copy a file into each of the subdirectories that you created in
the Data directory.
4 Copy a file from the home directory of the user secret0 into
your home directory.
5 Try to move a file from the home directory of the user
secret0 into your home directory.
6 Try to move a file from your directory into the home directory
of the user secret0.

Hidden files vs private files
Hidden files begin with a . and

will not be displayed using the
default ls command. Seeing
them requires the following
command:
$ ls -a
Hidden files are usually used by

programs to determine default
settings, e.g., .bashrc.

Hidden files vs private files
Hidden files begin with a . and Files have different levels of

will not be displayed using the permission to different users,
default ls command. Seeing which you can see in the first
them requires the following column from the following
command: command:
$ ls -a $ ls -l
Hidden files are usually used by Unless your unix system is

programs to determine default specially encrypted, there is
settings, e.g., .bashrc. no such thing as a truly
private file. Your system
administrator will always have
access to all of your files.
User, group, world
The first column of the ls -l command will return a string of the

type drwxrwxrwx.
d: directory
r: read access
w: write access
x: execute
access
-: no access

User, group, world
The first column of the ls -l command will return a string of the

type drwxrwxrwx.
d: directory User: accessible to yourself (and the

r: read access system administrator).
w: write access Group: accessible to others in your group
(check ls -l /home).
x: execute
access World: accessible to everyone with an
account on this machine.
-: no access

Changing file permissions (chmod)
The chmod can add permissions with the + symbol or remove

permissions with the - symbol.
$ chmod u,g,o,a±r,w,x file
E.g., to the following command will give everyone execute

permissions to file
$ chmod a+x file
The following command will remove read permissions from

non-group members to file
$ chmod o-r file

Changing file permissions (chmod)
The chmod can add permissions with the + symbol or remove

permissions with the - symbol.
$ chmod u,g,o,a±r,w,x file
u: user r: read
g: group w: write
o: other x: execute
a: all

Other chmod options
Even if you remove user read option of a file, if you are the
owner of that file you can always run the chmod command to
restore your write access.
Multiple chmod options can also be specified with a numeric
key. For example, $ chmod 755 file is commonly used to
give the user read, write, execute permissions and both group
and world only read and execute permissions. (see
$ man chmod ).
The chmod command can set all the files in a folder to have
the same permission using the -R option.

Exercises
1 Use the ls -l command to determine which files you can

access and which files you cannot for the user secret0. What
command should that user have to make their files
inaccessible to outsiders?
2 Create a new file with read and write access to other members
in your unixclass group, but no other users. Work with your
neighbor to check that you can both read and write this file
from their account. Log in as one of the users hopkins0-7
(password matches user name) and verify that you cannot
read or write this file from those accounts.
3 Discussion: Why are there different levels of security on unix
machines? Why is it so dangerous for a hacker to obtain the
login information for any user on a machine?

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Writing command output to files
command > file will output the

results of the command to file.
$ ls /usr/bin > binFiles

$ head binFiles
$ tail binFiles

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Writing and appending output to files
command > file will output the command >> file appends the
results of the command to file. output to file.
$ ls /usr/bin > binFiles $ ls /bin >> binFiles

$ head binFiles $ head binFiles
$ tail binFiles $ tail binFiles

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Redirecting output and the cut command
The cut command selects portions of the line of a file (see

$ man cut ).
For example, the following code will select columns 1 and 11 from
the annotation file and output it to a new file.
$ cut -f 1,11 ∼/HG-U133 Plus 2.na32.annot.txt > ∼/HG-U133 Plus 2.Unigene.txt

$ more ∼/HG-U133 Plus 2.Unigene.txt

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Combining commands through the pipe command (|)
command1 | command2 first runs command1 and then runs its

output through command2.
$ ls /usr/bin | head
$ ls /usr/bin | more
$ cut -f11 ∼/HG-U133 Plus 2.na32.annot.txt | grep Hs | sort | head
$ cut -f11 ∼/HG-U133 Plus 2.na32.annot.txt | grep Hs | sort | uniq | head

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Piping and grep
grep string will find “string” within the specified list. In files,
this can be used with piping as follows.
$ ls /usr/bin | grep vi
$ history | grep head
$ cat ∼/Data/HG-U133 Plus 2.na32.annot.csv | grep Notch

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Exercises
1 Create a file that contains all of the commands we applied to

files in the Data directory.
2 Create a file containing an alphabetical list of all the unique
programs in $PATH.

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Why use shell scripts instead of piping commands?

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Why use shell scripts instead of piping commands?
Reproducibility
Facilitates generalization and reuse of commands.

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Creating first shell script

Use pico to create a new script file
$ cd ∼/Scripts
$ pico HelloWorld.sh
Type in the following code

#!/bin/bash # the starting line of every script
# show “hello world” on the screen

echo hello world
To run the script

$ chmod a+x HelloWorld.sh
$ ./HelloWorld.sh

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Creating a script with a command
This script will print the first 10 lines of

HG-U133 Plus 2.na32.annot.txt.
outputFile=∼/Data/HG-U133 Plus 2.na32.annot.txt
# show the first ten lines

head $outputFile

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Creating a script with a command and input arguments
This script will print the first 10 lines of the file specified as
./myScript.sh file.
outputFile=$1
# remind the user of the file they selected

echo $outputFile
# show the first ten lines

head $outputFile

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Command line arguments
#!/bin/bash
$0 name of the script
$# number of arguments echo Running script $0
$@ vector of all the echo $# command line arguments
arguments echo $1 argument one
$1 first argument echo $2 argument two
$2 second argument
...

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Command line arguments
#!/bin/bash #!/bin/bash
echo Running script $0 echo Running script $0

echo $# command line arguments echo $# command line arguments
echo $1 argument one for argVar in “$@”
echo $2 argument two do
echo $argVar
done

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Looping
Loops facilitate repeating a task for some pre-specified criterion

(e.g., a set number of times, over a list of files, etc.)
For loop repeats task over a #!/bin/bash

specified list
for i in list echo Running script $0
do echo $# command line arguments
task(s) for argVar in “$@”
done do
echo $argVar
done

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Another example with loops

This script will output the number of lines of each file in the
specified directory.
#!/bin/bash
inputDirectory=$1
# looks at every element in a listing of inputDirectory

# and puts it into a variable called $afile
for afile in $(ls $inputDirectory)
do
echo The line count for $afile in the directory $inputDirectory is:
# use the full path for wc
wc -l $inputDirectory/$afile
done
Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Conditionals (if, then)
But what if the user forgets to specify an input directory or that

directory doesn’t exist?

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Conditionals (if, then)
This script will check the input directory

#!/bin/bash
inputDirectory=$1
if [ ”$inputDirectory” == ”” ]
then
echo You must give a directory as the first argument
exit
fi
if [ ! -d $inputDirectory ]
then
echo $inputDirectory is not a valid directory
exit
fi

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Structure of conditionals (if,then)
if [ condition ]
then
task(s)
elif [ condition ]
then
task(s)
else
task(s)
fi

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Structure of conditionals (if,then, while)
if [ condition ] While loops will run as long as

then the condition is true.
task(s)
elif [ condition ]
then while [ condition ]
task(s) do
else task(s)
task(s) done
fi Be careful of infinite loops!!!

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Conditionals
String conditionals (True File conditionals (True when. . .)

when. . .) [ -d file ] file is a directory
[ -z s ] s is of length zero [ -e file ] file exists
[ s1 == s2 ] s1 and s2 match [ -r file ] file is readable
[ s1 != s2 ] s1 and s2 do not match
[ s1 < s2 ] s1 is less than [ -w file ] file is writable
s2 alphabetically [ -x file ] file is executable
[ s1 > s2 ] s1 is greater than
s2 alphabetically
Combining expressions
Can also use -eq, -ne, -lt, -le, -gt, [ !EXPR1 ] not
or -ge to compare two binary [ EXPR1 -a EXPR2 ] and
arguments. [ EXPR1 -o EXPR2 ] or

Unix filesystem
Redirecting output
File processing
Piping commands
Shell scripts
Exercises
1 Write a script that goes into all of the folders in your home
directory and copies any executable files into a new directory
on your machine.
2 Write a script to go into a specified user’s home directory and
copies over all their readable files to a specified directory in
your folder.
3 Write a script that will go into as many directories as specified
in command line arguments and copy over all their readable
files. Extend the program to go through these folders
recursively if any of the command line arguments is
“recursive”.

Unix filesystem
File processing Useful commands
Redirecting commands and scripts Resources
diff
diff file1 file2 compares the contents of file1 to the contents of

file2. The command can also be used to compare the contents
of folders.

Unix filesystem
Converting characters
tr string1 string2 string3 replaces all instances of string1 with

string2 inside of string3. For example, we can regenerate the
tab delimited data file by
$ cat ∼/Data/HG-U133 Plus 2.na32.annot.csv | tr ’,’ ’\t’ >

∼/Data/HG-U133 Plus 2.na32.annot.new.txt
$ diff ∼/Data/HG-U133 Plus 2.na32.annot.new.txt ∼/Data/HG-
U133 Plus 2.na32.annot.txt
More controlled modifications to text can be made using the sed
or awk commands (see the additional resources).

Unix filesystem
which
Indicates directory within those listed in $PATH that contains a

program of interest.
$ which pico
Will also indicate if the program cannot be found.
$ which fakeProgram

Unix filesystem
clear
Will erase all commands from your current workspace. But, your
history will not be lost.

Unix filesystem
passwd
Will change your password.

Requires knowledge of your current password.
If you forget your current password, your system administrator
can help you to reset it.

Additional resources
Google is your friend!

http:
//loris.som.jhmi.edu/linux_course/resources.html

Introduction To LINUX UNIX

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To LINUX UNIX

Uploaded by

Copyright:

Available Formats

Introduction to Unix

Computational Genomics Workshop

3 Redirecting commands and scripts

E. J. Fertig Intro Unix

The “ssh” command enables secure login to a machine to which

You will then be prompted for your password.

E. J. Fertig Intro Unix

The most useful command: man

For help with the structure of a command, use the following:

E. J. Fertig Intro Unix

Less typing through tab, copy, and paste

If you start typing a command, the tab key will autocomplete

E. J. Fertig Intro Unix

Re-using past commands

The history command will display a list of the most recent

E. J. Fertig Intro Unix

Other special characters

* a wildcard for searching

E. J. Fertig Intro Unix

E. J. Fertig Intro Unix

The home directory ($HOME, ∼)

By default, will be your home directory

(where “echo” prints the command to the screen and $ represents

E. J. Fertig Intro Unix

Finding contents of directories (ls)

To see the contents your current path:

To see the contents of another directory (e.g., the Data directory):

The * command serves as a placeholder for any characters. E.g.,

E. J. Fertig Intro Unix

Changing directories (cd)

E. J. Fertig Intro Unix

Direct vs relative referencing

Relative referencing gives the Direct referencing gives the full

What would be reasons for using relative referencing? Direct

E. J. Fertig Intro Unix

Structure of the filesystem

Folders are separated by the “/” symbol (e.g., )

E. J. Fertig Intro Unix

1 List the folders contained in the base directory “/”.

E. J. Fertig Intro Unix

1 List the folders contained in the base directory “/”.

E. J. Fertig Intro Unix

Exercises solutions, cont.

4 Determine all possible programs that this system can run.

E. J. Fertig Intro Unix

Viewing text files

head: View the first 10 lines of a file.

E. J. Fertig Intro Unix

Examples of viewing files

E. J. Fertig Intro Unix

Standard unix file editors:

E. J. Fertig Intro Unix

Demo of file editing with pico

E. J. Fertig Intro Unix

Copying files onto server

The “scp” (secure copy) is the most standard command to

E. J. Fertig Intro Unix

Copying files onto server (scp)

$ scp [user1]@[host1]:file1 [user2]@[host2]:file2

To copy to the server:

$ scp file [user]@[host]:∼/

To copy from the server:

E. J. Fertig Intro Unix