Using

Rocks

Administrator: Common functions

1. Rocks Command Line The Rocks command line is the primary administrative interface for a Rocks Cluster. Rocks provides a very rich command set that can be used for system administration, configuration, and reporting. The command line follows a simple grammar to access information about the cluster from the database and use it to set cluster properties, set state of nodes, report state of cluster installation, and generate configuration and system files. The grammar for the rocks command line is
# rocks <verb> <object(s)>. <subject> <arguments> <parameters>

Some examples of the command line are a. List all hosts in a cluster
# rocks list host

b. List all network interfaces of the compute hosts in a cluster

# rocks list host interface compute # rocks list host attr compute-0-0

c. List attributes of a particular host d. Generate dhcp, dns, hosts files for the cluster
# rocks report host # rocks report host dhcpd frontend # rocks report resolv

e. Set a host to re-install on reboot

# rocks list host boot # rocks set host boot compute-0-0 action=install

Get a listing of all command in the system by running

# rocks

without any arguments. Get help for each of the command line utilities using
# rocks <command line> help

Some commands can only be run as root. a. Any command that changes information in the database, can only be run as root. b. Most other commands that list and report contents from the database, can be run as a normal user.

2. Adding Users to a Rocks Cluster Adding users to a Rocks cluster is done as a two-step process. a. Add the user to the Frontend normally, as you would on any other Linux/Unix machine
# useradd <username> # passwd <username> Changing password for user <username>. New UNIX password: Retype new UNIX password: passwd: all authentication tokens updated successfully.

b. Now sync the user to all the other nodes in the cluster This now adds the user to all the machines in the cluster. 3. Adding Software to a Rocks Cluster Adding software to a cluster means being able to access the software on all the machines that are a part of the cluster. There are 2 ways of adding software to a Rocks Cluster. a. We can install the software in a shared location. This means the software will be accessible on all machines of the cluster with the same path. i. /share/apps is a global shared location for the cluster. Just install the software in /export/apps on the frontend. ii. /home/username is a user-specific shared location for the cluster. b. We can use a software package (RPM), and use the Rocks software installation insfrastructure to install software on all machines of the cluster. i. We can download an RPM from a trusted source ii. We can create our own RPM using the Rocks build infrastructure. iii. We can create an RPM using rocks create package
# rocks create package <directory> <name> # ls /root/extra anaconda-ks.cfg install.log install.log.syslog rocks-post.log rocks-post.sh rocks-pre.log rocks-pre.sh # rocks create package /root/extra rocks-root-extra version=2 release=3 some output ... # ls *.rpm rocks-root-extra-2-3.x86_64.rpm # rpm -qlp rocks-root-extra-2-3.x86_64.rpm # rocks sync users

/ /root /root/extra /root/extra/anaconda-ks.cfg /root/extra/install.log /root/extra/install.log.syslog /root/extra/rocks-post.log /root/extra/rocks-post.sh /root/extra/rocks-pre.log /root/extra/rocks-pre.sh # rocks create package /root/extra rocks-root-extra version=2 release=3 prefix=/var/local # ls *.rpm rocks-root-extra-2-3.x86_64.rpm # rpm -qlp rocks-root-extra-2-3.x86_64.rpm / /var /var/local /var/local/extra /var/local/extra/anaconda-ks.cfg /var/local/extra/install.log /var/local/extra/install.log.syslog /var/local/extra/rocks-post.log /var/local/extra/rocks-post.sh /var/local/extra/rocks-pre.log /var/local/extra/rocks-pre.sh

After youve created your RPM file, youll need to perform the following steps to install the RPM on all your compute nodes. a. Place this RPM file in
/export/rocks/install/contrib/5.3/<arch>/RPMS

b. Create the node xml file to reflect new package

# cd /export/rocks/install/site-profiles/5.3/nodes # cp skeleton.xml extend-compute.xml

c. Inside extend-compute.xml file, put the line

<package>rpm name</package>

d. Re-create the rocks distribution

# cd /export/rocks/install # rocks create distro

e. Install the RPM on your compute nodes. There are 2 ways to achieve this. a. Run
# rocks run host compute command=yum install <rpmname>

b. Re-install your compute nodes

# rocks set host boot compute action=install # rocks run host compute command=reboot

User: Common Functions

SGE is the default queuing system provided by the Rocks group. SGE provides utilities to run user jobs on the cluster in a resource-sharing manner. Here are some of the ways to run jobs via SGE. Some of the utilities that SGE provides are a. qsub Submission script. This submits user jobs to the queuing system. b. qstat Reporting script. This reports the status of the queues on the system. c. qdel Deletion script. This deletes a running job from the queuing system. 1. Running a simple job via SGE a. Create a submission script. You can use any shell you want (bash, csh, python, etc.)
# vim date.sh #!/bin/bash #$ -j y #$ -S /bin/bash #$ -cwd date sleep 10 date

b. Submit the script to the queuing system

# qsub date.sh Your job 7 ("date.sh") has been submitted

c. Monitor the queue

# qstat

d. Check the output

# cat date.sh.o7 Sun Aug 1 19:40:39 PDT 2010 Sun Aug 1 19:40:49 PDT 2010

2. Running an MPI Job via SGE - Lets choose MPI-Blast as an example of running an MPI job. We will do this in 4 steps a. Download a FASTA formatted database file from NCBI ftp site. i. Navigate to ftp://ftp.ncbi.nih.gov/blast/db/FASTA/ ii. Select any database that you like. Ill choose drosoph.nt.gz (Drosophila Melanogaster) - Nucleotide database of the fruit-fly. iii. Download using wget.
# wget ftp://ftp.ncbi.nih.gov/blast/db/FASTA/drosoph.nt.gz

iv. Unzip to the BLASTDB location.

# gunzip c drosoph.nt.gz > $BLASTDB/drosoph.nt

b. Format the FASTA database to be usable by MPI-Blast. i. You can run mpiformatdb by hand ii. You can create an SGE job to run mpiformatdb
# vi mpiformatdb.sh #!/bin/bash #$ -cwd #$ -j y #$ -S /bin/bash export MPI_DIR=/opt/openmpi/ export BLASTDB=$HOME/bio/ncbi/db/ export BIOROLL=/opt/bio export BLASTMAT=$BIOROLL/ncbi/data/ $BIOROLL/mpiblast/bin/mpiformatdb --nfrags=4 \ -p F -i $BLASTDB/drosoph.nt # qsub mpiformatdb.sh

c. Create a search string

# vi test.txt >Test AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGC TTCTGAACTGGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATACTTTAACCAA TATAGGCATAGCGCACAGACAGATAAAAATTACAGAGTACACAACATCCATGAAACGCATTAGCACCACC ATTACCACCACCATCACCATTACCACAGGTAACGGTGCGGGCTGACGCGTACAGGAAACACAGAAAAAAG CCCGCACCTGACAGTGCGGGCTTTTTTTTTCGACCAAAGGTAACGAGGTAACAACCATGCGAGTGTTGAA GTTCGGCGGTACATCAGTGGCAAATGCAGAACGTTTTCTGCGTGTTGCCGATATTCTGGAAAGCAATGCC AGGCAGGGGCAGGTGGCCACCGTCCTCTCTGCCCCCGCCAAAATCACCAACCACCTGGTGGCGATGATTG AAAAAACCATTAGCGGCCAGGATGCTTTACCCAATATCAGCGATGCCGAACGTATTTTTGCCGAACTTTT

d. Run MPI-Blast on the database, using the search string as your input. i. Create a SGE submission script.
# vi mpiblast.sh #!/bin/bash #$ -cwd #$ -j y #$ -S /bin/bash export MPI_DIR=/opt/openmpi/ export BLASTDB=$HOME/bio/ncbi/db/ export BIOROLL=/opt/bio export BLASTMAT=$BIOROLL/ncbi/data/ $MPI_DIR/bin/mpirun -np $NSLOTS \ /opt/bio/mpiblast/bin/mpiblast \ -d drosoph.nt -i $HOME/test.txt \ -p blastn -o $HOME/result.txt

ii. Submit the script to the queuing system

# qsub pe orte 4 mpiblast.sh