You are on page 1of 14

How to Build a Linux Bewoulf Cluster

Home My Accounts | Newsletter | News Flash | Contact Us | Search

How to Build a Beowulf Linux Cluster

This document outlines the process required to install and configure Linux for
use in a cluster environment. While no prior knowledge of either Linux or
clusters is assumed, the reader should be forewarned that this is not a trivial

What is Linux?
What is a Cluster?
The First Step: Hardware Considerations
Clusters in an Ideal World
The Master Node
Processor Speed
Disk Speed
Network Speed
The Slave Nodes
Processor Speed
Network Speed
Other Hardware Components
The Second Step: Installing Linux
Gather Hardware Information
Video Card
Hard Drive
Network Card
Install Linux
The Third Step: Configuring the Nodes
General Linux System Files

1 of 14 07/12/2010 12:27 PM

The master can also participate in computation like the slave nodes do. This has lead to many implementations of GNU/Linux of which clustering is but one. Both are open source.How to Build a Linux Bewoulf Cluster http://www.. The message passing interface works to transmit data among the computers (commonly called nodes or hosts) in the GNU is a recursive acronym that stands for Gnu is Not Unix. The job scheduler is just what it sounds like.mcsr. The slave nodes are just that.. however. do most of the 2 of 14 07/12/2010 12:27 PM . The master node is the computer that users are most likely to interact with since it usually has the job scheduler running on it. in general. Since the source code is available. Some would argue that a cluster must at least consist of a message passing interface and a job scheduler. however. but it is not required or even recommended in large clusters. Consider a cluster built for a single purpose. /etc/fstab Enabling Required Services Testing the Configuration Adding a User General Notes What is GNU/Linux? GNU/Linux is a freely or inexpensively available operating system based around the Linux kernel which was developed by Linus Torvalds. meaning that the source code for the kernel and GNU applications are freely available to anyone. a master and a slave. a cluster must consist of at least two nodes. It is possible to have a cluster without either of these components. It takes job requests from user input or other means and schedules them to be run on the number of nodes required in the cluster. anyone who can code can make modifications to them. Back to Top What is a Cluster? A cluster is a group of computers which work together toward a final goal. GNU is basically a collection of open source implementations of common Unix utilities and programs written to provide and alternative to expensive commercial Unix software. They respond to the requests of the master node and. There would be no need for a job scheduler and data could be shared among the hosts with simple methods like a CORBA interface. By definition.

and RAM. For instance. Therefore. Clusters in an Ideal World To maximize the benefits of a cluster. They are: processor speed. It is generally accepted that for optimal performance. This is due to the fact that one node which takes longer to do its work can slow the entire cluster down as the rest of the nodes must stop what they are doing and wait for the slow node to catch up.olemiss. but it is a consideration that must be made. As stated earlier. The Master Node There are four main considerations when building the master node.. Strictly speaking. the right hardware must be used. It is possible in theory to mix architectures when building a cluster by using Java. but that is outside the scope of this document. This is far from ideal. it makes sense to cover this early in the process. Having identical hardware specs also simplifies the setup process a great deal as it will allow each hard drive to be imaged from a master instead of configuring each node individually. The only requirement is that they both share the same architecture. The master node will be handling many more tasks 3 of 14 07/12/2010 12:27 PM . It is not necessary that these machines have the same levels of performance.mcsr. one must have access to computers on which to install the software. This is not always the case.. computing. Processor Speed This is especially critical if the master node participates in it is necessary to have at least two machines when building a cluster.How to Build a Linux Bewoulf Cluster http://www. disk speed. however. the cluster should only consist of all Intel machines or all Apple machines but not a mixture of the two. the only hardware requirements when building a cluster is two computers and some type of networking hardware to connect them with. Back to Top The First Step: Hardware Considerations To build a cluster. all nodes except the master node must have identical hardware specifications. network speed.

A fast SCSI drive is recommended. the Linux kernel can and will cache its disk writes to memory and keep them there until they must be written to disk. Time spent transmitting data is time wasted. the more processes can be run without accessing the disk. This can be mitigated by a good deal if the programmer expressly tries to minimize the ratio of time on the network to time on the processor but it never hurts to have more network speed. made even more so due to the fact that most nodes make use of NFS which means that every node in the cluster will be competing for access of the master node's disk. These diskless 4 of 14 07/12/2010 12:27 PM . Fast Ethernet is the better the performance of the cluster. Network Speed This is critical as well.. In fact. Second. it is strongly recommended that a switch be used instead of a hub when designing the cluster network. than the slave nodes so a faster processor may be required to keep it from lagging behind. The faster the network.. While not part of the master node per se. Keep in mind that since the master node can be kept quite busy doling out work to the other nodes. Both of these increase the speed of the master node which is critical to good overall cluster performance.How to Build a Linux Bewoulf Cluster http://www. Disk Speed Since most work done on the cluster will be saved as files on a hard drive at some time or another. but an IDE drive will work as well. their disk performance is not critical. a RAID array of types 5 or 0+1 is ideal. disk speed for the master node is absolutely critical. the more RAM. First.olemiss.mcsr. For this reason. Slave Nodes The slave nodes need to accomplish two tasks: perform the computations assigned to them and then send that data back out over the network. a slowdown here can have a huge negative impact on the entire cluster as the slave nodes waste time waiting for their next instruction. it is common to have nodes without hard drives in a cluster. Gigabit Ethernet is ideal but basically any network speed will work. RAM RAM is crucial in the master node for two reasons.

assumes that the slave nodes will have hard drives. but SMP as well. a switch is more desirable than a hub when designing clusters due the increased speed that they offer.. SMP) can be desirable but add another degree of complexity to programming applications for the to Build a Linux Bewoulf Cluster http://www. however. Network Speed This affects the slave nodes in exactly the same way that it does the master node. As of the time of this writing. Other Hardware Components. Keep heat in mind as well. See that section above for more information. RAM This affects the slave nodes in exactly the same way that it does the master node. Not only must the programmer take distributed processing into consideration. Intel Pentium III's offer a good price/performance ration. This document.olemiss. consider using 1U cases when building a cluster to reduce the space requirements and for the increased organization they offer.. Processor Speed Since a nodes primary function is computation. Also. See that section above for more information. 50 or more 5 of 14 07/12/2010 12:27 PM . AMD processors. produce much more heat and use more power than the other two. however.mcsr. It is also a good idea to purchase a KVM (Keyboard Video Mouse switch) to allow easy access to each individual node.e. The more processing power the better. Pentium IV's offer good performance if the programmer includes SSE2 (Intel's special set of instructions designed to provide enhanced floating point performance) but AMD's Athlon processors offer an outstanding price/performance ratio and even better performance for most applications. it makes sense that the fastest possible processor is used. nodes further reduce the cost of building a cluster and eliminate some of the time required to set a cluster up. so make a decision based on all these factors. Multiple processors for each node (i. As mentioned earlier. The three most important hardware considerations for slave nodes are processor speed. network speed and RAM.

This document uses RedHat 7. Gather Hardware Information Before installing Linux.. There are many distributions of Linux available and different people prefer different distributions for different reasons. This is especially important when using AMD based machines in a small room.. it is a good idea to have this information handy as misconfiguration can actually physically damage the monitor. Back to Top The Second Step: Installing Linux After the machines are assembled. the next logical step is to install it is a good idea to gather as much information as possible about the PC Linux will be installed on. Hard Drive 6 of 14 07/12/2010 12:27 PM .How to Build a Linux Bewoulf Cluster http://www. however. It is possible that the installation process will detect the information it needs automatically or that the monitor will be included in a list the user can choose from during install time. Other versions of Linux can be used. Monitor Find out the maximum horizontal and vertical refresh rates as well as the maximum resolution and color depths it supports. The monitor manufacturer's website should have all the information required. Video Card As with the monitor. Most modern cards will not need a special RAMDAC setting or clockchip setting so these steps can be skipped when the time comes. and some are more or less desirable depending on the experience level of the cluster administrator or personal taste. nodes can produce a significant amount of heat and effect both the stability of the cluster and comfort of the operator.1 with the default kernel as a basis. Find out the make and model of the card as well as the amount of RAM it has. it is possible the the installer will detect and configure the card automatically but this is not always the case.mcsr. Nevertheless. The areas defined below should be considered the minimum a user should know before attempting to install Linux.olemiss.

4. Other choices can include "text expert" or just simply press return to boot into the graphical installer. The graphical mode is best for users with little Linux experience. Be sure the know the size of the drive as this will effect how the partitions are set up later in the install process. 5.2 kernel that comes with RedHat 7. Install Linux Linux will need to be installed on every node of the cluster. then press ENTER This documentation assumes that the user will choose "text" from the boot prompt. use the install boot diskettes that came with the version of Linux being used. Boot from the CD-ROM. 2. etc. and continue with the installation. It is very likely that the installer will detect this properly but there is always a chance that configuration will have to be done manually. Follow the directions on the screen. Thanks to the easy to use installers that ship with most Linux distributions.mcsr..olemiss. With the computer powered up. you may want to refer to these Red Hat guides: The Official RedHat Linux x86 Installation Guide The Official Red Hat Linux Reference Guide Red Hat's Official Linux Reference Guide: Appendix B--Introduction to Disk Partitions 1. the cluster at the Mississippi Center for Supercomputing Research where this document 7 of 14 07/12/2010 12:27 PM . say no. Note that some ATA100 drives or controllers may be problematic when using the 2. 3. like a monitor. Choosing text is not necessary but is faster and more but it can allow for easier administration in the long run. If the computer will not boot from CD-ROM. it is recommended to use separate partitions. When installing Linux on the master node. this task is becoming trivial.How to Build a Linux Bewoulf Cluster http://www.1 Network Card Know the make and model of the card. The process is fairly straightforward until partitioning.4. This is not necessary. insert the RedHat Binary CD #1 and restart the computer. The Master node of Mimosa. If you have a driver diskette for any special devices. Nevertheless. insert that diskette and press ENTER Otherwise. Choose an install mode... If the computer will not boot from the CD-ROM. check the BIOS settings of the computer and make sure that the first boot device is the CD-ROM. sound card.

How to Build a Linux Bewoulf Cluster http://www. Now choose the packages to install. install LILO. are 8. This is not a bad idea if disk space is not a concern. make sure that NFS. 6. Making a boot disk is always a very good idea. the user is presented with an option to make a boot disk.1G /usr/local Do not forget to add a swap partition when partitioning the drive. Generally. it is advisable to not install the firewall at all. you are actually going to the directory on the master node thanks to NFS (Network File System). is configured basically like this: Device Size Mount Point /dev/sda1 9. Once the packages have installed. At the minimum. Setting the default run level to 5. multi-user with networking. the Linux bootloader. you just the / partition and the swap partition. and since all user files will be stored on the master nodes /home partition. 9. 11. and RSH. Networked File System. The /home and /usr/local of the slave nodes are mounted to those on the master node.. For the slave nodes.6G / /dev/sda2 53M /boot /dev/sda5 16G /home /dev/sdb6 6. can cause some problems if it has not been fully tested. until it is certain that the 8 of 14 07/12/2010 12:27 PM . whichever is larger. One way to ensure that these packages get installed is to simply choose everything.olemiss. graphical mode. you must check the "Select Individual Packages" box on the "Package Group Selection" page. If you type "cd /home or cd /usr/local" on the slave node. The swap partition should be at least 128 megabytes. 7. If the firewall is installed. so they can be any size on the slave node.mcsr. is being written. the Red Hat Linux Reference Guide recommends 2x RAM or 32 MB. One of the last steps in installing Linux is deciding whether or not the computer should boot up into graphical mode. 10. Otherwise. Remote Shell. RedHat will now ask about installing and configuring their firewall. make sure that it allows connections via SSH and RSH. We need to add a step here about configuring X. since the applications will all be installed on the master node's /usr/local partition. Since it is assumed that Linux will be the only operating system on the master node and the slave nodes. If the cluster has no contact with the outside world or is behind a very good firewall. it is a better choice to have the system's default run level set for 3. It is also a good idea to install SSH as a backup in case RSH fails but this is not required. as you can always turn off unneeded services later. to the MBR(Master Boot Record) of the primary hard disk. Actually.

3. Remember that many of the files that will be created or edited in this process will grant access to the root user only. it is possible to set the computer to boot into graphical mode by editing /etc/inittab and changing the line that looks like: id:3:initdefault: to id:5:initdefault: 12. Spector recommends using a Net 10 special address class--that is. This allows these machines to be accessed by name 9 of 14 07/12/2010 12:27 PM . Be sure to read the documentation for the clustering software to be used as different software may require some different configuration procedures. The steps below outline the basic configuration steps clustering software will need to operate properly.2. giving each node an address of 10.2.0. Also make sure that NFS is running. or later. X window system will not cause any problems. etc. it is time to configure them to work with the clustering software. For mimosa. It is therefore prudent to log in as root to save time. 10.168. for mimosa. etc. This step is not covered in detail in this such as: 10.. 192. Type "setup" and make sure that rsh. adding the names and IP addresses of every node in the cluster. General Linux System Files /etc/hosts Edit this file on every cluster node.How to Build a Linux Bewoulf Cluster http://www.0..1. Keep in mind that these steps are just a guideline. Back to Top The Third Step: Configuring the Nodes Now that Linux is functioning properly on every node. You will need to decide what to IP addresses to use for the nodes of your cluster. 10.2. we used the Portland Group's Cluster Development Kit.0.2. rlogin.1. we elected to use: 192. This can be done now.X. However.olemiss.2. The clustering software must be installed on the master and nodes. Once it has been determined that the system functions properly in run level 5. For what it's worth.0. You might try the same. of course.X.0.168.0. Now Linux is installed. Reboot the computer and continue to the next step. 192. and rexec are checked.mcsr.3.X.168.

6 node1-5 192.equiv Hosts defined in this file are considered to be equivalent to the localhost for security purposes.0.168. A typical /etc/hosts file will look something like this.1 mimosa so check to see if it does. instead of by IP number.3 node1-2 192. All clustering software may not require this.0.0.9 node1-8 /etc/hosts..168.168. where mimosa is the hostname of the master node and the others are the hostnames of the slave nodes except localhost: 127.4 node1-3 192.How to Build a Linux Bewoulf Cluster http://www.9 node1-8 .equiv file looks something like this: 192.168.0. This means that users of these machines can access the localhost without supplying a password. localhost node1-6 This means that this file will be hidden.4 node1-3 192.6 node1-5 node1-7 192.1 mimosa 192.168..0.168. but is required so that RSH will be able to log into each machine without a password.2 node1-1 192.0.3 node1-2 192.2 node1-1 192.8 node1-7 192. Notice the "." at the beginning of the file name. 10 of 14 07/12/2010 12:27 PM .5 node1-4 192.6 node1-6 This file should exist in each user's home directory.mcsr.5 node1-4 192. This can be a significant security risk.0.168. A typical /etc/hosts. This file is also required so users can use RSH to connect to each node without supplying a password.

. This will allow every host to access these directories and will eliminate the need to replicate work on every node. Modify the rsh. /etc/pam. For example. and "rlogin" to the end of the file. This file determines which directories will be exported by service=system-auth session required /lib/security/pam_stack.txt will be accessible from every node that 11 of 14 07/12/2010 12:27 PM . rlogin.olemiss. if user joe adds a file named foo.How to Build a Linux Bewoulf Cluster http://www. Simply add "rsh".so auth required /lib/security/pam_securetty. A typical . An example of these files after modification is given below: auth required /lib/security/pam_rhosts_auth. "rexec".mcsr.d This directory contains configuration files that effect logins of the various services defined service=system-auth /etc/exports This file should only be modified on the master account required /lib/security/ and rexec files by rearranging the lines to have the line with "rhosts" as the first line and the line with "securetty" as the second line. foo.. This allows for easier administration of the nodes and is highly recommended.rhosts file looks something like this: mimosa node1-1 node1-2 node1-3 node1-4 node1-5 node1-6 node1-7 node1-8 /etc/securetty This file is a list of tty's from which root can log in.txt in his home directory on the master auth required /lib/security/ auth required /lib/security/pam_env.

255.. Choose "System Services" then select each of the items listed above. enable rsh.d/ and type . rlogin and rexec on all nodes. On each node. Restart the machines to force the changes to take effect.255.0/255. This is accomplished by adding the following line to /etc/exports. There are several ways to do this.0(rw.How to Build a Linux Bewoulf Cluster http://www. 2. it is a good idea to export each user's home directory. Testing the Configuration 1.0.no_root_squash) It is also a good idea to export /usr/local as a great deal of user programs will be installed here.mcsr.168. as root enter the command "setup". /home 192.olemiss./network start. Do this by adding the following line to /etc/exports..255.. Since NFS is being used. [hostname of master node]:/usr/local /usr/local nfs [hostname of master node]:/home /home nfs On each node. Ensure that rsh will work without a password by trying. On the master node. /usr/local 192. is configured to mount this export. it is necessary to modify /etc/fstab to mount these NFS directories.. /etc/fstab is a list of devices and directories that will be mounted at boot time. Enabling Required Services Now that the required modifications have been made. type .. On RedHat..no_root_squash) /etc/fstab This file should only be modified on the slave nodes.0(rw. When configuring the master node be sure to select "nfs" as well.168. also type . Typically. (Such as 192./network stop.168./nfs start On each node. go to /etc/rc.X).0. type .0. type ifconfig and make sure that the machine has its appropriate interior IP address. Select items by pressing the space bar.255.d/init./nfs stop On the master node.0/ The following lines should be added to mount the two exports set up above. from the 12 of 14 07/12/2010 12:27 PM .

If you have some problems with rcp. rsh and rlogin are considered insecure themselves. cd /etc rcp root@[masternode]:/etc/passwd . to rsh to a slave node. it can be changed with the hostname command like this: hostname host1 13 of 14 07/12/2010 12:27 PM .). master node. For these reasons. If rsh works as expected. system. you may have to type the full path (/usr/bin/rcp followed by the rest of the command). Go to user accounts and add your user. The easiest way is to use X windows. To clone this user to the slave nodes use these commands rsh to appropriate node. Find linuxconf under programs. rcp root@[masternode]:/etc/shadow . if the master node can be accessed from the Internet..olemiss. Allowing root to log in remotely to a node via rsh is not generally recommended in a general purpose Linux installation. Back to Top General Notes The most glaring issue with this process is the security implications it raises. That means to copy this file to the current directory. Make sure you notice the (. it is absolutely imperative to Build a Linux Bewoulf Cluster http://www. Using /etc/hosts. Remember that when a user is added to the system. The passwd and shadow files contain the information about all the users accounts on that particular machine.mcsr. follow the directions in the clustering software documentation to install and configure the clustering software. In fact.. Add a User To create a user log onto the master node. This is most easily accomplished by simply copying /etc/passwd and /etc/shadow from the master node to all other nodes.equiv and the other methods to allow logins without supplying a password is usually frowned on as well. Hostnames can be changed either with RedHat's setup utility under "Networking". great care is taken to ensure the security of the cluster. it will necessary to add that user to all nodes. Alternatively.

olemiss. [an error occurred while processing this directive] 14 of 14 07/12/2010 12:27 PM .How to Build a Linux Bewoulf Cluster http://www. Back to Top Last Modified: Tuesday. All Rights Reserved.mcsr.. 18-Dec-2001 15:55:11 CST Copyright © 1997-2005 The Mississippi Center for Supercomputing