Cloud - setup private cloud

My interest in cloud computing started when I worked on a project, to deploy application on Amazon cloud and setup the clusters for MySql, Apache and other technologies to load balance. While working on Amazon cloud I got some insights of the cloud computing and various underlying technologies. Here I am going to illustrate all the steps that I have followed to setup cloud. I will be describing the cloud packages, how to setup front end and back end setup, cloud implementation, ec2 or euca2ools setup, creating xen-supported guest machine, creating EMIs for eucalyptus, run/stop instances and some xen/eucalyptus/ec2 useful command.

Technical background
When I start on cloud setup, I was sure I will go for XEN rather than KVM. Then came the Ubuntu jaunty as this version was built for the virtualization and designed with built-in support. I used eucalyptus, an opensource tool that provide Amazon like features and is EC2 tools compliance. So I starts researching the contents on all these technologies. The more I study, the more I got confident to use these technologies for cloud setup. Though, Ubuntu prefers KVM for the visualization support and has added to the default packages. But limitation of KVM to run only virtualization extensions (Intel VT or AMD-V) processors only(see http://www.linux-kvm.org/page/Main_Page), makes my decision to use XEN for virtualization. Xen, is the underlaying technology used by eucalyptus. Xen hypervisor allows several guest operating systems to be executed on the same computer hardware concurrently. Xen partitions a single physical machine into multiple virtual machines, to provide server consolidation and utility computing. Existing applications and binaries run unmodified. The hypervisor controls the MMU, CPU scheduling, and interrupt controller, presenting a virtual machine to guests. The first guest operating system, called in Xen terminology "domain 0" (dom0), is booted automatically when the hypervisor boots and given special management privileges and direct access to all physical hardware by default. The system administrator can log into dom0 in order to manage any further guest operating systems, called "domain U" (domU) in Xen terminology. Eucalyptus has 3 major packages. Cluster Controller: provides support for the virtual network overlay Cloud Controller: includes the front-end services and the Walrus storage system. Node Controller: that interacts with XEN to manage individual VMs. The basic architecture how client, cloud controller, cluster controller and node controllers communicate with each other in a eucalyptus setup.

Postfix is a mail server. Front end setup I have setup a Dell machine as front end. In my private cloud setup. Install Ubuntu 9. Select ssh server to be installed on the machine by default. Step 2: Install postfix. So I install both packages on this machine. Front end runs both eucalyptus-cloud and eucalyptus-cc. Eucalyptus send mails to users that are added by admin on eucalyptus web interface.04 on a machine. eucalyptus-cc and eucalyptus-nc. Steps to install postfix and to setup configurations and testing successful Postfix installation command# apt-get install postfix Set configurations command# sudo dpkg-reconfigure postfix Set following parameters with given values. so We can connect to machine remotely. General type of mail configuration: Internet Site NONE doesn't appear to be requested in current config .----> eucalyptus-nc | | Client ---> eucalyptus-cloud ----> eucalyptus-cc ------> eucalyptus-nc | | ----> eucalyptus-nc Packages provided by Ubuntu are: eucalyptus-cloud. the front end runs both eucalyptus-cloud and eucalyptus-cc and 3 machines act as node controller and run eucalyptus-nc. Follow the steps for installing the front end: Step 1: Prepare a raw ubuntu OS system. So we install postfix that eucalyptus will use as mail server.

com Root and postmaster mail recipient: <admin_user_name> (Optional) Other destinations for mail: server1.example.conf' VNET_DHCPDAEMON="/usr/sbin/dhcpd3" Step 5 Eucalyptus configuration .d -f dhcdbd remove Set the DHCP server path in the eucalyptus config file '/etc/eucalyptus/eucalyptus.com.d/dhcdbd stop command# update-rc. localhost.System mail name: server1.0.com data subject: your sub (press enter twice) Type your message and finish with ".com.example.0. example. command# apt-get install dhcp3-server command# /etc/init. localhost Force synchronous updates on mail queue?: No Local networks: 127. command# apt-get install eucalyptus-cloud eucalyptus-cc Step 4 Install dhcp server Install the DHCP server which is used by eucalyptus for assigning the IPs to instances. all the dependencies will be installed automatically by package manager.com.0/8 Yes doesn't appear to be requested in current config Mialbox size limit (bytes): 0 Local address extension character: + Internet protocols to use: all Test postfix installation command# EHLO localhost mail from: root@localhost rcpt to: akashsharmaa@gmail." then Enter Finish with "Quit" Check your mail (if received your mail is working) Step 3 Install Cloud and Cluster controller Install the packages provided by Ubuntu. DHCP server should not be in running state and set the DHCP server not to run when system is booted up again.example.

conf and set the following parameters VNET_INTERFACE="peth0" VNET_BRIDGE="eth0" Locate 'kvm' and replace to 'xen' locate VNET_MODE="SYSTEM" and disable it locate VNET_MODE="STATIC" and enable it Add machine and IPs for instances to be used and change the netowork setting to current network settings VNET_MODE="STATIC" VNET_SUBNET="192.168.168.42" VNET_MACMAP="AA:DD:11:CE:FF:ED=192.Set the static mode for eucalyptus so that eucalyptus uses IP's from the pool of IP's provided in the config file and use the machine addresses provided along with IPs.255 Step 7 Rebbot the system to take effect Step 8 Cloud configuarions .168.145.145.1 netmask = 255.255.145.168.168.sxp (xend-http-server yes) gateway = 192.75 AA:DD:11:CE:FF:EF=192.conf' open /etc/eucalyptus/eucalyptus.255.145.0" VNET_NETMASK="255.145.73 AA:DD:11:CE:FF:EE=192.168.0 broadcast = 192.145.255.255.150.168.77" Step 6 Other configurations Edit : /usr/share/eucalyptus/gen_libvirt_xml Locate all 'sda' to 'xvda' Edit : /etc/xen/xend-config.145.145.1" VNET_DNS="192.168.255" VNET_ROUTER="192. Configuration file for eucalyptus is '/etc/eucalyptus/eucalyptus.0" VNET_BROADCAST="192.168.

command# egrep '(vmx|svm)' /proc/cpuinfo [if no print]. More info can be found at: http://libvirt.org/ To install 'libvirt-bin' package. we will install node controller and XEN 3. we can use the eucalyptus web interface and setup cluster and set other parameters.d/apparmor stop command# update-rc. (Procedure to create images will be described later in the page) Restart the eucalyptus daemon to take effects Back end setup On the systems where we want to run instances. I installed required dependencies using the package manager apt-get. libvirt-bin cant be installed Stop the apparmor for Ubuntu System command# /etc/init.Once eucalyptus front end has been installed. Eucalyptus cloud web URL: https://<front end ip>:8443/ Under configuration tab add cluster name Set host to 'localhost' Create kernel and ramdisk images and set in eucalyptus site. Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). Since I used the XEN as virtualization package.5 python2.0. I install the eucalyptus-nc dependencies manually. but the eucalyptus-nc was installed by donwloading and manually installing bypassing dependencies list. Step 1 Hardware compatibility eucalyptus-nc requires 'libvirt-bin' package. command# apt-get install iproute iptables module-init-tools python2.6 command# apt-get install xen-utils .d -f apparmor remove Step 2 Install XEN package I am using the XEN virtualization package instead of Ubuntu's default KVM. certain flags should be supported by CPU. I install the node controller package eucalyptus-nc manually.

com/jaunty/eucalyptus-nc Download the i386 package from URL: http://mirrors.deb Install lib-virt command# dpkg -i libvirt-bin_0. .deb command# sudo adduser $USER libvirtd Step 4 Install eucalyptus node package command# apt-get install apache2 eucalyptus-common eucalyptus-gl libapache2-modaxis2c command# apt-get install libaxis2c0 librampart0 dhcp3-server vlan aoetools command# apt-get install libc6 libcurl3-gnutls libvirt0 zlib1g Download eucalyptus-nc package from Ubuntu package repository.5~bzr266-0ubuntu2_i386. More info can be found at: http://libvirt. Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes).kernel.deb command# dpkg -i eucalyptus<version> Step 5 Grub loader for Xen Kernel Setup grub menus and load xen images to boot into XEN kernel.1-0ubuntu5_i386.ubuntu.1-0ubuntu5_i386.com/jaunty/libvirt-bin I used i386 package.org/ubuntu/pool/universe/e/eucalyptus/eucalyptusnc_1.ubuntu. eucalyptus-nc package URL: http://packages.kernel.command# apt-get install ubuntu-xen-server Step 3 Install libvirt-bin package eucalyptus-nc requires 'libvirt-bin' package.6.6.org/ubuntu/pool/main/libv/libvirt/libvirtbin_0.org/ command# apt-get install adduser bridge-utils dnsmasq-base iptables libsasl2-2 libxen3 logrotate command# apt-get install netcat-openbsd libavahi-client3 libavahi-common3 libc6 libdbus1-3 command# apt-get install libgcrypt11 libgnutls26 libhal1 libpolkit-dbus2 libpolkit2 libreadline5 libselinux1 command# apt-get install libtasn1-3 libvirt0 libxml2 zlib1g policykit Download the libvirt-bin package from URL: http://packages. http://mirrors.

26-2-xen-686_2.26-2-xen686_2.conf (set) VNET_BRIDGE="eth0" VNET_INTERFACE="peth0" HYPERVISOR="xen" VNET_MODE="STATIC" Edit : /usr/share/eucalyptus/gen_libvirt_xml locate 'sda' s and replace to 'xvda' s for static ips Edit : /etc/xen-tools/xen-tools.deb command# sudo dpkg -i linux-image-2.6/linux-image-2.255 netmask 255.1 broadcast 192.6.deb Step 6 Eucalyptus Configuerations Stop the eucalyptus daemon and set the configuration depending upon your installation.org/debian/pool/main/l/linux-2.26-15_i386.26-15_i386.d/eucalyptus-nc start .deb command# sudo dpkg -i linux-modules-2.26-15_i386.6.168.6.168.ipv4. command# /etc/init.debian.conf (uncomment net.26-2-xen-686_2.255.Download the modules and image from URLs: wget http://ftp.145.deb wget http://ftp.6.26-15_i386.ip_forward=1) Edit : /etc/eucalyptus/eucalyptus.6.6/linux-modules-2.6.6.org/debian/pool/main/l/linux-2.0 (xend-http-server yes) (xend-unix-server yes) command# /etc/init.debian.145.26-2-xen686_2.255.6.conf gateway 192.d/eucalytpus-nc stop Edit : /etc/sysctl.

. To check the cloud installation. so cloud know about the available node.0-* (Replace this with the actual directory name) sudo -s echo deb file://${PWD} ./ >> /etc/apt/sources. We will add the nodes list to the cloud.0-*. and node communication.com/downloads Steps command# command# command# command# command# tar zxvf euca2ools-1. Install these packages without verification [y/N]? y .gz cd euca2ools-1.eucalyptus.list apt-get update command# apt-get install euca2ools WARNING: The following packages cannot be authenticated! . we have front controller and node controllers ready.. command# sudo euca_conf -addnode <node ip> <node ip> <node ip> This will add nodes to cloud config file and replicates the permission and certificate files to the nodes to be able to have password-less communication between nodes and cloud.Cloud integration Now.tar. you can run 'euca-describeavailability-zones verbose' command and check if all the nodes are listed in output. We have to setup now the communication between cluster controller and node controllers. ec2 or euca2ools setup Euca2ools setup Install dependencies command# apt-get install curl libopenssl-ruby ruby Download the appropriate tarball from http://open.

The content of 'part' is following: root 400 . unzip these archives Copy the 'lib' and 'bin' folders under ~/. Export parameters export EC2_HOME=~/.0_12 export PATH=$PATH:<directory>/jdk1.6.04/serverguide/C/ubuntu-vm-builder.euca folder and export the eucarc file.html Steps Select a directory where all files will be stored.zip' file from cloud site at https://<front end ip>:8443/.euca folder.euca export JAVA_HOME=<directory>/jdk1. command# source ~/.ubuntu. Install dependencies Install jdk1. Steps Download ec2-ami-tools-1.0_12 java version.ec2 tools Setup Eucalyptus supports ec2-ami-tools-1.com/8. More info on vmbuilder can be found on: https://help.3-30349 versions of ec2 tools.3-30349. I used '/usr/akash/cloud' Create a 'part' file under directory.euca/bin Set URL and credentials Download 'euca2-admin-x509.0_12/bin export PATH=$PATH:~/.6.eucarc Create guest machine I create my image using vmbuilder. Unzip this archive under ~/.3-26357 and ec2-api-tools-1.3-26357 and ec2-api-tools-1.6.

168. command# xm list If you see your emi ID in the output. we should create images that can be run using eucalyptus./mnt/ephemeral 0 /dev/sda2 swap 1 /dev/sda3 Create 'firstboot' file. if its compatible to xen. your image can be run with XEN. cloud is setup.xml | awk '{print $2}'` command# echo $EKI .manifest.6.28-13-server -d . which is run first time.6.cfg Check the status of emi using xm list command.manifest. command# xm create /usr/akash/cloud/ubuntu-xen/xen./kernel --kernel true command# ec2-upload-bundle -b eucaKernel -m . Use xm create command.xml command# EKI=`ec2-register eucaKernel/vmlinuz-2. This image can be tested.6.87:9999/ubuntu --addpkg ssh --rootsize=4096 --arch i386 --verbose --debug It creates the image file that i used for creating emi. apt-get -y install openssh-server Create the image.152. We will create emis for kernel. command# vmbuilder xen ubuntu --firstboot ./firstboot -mirror=http://192.28-13-server./kernel/vmlinuz-2. ramdisk and instance. kernel emi Steps command# mkdir kernel command# ec2-bundle-image -i vmlinuz-2. Creating EMIs Now. You can found the image under 'ubuntu-xen' folder. instance boots up.28-13server.

28-13-server -d . I encounter many issues and face problems because of no/less help available.img.priv command# ec2-run-instances emi-73FD0EAD -k mykey --kernel eki-8D701398 -ramdisk eri-E47F14E8 --instance-type m1.xml command# EMI=`ec2-register plabImage/root./image command# ec2-upload-bundle -b plabImage -m .6.28-13-server./image/root.ramdisk emi Steps command# mkdir ramdisk command# ec2-bundle-image -i initrd.manifest./ramdisk --ramdisk true command# ec2-upload-bundle -b eucaRamdisk -m .xml | awk '{print $2}'` command# echo $EMI Run instance Add key pair command# ec2-add-keypair mykey > ~/.6.28-13server./ramdisk/initrd.com forums or post my issues in forums to get the solution.manifest.euca/mykey. . it might be of some help if you got similar issues.euca/mykey.img-2.priv command# chmod 0600 ~/.img-2.manifest.6.img -d .xml | awk '{print $2}'` command# echo $ERI instance emi Steps command# mkdir image command# ec2-bundle-image -i root.manifest. I will try to list some of the issues thats I face. To resolve the issues I have to google or search http://open.img-2.xlarge Troubleshoot Once I started installing eucalyptus.eucalyptus.img.xml command# ERI=`ec2-register eucaRamdisk/initrd.

nc.log On node controllers.m1.large 0001 / 0001 2 512 17 |.log file [EUCAINFO ] doDescribeResource() invoked [EUCAINFO ] doDescribeResource() invoked [EUCAERROR ] libvirt: Domain not found (code=42) which indicates. . While starting a instance. command# ec2-describe-availability-zones verbose Output of above command could be like this. Or all the resources has been consumed by the instances. we can clearly visualize the resource management. check euca_test_nc.71 certs[cc=true.log file for any errors. euca_test_nc.xlarge 0000 / 0000 4 2048 20 |.nc=true] @ Wed From the output.c1. AVAILABILITYZONE AVAILABILITYZONE AVAILABILITYZONE AVAILABILITYZONE AVAILABILITYZONE AVAILABILITYZONE AVAILABILITYZONE AVAILABILITYZONE Sep 30 02:12:15 EDT 2009 iCloud UP localhost |. Cloud resources Some times when you install the eucalyptus and cluster. While starting instance.192. my first area of interest was the logs generated by eucalyptus and its dependencies. Because of this issue.m1. Output also lists the nodes registered/available currently to cluster.vm types free / max cpu ram disk |.log file contains all the errors eucalytus-nc encounter while starting. So we can omit this error message in nc. You can check the resource status for cluster using 'ec2-describe-availability-zones verbose' command. this may be issue with the node resource registration with the cluster.log file. and keep on adding the logs for non-existing network while instance is currently in booting up.log nc.145. I got following in the nc. Still you can not run instances. euca_test_nc. domain can not be started.xlarge 0001 / 0001 2 1024 20 |.c1.small 0002 / 0002 1 128 15 |. but after some time this error is resolved automatically when network is started for the instance. free/used resources and number of instance we can run. cloud and nodes are running successfully on systems.log file contains all the logs generated in cluster-node communication.Eucalyptus Logs When I got any issue.medium 0002 / 0002 1 256 15 |.168. if there is any issue with starting eucalytus-nc command. Eucalyptus maintain all the logs under pre-defined directory '/var/log/eucalyptus'. initially I wasnt aware of buggy log handling of eucalyptus and it takes a lot time to understand this.m1.

.log or any other log file doesnt provide useful info.copy the image file used to create emi to any of the node. best way to investigate issue and save time to run the instance using xm command provided by xen.is the name option provided in xen.w' ] name = 'Ubuntu' dhcp = 'dhcp' vif = [''] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' extra = 'xencons=tty console=tty1 console=hvc0' .xvda1.xvda2.w'.img. check following xen config file content kernel = '/boot/vmlinuz-2. So.28-13-server' ramdisk = '/boot/initrd. command# xm console Ubuntu Ubuntu . I got stuck with instance related issues. 'file:/usr/akash/cloud/ubuntu-xen/swap.create xen conf file. .while xm create is starting instance.cfg . Common steps are.img-2.28-13-server' memory = 128 root = '/dev/xvda1 ro' disk = [ 'file:/usr/akash/cloud/ubuntu-xen/root. instance will die and nc.6. use console command to check the console output. command# xm create xen.img. When I start instance using ec2 tools.6.cfg file .run the instance using xm command.Instance issues Some times.

Useful commands Xen commands command# xm list list all the available Xen instances you're allowed to control command# xm create xen.cfg run the instance using xen. returns all the available nodes and resource available .cfg file options command# xm status current status of your Xen guest command# xm console <domain id> allow you to access your Xen guest's console command# xm restore <domain file> Restore a domain from a saved state command# xm save <domain id> Save a domain state to restore later command# xm pause <domain id> Pause execution of a domain command## xm shutdown <domain id> Shutdown a domain command# xm destroy <domain id> Terminate a domain immediately command# xm mem-max <domain id> Set the maximum amount reservation for a domain command# xm mem-set <domain id> Set the current memory usage for a domain EC2 command Delete image command# ec2-deregister emi-55150E20 command# euca-delete-bundle -a $EC2_ACCESS_KEY -s $EC2_SECRET_KEY -url $S3_URL -b eucaubuntu -p image --clear Check cloud status.

168.conf .145.168.71 Sync keys between nodes and cluster controller command# $EUCALYPTUS/usr/sbin/euca_sync_key -c $EUCALYPTUS/etc/eucalyptus/eucalyptus.command# euca-describe-availability-zones verbose EUCALYPTUS Commands Add new node command# euca_conf -addnode 192.145.71 Delete node command# euca_conf -delnode 192.

Sign up to vote on this title
UsefulNot useful