11/1/13

ubuntu cloud image | More Mind Spew-age from Harold Spencer Jr.

More Mind Spew-age from Harold Spencer Jr.

Random Information sometimes techie, sometimes not..

Tag Archives: ubuntu cloud image

Another Great Example of AWS Fidelity Neo4j, Cloud-Init and Eucalyptus

Posted by hspencer77 on December 4, 2012 I recently ran across a blog entry entitled Neo4j 1.9.M01 Self-managed HA (http://http://blog.neo4j.org/2012/10/neo4j-19m01-self-managed-ha.html). I found the concept of graph databases storing data (http://www.neo4j.org/learn/graphdatabase) really interesting and reached out to the guys at Neo4j (http://www.neo4j.org/) to get some insight on how to deploy their HA solution on Eucalyptus. Amongst the resources that they provided, they shared this little gem how to deploy Neo4j on EC2 (http://www.neo4j.org/develop/ec2). In order to run first, you need to know how to walk so before going down the path of standing up HA Neo4j, I decided to be influenced by the DIY on EC2 article provided by Neo4j and deploy Neo4j on Eucalyptus - with a little help from Cloud-Init (https://help.ubuntu.com/community/CloudInit). The follow-up blog will show how to use the same setup, and deploy an HA Neo4j environment.

The Setup

Eucalyptus
blogs.mindspew-age.com/tag/ubuntu-cloud-image/ 1/6

11/1/13

ubuntu cloud image | More Mind Spew-age from Harold Spencer Jr.

The Eucalyptus cloud I used is configured using Eucalyptus High-Availability (http://www.eucalyptus.com/docs/3.1/ig/planning_ha.html#planning_ha). Its running on CentOS 6.3, running KVM. Its also running in Managed networking mode (http://www.eucalyptus.com/docs/3.1/ig/nw_managed.html#nw_managed), so that we can take advantage of network isolation of the VMs, and the use of security groups - interacting very much in the same way as its done in the security groups provided in AWS EC2 (http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-network-security.html).

Ubuntu Cloud Image 12.04 LTS Precise Pangolin

The image that we will use is the Ubuntu 12.04 LTS Cloud image (http://cloudimages.ubuntu.com/precise/current/). The reasons for using this image is as follows: Ubuntu cloud images (http://cloud-images.ubuntu.com/) come pre-packaged with cloud-init, which helps with bootstrapping the instance. I wanted to have the solution work on AWS EC2 and Eucalyptus; since Ubuntu cloud images work on both, its a great choice.

Registering the Ubuntu Cloud Image with Eucalyptus

In order for us to get started, we need to get the Ubuntu Cloud image into Eucalyptus so that we can use it for our instance. To upload, bundle and register the Ubuntu Cloud image, ramdisk and kernel, do the following: 1. Download current version of Ubuntu Precise Server AMD64 (http://cloudimages.ubuntu.com/precise/current/precise-server-cloudimg-amd64.tar.gz) from the Ubuntu Cloud Image Precise (http://cloud-images.ubuntu.com/precise/current/) page, then unpack (ungzip, unarchive) the tar-gzipped file. $t a rz x v fp r e c i s e s e r v e r c l o u d i m g a m d 6 4 . t a r . g z xp r e c i s e s e r v e r c l o u d i m g a m d 6 4 . i m g xp r e c i s e s e r v e r c l o u d i m g a m d 6 4 v m l i n u z v i r t u a l xp r e c i s e s e r v e r c l o u d i m g a m d 6 4 l o a d e r xp r e c i s e s e r v e r c l o u d i m g a m d 6 4 f l o p p y xR E A D M E . f i l e s 2. Make sure to download and source your Eucalyptus credentials (http://www.eucalyptus.com/docs/3.1/ug/get_creds.html#get_cred).
blogs.mindspew-age.com/tag/ubuntu-cloud-image/ 2/6

11/1/13

ubuntu cloud image | More Mind Spew-age from Harold Spencer Jr.

3. We need to bundle, upload, and register precise-server-cloudimg-amd64-loader (ERI), preciseserver-cloudimg-amd64-vmlinuz-virtual (EKI), and precise-server-cloudimg-amd64.img (EMI). For more information regarding this, please refer to the Image Overview (http://www.eucalyptus.com/docs/3.1/ug/understanding_images.html#understanding_images) section of the Eucalyptus 3.1 User Guide (http://www.eucalyptus.com/docs/3.1/ug/index.html#welcome).

$e u c a b u n d l e i m a g eip r e c i s e s e r v e r c l o u d i m g a m d 6 4 l o a d e rr a m d i s kt r u e $e u c a u p l o a d b u n d l ebl a t e s t u b u n t u p r e c i s em/ t m p / p r e c i s e s e r v e r c l o u d i m g $e u c a r e g i s t e rax 8 6 _ 6 4l a t e s t u b u n t u p r e c i s e / p r e c i s e s e r v e r c l o u d i m g a m d 6 4 $e u c a b u n d l e i m a g eip r e c i s e s e r v e r c l o u d i m g a m d 6 4 v m l i n u z v i r t u a lk e r n e l $e u c a u p l o a d b u n d l ebl a t e s t u b u n t u p r e c i s em/ t m p / p r e c i s e s e r v e r c l o u d i m g $e u c a r e g i s t e rax 8 6 _ 6 4l a t e s t u b u n t u p r e c i s e / p r e c i s e s e r v e r c l o u d i m g a m d 6 4 $e u c a b u n d l e i m a g eip r e c i s e s e r v e r c l o u d i m g a m d 6 4 . i m g $e u c a u p l o a d b u n d l ebl a t e s t u b u n t u p r e c i s em/ t m p / p r e c i s e s e r v e r c l o u d i m g $e u c a r e g i s t e rax 8 6 _ 6 4l a t e s t u b u n t u p r e c i s e / p r e c i s e s e r v e r c l o u d i m g a m d 6 4

After bundling, uploading and registering the ramdisk, kernel and image, the latest-ubuntu-precise bucket in Walrus should have the following images:

$e u c a d e s c r i b e i m a g e s|g r e pl a t e s t u b u n t u p r e c i s e I M A G Ee k i 0 F 3 9 3 7 E 9l a t e s t u b u n t u p r e c i s e / p r e c i s e s e r v e r c l o u d i m g a m d 6 4 v m l i n u z -

I M A G Ee m i C 1 6 1 3 E 6 7l a t e s t u b u n t u p r e c i s e / p r e c i s e s e r v e r c l o u d i m g a m d 6 4 . i m g . m a n i

I M A G Ee r i 0 B E 5 3 B F Dl a t e s t u b u n t u p r e c i s e / p r e c i s e s e r v e r c l o u d i m g a m d 6 4 l o a d e r . m

Cloud-init Config File

Now that we have the image ready to go, we need to create a cloud-init config file to pass in using the user-data-file option that is part of euca-run-instances. For more examples of different cloud-init files, please refer to the cloud-init-dev/cloud-init repository on bazaar.launchpad.net (http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/examples/). Below is the cloud-init.config file I created for bootstrapping the instance with an install of Neo4j, using ephemeral disk for the application storage, and installing some other packages (i.e. latest euca2ools, mlocate, less, etc.). The script can be also accessed from github as well under the eucalptus/recipes repo (https://github.com/eucalyptus/recipes/blob/master/cloud-init/cloud-init-neo4j.config).

blogs.mindspew-age.com/tag/ubuntu-cloud-image/

3/6

11/1/13

ubuntu cloud image | More Mind Spew-age from Harold Spencer Jr.

# c l o u d c o n f i g a p t _ u p d a t e :t r u e a p t _ u p g r a d e :t r u e d i s a b l e _ r o o t :t r u e p a c k a g e _ r e b o o t _ i f _ r e q u i r e d :t r u e p a c k a g e s : -l e s s -b i n d 9 u t i l s -d n s u t i l s -m l o c a t e c l o u d _ c o n f i g _ m o d u l e s : -s s h -[a p t u p d a t e u p g r a d e ,a l w a y s] -u p d a t e s c h e c k -r u n c m d r u n c m d : -[s h ,x c ," i f[b/ d e v / s d a 2] ;t h e nt u n e 2 f sLe p h e m e r a l 0/ d e v / s d a 2 ; e l i f[ -[s h ,x c ," m k d i rp/ v a r / l i b / n e o 4 j "] -[s h ,x c ," m o u n tL A B E L = e p h e m e r a l 0/ v a r / l i b / n e o 4 j "] -[s h ,x c ," i f[z` l s/ v a r / l i b / n e o 4 j / * `] ;t h e ns e di n p l a c e' $i M E T A _ H O -[s h ,x c ," M E T A _ H O S T N A M E = ` c u r lsh t t p : / / 1 6 9 . 2 5 4 . 1 6 9 . 2 5 4 / l a t e s t / m e t a d a t a / l -[s h ,x c ," M E T A _ H O S T N A M E = ` c u r lsh t t p : / / 1 6 9 . 2 5 4 . 1 6 9 . 2 5 4 / l a t e s t / m e t a d a t a / l -[s h ,x c ," w g e tOc 1 2 4 0 5 9 6 e u c a l y p t u s r e l e a s e k e y . p u bh t t p : / / w w w . e u c a l y p t u -[a p t k e y ,a d d ,c 1 2 4 0 5 9 6 e u c a l y p t u s r e l e a s e k e y . p u b] -[s h ,x c ," e c h o' d e bh t t p : / / d o w n l o a d s . e u c a l y p t u s . c o m / s o f t w a r e / e u c a 2 o o l s / 2 . 1 -[s h ,x c ," e c h o' d e bh t t p : / / d e b i a n . n e o 4 j . o r g / r e p os t a b l e / '>/ e t c / a p t / s o u r c -[a p t g e t ,u p d a t e] -[a p t g e t ,i n s t a l l ,y ,f o r c e y e s ,e u c a 2 o o l s] -[a p t g e t ,i n s t a l l ,y ,f o r c e y e s ,n e o 4 j] -[s h ,x c ," s e di n p l a c e' s / # o r g . n e o 4 j . s e r v e r . w e b s e r v e r . a d d r e s s = 0 . 0 . 0 . 0 / o r -[s h ,x c ," s e r v i c en e o 4 j s e r v i c er e s t a r t "] -[s h ,x c ," e x p o r tL A N G U A G E = e n _ U S . U T F 8 "] -[s h ,x c ," e x p o r tL A N G = e n _ U S . U T F 8 "] -[s h ,x c ," e x p o r tL C _ A L L = e n _ U S . U T F 8 "] -[l o c a l e g e n ,e n _ U S . U T F 8] -[d p k g r e c o n f i g u r e ,l o c a l e s] -[u p d a t e d b] m o u n t s : -[e p h e m e r a l 0 ,/ v a r / l i b / n e o 4 j ,a u t o ," d e f a u l t s , n o e x e c "]

Now, we are ready to launch the instance.

blogs.mindspew-age.com/tag/ubuntu-cloud-image/

4/6

11/1/13

ubuntu cloud image | More Mind Spew-age from Harold Spencer Jr.

Putting It All Together

Before launching the instance, we need to set up our keypair and security group that we will use with the instance. 1. To create a keypair, run euca-create-keypair (http://www.eucalyptus.com/docs/3.1/cli/euca-createkeypair.html#euca-add-keypair). *NOTE* Make sure you change the permissions of the keypair to 0600 after its been created.

e u c a c r e a t e k e y p a i r n e o 4 j u s e r>n e o 4 j u s e r . p r i v ;c h m o d0 6 0 0n e o 4 j u s e r . p r i v

2. Next, we need to create a security group for our instance. To create a security group, use eucacreate-group (http://www.eucalyptus.com/docs/3.1/cli/euca-create-group.html#euca-add-group). To open any ports you need for the application, use euca-authorize (http://www.eucalyptus.com/docs/3.1/cli/euca-authorize.html#euca-authorize). The ports we will open up for the Neo4j application are SSH (22), ICMP, HTTP( 7474), and HTTPS (7473). Create security group: #e u c a c r e a t e g r o u pn e o 4 j t e s td" S e c u r i t yf o rN e o 4 jI n s t a n c e s " Authorize SSH: #e u c a a u t h o r i z ePt c pp2 2s0 . 0 . 0 . 0 / 0n e o 4 j t e s t Authorize HTTP: #e u c a a u t h o r i z ePt c pp7 4 7 4s0 . 0 . 0 . 0 / 0n e o 4 j t e s t Authorize HTTPS: #e u c a a u t h o r i z ePt c pp7 4 7 3s0 . 0 . 0 . 0 / 0n e o 4 j t e s t Authorize ICMP: #e u c a a u t h o r i z ePi c m pt1 : 1s0 . 0 . 0 . 0 / 0n e o 4 j t e s t 3. Finally, we use euca-run-instances (http://www.eucalyptus.com/docs/3.1/cli/euca-runinstances.html#euca-run-instances) to launch the Ubuntu Precise image, and use cloud-init to install Neo4j:

#e u c a r u n i n s t a n c e skn e o 4 j u s e ru s e r d a t a f i l ec l o u d i n i t n e o 4 j . c o n f i ge

blogs.mindspew-age.com/tag/ubuntu-cloud-image/

5/6

11/1/13

ubuntu cloud image | More Mind Spew-age from Harold Spencer Jr.

To check the status of the instance, use euca-describe-instances (http://www.eucalyptus.com/docs/3.1/cli/euca-describe-instances.html#euca-describe-instances).

#e u c a d e s c r i b e i n s t a n c e si A 9 E F 4 4 8 C R E S E R V A T I O Nr E D 8 E 4 6 9 93 4 5 5 9 0 8 5 0 9 2 0n e o 4 j t e s t I N S T A N C Ei A 9 E F 4 4 8 Ce m i C 1 6 1 3 E 6 7e u c a 1 9 2 1 6 8 5 5 1 0 4 . w u t a n g . e u c a h a s p . e u c a l y p t e u c a 1 0 1 0 6 6 9 1 5 4 . w u t a n g . i n t e r n a lr u n n i n ga d m i n0m 1 . s m a l l2 0 1 2 1 2 0 4 T 0 3 : 1 3 : 1 e n t e r t h e w ue k i 0 F 3 9 3 7 E 9e r i 0 B E 5 3 B F Dm o n i t o r i n g d i s a b l e e u c a 1 9 2 1 6 8 5 5 1 0 4 . w u t a n g . e u c a h a s p . e u c a l y p t u s s y s t e m s . c o me u c a 1 0 1 0 6 6 9 1 5 4

Because I added in the cloud-init config file to do an apt-get upgrade, it takes about 5 to 7 minutes until the instance is fully configured and Neo4j is running. Once you have it running, go to https://<ipaddress (https://<ip-address) of instance>:7473. It will direct you to the web administration page for monitoring and management of the Neo4j instance. In this example, the URL will be https://euca-192168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com:7473 (https://euca-192-168-55-104.wutang.euca-hasp.eucalyptus-systems.com:7473)

(http://blogs.mindspew-age.com/2012/12/04/anothergreat-example-of-aws-fidelity-neo4j-cloud-init-and-eucalyptus/neo4j-landing-page/) Neo4j Monitoring and Management Tool Thats it! The cool thing about this too, is that you can find an Ubuntu Precise AMI on AWS EC2, use the same cloud-init script, use euca2ools, and follow these instructions to get the same deployment on AWS EC2. As mentioned before, the follow-up blog will be how to deploy the HA solution of Neo4j on Eucalyptus. Enjoy! 2 Comments Posted in Infrastructure, Interests, Private Cloud Computing, Start-Up Culture, System Adminstration Tagged 12.04, ami, aws ec2, centos 6.3, cloud-init, emi, eucalyptus, kvm, neo4j, precise, ubuntu cloud image Blog at WordPress.com. | Customized Piano Black Theme.

blogs.mindspew-age.com/tag/ubuntu-cloud-image/

6/6