VMFest

Antoni Batchelli @tbatchelli a.k.a @disclojure

Friday, April 8, 2011

what is it?
• VMFest is a clojure library that wraps
VirtualBox (http://virtualbox.org)

• It wraps the remote API of VirtualBox • It provides different API layers • Works with standard images
Friday, April 8, 2011

Goals
• Ease the development and testing of pallet • Provide a local cloud to test multi-service
cloud apps in the cloud, e.g. test many OS concurrently

• Fast, easy, a library, clojurey
Friday, April 8, 2011

Demo Time!

Friday, April 8, 2011

a few challenges
• VBox is a COM app wrapped with a SOAP
layer. Resource management is a mess! e.g.,
Each client object has a server counterpart removing your local object does not remove the remote counterpart implicitly clojure does not really have explicit object destruction

Friday, April 8, 2011

a few challenges
• Hence, to avoid memory leaks on the
server side we should either:

-

Keep track of all objects created, destroy them explicitly when not used anymore Kill the session, the server will kill related objects Have vbox kill you objects server-side after a while orphan local objects, messy

Friday, April 8, 2011

a few challenges
• Networks are unreliable • The vbox server could be restarted • Checking for a ‘live’ session is costly

Friday, April 8, 2011

a few challenges
• The VBox API is quite non-orthogonal
Sometimes you interact with composite objects via the master object, sometimes via the slave object

• Need locking to access VMs, sometimes,
not sure, maybe, I am sure it is written somewhere, right? sure. Right?

Friday, April 8, 2011

design decisions
• A wet/dry system of objects
A ‘wet’ object is a ‘live’ VBox object that has a server-side counterpart A ‘dry’ object is a clojure data structure with no live counterpart, but with enough info to create a corresponding wet object

Friday, April 8, 2011

design decisions
• To use wet objects you need to have a
session, and sometimes a lock
Session wrappers

-

(with-session vb-m :write [session machine] (.setMemorySize machine (long 2048)) (.saveSettings machine))

Friday, April 8, 2011

design decisions
• To use wet objects you need to have a
session, and sometimes a lock
Session wrappers

-

dry object

(with-session vb-m :write [session machine] (.setMemorySize machine (long 2048)) (.saveSettings machine))

Friday, April 8, 2011

design decisions
• To use wet objects you need to have a
session, and sometimes a lock
Session wrappers

-

dry object lock type

(with-session vb-m :write [session machine] (.setMemorySize machine (long 2048)) (.saveSettings machine))

Friday, April 8, 2011

design decisions
• To use wet objects you need to have a
session, and sometimes a lock
Session wrappers

-

dry object lock type

(with-session vb-m :write [session machine] (.setMemorySize machine (long 2048)) (.saveSettings machine))

wet object

Friday, April 8, 2011

design decisions
(defprotocol vbox-object (as-map [this]) (soak [this vbox])) (defprotocol vbox-remote-object (dry [this server]))

Friday, April 8, 2011

(extend-type vmfest.virtualbox.model.Machine model/vbox-object (soak [this vbox] (let [vb-m (virtualbox/find-vb-m vbox (:id this))] (as-map [this] (session/with-vbox (:server this) [_ vbox] (try (let [machine (model/soak this vbox)] (merge this (map-from-IMachine machine (:server this)))) (catch Exception e (log/error (...) e) (merge this {:error "Machine not found" :exception e}))))))

Friday, April 8, 2011

(extend-type IMachine model/vbox-remote-object (dry [this server] (let [id (.getId this)] (vmfest.virtualbox.model.Machine. id server nil))))

Friday, April 8, 2011

design decision
• Session wrappers:
create a new session at the beginning create wet objects from dry ones destroys session at the end of the block There is a session Objects are cleaned up

• This ensures that

Friday, April 8, 2011

design issues

Each wrapper means a new session with the server

-

Expensive, but only in relative terms: every interaction with a local object also produces a SOAP call

Sometimes you need different types of lock in the same interaction (rare, but true)

Friday, April 8, 2011

We are forcing a programming model on the user

more design decisions
• API has 3 rings (layers)
Ring 0: Infrastructure to interact with vbox Ring 1: low level, wet objects, live session Ring 2: higher level, dry objects, no session

Friday, April 8, 2011

more design decisions
• When to use each ring?
Ring 2 should be the default, Ring 1 for performance reasons or adding new functionality Ring 0 is for hacking vmfest.

Friday, April 8, 2011

ring 0
(defmacro with-session [machine type [session vb-m] & body] `(try (with-vbox (:server ~machine) [mgr# vbox#] (let [~session (.getSessionObject mgr#) immutable-vb-m# (.findMachine vbox# (:id ~machine))] (.lockMachine immutable-vb-m# ~session (~type lock-type-constant)) (let [~vb-m (.getMachine ~session)] (try ~@body (finally (.unlockMachine ~session)))))) (catch VBoxException e# (conditions/wrap-exception e# {:message (format ...)}))))
Friday, April 8, 2011

ring 1
(defn stop [^IConsole c] {:pre [(model/IConsole? c)]} (conditions/with-vbox-exception-translation {:VBOX_E_INVALID_VM_STATE "Virtual machine not in Running state." :VBOX_E_PDM_ERROR "Controlled power off failed."} (log/trace (format "stop: machine-id: %s" (.getId (.getMachine c)))) (.powerButton c)))

Friday, April 8, 2011

ring 2
(defn stop [^Machine m] {:pre [(model/Machine? m)]} (session/with-session m :shared [s _] (machine/stop (.getConsole s))))

Friday, April 8, 2011

more issues with this design
• Chaining Ring 2 calls will result in multiple
sessions established

-

This doesn’t happen at ring 1

• There is some duplicity of functions
between Ring 2 and Ring 1

• What to do with errors? Not sure
Friday, April 8, 2011

architecture
• VBox has immutable images + differencing
disks

• VBox has no way to clone images • You can attach the same immutable image
to many VMs, each one will have a differencing disk

Friday, April 8, 2011

architecture
• VMFest creates VMs dynamically and
image cloning costs much longer attaches an immutable image to them

• The only cost is the differencing image; no • Currently this process is closed, but not for

Friday, April 8, 2011

vmfest has a user!
• Pallet
A cloud provisioning library that works in conjunction with jclouds (I am a contributor) Goal, develop locally, deploy on the cloud in ~400 lines of code, pallet builds a cloud provider on top of vmfest Pallet uses vmfest for live testing of crates on many OSs

Friday, April 8, 2011

pallet integration
• In ~/.pallet/config.clj
Create a :virtualbox provider Add your images in that provider
{:description "Ubuntu 10.10 (64bit)" :uuid "4f072132-7fbc-431b-b5bf-4aa6a807398a" :os-type-id "Ubuntu_64" :os-version "10.10" :username "user" :password "superduper" :os-family :ubuntu :os-64-bit true :no-sudo false :sudo-password "superduper"}
Friday, April 8, 2011

Demo Time!!!

Friday, April 8, 2011

future directions
• A DSL to specify VM configurations
{:cpu-count 2 :memory-size 555 :network [{:attachment-type :bridged :host-interface "en1: Airport 2"} nil {:attachment-type :bridged :host-interface "nothing"}] :storage [{:name "IDE Controller" :bus :ide :devices [nil nil {:device-type :dvd}]}]}

Friday, April 8, 2011

future directions
• Install a VM directly from the cloud
Currently setting up VMFest is somewhat involved Download images that are ready to be used as a cloud (std image + vm guest additions) Build a library of images

Friday, April 8, 2011

Thanks
• Next steps:
visit http://github.com/tbatchelli/vmfest drop by the #pallet channel on irc.freenode.net email me tbatchelli@gmail.com

Friday, April 8, 2011