5/30/09

djangointhecloud
1.PutyourWebappintheCloud 2.VirtualObservatory 3.WebServices
deployyourwebapptogoogleland
wherenoevilcanbedone!

djangointhecloud
GoogleAppEngine
RunonGooglemachines Yourowndomainorappspot.com JavaorPython Persistentstorage Freebutmanyquotas Scalableadvantageofthecloud Doesonethingwell:runningwebapps LAMPbased notsoeasywithdjango...
4

AppEngineDoesOneThingWell
AppEnginehandlesHTTP(S)requests,nothingelse
ThinkRPC:requestin,processing,responseout WorkswellforthewebandAJAX;alsoforotherservices

Appcongura^onisdeadsimple
Noperformancetuningneeded

Amazon

Everythingisbuilttoscale
innitenumberofapps,requests/sec,storagecapacity

Security
Preventthebadguysfrombreaking(into)yourapp ConstraindirectOSfunc^onality
noprocesses,threads,dynamiclibraryloading nosockets(useurlfetchAPI) cantwriteles(usedatastore) disallowunsafePythonextensions(e.g.ctypes)

WhyNotLAMP?
Linux,Apache,MySQL/PostgreSQL,Python/Perl/PHP/ Ruby LAMPistheindustrystandard Butmanagementisahassle:
Congura^on,tuning Backupandrecovery,diskspacemanagement Hardwarefailures,systemcrashes Sojwareupdates,securitypatches Logrota^on,cronjobs,andmuchmore Redesignneededonceyourdatabaseexceedsonebox

Limitresourceusage

Limit1000lesperapp,eachatmost1MB Hard^melimitof10secondsperrequest Mostrequestsmustuselessthan300msecCPU^me Hardlimitof1MBonrequest/responsesize,APIcallsize,etc. Quotasystemfornumberofrequests,APIcalls,emailssent,etc Everythinganappdoesislimitedbyquotas,forexample: requestcount,bandwidthused,CPUusage,datastorecallcount,diskspaceused,emailssent, evenerrors! Ifyourunoutofquotathatpar^cularopera^onisblocked(raisinganexcep^on)forawhile (~10min)un^lreplenished

Wecarrypagerssoyoudonthaveto
6

5/30/09

googleappsanddjango
easytoportdjangotogoogleapps addspecialmain.pytodjangoapp
launchesit

googleappsanddjango
django
urlpatterns = patterns('', (r'^polls/$', pollapp.polls.views.index'), (r'^polls/(?P<poll_id>\d+)/$', pollapp.poll.views.detail'), (r'^polls/(?P<poll_id>\d+)/results/$', pollapp.poll.views.results'), (r'^polls/(?P<poll_id>\d+)/vote/$', pollapp.poll.views.vote'), )

addapp.yaml djangodatabaseapiislikegoogledatastore
django classGree^ng(models.Model): author=models.ForeignKey(Event) content=models.TextField() date=models.DateTimeField() appengine classGree^ng(db.Model): author=db.UserProperty() content=db.StringProperty(mul^line=True) date=db.DateTimeProperty()

appengine

putyourwebappinthegooglecloud step1:gettheSDK

step2:porttogoogledatastore
(lookslikedjangomodel!)
fromgoogle.appengine.extimportdb classGree^ng(db.Model): author=db.UserProperty() content=db.StringProperty(mul^line=True) date=db.DateTimeProperty(auto_now_add=True) g=Gree^ng(author=users.get_current_user(),content=hello!,date=now()) gree^ngs=Gree^ng.all() g=Gree^ng.all().lter(author=users.get_current_user()).order(date) BUTalsohaveGQL: gree^ngs=db.GqlQuery("SELECT*FROMGree^ngORDERBYdateDESCLIMIT10)

step3:runninglocally
bluegreenhat/ main.py app.yaml applica^on:bluegreenhat version:1 run^me:python api_version:1 handlers: url:/.* script:main.py

main.py

$ dev_appserver.py myapp/ pointbrowsertolocalhost:8080

5/30/09

step4:register,upload,run

VirtualObservatory

Towardanewastronomy
Past: Observa^onsofsmall,carefullyselectedsamples

Mul^wavelengthdataformillionsofobjects DataMining,Outliers,Correla^ons,etc TheoryvsExperiment

Towardanewastronomy

Towardanewastronomy
Tools Visualiza^on DataMining

5/30/09

VirtualObservatoryis:
Uniforminterfaces Datadiscovery
&publica^on
Op^cal

OpenSkyQuery

standards

Datafedera^on
scienceisinthejoin

Usability Remoteanalysis
bigdata,services,storage
Op^cal+Xray

Ijustwantto....

Bigcompu^ng
terabytes&teraops HydraA

WebEnabledSourceIden^ca^on withCrossMatching(WESIX)
UploadimagestoSExtractorandcrosscorrelatetheobjectsfoundwithselected surveycatalogs.
ThisNVOservicedoessource extrac^onandcrossmatchingfor anyastrometricFITSimage.The useruploadsaFITSimage,andthe remoteservicerunstheSExtractor sojwareforsourceextrac^on. Theresul^ngcatalogcanbecross matchedwithanyofseveralmajor surveys,andtheresultsreturned asaVOTable.Thewebpagealso allowsuseofAladinorVOPlotto visualizeresults.

ExampleService
Client interface

http://archive.eso.org/dss/dss/image

name= M51 x= 10 y= 10 sky_survey=DSS2_red mime_type=download_gif

Server

GET Request: Keyword/value

Response: image/gif

WebServices
Principle:ClickorCode Protocol:simpleREST/GETorsophis^catedSOAP Standards
Basicserviceprole
formaldescrip^on(WSDL),inputparametrs,outputformats,capabili^es VOResource(whoandwhatisit)

ServiceOrientedArchitecture
request

client
response request

service
response

Security(singlesignon,authen^ca^onandauthoriza^on) Asynchronous(batch)services Distributeddatastorage(VOSpace)

CoordinatedwithIVOA,OpenGridForum
registry

5/30/09

ServiceOrientedArchitecture
3.bind
request

VODataServices
ConeSearch
FirststandardNVOservice:
radius+posi^onlistofobjects encodedasVOTable

client
response request

SimpleImageAccessProtocol
service

response

conesearchforimages imagesarereferencedbyURL

SimpleSpectrumAccessProtocol
service contract

2.nd

spectrahavesubtle^esprotocolmorecomplicated

registry

1.publish

VODataServices
AstronomicalDataQueryLanguage
Fordatabasequeries CoreSQLfunc^onsplusastronomyspecicextensions
Skyregion,Xmatch

Registry
publishndbind RegistryMetadata
Descrip^onsof
datacollec^ons datadeliveryservices organiza^ons,etc.

SkyNode
Exposesrela^onaldatabases AcceptsADQLquery FullSkyNodessupportposi^onalcrossmatchfunc^on OpenSkyQueryportal
showdatabasestructure querytools

BasedonDublinCorewith astronomyspecicextensions RepresentedasXMLschema; extensible ContentsstoredinResource Registries

exchangemetadatarecords throughtheOpenArchives Ini^a^veProtocol(OAIPMH)

Datasets,StandardServices,Organiza^ons
Dataset:2MASS
PI:Skrutskie,UVa Date:19972001
TableAccessService:
Creator:OMullane,JHU Date:2008

RegistryContent:

A mandatory form plus other supporting forms

Registryrecord

Organiza^on:IPAC
Director:Helou Loca^on:Pasadena

ImageService: Lockmanhole
Creator:Alexov,IPAC Date:2005

Iden^erformat

ivo://ipac.caltech/Lockman-Hole

5/30/09

DistributedRegistry

WebServiceGlossary
Webservice(thinka(code)libraryontheweb)
Apieceofsojwareavailableoveranetworkwithaformal descrip^onofhowitiscalledandwhatitreturnsthatacomputer canunderstand Anapproachtowebservicesthatusestheexis^nginfrastructure oftheweb AnapproachtowebservicesthatusesanXMLbasedmessaging framework AnXMLdescrip^onofawebservice(normallySOAP)andhowto interactwithit

REST(thinkHTTPGETandPOST)
Astrogrid CfA NCSA NOAO STScI/JHU HEASARC CDS ESO

Caltech

JapanVO

SOAP(thinkenvelopes)

Ongoing harvesting July 08

(CfA, ESO, NOAO soon)

WSDL(thinkacontract)

8Sep2008

NVOSS2008WebServices

32

Whatwewontcover
Security:
Iwanttoprotectmydata/resources Iwanttoupload/downloadale Iwanttheservicetorememberthings Itsgoingtotakesome^me Iwantmorethanjustrequest/response
NVOSS2008WebServices 33

WebservicesintheVO
SkyNodes,OpenSkyQueryandWESIX FootprintServices SpectrumServices RegistryInterface VOSpace CasJobs CDS VOServices Conesearch
NVOSS2008WebServices 34

Azachments State

Asynchrony Messaging

8Sep2008

8Sep2008

Numquamponendaestpluralitassine necessitate
REST(Representa^onStateTransfer)isan architecturalstylenotanimplementa^on:
Dis^nguishesresourcesandopera^ons Eachresourceisiden^edbyaURI Theonlyopera^onsaretheHTTPmethods
HTTPMethod PUT GET POST

RESTbyanalogy
Think CREATE RETRIEVE UPDATE Descrip:on Createaresourcewiththe userspeciedid Retrievearesource representa^on Updatearesource/Append toaresource/Createa resourcewithaserver assignedid Deletearesource /procanalogy Startanew process Getthestatusofa process Amendaprocess

DELETE

DELETE

Killaprocess

8Sep2008

NVOSS2008WebServices

35

8Sep2008

NVOSS2008WebServices

36

5/30/09

Byaccident,notbydesign
ParametersinHTTPGETURLs:
hzp://processes.com/123/status hzp://processes.com/services?ac^on=getStatus&pid=123

WhentouseREST
RESTfulservicesaregoodforWWWtypefunc^onality Limita^ons:
Noformaldescrip^on:
WADLnotwidelyusedyet NotanissueforpureREST(noparametersinURL) Reliablemessaging Messagerou^ng Digitalsignatures

HTTPGETissafeitdoesnothingelsethanretrieval HTTPGET,PUTandDELETEareidempotenttheeectson thesystemofoneorNiden^calrequestsarethesame ConsiderHTTPGETwith: hzp://processes.com/services/ac^on=cancelJob&pid=123 Serviceswhichmaintainidempotencywhilstallowing parameterizedURLsareaccidentallyRESTful 85%ofwebservicetrac,6xfasterallegedly

Nomessaginginfrastructure: Nomessagelevelsecurity:

Noresourcelifecyclemanagement Notransac^onsupport Noasynchronouseventno^ca^on

8Sep2008

NVOSS2008WebServices

37

8Sep2008

NVOSS2008WebServices

38

WhatisSOAP?
SimpleObject/ServiceOrientedAccessProtocol W3Cspecica^on
hzp://wanderingbarque.com/nonintersec^ng/2006/11/15/thesstandsfor simple/ hzp://www.w3.org/TR/soap

AnatomyofaSOAPmessage
SOAPenvelope
SOAPheader(op^onal) SOAPbody
Messagepayload(actualdata)

AnXMLbasedmessagingframeworkforexchanginginforma^on betweenpeersinadecentralized,distributedenvironment Denesthemessagestructurebutnotthemessagecontent(needs othertechnologies) Fundamentallystateless(nomemoryofwhathashappened previously) Onewaymessageexchangeparadigm

XML namespace and schema

8Sep2008

NVOSS2008WebServices

39

8Sep2008

NVOSS2008WebServices

40

Request: <soap:Envelope xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd=http://www.w3.org/2001/ XMLSchema xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body> <ComovingLineOfSight xmlns="http://skyservice.pha.jhu.edu"> <z>float</z> <hubble>float</hubble> <omega>float</omega> <lambda>float</lambda> </ComovingLineOfSight> </soap:Body> </soap:Envelope> Response: <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance xmlns:xsd="http://www.w3.org/2001/ XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body> <ComovingLineOfSightResponse xmlns=http://skyservice.pha.jhu.edu> <ComovingLineOfSightResult>float</ComovingLineOfSightResult> </ComovingLineOfSightResponse> </soap:Body> </soap:Envelope>

SOAPexample

WhatisWSDL?
WebServicesDescrip^onLanguage W3Cspecica^on AnXMLgrammarfordescribingthepublicinterfaceof awebserviceintermsof:
itsexposedopera^ons themessageformats protocolbindings(e.g.SOAP,HTTP) hzp://www.w3.org/TR/wsdl

MostcommonlyusedwithSOAP

denesformatofmessagecontentintheSOAPbody

8Sep2008

NVOSS2008WebServices

41

8Sep2008

NVOSS2008WebServices

42

5/30/09

AnatomyofaWSDL
<deni^ons> <types> <message> <portType> <opera^on> <input> <output> <fault> </opera^on> </portType> <binding> <service> </deni^ons>
AWSDLdocumentdescribes: asetofserviceswhichexchangemessages thedatainthemessagesisdenedinasetoftypes eachmessagetransferinaservicedenesan opera^on opera^onshavebindingstospecicimplementa^ons usingprotocolslikeHTTP

WhatisWSDL
Itistheshareddocumentthat allowsarequestresponse conversa^on

8Sep2008

NVOSS2008WebServices

43

UsingSOAPandWSDL
StartwithWSDL(contractrst)vs.startwithcode (contractlast) FrameworksprovideAPIs:
tohandleSOAP:
(De)serializaFon:conver^ngcodeobjectto/fromXML representa^on Stub:clientsideproxytoservice Skeleton:serversidetohandlerequestandresponse

andcodebindingsderivedfromWSDL:

Alternateapproach:

SOAP(andWSDL)arejustXML UseXSLT,curl,JavaScript,etc.
NVOSS2008WebServices 45

8Sep2008