P. 1
Pick n Mix – DJUGL Sep 09

Pick n Mix – DJUGL Sep 09

|Views: 20|Likes:
Published by Ben Firshman

More info:

Categories:Types, Speeches
Published by: Ben Firshman on Oct 30, 2010
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

05/12/2014

pdf

text

original

Pick ‘n’ Mix

Ben Firshman

Elegant Queues

Performance: * Incredibly important, especially now web applications doing increasingly complex things. * Amazon found that 100ms of latency caused 1% loss in sales * Extra half a second on Google search results caused 20% drop in traffic * Especially when altering data, it still needs to be responsive: * Posting a tweet to followers * Uploading images * Importing API data * Running stuff asynchronously

Tasks
* Running TASKS asynchronously

Tasks

❖ Independent ❖ Loosely ❖ Easily

components

coupled

scalable

* Independent components that can run concurrently * Loosely coupled: only depends on initial input * Thanks to which, they’re easily scalable. * Essentially: * Stuff you can send off to run in the background. * Stuff the user doesn’t immediately care about.

Application

* Say we have our application * and a bunch of tasks that need running * What runs these tasks?

Application

* Say we have our application * and a bunch of tasks that need running * What runs these tasks?

Application

? ? ?

?

* Say we have our application * and a bunch of tasks that need running * What runs these tasks?

Application

Worker

Worker

* Workers process the tasks. * Unlike “ghetto” queues done with cron jobs and databases, these are processes that have tasks pushed to them * These could either be on the same server as the application, or, thanks to independence, spread out onto many * Some way for application to publish tasks that need to be completed and distributing to workers

Application

Message Queue

* Message Queues are useful for this. (explain) * The messages can just be pickled tasks

Application

Message Queue

* Message Queues are useful for this. (explain) * The messages can just be pickled tasks

Application

Message Queue

Application

Application

Message Queue

* Nice thing with message queues: we can have multiple applications push things to the queue. * Means the frontend servers can be scaled horizontally without any of the task infrastructure caring

Application

Application

Message Queue

* Nice thing with message queues: we can have multiple applications push things to the queue. * Means the frontend servers can be scaled horizontally without any of the task infrastructure caring

Application

Application

Message Queue

Worker

Worker

Application

Application

Message Queue

Worker

Worker

Application

Application

Message Queue

Worker

Worker

Application

Application

Message Queue

Worker

Worker

Application

Application

Message Queue

Worker

Worker

Application

Application

Message Queue

Worker

Worker

* This is where celery comes in * Ask Solem Hoel has written an excellent piece of software that provides everything you need for a task queue

Celery

Worker daemon

Celery

❖ ❖

Worker daemon API for defining and executing tasks

Celery

❖ ❖ ❖

Worker daemon API for defining and executing tasks Handles communication with message queue (AMPQ or STOMP)

Code!
from celery.task import Task from celery.registry import tasks import twitter class ImportTweetsTask(Task): def run(self, username, **kwargs): return twitter.Api().GetUserTimeline(username) tasks.register(ImportTweetsTask)

* Suppose you’re building a service which lets you update your status, and you want to import your latest tweets

Code!

>>> from myapp.tasks import ImportTweetsTask >>> ImportTweetsTask.delay(username=’londonpython’)

* This is the way a simple task is run. * If you’re just writing something that writes the result to the database, this is fine

Code!
>>> result = ImportTweetsTask.delay( username=’londonpython’) >>> result.ready() False >>> result.ready() True >>> [t.text for t in result.result] [u'I will also be looking into paying for pizza for DJUGL attendees next Thursday at Global Radio offices in Leicester Square.', ...]

map

>>> import operator >>> from celery.task import dmap >>> dmap(operator.add, [[1, 2], [4, 2], [3, 8]]) [3, 6, 11]

* Callable is pickled, so it must be available on the worker’s path

More goodies
❖ ❖ ❖ ❖ ❖

Periodic tasks as a distributed replacement for cron View for Ajax to check status Statistics Task sets for progress bars Retry tasks on failure

* Stats: how many tasks have been executed by type, time taken etc

http://www.celeryq.org/

py.test

py.test

❖ ❖ ❖

No boilerplate tests Useful test failure information Extensibility

* No boilerplate: Automatic detection of tests and use of built in Python features like assert

Code!
def test_equality(): x = 5 assert x == 5 def test_not_equal(): y = 9 assert y != 2 def test_articles_exist(articles): assert articles.objects.count() > 0

* “funcargs” provide fixtures

pytest_django

❖ ❖ ❖

Creates test database and environment Ensures every test runs within a transaction Provides a few useful funcargs and utilities

Funcargs
def test_something(client): assert 'Success!' in client.get('/path/').content def test_article_view(rf): request = rf.get('/article/4/') response = article_view(request, 4) assert 'Test title' in response.content

Funcargs cont.

def test_middleware(settings, client): settings.MIDDLEWARE_CLASSES = ( 'app.middleware.SomeMiddleware', ) assert 'Middleware works!' in client.get('/')

Decorators
@py.test.params([dict(a=1, b=2), dict(a=3, b=3)]) def test_equals(a, b): assert a == b @py.test.urls('myapp.test_urls') def test_something(client): assert 'Success!' in client.get('/some_path/')

http://pytest.org/ http://github.com/bfirsh/pytest_django/

Feeds

syndication-view

An attempt to clean up django.contrib.syndication

syndication-view
❖ ❖

An attempt to clean up django.contrib.syndication Feeds as class based views

syndication-view
❖ ❖ ❖

An attempt to clean up django.contrib.syndication Feeds as class based views Backwards compatible

syndication-view
❖ ❖ ❖ ❖

An attempt to clean up django.contrib.syndication Feeds as class based views Backwards compatible Eventually, refactoring generation of feeds including full Atom support

Code!
from django.contrib.syndication.feeds import Feed from articles.models import Article class ArticleFeed(Feed): title = "example.com news" link = "/sitenews/" description = "Latest news from example.com." def items(self): return Article.objects.all()[:15]

Yuck.
from articles.feeds import ArticleFeed feeds = { 'articles': ArticleFeed, } urlpatterns = patterns('', # ... (r'^feeds/(?P<url>.*)/$', 'django.contrib.syndication.views.feed', {'feed_dict': feeds}), # ... )

Code!
from syndication.views import Feed from articles.models import Article class ArticleFeed(Feed): title = "example.com news" link = "/sitenews/" description = "Latest news from example.com." def items(self): return Article.objects.all()[:15]

Much better.
from articles.feeds import ArticleFeed urlpatterns = patterns('', # ... (r'^articles/feed/$', ArticleFeed()), # ... )

http://github.com/bfirsh/syndication-view/

If you have any pet hates of contrib.syndication, let me know!

django-mptt

django-mptt

❖ ❖ ❖

Making trees out of Django models. Efficient retrieval. Methods added to models for traversing trees (get_children(), get_root(), get_decendants() etc)

* Efficient retrieval as opposed to a simple parent field and retrieving children with related name

Code!
from django.db import models import mptt class Category(models.Model):     name = models.CharField(max_length=50)     parent = models.ForeignKey('self', null=True, blank=True, related_name='children') mptt.register(Category)

Code!

from django.db import models import mptt class Category(mptt.Model):     name = models.CharField(max_length=50)     parent = models.ForeignKey('self', null=True, blank=True, related_name='children')

Code!

from django.db import models import mptt class Category(mptt.Model):     name = models.CharField(max_length=50)     parent = models.ForeignKey('self', null=True, blank=True, related_name='children')

http://github.com/bfirsh/django-mptt/

http://benfirshman.com/ @bfirsh

Credits
http://www.flickr.com/photos/ilovetechnology/3048585444/ http://www.flickr.com/photos/tomhakase/312785554/ http://www.flickr.com/photos/pensive-reflections/3829587788/ http://www.flickr.com/photos/purpleslog/183842413/ http://www.flickr.com/photos/chrissie64/1425138918/ http://www.flickr.com/photos/patlejch/2951799028/ http://www.flickr.com/photos/mckenzieo/1601233059/ http://www.flickr.com/photos/webel/2429902979/

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->