P. 1
Celery - An A Synchronous Task Queue (Not Only) for Django

Celery - An A Synchronous Task Queue (Not Only) for Django

|Views: 1,936|Likes:
Celery is an asynchronous task queue. It can be used to process large amounts of messages in real-time.
Celery is an asynchronous task queue. It can be used to process large amounts of messages in real-time.

More info:

Published by: Markus Zapke-Gründemann on Jun 07, 2011
Copyright:Attribution Share Alike

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

11/22/2012

pdf

text

original

An asynchronous task queue (not only) for Django

Markus Zapke-Gründemann DjangoCon Europe 2011

Celery

Image (cc-by-sa): http://www.flickr.com/photos/artdrauglis/3747272595/

Overview
• Why should I use a task queue? • Celery • Python Task • Django Task • Webhooks

Why should I use a task queue?

Users have to wait…
Action Add a comment Upload an image Order a product Mark a favorite Task Check for spam Create thumbnails Process payment, send emails Update database

A task queue can help!
• Decoupling of information producers and
consumers

• Profit from asynchronous processing • Improve scalability • Replace cronjobs

Client

Broker

Worker

Client

Worker

Client

Broker

Client

Worker

Client

Worker

2
Client Broker

Client

1

Worker

Client

4
Client Broker

3 2

Worker

Client

1

Worker

Database Filesystem API

3
Client

5
Client Broker

4 2

Worker

Client

1

Worker

Celery

Celery
• Author: Ask Solem Hoel • Written in Python • Synchronous, asynchronous and scheduled
tasks

• Broker: RabbitMQ, NoSQL and Ghetto
Queue

RabbitMQ
• Message Broker • Written in Erlang • AMQP (Advanced Message Queuing
Protocol)

• Clustering support

Celery Features
• Serialization (pickle, JSON,YAML or custom
code)

• Task sets and subtasks • Retry failed tasks • Routing (only AMQP brokers)

Celery Features
• Different result stores • Easy logging from tasks • Webhooks • Integration with Python Web Frameworks
(Django, Flask, Pylons)

Celery History
• Until 1.0.6 the Django ORM was used • Since 2.0 SQLAlchemy is used and Django
integration is provided by the „djangocelery“ package

Install Celery
$ pip install celery

Behind the scenes
• kombu - An AMQP messaging framework
for Python

• amqplib - Python AMQP client. • anyjson - A uniform JSON API

Setting up RabbitMQ
$ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Python Task

Python Task
# tasks.py from celery.task import task @task def add(x, y): return x + y

Python Task Configuration
# celeryconfig.py BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "myuser" BROKER_PASSWORD = "mypassword" BROKER_VHOST = "myvhost" CELERY_RESULT_BACKEND = "amqp" CELERY_IMPORTS = ("tasks", )

Python Task Execution
>>> from tasks import add >>> result = add.delay(1, 2) >>> result.ready() False $ celeryd --loglevel=INFO >>> result.get() 3 >>> result.result 3 >>> result.successful() True

Django Task

Install Celery for Django
$ pip install django-celery

Django Task
Client Broker Worker

Django Task
blog.views.add_comment

Client

Broker

Worker

Django Task
blog.views.add_comment blog.tasks.spam_filter

Client

Broker

Worker

Django Task
blog.views.add_comment blog.tasks.spam_filter

Client

Broker

Worker

Comment Model

Django Task
blog.views.add_comment blog.tasks.spam_filter

Client

Broker

Worker

comment.save() tasks.spam_filter.delay(comment.id, remote_addr)

Comment Model

Django Task
blog.views.add_comment blog.tasks.spam_filter

Client

Broker

Worker

comment.save() tasks.spam_filter.delay(comment.id, remote_addr)

comment.is_spam = True comment.save()

Comment Model

Django Task
# blog/tasks.py @task def spam_filter(comment_id, remote_addr=None): logger = spam_filter.get_logger() logger.info("Running spam filter for comment %s" % comment_id) comment = Comment.objects.get(pk=comment_id) current_domain = Site.objects.get_current().domain akismet = Akismet(settings.AKISMET_KEY, "http://%s" % current_domain) if not akismet.verify_key(): raise ImproperlyConfigured("Invalid AKISMET_KEY")

is_spam = akismet.comment_check(user_ip=remote_addr, comment_content=comment.comment, comment_author=comment.name, comment_author_email=comment.email_address) if is_spam: comment.is_spam = True comment.save() return is_spam

Django Task
# blog/tasks.py @task def spam_filter(comment_id, remote_addr=None): logger = spam_filter.get_logger() logger.info("Running spam filter for comment %s" % comment_id) comment = Comment.objects.get(pk=comment_id) current_domain = Site.objects.get_current().domain akismet = Akismet(settings.AKISMET_KEY, "http://%s" % current_domain) if not akismet.verify_key(): raise ImproperlyConfigured("Invalid AKISMET_KEY")

is_spam = akismet.comment_check(user_ip=remote_addr, comment_content=comment.comment, comment_author=comment.name, comment_author_email=comment.email_address) if is_spam: comment.is_spam = True comment.save() return is_spam

Django Task Configuration
# settings.py INSTALLED_APPS += ("djcelery", ) import djcelery djcelery.setup_loader() BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "myuser" BROKER_PASSWORD = "mypassword" BROKER_VHOST = "myvhost" $ python manage.py syncdb $ python manage.py celeryd -l info

Webhooks

Webhooks
# POST >>> from celery.task.http import URL >>> res = URL("http://example.com/multiply").get_async(x=10, y=10) >>> res.get() # {"status": "success", "retval": 100} 100 # GET >>> from celery.task.http import HttpDispatchTask >>> url = "http://example.com/multiply" >>> res = HttpDispatchTask.delay(url, method="GET", x=10, y=10) >>> res.get() # {"status": "success", "retval": 100} 100

Links
• http://celeryproject.org/ • http://www.rabbitmq.com/ • http://github.com/ask • http://pypi.python.org/pypi/celery • http://pypi.python.org/pypi/django-celery • http://pypi.python.org/pypi/celerymon

Thanks!
@keimlink http://www.keimlink.de

License
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/ licenses/by-sa/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->