Advanced Django

Simon Willison
http://simonwillison.net/ PyCon UK, 8th September 2007

Today’s topics
Unit testing newforms Ajax And if we have time... OpenID

Unit testing

Hard core TDD
Test Driven Development as taught by Kent Beck “Write new code only if an automated test has failed” Revolutionises the way you write code Pretty hard to adopt

I don’t do test driven development. I do stupidity driven testing... I wait until I do something stupid, and then write tests to avoid doing it again.

Titus Brown

What NOT to do
No matter how tough the deadline, don't let your test suite start breaking and say “I’ll fix it later” This will hurt. A lot.

Testing in Django
Testing web apps is HARD, but Django helps out with a bunch of features: Fixtures Doctests Test Client E-mail capture

Doctests
Used by Django for both ORM testing and generated documentation You are encouraged to add them to your own models.py manage.py test to execute them Great for regression tests - just copy and paste from an interpreter session

>>> >>> 0 >>> -1 >>> 1 >>> 20 >>> 19 >>> >>> 0 >>> 0 >>> 1 >>> 1

p = Person(name="Bob", dob=date(1980, 1, 1)) p.age(date(1980, 1, 1)) p.age(date(1979, 1, 1)) p.age(date(1981, 6, 1)) p.age(date(2000, 1, 1)) p.age(date(1999, 12, 31)) p2 = Person(name="Barry", dob=date(1981, 5, 5)) p2.age(date(1981, 5, 5)) p2.age(date(1982, 5, 4)) p2.age(date(1982, 5, 5)) p2.age(date(1982, 5, 6))

class Person(models.Model): """ ... tests here ... """ name = models.CharField(maxlength=100) dob = models.DateField() def age(self, age=False): return 1

$ python manage.py test ...F ====================================================================== FAIL: Doctest: peopleage.models.Person ---------------------------------------------------------------------Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 2161, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for peopleage.models.Person File "/home/simon/Projects/Django/oscon07/peopleage/models.py", line 4, in Person ---------------------------------------------------------------------File "/home/simon/Projects/Django/oscon07/peopleage/models.py", line 7, in peopleage.models.Person Failed example: p.age(date(1980, 1, 1)) Expected: 0 Got: 1

def age(self, age=False): if not age: age = date.today() delta = age - self.dob return int(math.floor(delta.days / float(365)))

File "/home/simon/Projects/Django/oscon07/peopleage/models.py", line 16, in peopleage.models.Person Failed example: p.age(date(1999, 12, 31)) Expected: 19 Got: 20

def age(self, age=False): if not age: age = date.today() years = age.year - self.dob.year this_year_birthday = self.dob.replace(year=age.year) birthday_has_passed = age >= this_year_birthday if not birthday_has_passed: years = years - 1 return years

Fixtures
It’s useful to be able to clear and populate your test database in between tests Fixtures let you do that (they also let you populate your database with real data when you deploy your application)

Denormalisation

Normalised data is for sissies

Cal Henderson

A forum, where each thread can have one or more replies Maintain a separate counter in the Forum table of number of replies, to speed up queries

class Thread(models.Model): subject = models.CharField(maxlength=100) num_replies = models.IntegerField(default=0) class Reply(models.Model): thread = models.ForeignKey(Thread) message = models.TextField()

[{ "model": "forum.thread", "pk": "1", "fields": { "num_replies": 0, "subject": "First thread" } },{ "model": "forum.thread", "pk": "2", "fields": { "num_replies": 1, "subject": "Second thread" } },{ "model": "forum.reply", "pk": "1", "fields": { "thread": 2, "message": "First post!1" } }]

from django.test import TestCase from models import Thread, Reply class ThreadCountTestCase(TestCase): fixtures = ['threadcount.json'] def test_add_reply(self): thread = Thread.objects.get(pk=2) self.assertEqual(thread.num_replies, 1) thread.reply_set.create(message="Another post") thread = Thread.objects.get(pk=2) self.assertEqual(thread.reply_set.count(), 2) self.assertEqual(thread.num_replies, 2) def test_delete_reply(self): thread = Thread.objects.get(pk=2) self.assertEqual(thread.num_replies, 1) Reply.objects.get(pk=1).delete() thread = Thread.objects.get(pk=2) self.assertEqual(thread.reply_set.count(), 0) self.assertEqual(thread.num_replies, 0)

====================================================== FAIL: test_add_reply (forum.tests.ThreadCountTestCase) ---------------------------------------------------------------------Traceback (most recent call last): File "/home/simon/Projects/Django/oscon07/forum/tests.py", line 16, in test_add_reply self.assertEqual(thread.num_replies, 2) AssertionError: 1 != 2 ====================================================== FAIL: test_delete_reply (forum.tests.ThreadCountTestCase) ---------------------------------------------------------------------Traceback (most recent call last): File "/home/simon/Projects/Django/oscon07/forum/tests.py", line 23, in test_delete_reply self.assertEqual(thread.num_replies, 0) AssertionError: 1 != 0 ----------------------------------------------------------------------

class Reply(models.Model): ... def save(self): super(Reply, self).save() self.thread.num_replies = self.thread.reply_set.count() self.thread.save() def delete(self): super(Reply, self).delete() self.thread.num_replies = self.thread.reply_set.count() self.thread.save()

....... ----------------------------------------------------------Ran 7 tests in 0.372s OK

Testing views
Django’s TestClient lets you simulate a browser interacting with your site It also provides hooks in to the underlying application, so you can test against the template and context that was used to generate a page

from django.test import TestCase class RegistrationTest(TestCase): def test_slash(self): response = self.client.get('/register') self.assertEqual(response.status_code, 301) response = self.client.get('/register/') self.assertEqual(response.status_code, 200) def test_register(self): response = self.client.get('/register/') self.assertEqual(response.template[0].name, 'register.html') self.assertTemplateUsed(response, 'register.html')

def test_signup_done_page(self): self.assertEqual(len(mail.outbox), 0) data = { 'email': 'simon@example.com', 'username': 'example', 'firstname': 'Example', 'lastname': 'User', 'password': 'password1', 'password2': 'password1', } response = self.client.post('/signup/', data) self.assertEquals(response.status_code, 302) self.assertEquals(response['Location'], '/welcome/') # Check that the confirmation e-mail was sent self.assertEqual(len(mail.outbox), 1) sent = mail.outbox[0] self.assertEqual(sent.subject, 'Welcome to example.com')

More on testing with Django:
http://www.djangoproject.com/documentation/testing/

newforms

The perfect form
Display a form User fills it in and submits it Validate their entered data If errors, redisplay form with previously entered data and contextual error messages Continue until their submission is valid Convert submission to appropriate Python types

Manipulators

Manipulators newforms

Forms are declarative
from django import newforms as forms class UserProfileForm(forms.Form): name = forms.CharField(max_length=100) email = forms.EmailField() bio = forms.CharField(widget=forms.Textarea) dob = forms.DateField(required=False) receive_newsletter = forms.BooleanField(required=False)

A form instance can... validate a set of data against itself render itself (and its widgets) re-render itself with errors convert to Python types

Simple form handling view
def create_profile(request): if request.method == 'POST': form = UserProfileForm(request.POST) if form.is_valid(): # ... save the user’s profile return HttpResponseRedirect('/profile/saved/') else: form = UserProfileForm() return render_to_response('profile.html', {'form': form})

Initial data
form = UserProfileForm( initial = { 'name': 'Simon Willison', 'email': 'simon@simonwillison.net', } )

Simple template
<style type="text/css"> ul.errorlist { color: red; } </style> ... <form action="/profile/create/" method="POST"> {{ form.as_p }} <input type="submit" value="Submit" /> </form> ...

Output
<form action="/profile/create/" method="POST"> <p><label for="id_name">Name:</label> <input id="id_name" type="text" name="name" maxlength="100" /></p> <p><label for="id_email">Email:</label> <input type="text" name="email" id="id_email" /></p> <p><label for="id_bio">Bio:</label> <textarea id="id_bio" rows="10" cols="40" name="bio"></textarea></p> <p><label for="id_dob">Dob:</label> <input type="text" name="dob" id="id_dob" /></p> <p><label for="id_receive_newsletter">Receive newsletter:</label> <input type="checkbox" name="receive_newsletter" id="id_receive_newsletter" /></p> <input type="submit" value="Submit" /> </form>

Output
<form action="/profile/create/" method="POST"> <p><label for="id_name">Name:</label> <input id="id_name" type="text" name="name" maxlength="100" /></p> <p><label for="id_email">Email:</label> <input type="text" name="email" id="id_email" /></p> <p><label for="id_bio">Bio:</label> <textarea id="id_bio" rows="10" cols="40" name="bio"></textarea></p> <p><label for="id_dob">Dob:</label> <input type="text" name="dob" id="id_dob" /></p> <p><label for="id_receive_newsletter">Receive newsletter:</label> <input type="checkbox" name="receive_newsletter" id="id_receive_newsletter" /></p> <input type="submit" value="Submit" /> </form>

Output
<form action="/profile/create/" method="POST"> <p><label for="id_name">Name:</label> <input id="id_name" type="text" name="name" maxlength="100" /></p> <p><label for="id_email">Email:</label> <input type="text" name="email" id="id_email" /></p> <p><label for="id_bio">Bio:</label> <textarea id="id_bio" rows="10" cols="40" name="bio"></textarea></p> <p><label for="id_dob">Dob:</label> <input type="text" name="dob" id="id_dob" /></p> <p><label for="id_receive_newsletter">Receive newsletter:</label> <input type="checkbox" name="receive_newsletter" id="id_receive_newsletter" /></p> <input type="submit" value="Submit" /> </form>

Custom validation
from django import newforms as forms from django.newforms.util import ValidationError class UserProfileForm(forms.Form): name = forms.CharField(max_length=100) email = forms.EmailField() bio = forms.CharField(widget=forms.Textarea) dob = forms.DateField(required=False) receive_newsletter = forms.BooleanField(required=False) def clean_email(self): if self.cleaned_data['email'].split('@')[1] == 'hotmail.com': raise ValidationError, "No hotmail.com emails, please."

Custom validation
from django import newforms as forms from django.newforms.util import ValidationError class UserProfileForm(forms.Form): name = forms.CharField(max_length=100) email = forms.EmailField() bio = forms.CharField(widget=forms.Textarea) dob = forms.DateField(required=False) receive_newsletter = forms.BooleanField(required=False) def clean_email(self): if self.cleaned_data['email'].split('@')[1] == 'hotmail.com': raise ValidationError, "No hotmail.com emails, please."

Custom rendering
<ol class="formItems longForm"> <li{% if form.email.errors %} class="errors"{% endif %}> <label for="id_email">Email: </label> {{ form.email }} {{ form.email.errors }} <p class="info">Your e-mail address.</p> </li> ... </ol>

Model shortcuts
DRY: You’ve already declared your models; you shouldn’t have to repeat yourself in your forms
UserForm = form_for_model(User) ############################### page = Page.objects.get(pk=1) PageForm = form_for_instance(page) form = PageForm(request.POST) ... if form.is_valid(): form.save()

Full documentation:
http://www.djangoproject.com/documentation/newforms/ django/trunk/tests/regressiontests/forms/tests.py

Ajax

First things first
If you're going to do Ajax, you need a JavaScript library You could use Yet Another XMLHttpRequest abstraction... but the popular libraries offer fantastic convenience Good libraries include YUI, Dojo, MochiKit and (controversial) Prototype...

.. and jQuery
I'm going to be using jQuery Almost everything is done in terms of CSS selectors and chained methods It looks like a gimmick, but it isn't http://simonwillison.net/2007/Aug/15/jquery/

Ajax formats...
Django has great support for any and every Ajax format HTML fragments XML JSON

Username available?
from django.contrib.auth.models import User def check_username(request): reply = "" username = request.GET.get('username', '') if username: if User.objects.filter(username=username).count(): reply = 'Unavailable' else: reply = 'Available' return HttpResponse(reply)

jQuery('span#msg').load( '/check_username/?username=' + input.val() );

$('span#msg').load( '/check_username/?username=' + input.val() );

var input = $('input#id_username') input.keyup(function() { $('span#msg').load( '/check_username/?username=' + input.val() ); });

$(document).ready(function() { var input = $('input#id_username') input.keyup(function() { $('span#msg').load( '/check_username/?username=' + input.val() ); }); });

$(function() { var input = $('input#id_username') input.keyup(function() { $('span#msg').load( '/check_username/?username=' + input.val() ); }); });

Recycling server-side form validation

from django import newforms as forms class ContactForm(forms.Form): subject = forms.CharField(max_length=100) message = forms.CharField(widget=forms.Textarea()) sender = forms.EmailField()

def validate_contact(request): "Validate post data, return errors as json" form = ContactForm(request.POST) if (request.GET.has_key('field')): # Validate a single field errors = form.errors[request.GET['field']] else: errors = form.errors return JsonResponse({ 'valid': not errors, 'errors': errors })

from django.utils import simplejson class JsonResponse(HttpResponse): def __init__(self, data): HttpResponse.__init__( self, simplejson.dumps(data), mimetype='application/json' )

function validateInput(input) { $.post('/contact/validate/?field=' + input.attr('id').replace('id_', ''), $('form').formToArray(), function(data) { var json = eval('(' + data + ')'); showErrors(input, json.errors); } ); } $(function() { $(':input').blur(function() { validateInput($(this)); }); });

function relatedErrorList(input) { var prevUL = $(input).parent().prev(); if (prevUL && prevUL.attr('class') == 'errorlist') { return prevUL; } var errorlist = $('<ul class="errorlist"></ul>'); input.parent().before(errorlist); return errorlist; } function showErrors(input, errors) { var errorlist = relatedErrorList(input); errorlist.empty(); $.each(errors, function(i, error) { errorlist.append('<li>' + error + '</li>'); }); }

Django philosophy
Django often gets marked down in “framework comparisons” due to the lack of built in Ajax support Personally I think that shipping without a recommended library is a feature, not a bug

(bonus section)

What is OpenID?

OpenID is a decentralised mechanism for Single Sign On

An OpenID is a URL
http://simonwillison.myopenid.com/ http://simonwillison.net/ http://swillison.livejournal.com/ http://openid.aol.com/simonwillison

How it works
You enter your OpenID on a site (instead of the usual username and password) It redirects you back to your OpenID provider They authenticate you in some way They redirect you back to the original site

Simple registration
(an optional but useful extension)

Consumers can also ask...
Your preferred username Your e-mail address Your first and last name Your date of birth Your language, country and timezone

How do you use OpenID in a Django application?

The hard way
Use the JanRain OpenID library Pretty much a reference implementation for the OpenID spec Well written, well tested but takes a while to get the hang of
www.openidenabled.com/openid/libraries/python/

The easy way
Use django-openid A simple wrapper around JanRain A middleware component, some models and a few pre-written views
http://code.google.com/p/django-openid/

Installation
Add 'django_openidconsumer' to your INSTALLED_APPS setting manage.py syncdb Add the OpenIDMiddleware to your MIDDLEWARE_CLASSES setting Add the views to your URLconf

In urls.py
... (r'^openid/$', 'django_openidconsumer.views.begin'), (r'^openid/complete/$', 'django_openidconsumer.views.complete'), (r'^openid/signout/$', 'django_openidconsumer.views.signout'), ...

request.openid
The middleware adds an openid property to the Django request object If the user is not signed in, this will be None Otherwise, it will be an OpenID object; the str() representation will be the OpenID (or use request.openid.openid)

def example_view(request): if request.openid: return HttpResponse("OpenID is %s" % escape(request.openid)) else: return HttpResponse("No OpenID")

request.openids
The module supports users signing in with more than one OpenID at a time request.openids provides a list of all authenticated OpenIDs request.openid merely returns the most recent from this list

For simple registration
(r'^openid/$', 'django_openidconsumer.views.begin', { 'sreg': 'email,nickname' }), def example_sreg(request): if request.openid and request.openid.sreg.has_key('email'): return HttpResponse("Your e-mail address is: %s" % escape( request.openid.sreg['email'] )) else: return HttpResponse("No e-mail address")

Coming soon
django_openidauth, providing tools to associate OpenIDs with existing django.contrib.auth accounts django_openidserver, to make it easy to provide OpenIDs for users of your Django application

More information
http://openid.net/ Also home to the OpenID mailing lists http://www.openidenabled.com/ http://simonwillison.net/tags/openid/ http://code.google.com/p/django-openid/

Thank you!