Since DjangoCon, there seems to be a fair amount of discussion centered around the comparison of, drawbacks of, and paths to improving Django's templating engine.

This write is not intended to be a discussion on the principals or philosophies of Django's approach to templates verses the alternatives. I will, however, that I fully understand and appreciate the desire for the separation of concerns (i.e., developer vs. designer), although working mostly as a team of one, I personally find this approach to be a bit restricting. I don't know that "jumping ship" to Jinja is necessarily the best option for me. Instead, finding ways to improve Django templates and contributing back to the Django community feels right to me.

Having said that, I must admit that I find writing template extensions to be the single most tedious and cumbersome aspects of using Django. It feels like I am constantly repeating myself - DRY alert! - by writing new template tags: start by writing the micro-parser for the tag syntax, follow that by creating the wrapper Node class extension, and only then getting to the point where I can actually write the business or domain-specific logic that is needed. I am forever looking up references and consulting past work because there seems to be nothing intuitive or natural about the process. This is major put-off, and instead I would find myself putting the work of populating my context data into the individual views - which also didn't seem very DRY, nor very reusable.

I've seen a few suggestions for patches to help minimize the effort required to extend tags template. Until one of these patches - past, present, or future - finds it's way into trunk - or a release, in many developer's situation - I thought that I would share a solution that I have been using for awhile now that can be used in the interim.

Rather than list all the source code here, I've zipped up a simple project that exposes a Library extension and a sample application that takes advantage of it using various examples. The download can be found here.

The basic idea is this: provide an means of accessing something (a queryset, any arbitrary Python object) and injecting it into the template context as a named variable via a simple function that has been automagically been made into a new template tag by way of smart decorator. For flexibility, passing arguments into the function is an optional feature. I like to think of these new template-tagged functions as tag getters.

A somewhat contrived example might help to illustrate. First, the template code itself: I want to be able to retrieve a list of active users, possibly filtered. Currently, my most direct option would be to perform the query within the view and pass along the resulting queryset to the template through the Context instance. It's easy to see that this can become quite irksome: perhaps I have multiple views where this is necessary, or the variations on the filter are specific enough to make it overly complex to all the possible results to the template in manner where it can be consumed in a sensible fashion

Consider the following example that filters active users by applying some regular expressions (I warned that this would be contrived):

1
2
3
4
5
6
7

{% get_active_users as active_users using "^L" "(x|z)" %}
<p>Active users: {{ active_users.count }}</p>
<ul>
    {% for active_user in active_users %}
    <li>{{ active_user.get_full_name }}</li>
    {% endfor %}
</ul>

Get me the active users whose first or last name begins with L or contains either x or z (note: I could have easily done this by using a single regex argument, but I wanted to show multiple arguments being passed). The result, in this case a queryset, is assigned to the variable active_users. using and the subsequent arguments are optional in this case, as we'll see shortly.

Creating the get_active_users template tag is very simple and pretty much only requires thought to be given to the actual problem of user retrieval, allowing the developer to bypass most of the mechanics and build up of necessary template extension foo:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import operator
from django.db.models import Q
from django.contrib.auth.models import User
from template_utils import GetterLibrary

register = GetterLibrary()

@register.getter_tag
def get_active_users(*regex_filters):
    qs = User.objects.filter(is_active=True)
    if regex_filters:
        filter_by = [
            Q(first_name__regex=regex_filter) | Q(last_name__regex=regex_filter)
            for regex_filter in regex_filters
        ]
        qs = qs.filter(reduce(operator.or_, filter_by))

    return qs

So, we have a shortcut to producing a new template involving really only three steps, 2 of which we would have had to do regardles:

1. template_utils.GetterLibrary is a simple class derived from django.template.Library. I have point out here that my naming skills are lame at best, so please forbear from snickering - I welcome suggestions for a better choice of labels. GetterLibrary does everything we know and love from the base Library class, plus:

2. The getter_tag decorator method of the GetterLibrary instance, which provides the bit of function-to-tag automation, and:

3. Our "worker" function, whose sole job is to return any result we wish to assign to our new context variable.

There are several more examples to be found in the test project, so please download it it and poke about through the code and demo app.

Even though Leopard ships with a fairly recent version of Python (2.5.1), there are several reasons one would want to use an alternative installation. For instance, Python 2.3.5 shipped with Tiger and was growing pretty long in the tooth by the time Leopard arrived; 2.3 was the default installation on Panther.

I've tried a few approaches to installing newer versions on the Mac: several years ago, I played around with Fink, then switched to MacPorts (formerly DarwinPorts). While either of these package managers make it very easy to install various software pacakages on the Mac, they each presented certain problems or limitations with respect to Python and the other web development packages I need.

My current preferred solution is to go with the MacPython distribution found on the Python website. There is also a great SIG (http://mail.python.org/mailman/listinfo/pythonmac-sig) for getting help.

From http://www.python.org/download/mac/:

Python comes pre-installed on Mac OS X, but due to Apple's release cycle, it's often one or even two years old. The overwhelming recommendation of the "MacPython" community is to upgrade your Python by downloading and installing a newer version.

Listed below are the steps to setting up some of the most common Python packages used to for web development, particularly with respect to Django.

Getting started

Basic assumptions

• Mac OS X Leopard (10.5) - as of the date of this posting, I am on 10.5.3

• Subversion - you can find an installer at http://subversion.tigris.org/

• I use MySQL, so instructions for installing it and the necessary Python bindings (MySQLdb) are shown. If anyone has detailed instructions for Postgres installation, please send them to me at python [at] dakrauth.com

• An installation directory to use for these steps. For instance, here's how I began (from a Terminal window, bash shell):

  1 2 3

  mkdir -p ~/python/install mkdir ~/python/site-packages cd ~/python/install

  I use ~/python/site-packages as an easy way of referencing packages that will change frequently or for creating a pseudo-virtual environment. More on this below.

Installations

Python 2.5.2

• MacPython installation:

  1 2 3 4

  curl -O http://www.python.org/ftp/python/2.5.2/python-2.5.2-macosx.dmg open python-2.5.2-macosx.dmg open /Volumes/Universal\ MacPython\ 2.5.2/MacPython.mpkg hdiutil unmount /Volumes/Universal\ MacPython\ 2.5.2/

  This installation will create the site-packages directory at:

  /Library/Frameworks/Python.framework/Versions/Current/lib/python2.5/site-packages
  

  Additionally, your .profile file will be updated to place the above path at the front of you $PATH environment variable. I usually remove this change, since the only thing I like to have my .profile file a source line to my .bashrc file. However, I ensure that /usr/local/bin is place ahead /usr/bin in my $PATH string.

• Now I set up the ties to the aforementioned local site-packages directory:

  1 2 3

  pushd /Library/Frameworks/Python.framework/Versions/Current/lib/python2.5/site-packages echo /Users/your-username/python/site-packages > my.pth popd

MySQL

• Lines 4, 5, and 6 below are a quick hack around an odd bug in the installer.

  1 2 3 4 5 6

  curl -O http://mirror.services.wisc.edu/mysql/Downloads/MySQL-5.0/mysql-5.0.51b-osx10.5-x86.dmg open mysql-5.0.51b-osx10.5-x86.dmg open /Volumes/mysql-5.0.51b-osx10.5-x86/mysql-5.0.51b-osx10.5-x86.pkg/ pushd /usr/local/mysql/lib sudo ln -s /usr/local/mysql/lib mysql popd

• Add the preference pane to your System Preferences and start the server from the preference pane (I also prefer to select Automatically Start MySQL Server on Startup):

  1 2	open /Volumes/mysql-5.0.51b-osx10.5-x86/MySQL.prefPane/ hdiutil unmount /Volumes/mysql-5.0.51b-osx10.5-x86/

• To setup a sample database to use with a Django project, here are the commands I issue from within the mysql shell:

  1 2 3 4 5 6

  CREATE DATABASE spam_db CHARACTER SET utf8 COLLATE utf8_general_ci; GRANT ALL PRIVILEGES ON spam_db.* TO 'spam_db_user'@'localhost' IDENTIFIED BY 'spam_db_password';

MySQLdb

• MySQLdb provides the Python bindings to interface with the database.

  1 2 3

  curl -O http://internap.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.2.tar.gz tar -xvzf MySQL-python-1.2.2.tar.gz cd MySQL-python-1.2.2

• Edit the site.cfg file:

  • Uncomment the line that begins #mysql_config =

  • Edit the path to point to /usr/local/mysql/bin/mysql_config:

    mysql_config = /usr/local/mysql/bin/mysql_config
    

• Complete the build and installation:

  1 2 3

  python setup.py build sudo python setup.py install cd ..

PIL

The Python Imaging Library adds image processing capabilities and supports many file formats. Probably on of the most common applications I use it for are resizing and cropping images, and generating thumbnail images. This installation seems to cause many people grief, particularly JPEG handling (which is an optional install that depends upon libjpeg).

• freetype (optional) - a software font engine

  1 2 3 4 5 6 7

  curl -O http://download.savannah.gnu.org/releases/freetype/freetype-2.3.6.tar.gz tar -xvzf freetype-2.3.6.tar.gz cd freetype-2.3.6 ./configure make sudo make install cd ..

• libjpeg

  1 2 3 4 5 6 7 8

  curl -O http://www.ijg.org/files/jpegsrc.v6b.tar.gz tar -xvzf jpegsrc.v6b.tar.gz cd jpeg-6b ./configure make make test sudo make install-lib cd ..

• PIL build

  1 2 3 4

  curl -O http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz tar -xvzf Imaging-1.1.6.tar.gz cd Imaging-1.1.6 python setup.py build_ext -i

  Checking for the following lines near the end of the output from the previous command:

  --- TKINTER support ok
  --- JPEG support ok
  --- ZLIB (PNG/ZIP) support ok
  --- FREETYPE2 support ok
  

  Complet install and perform a last sanity check:

  1 2 3 4

  python selftest.py sudo python setup.py install python -c "import Image; Image.open('Images/lena.jpg').show()" cd ..

  You should see a Preview window pop up with the picure of a woman wearing a hat.

Common Packages

• IPython - An enhanced Python shell. You will want it - and be lost without it very quickly. Django will look for IPython when you run the shell command extension.

  1 2 3 4 5 6 7

  curl -O http://ipython.scipy.org/dist/ipython-0.8.4.tar.gz tar -xvzf ipython-0.8.4.tar.gz cd ipython-0.8.4 python setup.py build sudo python setup.py install python setup.py install_scripts --install-dir=/usr/local/bin cd ..

• docutils

  1 2 3 4 5 6

  curl -O http://internap.dl.sourceforge.net/sourceforge/docutils/docutils-0.5.tar.gz tar -xvzf docutils-0.5.tar.gz cd docutils-0.5 python setup.py install sudo cp tools/buildhtml.py tools/rst2html.py /usr/local/bin cd ..

• Markdown

  1 2 3 4 5

  curl -O http://internap.dl.sourceforge.net/sourceforge/python-markdown/markdown-1.7.tar.gz tar -xvzf markdown-1.7.tar.gz cd markdown-1.7 sudo python setup.py install cd ..

Django

Installation of Django, using either Apache with mod_python, mod_wsgi, or FastCGI has been written about in numerous locations. However, here is my personal strategy for managing Django across various revisions:

• First, I set up a directory to store the revisions of Django that I need across the sites I manage:

  1 2	mkdir ~/python/django pushd ~/python/django

• Next, I have two helper scripts I use to manage my revisions:

  A revision "getter" called djget:

  1 2 3 4 5 6 7

  #!/bin/bash svn export -r $1 http://code.djangoproject.com/svn/django/trunk $1 pushd $1/docs buildhtml.py --local mkdir html mv *.html html popd

  Run djget to export your desired revisions (why not just to stick to one revision? I manage a few sites and not all are on the same revision. Also, I like to play around with the trunk tip on my local machine first before deploying it live).

  Example:

  djget 7728
  

  This will export the desired revision into a directory named for the revision number (7728, in this case). It also generates HTML files from the reStructuredText documentation and moves those files into a new html directory

  A revision switcher called switch:

  1 2 3 4 5 6

  #!/bin/bash if [ -e django ]; then rm django; fi if [ -e html ]; then rm html; fi ln -s $1/django ln -s $1/docs/html

  Example:

  switch 7728
  

  This deletes existing symlinks to another revision and recreates them accordingly.

• I add the html symlinked directory to my Bookmarks Bar in Safari

• Finally, I symlink the django symlink in my local site-packages:

  1 2 3

  pushd ~/python/site-pacakges ln -s ~/python/django/django popd

Optional packages

• extensions - a repository for collecting global custom management extensions for the Django Framework. As a contributor to the repository, I would be remiss I neglected to mention it.

  1 2 3 4 5

  curl -O http://django-command-extensions.googlecode.com/files/django-command-extensions-0.3.tgz tar -xvzf django-command-extensions-0.3.tgz cd django-command-extensions-0.3 sudo python setup.py install cd ..

  - or -

  An example of how I use symlinks in my local site-packages to handle scenarios where I am frequently updating packages:

  1 2 3 4

  svn co http://django-command-extensions.googlecode.com/svn/trunk/ django-command-extensions pushd ~/python/site-packages/ ln -s ~/python/install/django-command-extensions/extensions popd

  To use in your projects, just add 'extensions' to your INSTALLED_APPS settings.py file (details).

• Pygments - a generic syntax highlighter for general use in all kinds of software such as forum systems, wikis or other applications that need to prettify source code

  1 2 3 4 5

  curl -O http://pypi.python.org/packages/source/P/Pygments/Pygments-0.10.tar.gz tar -xvzf Pygments-0.10.tar.gz cd Pygments-0.10 sudo python setup.py install cd ..

• mdx_codehilite - markdown code highlighting extension can be installed as well (requires Pygments):

  1 2 3 4 5 6

  curl -o mdx_codehilite-0.2.tar.gz \ http://gitorious.org/projects/python-markdown-extra/repos/codehilite/\ archive/f30802583958c9f54e8b4390d8dafa8dd4a147fa.tar.gz tar -xvzf mdx_codehilite-0.2.tar.gz cp -v python-markdown-extra-codehilite/mdx_codehilite.py \ /Library/Frameworks/Python.framework/Versions/Current/lib/python2.5/site-packages

  mdx_codehilite is used to added syntax highlighting markup to this blog

• dateutil - provides powerful extensions to the standard datetime module

  1 2 3 4 5

  curl -O http://labix.org/download/python-dateutil/python-dateutil-1.4.tar.bz2 tar -xvjf python-dateutil-1.4.tar.bz2 cd python-dateutil-1.4 sudo python setup.py install cd ..

  
  

• path - provides a class (path) for working with files and directories (less typing than os.path)

  1 2 3 4 5

  curl -O http://pypi.python.org/packages/source/p/path.py/path-2.2.zip unzip path-2.2.zip cd path-2.2 sudo python setup.py install cd ..

  I've used the path package for years, but many people are adopting Unipath

• twill - a scripting system for automating Web browsing. Useful for testing Web pages or grabbing data from password-protected sites automatically.

  1 2 3 4 5

  curl -O http://darcs.idyll.org/~t/projects/twill-0.9.tar.gz tar -xvzf twill-0.9.tar.gz cd twill-0.9 sudo python setup.py install cd ..

• feedparser - Parse RSS and Atom feeds in Python

  1 2 3 4 5

  curl -O http://pypi.python.org/packages/source/F/FeedParser/feedparser-4.1.tar.gz tar -xvzf feedparser-4.1.tar.gz cd feedparser-4.1 sudo python setup.py install cd ..

Miscellany

• Here are a few aliases I have set in my ~/.bashrc file for developing on Leopard:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

  #!/bin/bash # System helpers # Flush DNS - after making changes to your `/etc/hosts` file: alias flushdns='dscacheutil -flushcache' # List of open ports alias ports='sudo lsof -i -P' # Python helpers: alias site-packages='cd /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/' alias pyclean='find . -name "*.pyc" -exec rm {} \;' # Django helpers: alias runserver='python manage.py runserver' alias runserver0='python manage.py runserver 0.0.0.0:8888' alias djshell='python manage.py shell' # TextMate helpers: export SVN_EDITOR="mate -w" export EDITOR="mate -w"

From xkcd

Python is known for it's "batteries include" philosophy, meaning that the standard library that "ships" with the basic installation of the language is feature rich and ready for immediate consumption to tackle most garden variety problems from the get-go.

However, there are numerous third party modules and packages that I feel should be part of the standard distro, or at least treated as such - PEP 206 just scratches the surface of identifying some of the more useful modules that are becoming commonplace).

So here are my 11 most useful extensions that I have installed before I begin doing anything else:

1. IPython

   If you are going to be using the Python interactive interpreter environment (and you are, right?), then IPython is a must. It's not just the normal the interactive environment on steriods, it's the Barry Bonds of interactive environments. After getting used to the boat-load of features, you'll wonder how you ever got by without it.

2. MySQLdb

   If you use a MySQL database and want to interactive with it via Python APIs, you'll be needing this.

3. path

   From what I've read, Jason Orendorff's path model nearly made into the standard Python library. It's a shame it didn't, because one of the uglier Python warts in this author's opinion is the mess of file and path handler interfaces. Do I look in the os, os.path, shutil, fnmatch, or codecs module to find that file / path handling function? Forget it - just look in the path module.

4. dateutil

   For working with dates, date ranges, or recurrences of dates, this is the only library you will need. Excellent documentation with extensive example code.

   As an example, create a list of the next 3 occurrences of Friday the 13th:

   1 2 3 4 5 6 7 8 9 10 11 12

   from datetime import datetime from dateutil import rrule f13 = rrule.rrule( rrule.YEARLY, count=3, byweekday=rrule.FR, bymonthday=13, dtstart=datetime.now() ) list([d.date() for d in f13]) # [datetime.date(2008, 6, 13), datetime.date(2009, 2, 13), datetime.date(2009, 3, 13)]

5. PIL

   Working with images? Done. One caveat: you might want to save yourself some trouble and install the last JPEG library from www.ijg.org (version 6b currently).

6. wxPython

   Thinking of using Python's Tkinter interface to bestow another hideous, malformed GUI app on us all? Well, 1995 called and asked for it's cheesy look and feel back.

   Come on, wxPython is free, easy to install and use, and has great documentation in WxPython In Action.

7. BeautifulSoup

   An odd name, but great for parsing both kinds of HTML: the good and the bad.

   An example direct from the documentation:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   from BeautifulSoup import BeautifulSoup html = '<html><p>Para 1<p>Para 2<blockquote>Quote 1<blockquote>Quote 2' soup = BeautifulSoup(html) print soup.prettify() # <html> # <p> # Para 1 # </p> # <p> # Para 2 # <blockquote> # Quote 1 # <blockquote> # Quote 2 # </blockquote> # </blockquote> # </p> # </html>

8. docutils

   docutils has become, in my opinion, the de facto method for marking up Python documentation, to include __doc__ strings. It excels at processing reStructuredText for HTML or LaTex output.

9. PyXML

   I'm not quite sure why half of the XML functionality was pruned from the default Python standard library, but this is the other half. There are other XML libraries (libxml, pyRXP, 4Suite) that may suit your particular needs better, but as a minimum you should consider getting at least the full PyXML suite.

10. py2app / py2exe

    These libraries extend the distutils package to produce executable binaries for Mac OS X and Windows, respectively.

11. Markdown

    Markdown has become a daily part of my development and online life. Many bloggers have begun offering comments in Markdown format (this site included). I find extremely useful for purposes of documentation and more formally structured email because even unprocessed, it remains very readable and elegent as opposed to raw wiki syntax (which I find to be less decipherable than HMTL in many cases, particularly MediaWiki) or even reStructuredText (a bit too structured and esoterica for common everyday usage).

    John Gruber of Daring Fireball started Markdown and provides nice documentation and tools.