Jannis Leidel

Dear Djangonauts,

I’m a little baffled right now and I guess you all have a lot to do, so please forgive me if I’m disturbing. Going back to the begin of the Google summer of code, my mentor James and I discussed the best strategy to start implementing reusable Django-apps and the so-called “Hotclub of France” conventions (e.g. a public web repository for reusable apps). Unfortunately, a real discussion did not really take off, neither on the discussion list nor on IRC.

It now seems to me as it could just be to early for Django to have such a repository. Everybody, including me, loves to fiddle with Django in some degree and is suspicious of automatic and ready-made products. After creating ticket #6080 I’m a little tired of bringing up the same arguments again and again on the net. That’s why I wanted to ask you, if you generally see a chance for this to come before Django 1.0 and how you handle this at your projects if at all.

To continue my last post about reusable Django apps, I’d like to talk about setuptools, show you a way to install Django by using setuptools and propose the reintroduction into Django’s code, even if setuptools have been dumped by the developers not long ago due to manageability reasons. Features like entry points, egg-files, the develop command, automatic versioning and tight integration with the Python Package Index make it worth installing.

Click here to read more

django-reusableapps is yet another approach on enabling Django to load reusable, pluggable, egg-based applications without changing the Django sourcecode. Think of plugins or components, e.g. django-registration, django-voting, django-tagging etc.

It uses setuptools for finding, handling and loading egg-based Python modules with a certain “entry point” ('django.apps').

Egg-based Python modules (a.k.a. eggs) are compressed packaged Python modules like Django apps (and “[..] to Pythons as Jars are to Java…”). Every Django app can be converted to an egg distribution by using a special setup.py file.

I moved my code repository from code.google.com/p/django-package/ to websushi.org/trac/

Please also feel free to test django-package.diff with your Django trunk checkout:

1. wget http://websushi.org/bzr/django-package/django-package.diff
2. cp django-package.diff /path/to/django_src
3. cd /path/to/django_src
4. patch -p0 < django-package.diff

django-admin.py startapp now behaves differently, asking for metadata when creating a new setuptools affine Django application. Editing is done by running django-admin.py editapp in the application directory. The old way of creating “apps” is still valid via django-admin.py startapp --noskeleton.

I also started working on the Django application repository application itself, wondering what Ross is doing at djangoapps.org?

I’m calling for your ideas about the name of the Django application repository:

My proposals:

• djangopackages.com
• djangoapps.com
• djangoapps.org
• djangoapplications.com
• djangocontrib.com
• djangoforge.com
• djangoshop.com
• djangoproject.com/packages/
• djangoproject.com/apps/

just dumping ideas about the realization of a public application repository:

goals:

• central Django application repository
• integration with setuptools via django-admin.py
• easy download and searchable

requirements:

• file management (aka upload)
• community features
• style
• application based (eat your own..)

future:

• Paste hooks
• SVN/Bazaar/.. hooks

implementation:

1. standard setuptools procedures are used to register & upload the release of a django app to the Cheesshop
2. uploaded files have all the same keyword “django.contrib”
3. the homepage URL in the release.py of each django app is a sluggified url to the public repository entry (e.g. http://djangopackages.com/p/django-registration/)
4. a standalone Django application scans periodically the Cheesshop and adds new applications with the keyword “django.contrib” to the repository database (can be cron, signals and/or something else) see: cheeseshop_import.py (cheeserater.com, thanks to Jakob)
5. built with the common django-* application to provide standard community features (now on Google Code):
   django-registration
   django-openid
   django-voting or django-score-voting
   django-tagging
   django-utils
   django-contact-form
   typogrify
   cab (?)
   django-captcha (?)
   django-discussion (?)
   django-profile-images (?)
6. needed models: Package, Owner

questions

• would it make sense to wrap the register and upload process? (e.g. django-admin.py uploadapp)
• encourage BestPracticesToWorkWith3rdPartyAppsAndMakingYoursPortable !/?
• should dependencies be shown in the repository? are owners allowed to pick?
• if needed, what is the best way to administrate the repository editorially?

I’m catching up right now with my Summer of Code project after some horrible exams and have now the time to read the very interesting discussions on the Django developers list.

First, as you may know I wrote some code which changes django.core.management (django-admin.py/manage.py) to ask for basic metadata as soon as the user creates a new django application with “startapp”. The metadata gets stored in a release.py for easy manipulation. A setup.py is also provided to use any functionality of the setuptools.

Unfortunately the problem isn’t just a matter of the actual process of packaging. As pointed out in the discussion about best practices and default application layouts it’s mere the problem of giving the not-so-pro-developers the chance to use pre-built applications without forgetting the needs of the pros.

One proposal was a better documentation of best practices to help beginners understand why “projects” are good to dive into django but are awkward when it comes to distribution and deployment. A great default application scheme was written by Sebastian Macias, even if it confused me a little at first because it imitates Python’s “site-packages”..

I’m sure some of you already have solutions for distribution/deployment because it is obligatory for high traffic sites (or is it?). Server arrangements are pretty cool too, but there must be more. Django applications should be treated like applications not like websites.

Anyway the fact remains: a standard system to manage Django-based modules is required, whether we call them apps, snippets, components, extensions or plugins (think of Wordpress). I proposed to use setuptools even if a lot of people seem to be annoyed about it (why exactly?).

Turbogears and Pylons are using Paste and Setuptools in combination to provide reusability and deployment - considering the different code structure and software strategy a smart decision. Django may need some own handlers to solve the different aspects of this whole topic. Why not rethink the way Django gets deployed?

UPDATE:
You may be interested in reading this: Ian Bicking’s view on the failures of setuptools and python packaging. Well then.. so then just don’t do for Django either? Naah..

This week I continued to work on the startapp command of
django.core.management and changed its default behaviour to create a
standalone application (with skeleton files) when you run: django-
admin.py startapp myapp. Current skeleton files are: release.py,
setup.py, MANIFEST.in, docs/, test/, myapp/, myapp/templates/myapp.
Are there any generic files/folders to add?

Besides editing release.py manually you can now edit the meta
information by running inside an application dir: django-admin.py
editapp. This also moves the app directory on a name change.

You can find a ReleaseWrapper dictionary object at
django.utils.package which can be used to read and edit your release
manually:

>> import os
>>> from django.utils.package import ReleaseWrapper, DEFAULT_DIRECTORIES, DEFAULT_FILES
>>> release = ReleaseWrapper(os.getcwd())
>>> print (release.NAME, release.VERSION)
('myapp', '0.1')
>>> release.previous_name = release.NAME
>>> release.NAME = "myapp2"
>>> release['VERSION'] = “0.2″
>>> release.update(os.getcwd(), DEFAULT_DIRECTORIES, DEFAULT_FILES)
Moved: /Users/Jannis/Desktop/test_app/myapp to /Users/Jannis/Desktop/
test_app/myapp2
Created: /Users/Jannis/Desktop/test_app/myapp2/templates/myapp2
Written: /Users/Jannis/Desktop/test_app/release.py
Written: /Users/Jannis/Desktop/test_app/setup.py
Written: /Users/Jannis/Desktop/test_app/MANIFEST.in

The application is ready to be used by setuptools, e.g. creating a zip-
like “egg” file (python setup.py bdist_egg) which then can be
distributed and installed with setuptools’ easy_install command.

Creating a stripped down project-based application (__init__.py, models.py,
views.py), which is currently recommended by the tutorials, run:

django-admin.py --noskeleton startapp mysimpleapp

Please have a look at http://code.google.com/p/django-package/ for
further details, full installation instructions and of course the
patch.

And please, tell me what you think :)

Here is the first update on my Google Summer of Code 2007 project-a simple rip-off of my mail to the developer list.
Here you can find the new django-package project with the current patch:

http://code.google.com/p/django-package/

For now the patch adds a command line switch to the startapp command of django.core.management:
# django-admin.py --package startapp myapp
which asks for the meta information required for packaging. Please try it out!

It creates a basic skeleton application in the current directory, ready to be used by setuptools.
For testing purposes you can use setuptools standard functions like running python setup.py develop to add it to the python path, uninstalling with python setup.py develop --uninstall or wrapping it in an egg file with python setup.py bdist_egg.

The application skeleton

I would like to collect some of your ideas about the common file names and directories which should be created by default. This is also needed when it comes to packaging non-python files like template html-files, sql-files and so on. What are the common filenames? How do you manage your django projects?

As suggested by Neil there is already a release.py for easier manipulation of the meta data. The current code creates this:

# ls
MANIFEST.in
docs/
release.py
setup.py
test/
myapp
myapp/templates/

The new workflow

When introducing something like package management we also have to think about changing the workflow.

Current workflow:

1. startproject
2. startapp
3. development and testing
4. semi-automatic deployment

Proposed workflow:

1. startapp (with package skeleton and meta information)
2. develop application and test with “python setup.py develop”
3. egg deployment with “python setup.py bdist_egg” and “easy_install myapp-0.1.egg”

Generic applications like django-registration, django-tagging, django-voting, django-utils, django-contact-form, django-forum.. (more) can be installed system wide and therefore live on the python path (in python’s standard site-packages directory).

I’d like to encourage new developers to follow my workflow proposal as it makes development, packaging and deployment much easier.

Hence I suggest making it the default behaviour of startapp to create standalone packages, while requiring a command-line switch to create an application without the skeleton files (the current “develop inside a project” style, like in the tutorials).

One more thing..

One hypothetic idea is to introduce a whole new management command package to wrap most of the current functions like startproject, startapp and the ideas before. This would wrap some of setuptools functions and could have a feature-set like: package project, package app, package egg, package upload and so on. I actually do think this is evil since it wraps functions of setuptools but your mileage may vary.
Tell me what you think!

UPDATE: Err, wOOOt! My application was today officially accepted by Google and Django’s main developers. Thank you for your support and the confidence you have in me.
–
Hi. This weblog is about my summer of code 2007. Please look at my curriculum vitae to get to know me better.

My formal application for Google’s Summer of Code is as follows:

Django is one of the major web frameworks that emerged within the last years and created a solid user and developer base. It simplifies the development process by providing tools to reduce repetition, abstracting common web paradigms and still being hackable. Until today that attracted a wide section of programmers, from novices to pros.

As the code base is constantly moving towards the 1.0 release, more and more people are using Django to build small projects and websites, just as well as commercial applications like intranet services and content managment systems are realized with it, too. Good for them:

Reinventing the wheel with Django is perfectly easy – every Django beginner wrote a todo list or weblog applicaton :)

But this is not the end of the road for user participation. The community should be able to uncover more web ideas for contrib apps, combine the already written views and templates to more effecient apps and simply share their products with other users.

The implementation of a package system would lower the threshold for Django and Python beginners significantly because it reduces the hassle of the current installation procedure.

My task is to do the groundwork for an application which manages Django application packages and integrates tightly with the Django code.

The following steps should illustrate my considerations:

1. Infrastructure (6 weeks)

   Define hooks in the Django code to standardized python modules (e.g. setuptools, distutils, py2exe, py2app etc.) because they solve a lot of software management problems such as dependency tracking and version management. Then implement generic packaging functionality for use via django-admin.py/manage.py: CRUD, upload, download, install package.

   Example of a first-time installation of a Django app:

   tichy:~# django-admin.py package search Satchmo
Found 1 package on djangoapps.com:
1. satchmo - The webshop for perfectionists with deadlines.
tichy:~# django-admin.py package install satchmo
Installation directory for satchmo? [/home/jannis/django/satchmo/]
Installing satchmo (/home/jannis/django/satchmo/)……..Done.
Syncing sqlite database with initial data…Done.
Running tests…. 0 errors found.
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.


2. Website (4 weeks)

   Build a community website with a searchable application repository (e.g. title, description, rating, user) and provide several download options (egg, tar.gz, zip, exe, app, deb, rpm etc.). Then build interface to commandline package management as defined by setuptools and distutils.

3. Documentation (2 weeks)

   Add documentation about using the package function of manage.py, the community website (e.g. djangoapps.com) and write a tutorial about making Django applications packable (e.g. by including package information in a package.py file in the root directory of a Django project or app).

4. Bonus

   Include support for version control systems like subversion or bazaar (bzr) to encourage active development on the community website. Nightly builds and application documentation are some of the possible features.