← An autocomplete widget for django-tagging ...

An autocomplete form widget for ForeignKey model fields

In my last post I showed you how to write your own widget for django-tagging form fields that uses the nifty jQuery autocomplete plugin to simplify entering tags. This time I’d like to take on a suggestion PatricK made in the comments:

my question is, if this could be done on a more abstract level for enhancing relations. when having a foreignkey, you get the browse-icon (lens) right near the input-field. it´d be awesome to search the relations just by typing something into the field … the autocomplete-functionality should then search the related entries on the basis of the defined search-fields.

Well, this is what I’m going to do today :)

Preface

Thanks to the way Django’s forms work, the following implementation should be applicable to your own Django code — but please bare in mind that this isn’t a copy and paste how-to. I rather encourage you to learn about the flexibility of forms, fields and widgets and how it’s used in the automatic Admin interface. You’ll be able to — and sometimes required — to customize the code posted below.

As an example I’m going to use the message system that is included in Django’s auth contrib app. As far as I know Django doesn’t provide an admin representation for the Message class by default because it’s supposed to be used programmatically (and is used by the admin itself, too). The model consists of two fields, a ForeignKey to a User and a text field for the message.

Use case

Since ForeignKey fields are rendered as select fields by default there is a good chance that they become unusable if there are a lot of possible choices. So in case you have a lot of users in your database for example it might be a bit cumbersome to choose a user from the ForeignKey dropdown menu. Django provides an option that allows specifying a raw_id_fields attribute in your ModelAdmin subclass that will tell Django to render the ForeignKey as a simple input taking the primary key of the refered object. The actual selection is done by clicking the browse icon next to the field and browsing the model with the standard admin interface in a popup. What I want to provide here though is a way to do that directly in place, in the original form.

Widget

We are going to use the jQuery and the great Autocomplete plugin again which does all the hard work. Please download the sources of the autocomplete plugin if you don’t have it already. It includes everything we need to get started. Copy all files and folders of the extracted jQuery Autocomplete zip file to the directory you specified in the MEDIA_ROOT setting. It should then contain: ‘jquery.autocomplete.js’, ‘jquery.autocomplete.css’ and a ‘lib’ directory with the other necessary files. In case you arrange your files differently don’t hesitate to change the paths below in the inner Media class.

It has been proven to be a good idea to put something like the follwing widget in the widgets.py of your Django app, but any place will do if you change the import paths. 

from django import forms
from django.conf import settings
from django.utils.safestring import mark_safe
from django.utils.text import truncate_words

class ForeignKeySearchInput(forms.HiddenInput):
    """
    A Widget for displaying ForeignKeys in an autocomplete search input 
    instead in a ``select`` box.
    """
    class Media:
        css = {
            'all': ('jquery.autocomplete.css',)
        }
        js = (
            'lib/jquery.js',
            'lib/jquery.bgiframe.min.js',
            'lib/jquery.ajaxQueue.js',
            'jquery.autocomplete.js'
        )

    def label_for_value(self, value):
        key = self.rel.get_related_field().name
        obj = self.rel.to._default_manager.get(**{key: value})
        return truncate_words(obj, 14)

    def __init__(self, rel, search_fields, attrs=None):
        self.rel = rel
        self.search_fields = search_fields
        super(ForeignKeySearchInput, self).__init__(attrs)

    def render(self, name, value, attrs=None):
        if attrs is None:
            attrs = {}
        rendered = super(ForeignKeySearchInput, self).render(name, value, attrs)
        if value:
            label = self.label_for_value(value)
        else:
            label = u''
        return rendered + mark_safe(u'''
            <style type="text/css" media="screen">
                #lookup_%(name)s {
                    padding-right:16px;
                    background: url(
                        %(admin_media_prefix)simg/admin/selector-search.gif
                    ) no-repeat right;
                }
                #del_%(name)s {
                    display: none;
                }
            </style>
<input type="text" id="lookup_%(name)s" value="%(label)s" />
<a href="#" id="del_%(name)s">
<img src="%(admin_media_prefix)simg/admin/icon_deletelink.gif" />
</a>
<script type="text/javascript">
            if ($('#lookup_%(name)s').val()) {
                $('#del_%(name)s').show()
            }
            $('#lookup_%(name)s').autocomplete('../search/', {
                extraParams: {
                    search_fields: '%(search_fields)s',
                    app_label: '%(app_label)s',
                    model_name: '%(model_name)s',
                },
            }).result(function(event, data, formatted) {
                if (data) {
                    $('#id_%(name)s').val(data[1]);
                    $('#del_%(name)s').show();
                }
            });
            $('#del_%(name)s').click(function(ele, event) {
                $('#id_%(name)s').val('');
                $('#del_%(name)s').hide();
                $('#lookup_%(name)s').val('');
            });
            </script>
        ''') % {
            'search_fields': ','.join(self.search_fields),
            'admin_media_prefix': settings.ADMIN_MEDIA_PREFIX,
            'model_name': self.rel.to._meta.module_name,
            'app_label': self.rel.to._meta.app_label,
            'label': label,
            'name': name,
        }

This widget renders ForeignKeys as hidden inputs and adds a second input field to search in the related model using asynchronous requests, sometimes refered to as AJAX. Once the user selects a result the widget will automatically set the hidden field to the primary key. When editing an existing entry this will automatically populate the search field with the correct verbose value.

Admin integration

As already stated we are going to create an admin class for the Message model to enable admins to create messages for other users that will be displayed the next time they use the admin (or any other site that uses User.get_and_delete_messages).

The best place for the admin class is an admin.py file in one of your own app directories because it’s then picked up by Django’s autodiscover() function. 

import operator
from django.db import models
from django.contrib.auth.models import Message
from django.http import HttpResponse, HttpResponseNotFound
from django.contrib import admin
from django.db.models.query import QuerySet
from django.utils.encoding import smart_str

from yourapp.widgets import ForeignKeySearchInput

class MessageAdmin(admin.ModelAdmin):
    list_display = ('user', 'message')
    related_search_fields = {
        'user': ('username', 'email'),
    }

    def __call__(self, request, url):
        if url is None:
            pass
        elif url == 'search':
            return self.search(request)
        return super(MessageAdmin, self).__call__(request, url)

    def search(self, request):
        """
        Searches in the fields of the given related model and returns the 
        result as a simple string to be used by the jQuery Autocomplete plugin
        """
        query = request.GET.get('q', None)
        app_label = request.GET.get('app_label', None)
        model_name = request.GET.get('model_name', None)
        search_fields = request.GET.get('search_fields', None)

        if search_fields and app_label and model_name and query:
            def construct_search(field_name):
                # use different lookup methods depending on the notation
                if field_name.startswith('^'):
                    return "%s__istartswith" % field_name[1:]
                elif field_name.startswith('='):
                    return "%s__iexact" % field_name[1:]
                elif field_name.startswith('@'):
                    return "%s__search" % field_name[1:]
                else:
                    return "%s__icontains" % field_name

            model = models.get_model(app_label, model_name)
            qs = model._default_manager.all()
            for bit in query.split():
                or_queries = [models.Q(**{construct_search(
                    smart_str(field_name)): smart_str(bit)})
                        for field_name in search_fields.split(',')]
                other_qs = QuerySet(model)
                other_qs.dup_select_related(qs)
                other_qs = other_qs.filter(reduce(operator.or_, or_queries))
                qs = qs & other_qs
            data = ''.join([u'%s|%s\n' % (f.__unicode__(), f.pk) for f in qs])
            return HttpResponse(data)
        return HttpResponseNotFound()

    def formfield_for_dbfield(self, db_field, **kwargs):
        """
        Overrides the default widget for Foreignkey fields if they are
        specified in the related_search_fields class attribute.
        """
        if isinstance(db_field, models.ForeignKey) and \
                db_field.name in self.related_search_fields:
            kwargs['widget'] = ForeignKeySearchInput(db_field.rel,
                                    self.related_search_fields[db_field.name])
        return super(MessageAdmin, self).formfield_for_dbfield(db_field, **kwargs)
admin.site.register(Message, MessageAdmin)

So this code does several things:

  • It inherits from the default ModelAdmin to be able to override the default widget.
  • A new related_search_fields class attribute that is a mapping between lowercase model names and a list of fields to searched in. It uses the same syntax as the search_fields attribute to speed up the lookups or restrict them if needed. The following example would only match the beginning of the first and last name of users: 
related_search_fields = {
    'user': ('^first_name', '^last_name'),
}

  • It overrides the __call__ method to add additional URL handler to the admin to be used by the widget for searching. It will be waiting for requests on /<root-admin-path>/auth/message/search/.
  • The search view that constructs the querysets depending on the provided model and search fields and returns the data as a simple string. The Autocomplete plugin can parse that and will display it in the selection.
  • A formfield_for_dbfield method that will override the widget of any ForeignKey field whose name is specified in the related_search_fields class attribute.

Once assembled the result will mostly look like that:

Please note that I use a different stylesheet for the Autocomplete plugin here.

Django, Python Nov. 14, 2008, 11:57 a.m. comments (31)

comments

Martin Geber Nov. 14, 2008, 11:25 a.m.

Great post, thanks for that.

Martin Nov. 14, 2008, 12:13 p.m.

That's awesome. I really like the way that you handle the ajax-search through model-admin, not through an "external" view.

Unfortunately, so it's impossible to use that on frontend-forms (forms.Form) but I'm sure that wasn't your target.

You should swap this snippets out in an own mini-application. .. and I like screencasts. :-)

Martin Nov. 14, 2008, 12:18 p.m.

Correction. It's not impossible, I just have to change the url-path to the search view from "../search/" to an absolute url. Should be easy.

Thanks a lot for this, I'll going to check this out on a real-world-example.

Marc Fargas Nov. 14, 2008, 1:02 p.m.

Pretty nice, Thanks.
Just a comment: I'd rather put all this render() stuff in a js function, and just create the input box in the render().

The js function could be fired on document.ready() and look for a special class. That leave the HTML source much cleaner ;))

Jannis Leidel Nov. 14, 2008, 1:14 p.m.

@Martin Thanks! Yeah, I'll think about an own app for those kind of things :)

@Marc Indeed, that would make it even more reusable. And as I said, the code above is an example and should only be seen as an starting point.

Arthur Nov. 14, 2008, 1:45 p.m.

Awesome tutorial, thanks a lot! Definately going to implement this in some projects.

patrickk Nov. 14, 2008, 2:06 p.m.

great stuff.

I´ve just implemented the script(s) and here are my notes:
1. it´d be nice if the search-icon (lens) could still be used.
2. currently, it only works if the name of the related model (model_name) is the same as the name of the field. in my case, the field "author" refers to the model "user".
this can be solved with
model_name: '%(model_name)s'
and
'model_name': self.rel.to._meta.module_name
within the render-function.
well, at least it works for me.

Jannis Leidel Nov. 14, 2008, 2:29 p.m.

@patrickk: Regarding 1., do you mean like the raw-id widget Django is providing?

2. You are right about the second point, I just added those to the code above. Thanks!

patrickk Nov. 14, 2008, 2:38 p.m.

@jannis: yes, I mean like the raw-id widget. I´m not sure it´s really necessary, but sometimes it might be easier to use the search-icon instead of typing something.

another note: wouldn´t it be helpful if there is a helptext below the input-field which says something like "Searching: First Name, Last Name ...". I have to deal with this feedback (from my customers) on a regular basis when they complain about the change-list, because it´s not obvious what fields are going to be searched ...

AJ Nov. 14, 2008, 5:38 p.m.

To expand on MARC's comment, make sure the result is still usable on a browser that has Javascript disabled; the autocompletion obviously won't work, but as long as the user types in a valid username the form should still be able to operate normally. I haven't studied your code closely enough to tell whether it will or not.

Ben Nov. 16, 2008, 5:40 p.m.

Very helpful! I looked through several other pages, but after reading your explanation, everything made sense and I've got it working on my site now.

One question. When adding a new foreign key by clicking on the "+" the hidden field is correctly updated with the primary key of the new object, but the autocomplete field still shows the previous text which is confusing to the user. The insertion still works correctly of course as it's the hidden field that is actually used. Any ideas on how to correct this? It seems the JavaScript code RelatedObjectLookups.js controls this behavior, but I can't quite get my head around what to do.

Also, what is the purpose of the label_for_value() function. It's seems that you are just removing the <strong> tag is that all?

Thanks again for helping us all out!

Cheers,
Ben

Vermus Nov. 20, 2008, 8:56 p.m.

Thanks!
What do you say about ManyToManyField of model?
f.e. with Autocomplete.js options={'multiple': True, 'multipleSeparator':', ' } ...
Is too hard to do this?

Jannis Leidel Nov. 21, 2008, 2:25 p.m.

@VERMUS: Yeah, that could work, I had this in mind while working on the ForeignKeyWidget, too.

Orcun Nov. 23, 2008, 5:02 a.m.

its really very helpful. it would also be great using a smilar widget with ManyToManyFields by putting commas between values.

Orcun Nov. 23, 2008, 5:21 a.m.

i think best usage is to create a WidgetMedia and multiple inherit WidgetMedia and admin.ModelAdmin. its fast for using it for many models and putting all code together.

by the way the code doesn't work if you define model with __str__ method instead of __unicode__ representation.

varikin March 3, 2009, 10:37 p.m.

Very cool

Philipp Bosch March 4, 2009, 1:55 a.m.

This approach will not work with the new way of URL handling in the admin as of changeset 9739 if you go for the new recommended style of using include(admin.site.urls) instead of admin.site.root in urls.py. I think the way you handle the search URL in __call__() in the MessageAdmin class would need to replaced by something else. But I'm not clever enough to come up with a solution :) Maybe someone else is?

CE Lopes March 13, 2009, 3:54 p.m.

I second what Philipp is saying. Not only it will be incompatible soon enough, but it actually won't work with tabularinline models as of the most recent SVN...

Eric March 13, 2009, 7:25 p.m.

Would it be possible to do something siilar with InlineAdminModelForms??

Margie May 13, 2009, 4:28 a.m.

Hi, I am trying to work through this tutorial using the downloaded 1.1 Beta. It doesn't seem to work for me and I'm wondering if it's bcause I am already past changeset 9739? It would be great to have a version that works on the current 1.1 Beta, I'm too much of a newbie at this stuff to understand it without being able to have it working and step through it.

Regardless - Jannis, I really learned a lot from your other examples, this is great stuff. Thank you!
Margie

Margie May 13, 2009, 3:57 p.m.

Ignore my previous question/comments about problems on 1.1 Beta - your code works fine. I really learned a ton from this, thanks so much for posting it.

Margie

Christian Abbott June 24, 2009, 3:11 a.m.

As Philipp pointed out, __call__() is now deprecated. The new function which handles URLs is the aptly named get_urls(). So, long story short, replace the __call__() function in the MessageAdmin class with::

def get_urls(self):
urls = super(MessageAdmin,self).get_urls()
search_url = patterns('',
(r'^search/$', self.search)
)
return search_url + urls

Also, the view passed to the patterns function ("self.search" in the above code) will not be limited to only authorized users unless you wrap it in this function::
self.admin_site.admin_view()

So if you want to create this restriction (and you probably do, since this is in the AdminSite), change the line in the new code above from::
(r'^search/$', self.search)
to::
(r'^search/$', self.admin_site.admin_view(self.search))

The django documentation for this change is located on this webpage:
http://docs.djangoproject.com/en/dev/ref/contrib/admin/

Hope this helps others!

bob Aug. 16, 2009, 8:10 a.m.

Is there any reason why theres a </select> at the end of the file like that?

Jannis Leidel Aug. 17, 2009, 12:26 a.m.

Huh, that was a Pygments "feature" closing a <select> in the code. It's fixed now. Thanks.

daria Aug. 18, 2009, 11:35 p.m.

czOoST Gra7noI59Unral92Bb7wf

butm Aug. 22, 2009, 1:32 p.m.

Great, thanx!

Only one note, I had to include 'from django.conf.urls.defaults import *' in my admin.py in order to use "patterns".

akaihola Oct. 13, 2009, 10:53 a.m.

For other auto-complete solutions, see http://code.djangoproject.com/wiki/AutoCompleteSolutions

eka Nov. 23, 2009, 9:10 p.m.

How this can be applied to a GenericStackedInline ?

Sameer Nov. 25, 2009, 7:46 a.m.

Just wanted to let everyone who is still struggling with this solution know that...
CHRISTIAN ABBOTT's solution above works like a charm for Django 1.1

Thanks Jannis for the original solution & thank you Christian for the Django 1.1 fix.

eka Nov. 25, 2009, 8:32 p.m.

Any hint on how to make this work on an Inline?

Regards

Alexandre Dec. 1, 2009, 8:39 p.m.

I don't know what happened but I had changed the code as Christian Abbott said (after upgrading to 1.1) but all of a sudden it stop working...

so I went back to the original code...

And everything works now...

Lifefeed