Python 2.4 Decorators

Phillip is the author of the open-source Python libraries PEAK and PyProtocols, and has contributed fixes and enhancements to the Python interpreter. He is the author of the Python Web Server Gateway Interface specification (PEP 333). He can be contacted at [email protected].

As software environments become more complex and programs get larger, it becomes more and more necessary to find ways to reduce code duplication and scattering of knowledge. While simple code duplication is easy to factor out into functions or methods, more complex code duplication is not. For example, if a method needs to be wrapped in a transaction, synchronized in a lock, or have its calls transmitted to a remote object, there often is no simple way to factor out a function or method to be called, because the part of the behavior that varies needs to be wrapped inside the common behavior.

A second and related problem is scattering of knowledge. Sometimes a framework needs to be able to locate all of a program's functions or methods that have a particular characteristic, such as "all of the remote methods accessible to users with authorization X." The typical solution is to put this information in external configuration files, but then you run the risk of configuration being out of sync with the code. For example, you might add a new method, but forget to also add it to the configuration file. And of course, you'll be doing a lot more typing, because you'll have to put the method names in the configuration file, and any renaming you do requires editing two files.

So no matter how you slice it, duplication is a bad thing for both developer productivity and software reliability—which is why Python 2.4's new "decorator" feature lets you address both kinds of duplication. Decorators are Python objects that can register, annotate, and/or wrap a Python function or method.

For example, the Python atexit module contains a register function that registers a callback to be invoked when a Python program is exited. Without the new decorator feature, a program that uses this function looks something like Listing One(a).

When Listing One(a) is run, it prints "Goodbye, world!" because when it exits, the goodbye() function is invoked. Now look at the decorator version in Listing One(b), which does exactly the same thing, but uses decorator syntax instead—an @ sign and expression on the line before the function definition.

This new syntax lets the registration be placed before the function definition, which accomplishes two things. First, you are made aware that the function is an atexit function before you read the function body, giving you a better context for understanding the function. With such a short function, it hardly makes a difference, but for longer functions or methods, it can be very helpful to know in advance what you're looking at. Second, the function name is not repeated. The first program refers to goodbye twice, so there is more duplication—precisely the thing we're trying to avoid.

Why Decorate?

The original motivation for adding decorator syntax was to allow class methods and static methods to be obvious to someone reading a program. Python 2.2 introduced the classmethod and staticmethod built-ins, which were used as in Listing Two(a). Listing Two(b) shows the same code using decorator syntax, which avoids the unnecessary repetitions of the method name, and gives you a heads-up that a classmethod is being defined.

While this could have been handled by creating a syntax specifically for class or static methods, one of Python's primary design principles is that: "Special cases aren't special enough to break the rules." That is, the language should avoid having privileged features that you can't reuse for other purposes. Since class methods and static methods in Python are just objects that wrap a function, it would not make sense to create special syntax for just two kinds of wrapping. Instead, a syntax was created to allow arbitrary wrapping, annotation, or registration of functions at the point where they're defined.

Many syntaxes for this feature were discussed, but in the end, a syntax resembling Java 1.5 annotations was chosen. Decorators, however, are considerably more flexible than Java's annotations, as they are executed at runtime and can have arbitrary behavior, while Java annotations are limited to only providing metadata about a particular class or method.

Creating Decorators

Decorators may appear before any function definition, whether that definition is part of a module, a class, or even contained in another function definition. You can even stack multiple decorators on the same function definition, one per line.

But before you can do that, you first need to have some decorators to stack. A decorator is a callable object (like a function) that accepts one argument—the function being decorated. The return value of the decorator replaces the original function definition. See the script in Listing Three(a), which produces the output in Listing Three(b), demonstrating that the mydecorator function is called when the function is defined.

For the first example decorator, I had it return the original function object unchanged, but in practice, it's rare that you'll do that (except for registration decorators). More often, you'll either be annotating the function (by adding attributes to it), or wrapping the function with another function, then returning the wrapper. The returned wrapper then replaces the original function. For example, the script in Listing Four prints "Hello, world!" because the does_nothing function is replaced with the return value of stupid_decorator.

Objects as Decorators

As you can see, Python doesn't care what kind of object you return from a decorator, which means that for advanced uses, you can turn functions or methods into specialized objects of your own choosing. For example, if you wanted to trace certain functions' execution, you could use something like Listing Five.

When run, Listing Five prints "entering" and "exiting" messages around the "Hello, world" function. As you can see, a decorator doesn't have to be a function; it can be a class, as long as it can be called with a single argument. (Remember that in Python, calling a class returns a new instance of that class.) Thus, the traced class is a decorator that replaces a function with an instance of the traced class.

So after the hello function definition in Listing Five, hello is no longer a function, but is instead an instance of the traced class that has the old hello function saved in its func attribute.

When that wrapper instance is called (by the hello() statement at the end of the script), Python's class machinery invokes the instance's __call__() method, which then invokes the original function between printing trace messages.

Stacking Decorators

Now that we have an interesting decorator, you can stack it with another decorator to see how decorators can be combined.

The script in Listing Six prints "Called with <class '__main__.SomeClass'>", wrapped in "entering" and "exiting" messages. The ordering of the decorators determines the structure of the result. Thus, someMethod is a classmethod descriptor wrapping a traced instance wrapping the original someMethod function. So, outer decorators are listed before inner decorators.

Therefore, if you are using multiple decorators, you must know what kind of object each decorator expects to receive, and what kind of object it returns, so that you can arrange them in a compatible wrapping order, so that the output of the innermost decorator is compatible with the input of the next-outer decorator.

Usually, most decorators expect a function on input, and return either a function or an attribute descriptor as their output. The Python built-ins classmethod, staticmethod, and property all return attribute descriptors, so their output cannot be passed to a decorator that expects a function. That's why I had to put classmethod first in Listing Four. As an experiment, try reversing the order of @traced and @classmethod in Listing Four, and see if you can guess what will happen.

Functions as Decorators

Because most decorators expect an actual function as their input, some of them may not be compatible with our initial implementation of @traced, which returns an instance of the traced class. Let's rework @traced such that it returns an actual function object, so it'll be compatible with a wider range of decorators.

Listing Seven provides the same functionality as the original traced decorator, but instead of returning a traced object instance, it returns a new function object that wraps the original function. If you've never used Python closures before, you might be a little confused by this function-in-a-function syntax.

Basically, when you define a function inside of another function, any undefined local variables in the inner function will take the value of that variable in the outer function. So here, the value of func in the inner function comes from the value of func in the outer function.

Because the inner function definition is executed each time the outer function is called, Python actually creates a new wrapper function object each time. Such function objects are called "lexical closures," because they enclose a set of variables from the lexical scope where the function was defined.

A closure does not actually duplicate the code of the function, however. It simply encloses a reference to the existing code, and a reference to the free variables from the enclosing function. In this case, that means that the wrapper closure is essentially a pointer to the Python bytecode making up the wrapper function body, and a pointer to the local variables of the traced function during the invocation when the closure was created.

Because a closure is really just a normal Python function object (with some predefined variables), and because most decorators expect to receive a function object, creating a closure is perhaps the most popular way of creating a stackable decorator.

Decorators with Arguments

Many applications of decorators call for parameterization. For example, say you want to create a pair of @require and @ensure decorators so that you can record a method's precondition and postcondition. Python lets us specify arguments with our decorators; see Listing Eight. (Of course, Listing Eight is for illustration only. A full-featured implementation of preconditions and postconditions would need to be a lot more sophisticated than this to deal with things like inheritance of conditions, allowing postconditions to access before/after expressions, and allowing conditions to access function arguments by name instead of by position.)

You'll notice that the require() decorator creates two closures. The first closure creates a decorator function that knows the expr that was supplied to @require(). This means require itself is not really the decorator function here. Instead, require returns the decorator function, here called decorator. This is very different from the previous decorators, and this change is necessary to implement parameterized decorators.

The second closure is the actual wrapper function that evaluates expr whenever the original function is called. Try calling the test() function with different numbers of arguments, and see what happens. Also, try changing the @require line to use a different precondition, or stack multiple @require lines to combine preconditions. You'll also notice that @require(expr="len(__args)==1") still works. Decorator invocations follow the same syntax rules as normal Python function or method calls, so you can use positional arguments, keyword arguments, or both.

Function Attributes

All of the examples so far have been things that can't be done quite so directly with Java annotations. But what if all you really need is to tack some metadata onto a function or method for later use? For this purpose, you may wish to use function attributes in your decorator.

Function attributes, introduced in Python 2.1, let you record arbitrary values as attributes on a function object. For example, suppose you want to track the author of a function or method, using an @author() decorator? You could implement it as in Listing Nine. In this example, you simply set an author_name attribute on the function and return it, rather than creating a wrapper. Then, you can retrieve the attribute at a later time as part of some metadata-gathering operation.

Practicing "Safe Decs"

To keep the examples simple, I've been ignoring "safe decorator" practices. It's easy to create a decorator that will work by itself, but creating a decorator that will work properly when combined with other decorators is a bit more complex. To the extent possible, your decorator should return an actual function object, with the same name and attributes as the original function, so as not to confuse an outer decorator or cancel out the work of an inner decorator.

This means that decorators that simply modify and return the function they were given (like Listings Three and Nine), are already safe. But decorators that return a wrapper function need to do two more things to be safe:

• Set the new function's name to match the old function's name.
• Copy the old function's attributes to the new function.

These can be accomplished by adding just three short lines to our old decorators. (Compare the version of @require in Listing Ten with the original in Listing Eight.)

Before returning the wrapper function, the decorator function in Listing Ten changes the wrapper function's name (by setting its __name__ attribute) to match the original function's name, and sets its __dict__ attribute (the dictionary containing its attributes) to the original function's __dict__, so it will have all the same attributes that the original function did. It also changes the wrapper function's documentation (its __doc__ attribute) to match the original function's documentation. Thus, if you used this new @require() decorator stacked over the @author() decorator, the resulting function would still have an author_name attribute, even though it was a different function object than the original one being decorated.

Putting It All Together

To illustrate, I'll use a few of these techniques to implement a complete, useful decorator that can be combined with other decorators. Specifically, I'll implement an @synchronized decorator (Listing Eleven) that implements Java-like synchronized methods. A given object's synchronized methods can only be invoked by one thread at a time. That is, as long as any synchronized method is executing, any other thread must wait until all the synchronized methods have returned.

To implement this, you need to have a lock that you can acquire whenever the method is executing. Then you can create a wrapping decorator that acquires and releases the lock around the original method call. I'll store this lock in a _sync_lock attribute on the object, automatically creating a new lock if there's no _sync_lock attribute already present.

But what if one synchronized method calls another synchronized method on the same object? Using simple mutual exclusion locks would result in a deadlock in this case, so we'll use a threading.RLock instead. An RLock may be held by only one thread, but it can be recursively acquired and released. Thus, if one synchronized method calls another on the same object, the lock count of the RLock simply increases, then decreases as the methods return. When the lock count reaches zero, other threads can acquire the lock and can, therefore, invoke synchronized methods on the object again.

There are two little tricks being done in Listing Eleven's wrapper code that are worth knowing about. First, the code uses a try/except block to catch an attribute error in the case where the object does not already have a synchronization lock. Since in the common case the lock should exist, this is generally faster than using an if/then test to check whether the lock exists (because the if/then test would have to execute every time, but the AttributeError will occur only once).

Second, when the lock doesn't exist, the code uses the setdefault method of the object's attribute dictionary (its __dict__) to either retrieve an existing value of _sync_lock, or to set a new one if there was no value there before. This is important because it's possible that two threads could simultaneously notice that the object has no lock, and then each would create and successfully acquire its own lock, while ignoring the lock created by the other! This would mean that our synchronization could fail on the first call to a synchronized method of a given object.

Using the atomic setdefault operation, however, guarantees that no matter how many threads simultaneously detect the need for a new lock, they will all receive the same RLock object. That is, one setdefault() operation sets the lock, then all subsequent setdefault() operations receive that lock object. Therefore, all threads end up using the same lock object, and thus only one is able to enter the wrapped method at a time, even if the lock object was just created.

Conclusion

Python decorators are a simple, highly customizable way to wrap functions or methods, annotate them with metadata, or register them with a framework of some kind. But, as a relatively new feature, their full possibilities have not yet been explored, and perhaps the most exciting uses haven't even been invented yet. Just to give you some ideas, here are links to a couple of lists of use cases that were posted to the mailing list for the developers working on the next version of Python: http://mail.python.org/pipermail/python-dev/2004-April/043902.html and http://mail.python.org/pipermail/python-dev/ 2004-April/044132.html.

Each message uses different syntax for decorators, based on some C#-like alternatives being discussed at the time. But the actual decorator examples presented should still be usable with the current syntax. And, by the time you read this article, there will likely be many other uses of decorators out there. For example, Thomas Heller has been working on experimental decorator support for the ctypes package (http://ctypes.sourceforge.net/), and I've been working on a complete generic function package using decorators, as part of the PyProtocols system (http://peak.telecommunity.com/ PyProtocols.html).

So, have fun experimenting with decorators! (Just be sure to practice "safe decs," to ensure that your decorators will play nice with others.)

DDJ

Listing One
(a)

import atexit

def goodbye():
    print "Goodbye, world!"

atexit.register(goodbye)

import atexit

@atexit.register
def goodbye():
    print "Goodbye, world!"

class Something(object):
    def someMethod(cls,foo,bar):
        print "I'm a class method"
    someMethod = classmethod(someMethod)

class Something(object):
    @classmethod
    def someMethod(cls,foo,bar):
        print "I'm a class method"

def mydecorator(func):
    print "decorating", func
    return func
print "before definition"
@mydecorator

def some_function():
    print "I'm never called, so you'll never see this message"
print "after definition"

before definition
decorating <function some_function at 0x00A933C0>
after definition

def stupid_decorator(func):
    return "Hello, world!"
@stupid_decorator
def does_nothing():
    print "I'm never called, so you'll never see this message"
print does_nothing

class traced:
    def __init__(self,func):
        self.func = func

    def __call__(__self,*__args,**__kw):
        print "entering", __self.func
        try:
            return __self.func(*__args,**__kw)
        finally:
            print "exiting", __self.func
@traced
def hello():
    print "Hello, world!"
hello()

class SomeClass(object):
    @classmethod
    @traced
    def someMethod(cls):
        print "Called with class", cls            
Something.someMethod()

def traced(func):
    def wrapper(*__args,**__kw):
        print "entering", func
        try:
            return func(*__args,**__kw)
        finally:
            print "exiting", func
    return wrapper

def require(expr):
    def decorator(func):
        def wrapper(*__args,**__kw):
            assert eval(expr),"Precondition failed"
            return func(*__args,**__kw)
        return wrapper
    return decorator
@require("len(__args)==1")
def test(*args):
    print args[0]
test("Hello world!")

def author(author_name):
    def decorator(func):
        func.author_name = author_name
        return func
    return decorator

@author("Lemony Snicket")
def sequenceOf(unfortunate_events):
    pass
print sequenceOf.author_name    # prints "Lemony Snicket"

def require(expr):
    def decorator(func):
        def wrapper(*__args,**__kw):
            assert eval(expr),"Precondition failed"
            return func(*__args,**__kw)
        wrapper.__name__ = func.__name__
        wrapper.__dict__ = func.__dict__
        wrapper.__doc__ = func.__doc__
        return wrapper
    return decorator

def synchronized(func):
    def wrapper(self,*__args,**__kw):
        try:
            rlock = self._sync_lock
        except AttributeError:
            from threading import RLock
            rlock = self.__dict__.setdefault('_sync_lock',RLock())
        rlock.acquire()
        try:
            return func(self,*__args,**__kw)
        finally:
            rlock.release()
    wrapper.__name__ = func.__name__
    wrapper.__dict__ = func.__dict__
    wrapper.__doc__ = func.__doc__
    return wrapper
class SomeClass:
    """Example usage"""
    @synchronized
    def doSomething(self,someParam):
        """This method can only be entered 
           by one thread at a time"""