Building Hybrid Systems with Boost.Python

Introduction

Still, for many programmers, these very differences mean that Python and C++ complement one another perfectly. Performance bottlenecks in Python programs can be rewritten in C++ for maximal speed, and authors of powerful C++ libraries choose Python as a middleware language for its flexible system integration capabilities. Furthermore, the surface differences mask some strong similarities:

• 'C'-family control structures (if, while, for...)
• Support for object-orientation, functional programming, and generic programming (both are multi-paradigm programming languages.)
• Support for operator overloading
• High-level components such as collections and iterators.
• Encapsulation for reusable libraries (modules, namespaces)
• High-level error handling with exceptions
• C++ idioms in common use, such as handle/body classes and reference-counted smart pointers mirror Python reference semantics.

These limitations have lead to the development of a variety of high-level wrapping systems, most of which introduce their own specialized languages to control the process. In contrast, Boost.Python presents the user with a high-level C++ interface for wrapping C++ classes and functions, managing complexity behind-the-scenes with static metaprogramming. Boost.Python also goes beyond the scope of earlier systems by providing:

• Support for overriding C++ virtual functions in Python.
• Comprehensive lifetime management facilities for low-level C++ pointers and references.
• Support for "component-based" independent development of interacting extensions
• Integration with Python's powerful serialization engine (pickle).
• Coherence with the C++ rules for type conversion.

Development with Boost.Python is "user-guided": as much information is extracted directly from the source code to be wrapped as is possible within the limits of pure C++, and some additional information is supplied explicitly by the user. Mostly the process is mechanical and little intervention is required. Because the interface specification is written in the same full-featured language as the code being exposed, the user has unprecedented power available when she does need to take control.

Boost.Python Design Goals

However, it's also important not to translate all interfaces too literally: the idioms of each language must be respected. For example, though C++ and Python both have an iterator concept, the concepts are expressed very differently. Boost.Python has to be able to bridge the interface gap.

It must be possible to insulate Python users from crashes resulting from trivial misuses of C++ interfaces, such as accessing already-deleted objects. By the same token, the library should insulate C++ users from the low-level Python C API, replacing error-prone C interfaces like manual reference-count management and raw PyObject pointers with more-robust alternatives.

Support for component-based development is crucial so that C++ types exposed in one extension module can be passed to functions exposed in another extension without loss of crucial information like C++ inheritance relationships.

Finally, all wrapping must be non-intrusive. In other words, the wrapping must occur without modifying or even seeing the original C++ source code. Existing C++ libraries have to be wrappable by third parties who only have access to header files and binaries.

Hello Boost.Python World

char const* greet(unsigned x)
{
   static char const* const msgs[] =
   { "hello", "Boost.Python", "world!" };

   if (x > 2)
       throw std::range_error("greet: index out of range");

   return msgs[x];
}

extern "C" // all Python interactions use 'C' linkage
           // and calling convention
{
    // Wrapper to handle argument/result conversion
    // and checking
    PyObject* greet_wrap(PyObject* args, PyObject * keywords)
    {
         int x;
         // extract/check arguments
         if (PyArg_ParseTuple(args, "i", &x))
         {
             // invoke wrapped function
             char const* result = greet(x);
             // convert result to Python
             return PyString_FromString(result);
         }
         // error occurred
         return 0;
    }

    // Table of wrapped functions to be exposed by the module
    static PyMethodDef methods[] = {
        { "greet", greet_wrap, METH_VARARGS,
            "return one of 3 parts of a greeting" }
        , { NULL, NULL, 0, NULL } // sentinel
    };

    // module initialization function
    DL_EXPORT init_hello()
    {
        // add the methods to the module
        (void) Py_InitModule("hello", methods);
    }
}

#include <boost/python.hpp>
using namespace boost::python;
BOOST_PYTHON_MODULE(hello)
{
    def("greet", greet, "return one of 3 parts of a greeting");
}

>>> import hello
>>> for x in range(3):
...     print hello.greet(x)
...
hello
Boost.Python
world!

• The original function accepts an unsigned integer, and the Python C API only gives us a way of extracting signed integers. The Boost.Python version will raise a Python exception if we try to pass a negative number to hello.greet, but the C API version will proceed to do whatever the C++ implementation does when converting a negative integer to unsigned (usually wrapping to some very large number) and pass the incorrect translation on to the wrapped function.
• That brings us to the second problem: if the C++ greet() function is called with a number greater than 2, it will throw an exception. Typically, if a C++ exception propagates across the boundary with code generated by a C compiler, it will cause a crash. As you can see in the first version, there's no C++ scaffolding to prevent this from happening. Functions wrapped by Boost.Python automatically include an exception-handling layer that protects Python users by translating unhandled C++ exceptions into a corresponding Python exception.
• A slightly more-subtle limitation is that the argument conversion used in the Python C API case can only get that integer x in one way. PyArg_ParseTuple can't convert Python long objects (arbitrary-precision integers) that happen to fit in an unsigned int but not in a signed long, nor will it ever handle a wrapped C++ class with a user-defined implicit operator unsigned int() conversion. The Boost.Python's dynamic type conversion registry allows users to add arbitrary conversion methods.

Library Overview

Exposing Classes

Given:

struct World
{
    void set(std::string msg) { this->msg = msg; }
    std::string greet() { return msg; }
    std::string msg;
};

#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello)
{
    class_<World>("World")
        .def("greet", &World::greet)
        .def("set", &World::set)
    ;
}

class_<World>("World")

class_<World> w("World");

    w.def("greet", &World::greet)

So the example is equivalent to:

class_<World> w("World");
w.def("greet", &World::greet);
w.def("set", &World::set);

For completeness, here's the wrapped class in use:

>>> import hello
>>> planet = hello.World()
>>> planet.set('howdy')
>>> planet.greet()
'howdy'

Constructors

>>> planet = hello.World()

Given:

struct World
{
    World(std::string msg); // added constructor
    ...

class_<World>("World", init<std::string>())
    ...

class_<World>("World", init<std::string>())
    .def(init<double, double>())
    ...

Data Members and Properties

class_<World>("World", init<std::string>())
    .def_readonly("msg", &World::msg)
    ...

>>> planet = hello.World('howdy')
>>> planet.msg
'howdy'

In C++, publicly-accessible data members are considered a sign of poor design because they break encapsulation, and style guides usually dictate the use of getter and setter functions instead. In Python, however, __getattr__, __setattr__, and since 2.2, property mean that attribute access is just one more well-encapsulated syntactic tool at the programmer's disposal. Boost.Python bridges this idiomatic gap by making Python property creation directly available to users. If msg were private, we could still expose it as an attribute in Python as follows:

class_<World>("World", init<std::string>())
    .add_property("msg", &World::greet, &World::set)
    ...

>>> class World(object):
...     __init__(self, msg):
...         self.__msg = msg
...     def greet(self):
...         return self.__msg
...     def set(self, msg):
...         self.__msg = msg
...     msg = property(greet, set)

Operator Overloading

class_<rational<int> >("rational_int")


  .def(init<int, int>()) // constructor, e.g. rational_int(3,4)


  .def("numerator", &rational<int>::numerator)


  .def("denominator", &rational<int>::denominator)


  .def(-self)        // __neg__ (unary minus)


  .def(self + self)  // __add__ (homogeneous)


  .def(self * self)  // __mul__


  .def(self + int()) // __add__ (heterogenous)


  .def(int() + self) // __radd__


  ...

Inheritance

class_<Derived, bases<Base1,Base2> >("Derived")
     ...

1. When the class_<...> is created, Python type objects corresponding to Base1 and Base2 are looked up in the Boost.Python registry and are used as bases for the new Python Derived type object, so methods exposed for the Python Base1 and Base2 types are automatically members of the Derived type. Because the registry is global, this works correctly even if Derived is exposed in a different module from either of its bases.

2. C++ conversions from Derived to its bases are added to the Boost.Python registry. Thus wrapped C++ methods expecting (a pointer or reference to) an object of either base type can be called with an object wrapping a Derived instance. Wrapped member functions of class T are treated as though they have an implicit first argument of T&, so these conversions are neccessary to allow the base class methods to be called for derived objects.

Of course it's possible to derive new Python classes from wrapped C++ class instances. Because Boost.Python uses the newstyle class system, derivation works very much as for the Python built-in types. There is, however, one significant difference: the built-in types generally establish their invariants in their __new__ function, so that derived classes do not need to call __init__ on the base class before invoking its methods:

>>> class L(list):
...      def __init__(self):
...          pass
...
>>> L().reverse()
>>>

>>> class D(SomeBPLClass):
...      def __init__(self):
...          pass
...
>>> D().some_bpl_method()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: bad argument type for built-in operation

Virtual Functions

//
// interface to wrap:
//
class Base
{
public:
    virtual int f(std::string x) { return 42; }
    virtual ~Base();
};

int calls_f(Base const& b, std::string x) { return b.f(x); }

//
// Wrapping Code
//

// Dispatcher class
struct BaseWrap : Base
{
    // Store a pointer to the Python object
    BaseWrap(PyObject* self_) : self(self_) {}
    PyObject* self;

    // Default implementation, for when f is not overridden
    int f_default(std::string x) { return this->Base::f(x); }
    // Dispatch implementation
    int f(std::string x)
    { return call_method<int>(self, "f", x); }
};

...
    def("calls_f", calls_f);
    class_<Base, BaseWrap>("Base")
        .def("f", &Base::f, &BaseWrap::f_default)
        ;

>>> class Derived(Base):
...     def f(self, s):
...          return len(s)
...
>>> calls_f(Base(), 'foo')
42
>>> calls_f(Derived(), 'forty-two')
9

• The key element that allows overriding in Python is the call_method invocation, which uses the same global type conversion registry as the C++ function wrapping does to convert its arguments from C++ to Python and its return type from Python to C++.
• Any constructor signatures you wish to wrap must be replicated with an initial PyObject* argument.
• The dispatcher must store this argument so that it can be used to invoke call_method.
• The f_default member function is needed when the function being exposed is not pure virtual; there's no other way Base::f can be called on an object of type BaseWrap, since it overrides f.

Object Interface

Boost.Python provides a class object that automates reference counting and provides conversion to Python from C++ objects of arbitrary type. This feature significantly reduces the learning effort for prospective extension module writers.

Creating an object from any other type is extremely simple:

object s("hello, world");  // s manages a Python string

object ten_Os = 10 * s[4]; // -> "oooooooooo"

The extract<T> class template can be used to convert Python objects to C++ types:

double x = extract<double>(o);

The object type is accompanied by a set of derived types that mirror the Python built-in types such as list, dict, tuple, etc. as much as possible. This enables convenient manipulation of these high-level types from C++:

dict d;
d["some"] = "thing";
d["lucky_number"] = 13;
list l = d.keys();

Thinking hybrid

Boost.Python enables us to "think hybrid." Python can be used for rapidly prototyping a new application; its ease of use and the large pool of standard libraries give us a head start on the way to a working system. If necessary, the working code can be used to discover rate-limiting hotspots. To maximize performance these hotspots can be reimplemented in C++, together with the Boost.Python bindings needed to tie them back into the existing higher-level procedure.

Of course, this top-down approach is less attractive if it is clear from the start that many algorithms will eventually have to be implemented in C++. Fortunately Boost.Python also enables us to pursue a bottom-up approach. We have used this approach very successfully in the development of a toolbox for scientific applications. The toolbox started out mainly as a library of C++ classes with Boost.Python bindings, and for a while the growth was mainly concentrated on the C++ parts. However, as the toolbox is becoming more complete, more and more newly added functionality can be implemented in Python.

Figure 1 shows the estimated ratio of newly added C++ and Python code over time as new algorithms are implemented. We expect this ratio to level out near 70% Python. Being able to solve new problems mostly in Python rather than a more difficult statically typed language is the return on our investment in Boost.Python. The ability to access all of our code from Python allows a broader group of developers to use Python in the rapid development of new applications.

Conclusions

Computationally intensive tasks play to the strengths of C++ and are often impossible to implement efficiently in pure Python, while jobs like serialization that are trivial in Python can be very difficult in pure C++. Given the luxury of building a hybrid software system from the ground up, we can approach design with new confidence and power.

Citations

About the Authors

Ralf Grosse-Kunstleve is a scientist in the Physical Biosciences Division of the Lawrence Berkeley National Laboratory in California. He is part of the Computational Crystallography Initiative (cci.lbl.gov) which is leading an international effort aimed at advancing the automation of protein structure determination. The software system being developed by the collaboration has evolved together with the Boost.Python library and was designed from its inception as a hybrid Python/C++ system. The core components are available as an open source toolbox at cctbx.sourceforge.net.