Introduction: Bridging C++ and Python

Python and C++ are both powerful in their domains: Python excels in ease of use and rapid development, while C++ offers high performance and fine-grained control. In many projects, it is desirable to combine the two, leveraging C++ for performance-critical components and Python for high-level orchestration. Pybind11 is a modern solution that makes this integration straightforward by allowing developers to expose C++17 code to Python with minimal boilerplate. It is a lightweight, header-only library inspired by Boost.Python but without the heavy dependency on Boost. In this post, I will share how to create Python bindings for C++17 code using Pybind11, focusing on its simplicity and illustrating with examples of binding custom data structures and dummy APIs (without diving into performance details).

Why Pybind11? Simplicity and Key Features

Pybind11’s design philosophy emphasises ease-of-use and succinctness. Unlike writing a Python C extension from scratch or using older tools, Pybind11 dramatically reduces the amount of glue code required. In my experience, the following benefits make Pybind11 stand out:

  • Minimal Boilerplate: Pybind11 infers function signatures and types at compile time using template metaprogramming. This means you write very little explicit code to expose a function or class – the library deduces argument and return types automatically. As the official docs note, “notice how little code was needed to expose our function to Python” because Pybind11 handles the details for you.

  • Header-Only and Easy Build: Being header-only, no separate linking step is needed against a Pybind11 library. You include the headers and compile your C++ code into a Python module. Pybind11 works with modern C++ standards (C++11 and up) and leverages new language features for simplicity. This compact implementation avoids the complexity of Boost and keeps compile times and binary sizes small. For example, without any special build system, one can compile a Pybind11 module on Linux with a single command using g++ and Python’s build flags.

  • Automatic Conversions: Pybind11 automatically converts many C++ types to Python types and vice versa. It has built-in support for standard types (numbers, strings), STL containers, and more. For instance, if a C++ function returns a std::set<std::pair<std::string, std::vector<int>>>, Pybind11 will present it in Python as a set of tuples (str, list[int]) without extra work from the developer. This seamless mapping between C++ STL and Python data structures (and support for NumPy arrays, etc.) greatly simplifies integration.

  • Modern C++17 Features: Since we target C++17, it’s worth noting Pybind11 supports and benefits from C++17 features. It can, for example, handle std::optional, structured bindings, and other modern types via its type caster system. The library’s newer cousin Nanobind (by the same author) even takes C++17 further for efficiency, but Pybind11 itself is fully capable with C++17 and widely used.

  • No External Dependencies: Aside from Python development headers, Pybind11 doesn’t require linking against libpython or Boost. It simply needs a C++ compiler and Python’s API. This makes integration less error-prone. As the PyPI description highlights, Pybind11 aims to “minimize boilerplate code… by inferring type information using compile-time introspection”, essentially automating the hard parts of extension writing.

Alternatives: It’s useful to be aware of other binding tools. Boost.Python is the historic predecessor to Pybind11, offering similar functionality but with the heavy Boost dependency and older design. SWIG can generate bindings for multiple languages (including Python) but introduces an extra interface file and can be more cumbersome for modern C++ features. Cython allows writing Python-like syntax that gets compiled to C, which is great for gradually optimising Python code, but it’s less straightforward when you have an existing C++ codebase. Nanobind, as mentioned, is a newer project by Pybind11’s creator that uses C++17 to achieve even more efficient bindings in terms of speed and memory footprint. Each of these has its use cases, but for simplicity and quick integration, Pybind11 is often my go-to choice.

Setting Up Pybind11 Integration (Linux)

Setting up Pybind11 is relatively straightforward. For this demonstration, I’ll assume a Linux environment with GCC and Python 3.10. (Compiling on Windows is equally trivial using Visual Studio or MinGW, but here we focus on Linux steps for brevity.)

Prerequisites: You need Python’s development headers and Pybind11 available. On Linux, this can be done by installing the python3-dev package and the Pybind11 library (e.g. via pip install pybind11). The Pybind11 headers will be accessible either in your Python’s site-packages or from an include directory if installed system-wide.

Once you have the environment ready, you can write your C++ code and compile it into a Python extension module:

  1. Write the C++ Code with Pybind11: Create a C++ source file (e.g. mymodule.cpp). This will contain your C++ functions/classes and the Pybind11 binding code. We’ll see an example in the next section.

  2. Compile the code into a Python module: Use a compiler command that includes Python and Pybind11 headers and produces a shared library (.so file). Pybind11 provides a convenient way to get the necessary compiler flags. For instance, the following command compiles mymodule.cpp into a Python 3.10 module on Linux:

c++ -O3 -Wall -shared -std=c++17 -fPIC \
    $(python3.10 -m pybind11 --includes) \
    mymodule.cpp -o mymodule$(python3.10-config --extension-suffix)

Let’s break this down:

- `-std=c++17` enables C++17 standard (Pybind11 works with C++11 and above; here we use C++17 features).
    
- `-fPIC` and `-shared` are needed to create a shared library object.
    
- `$(python3.10 -m pybind11 --includes)` calls Python to retrieve the include paths for Pybind11 and Python headers, saving you from manually specifying `-I` paths. (If Pybind11 is installed via pip, this ensures the compiler knows where to find `<pybind11/pybind11.h>` and `<Python.h>`.)
    
- `mymodule.cpp -o mymodule$(python3.10-config --extension-suffix)` specifies the input file and output file. `python3.10-config --extension-suffix` will produce the appropriate extension for Python modules (e.g. `.cpython-310-x86_64-linux-gnu.so` or similar). This ensures your compiled module has the correct name for Python to import.
    

This one-liner is essentially the same approach suggested in Pybind11’s documentation for building modules on Linux. After running it, you should get a file like `mymodule.cpython-310-x86_64-linux-gnu.so`.

Note: On Windows, you would compile to a .pyd file (Python’s DLL equivalent) using MSVC with the appropriate flags (and no -fPIC). The Pybind11 docs cover cross-platform builds, and using a CMake script or Python’s setuptools can simplify multi-platform compilation. For example, Pybind11 provides a CMake function pybind11_add_module that abstracts these details. But using GCC on Linux as above is a quick way to get started.

Example: Binding a C++ Function and a Custom Class

Let’s walk through a concrete example. Suppose we have a simple C++17 class and function we want to use from Python. We’ll define:

  • A free function add(int, int) that returns the sum of two integers.

  • A custom data structure DataCounter that stores an integer count and provides methods to manipulate it (a dummy API: incrementing the count, retrieving the value, etc.), as well as a static method to demonstrate exposing class methods.

Here’s the C++ code (mymodule.cpp):

#include <pybind11/pybind11.h>
#include <string>

namespace py = pybind11;
// A free function to add two numbers int add(int a, int b) {
    return a + b;
} // A custom data structure with a dummy API

class DataCounter {
    public: DataCounter(int initial) : count(initial) {} void increment(int value) {
        count += value;
    } int value() const {
    return count;
} static std::string greet() {
return "Hello from C++";
}
private: int count;
};
// Pybind11 module definition PYBIND11_MODULE(mymodule, m) {
    m.doc() = "Example Pybind11 module (C++17)";
    // Optional module docstring // Bind the free function m.def("add", &add, "Add two integers");
    // Bind the DataCounter

    class py::class_<DataCounter>(m, "DataCounter") .def(py::init<int>(), py::arg("initial")) // bind constructor taking int .def("increment", &DataCounter::increment, py::arg("value")) .def("value", &DataCounter::value) .def_static("greet", &DataCounter::greet);
}

Let’s unpack what’s happening in the PYBIND11_MODULE block:

  • PYBIND11_MODULE(mymodule, m) is a macro that declares the entry-point of the Python module. The first argument mymodule is the name of the module as Python will see it (so Python will do import mymodule). The second argument m is a py::module_ object which we use to add our bindings.

  • m.doc() sets a docstring for the module (optional but good practice).

  • m.def("add", &add, "Add two integers") exposes the C++ function add to Python. The first argument is the name Python will use (add), the second is the function pointer, and the third is an optional documentation string. Pybind11 uses the function pointer to infer the argument and return types, automatically creating a Python callable. After this, in Python we can call mymodule.add(1, 2) and get 3 as result.

  • py::class_<DataCounter>(m, "DataCounter") begins the binding of the DataCounter class. The string "DataCounter" is the name of the Python class that will be created. We then chain methods to define how the class works in Python:

    • .def(py::init<int>(), py::arg("initial")) says that DataCounter can be constructed in Python by calling DataCounter(initial) with an int argument. We use py::init<int>() to select the constructor, and py::arg("initial") to name the parameter (so Python knows the parameter name for introspection).

    • .def("increment", &DataCounter::increment, py::arg("value")) exposes the increment() method. After binding, Python code can call obj.increment(5) to execute the C++ method, which will add 5 to the internal count. We provide the parameter name for clarity.

    • .def("value", &DataCounter::value) binds the value() method, which returns the current count. This allows Python to do obj.value() to retrieve the internal state.

    • .def_static("greet", &DataCounter::greet) binds the static method greet(). This shows how to expose static class methods; Python can call DataCounter.greet() directly on the class to get a greeting string from C++.

  • Note that we didn’t explicitly mention anything about data types beyond using them in function signatures. Pybind11 automatically knows how to convert the C++ int and std::string types to Python integers and strings. If we had more complex types (like std::vector<int> or custom structs), we might include extra Pybind11 headers (e.g. <pybind11/stl.h>) or write custom type casters, but basic types and many STL containers are supported out-of-the-box.

This example demonstrates a dummy API: we can increment a counter and fetch its value. It’s “dummy” in the sense that it’s simple, but in a real scenario DataCounter could represent some complex C++ object whose methods perform computationally heavy tasks. The Python user wouldn’t need to know the details – they just call increment() or value() as if it were a normal Python object method, but behind the scenes, C++ is doing the work.

Building and Using the Python Module

After writing the C++ code, you compile it as described earlier. For instance, running the g++ command shown in the setup section will produce mymodule.cpython-310-x86_64-linux-gnu.so (the exact filename may vary). Make sure this compiled file is either in your current working directory or in Python’s sys.path.

Now, let’s see how we use this module from Python:

import mymodule

# Use the bound C++ function
result =mymodule.add(3, 4)
print(f"3 + 4 = {result}")  # Expected output: 3 + 4 = 7

# Use the bound C++ class

counter =mymodule.DataCounter(initial=10) # Create a DataCounter object with initial value 10
counter.increment(5)
print(counter.value())  # Expected output: 15 # Call the static method
print(mymodule.DataCounter.greet())  # Expected output: Hello from C++

When you run a Python script like the above (in the same folder as the .so module, or after installing the module), you should see the expected outputs. The mymodule.add function behaves like a normal Python function that adds numbers, and DataCounter behaves like a normal Python class (you can even use Python’s help(mymodule.DataCounter) to see its docstring and methods). However, all the heavy lifting is being done in C++ under the hood. This means you could, for example, implement complex algorithms inside increment or other methods in C++ for efficiency, and still call them from Python naturally.

One of the great things I find with Pybind11 is that the integration feels seamless. In Python, the objects are full-fledged Python objects (you can print them, inspect them, etc.), but they are actually backed by C++ implementations. The simplicity of the binding code does not mean it’s a toy – you can expose very sophisticated C++ library code in the same manner, with classes, exceptions, overloads, and more, and Pybind11 will handle most of the translation work for you.

Discussion: Simplicity vs. Performance Trade-off

Pybind11’s ease-of-use does come with some overhead, but it’s generally a worthwhile trade-off. By design, Pybind11 provides zero-cost abstractions where possible, meaning it tries not to add runtime cost beyond what the Python C API requires. In fact, it’s advertised as having “minimal overhead, designed for zero-cost abstractions that keep C++ speed intact”. In practice, calling a C++ function through Pybind11 is very fast – the conversion of arguments and return values is quite efficient. If your C++ function does a significant amount of work (e.g. heavy computations on an array), the call overhead is negligible compared to the computation time.

However, if you were to call a trivial C++ function millions of times from Python (e.g. a tight loop calling an addition function), the overhead of crossing the language boundary could become noticeable. This is not a limitation of Pybind11 alone but of any approach that interfaces Python with C++ – each call has to go through the Python C API. Tools like Cython can sometimes achieve lower overhead for very fine-grained interactions by embedding C logic more directly, but they require a different development approach. In my experience, for the majority of use cases (where each C++ call does non-trivial work), Pybind11’s performance is more than sufficient. It lets you write high-performance extensions while maintaining a high level of code clarity.

For those extreme scenarios where you need to squeeze out every bit of performance (or reduce binary size), one might consider Nanobind or even writing custom CPython extensions. Nanobind, created by Pybind11’s author, uses C++17 and some new techniques to reduce overhead and memory usage, offering “more efficient bindings in space and speed”. The trade-off is that it’s a bit more opinionated (for example, it encourages a specific CMake build setup and has a slightly different API). Boost.Python is another alternative, but it’s largely superseded by Pybind11 due to Pybind11’s lighter weight and the fact that it doesn’t require linking against Boost. For most intermediate to advanced developers, Pybind11 hits a sweet spot between developer productivity and runtime performance, especially when integrating with C++17 codebases that can fully benefit from modern C++ features.

Conclusion

Interfacing C++17 with Python is no longer an arcane art reserved for CPython gurus – Pybind11 makes it accessible and even enjoyable. In this post, we demonstrated how to expose a C++ function and a custom data structure to Python using Pybind11, with only a few lines of straightforward code. We discussed how Pybind11’s approach prioritises simplicity and leverages modern C++ techniques to automatically handle type conversions and minimise boilerplate. The example with DataCounter and the add function shows the basics, but Pybind11 can handle much more complex scenarios (multiple classes, inheritance, overloaded functions, custom types, etc.) with similar elegance.

In my experience, using Pybind11 allows one to write C++ and Python side by side, using each for what it does best: compute-intensive tasks in C++, and high-level logic or scripting in Python. The result is often the “best of both worlds” – you get Python’s ease of use without sacrificing the performance of critical sections of code. By following the steps outlined (and consulting Pybind11’s excellent documentation for advanced needs), you can integrate C++17 code into your Python projects confidently. This opens up possibilities to reuse existing C++ libraries in Python, accelerate Python code by moving bottlenecks to C++, or simply to share data structures and algorithms between the two languages in a clean way.

References

  • Pybind11 Official Documentation – Basics and Introductionpybind11.readthedocs.iopybind11.readthedocs.io

  • Pybind11 PyPI Description – Project Goals and Comparison to Boost.Pythonpypi.org

  • Lena’s DEV Community Article – Creating Python Bindings with Pybind11 (mentions alternatives like Boost.Python & Nanobind)dev.to

  • Dan Vanderkam, Python and C++ with pybind11: A Great Combination – Blog post illustrating Pybind11 usage and STL conversion exampledanvk.org

  • Hacker News Discussion – Comment on Pybind11 ease-of-use vs runtime performance trade-offnews.ycombinator.com

  • Ahmed Gad, Pybind11 Tutorial – Tutorial demonstrating class binding and concluding remarks on combining C++ performance with Python simplicitymedium.com