misc.rst revision 14299:2fbea9df56d2
1Miscellaneous 2############# 3 4.. _macro_notes: 5 6General notes regarding convenience macros 7========================================== 8 9pybind11 provides a few convenience macros such as 10:func:`PYBIND11_DECLARE_HOLDER_TYPE` and ``PYBIND11_OVERLOAD_*``. Since these 11are "just" macros that are evaluated in the preprocessor (which has no concept 12of types), they *will* get confused by commas in a template argument; for 13example, consider: 14 15.. code-block:: cpp 16 17 PYBIND11_OVERLOAD(MyReturnType<T1, T2>, Class<T3, T4>, func) 18 19The limitation of the C preprocessor interprets this as five arguments (with new 20arguments beginning after each comma) rather than three. To get around this, 21there are two alternatives: you can use a type alias, or you can wrap the type 22using the ``PYBIND11_TYPE`` macro: 23 24.. code-block:: cpp 25 26 // Version 1: using a type alias 27 using ReturnType = MyReturnType<T1, T2>; 28 using ClassType = Class<T3, T4>; 29 PYBIND11_OVERLOAD(ReturnType, ClassType, func); 30 31 // Version 2: using the PYBIND11_TYPE macro: 32 PYBIND11_OVERLOAD(PYBIND11_TYPE(MyReturnType<T1, T2>), 33 PYBIND11_TYPE(Class<T3, T4>), func) 34 35The ``PYBIND11_MAKE_OPAQUE`` macro does *not* require the above workarounds. 36 37.. _gil: 38 39Global Interpreter Lock (GIL) 40============================= 41 42When calling a C++ function from Python, the GIL is always held. 43The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be 44used to acquire and release the global interpreter lock in the body of a C++ 45function call. In this way, long-running C++ code can be parallelized using 46multiple Python threads. Taking :ref:`overriding_virtuals` as an example, this 47could be realized as follows (important changes highlighted): 48 49.. code-block:: cpp 50 :emphasize-lines: 8,9,31,32 51 52 class PyAnimal : public Animal { 53 public: 54 /* Inherit the constructors */ 55 using Animal::Animal; 56 57 /* Trampoline (need one for each virtual function) */ 58 std::string go(int n_times) { 59 /* Acquire GIL before calling Python code */ 60 py::gil_scoped_acquire acquire; 61 62 PYBIND11_OVERLOAD_PURE( 63 std::string, /* Return type */ 64 Animal, /* Parent class */ 65 go, /* Name of function */ 66 n_times /* Argument(s) */ 67 ); 68 } 69 }; 70 71 PYBIND11_MODULE(example, m) { 72 py::class_<Animal, PyAnimal> animal(m, "Animal"); 73 animal 74 .def(py::init<>()) 75 .def("go", &Animal::go); 76 77 py::class_<Dog>(m, "Dog", animal) 78 .def(py::init<>()); 79 80 m.def("call_go", [](Animal *animal) -> std::string { 81 /* Release GIL before calling into (potentially long-running) C++ code */ 82 py::gil_scoped_release release; 83 return call_go(animal); 84 }); 85 } 86 87The ``call_go`` wrapper can also be simplified using the `call_guard` policy 88(see :ref:`call_policies`) which yields the same result: 89 90.. code-block:: cpp 91 92 m.def("call_go", &call_go, py::call_guard<py::gil_scoped_release>()); 93 94 95Binding sequence data types, iterators, the slicing protocol, etc. 96================================================================== 97 98Please refer to the supplemental example for details. 99 100.. seealso:: 101 102 The file :file:`tests/test_sequences_and_iterators.cpp` contains a 103 complete example that shows how to bind a sequence data type, including 104 length queries (``__len__``), iterators (``__iter__``), the slicing 105 protocol and other kinds of useful operations. 106 107 108Partitioning code over multiple extension modules 109================================================= 110 111It's straightforward to split binding code over multiple extension modules, 112while referencing types that are declared elsewhere. Everything "just" works 113without any special precautions. One exception to this rule occurs when 114extending a type declared in another extension module. Recall the basic example 115from Section :ref:`inheritance`. 116 117.. code-block:: cpp 118 119 py::class_<Pet> pet(m, "Pet"); 120 pet.def(py::init<const std::string &>()) 121 .def_readwrite("name", &Pet::name); 122 123 py::class_<Dog>(m, "Dog", pet /* <- specify parent */) 124 .def(py::init<const std::string &>()) 125 .def("bark", &Dog::bark); 126 127Suppose now that ``Pet`` bindings are defined in a module named ``basic``, 128whereas the ``Dog`` bindings are defined somewhere else. The challenge is of 129course that the variable ``pet`` is not available anymore though it is needed 130to indicate the inheritance relationship to the constructor of ``class_<Dog>``. 131However, it can be acquired as follows: 132 133.. code-block:: cpp 134 135 py::object pet = (py::object) py::module::import("basic").attr("Pet"); 136 137 py::class_<Dog>(m, "Dog", pet) 138 .def(py::init<const std::string &>()) 139 .def("bark", &Dog::bark); 140 141Alternatively, you can specify the base class as a template parameter option to 142``class_``, which performs an automated lookup of the corresponding Python 143type. Like the above code, however, this also requires invoking the ``import`` 144function once to ensure that the pybind11 binding code of the module ``basic`` 145has been executed: 146 147.. code-block:: cpp 148 149 py::module::import("basic"); 150 151 py::class_<Dog, Pet>(m, "Dog") 152 .def(py::init<const std::string &>()) 153 .def("bark", &Dog::bark); 154 155Naturally, both methods will fail when there are cyclic dependencies. 156 157Note that pybind11 code compiled with hidden-by-default symbol visibility (e.g. 158via the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is 159required for proper pybind11 functionality, can interfere with the ability to 160access types defined in another extension module. Working around this requires 161manually exporting types that are accessed by multiple extension modules; 162pybind11 provides a macro to do just this: 163 164.. code-block:: cpp 165 166 class PYBIND11_EXPORT Dog : public Animal { 167 ... 168 }; 169 170Note also that it is possible (although would rarely be required) to share arbitrary 171C++ objects between extension modules at runtime. Internal library data is shared 172between modules using capsule machinery [#f6]_ which can be also utilized for 173storing, modifying and accessing user-defined data. Note that an extension module 174will "see" other extensions' data if and only if they were built with the same 175pybind11 version. Consider the following example: 176 177.. code-block:: cpp 178 179 auto data = (MyData *) py::get_shared_data("mydata"); 180 if (!data) 181 data = (MyData *) py::set_shared_data("mydata", new MyData(42)); 182 183If the above snippet was used in several separately compiled extension modules, 184the first one to be imported would create a ``MyData`` instance and associate 185a ``"mydata"`` key with a pointer to it. Extensions that are imported later 186would be then able to access the data behind the same pointer. 187 188.. [#f6] https://docs.python.org/3/extending/extending.html#using-capsules 189 190Module Destructors 191================== 192 193pybind11 does not provide an explicit mechanism to invoke cleanup code at 194module destruction time. In rare cases where such functionality is required, it 195is possible to emulate it using Python capsules or weak references with a 196destruction callback. 197 198.. code-block:: cpp 199 200 auto cleanup_callback = []() { 201 // perform cleanup here -- this function is called with the GIL held 202 }; 203 204 m.add_object("_cleanup", py::capsule(cleanup_callback)); 205 206This approach has the potential downside that instances of classes exposed 207within the module may still be alive when the cleanup callback is invoked 208(whether this is acceptable will generally depend on the application). 209 210Alternatively, the capsule may also be stashed within a type object, which 211ensures that it not called before all instances of that type have been 212collected: 213 214.. code-block:: cpp 215 216 auto cleanup_callback = []() { /* ... */ }; 217 m.attr("BaseClass").attr("_cleanup") = py::capsule(cleanup_callback); 218 219Both approaches also expose a potentially dangerous ``_cleanup`` attribute in 220Python, which may be undesirable from an API standpoint (a premature explicit 221call from Python might lead to undefined behavior). Yet another approach that 222avoids this issue involves weak reference with a cleanup callback: 223 224.. code-block:: cpp 225 226 // Register a callback function that is invoked when the BaseClass object is colelcted 227 py::cpp_function cleanup_callback( 228 [](py::handle weakref) { 229 // perform cleanup here -- this function is called with the GIL held 230 231 weakref.dec_ref(); // release weak reference 232 } 233 ); 234 235 // Create a weak reference with a cleanup callback and initially leak it 236 (void) py::weakref(m.attr("BaseClass"), cleanup_callback).release(); 237 238.. note:: 239 240 PyPy (at least version 5.9) does not garbage collect objects when the 241 interpreter exits. An alternative approach (which also works on CPython) is to use 242 the :py:mod:`atexit` module [#f7]_, for example: 243 244 .. code-block:: cpp 245 246 auto atexit = py::module::import("atexit"); 247 atexit.attr("register")(py::cpp_function([]() { 248 // perform cleanup here -- this function is called with the GIL held 249 })); 250 251 .. [#f7] https://docs.python.org/3/library/atexit.html 252 253 254Generating documentation using Sphinx 255===================================== 256 257Sphinx [#f4]_ has the ability to inspect the signatures and documentation 258strings in pybind11-based extension modules to automatically generate beautiful 259documentation in a variety formats. The python_example repository [#f5]_ contains a 260simple example repository which uses this approach. 261 262There are two potential gotchas when using this approach: first, make sure that 263the resulting strings do not contain any :kbd:`TAB` characters, which break the 264docstring parsing routines. You may want to use C++11 raw string literals, 265which are convenient for multi-line comments. Conveniently, any excess 266indentation will be automatically be removed by Sphinx. However, for this to 267work, it is important that all lines are indented consistently, i.e.: 268 269.. code-block:: cpp 270 271 // ok 272 m.def("foo", &foo, R"mydelimiter( 273 The foo function 274 275 Parameters 276 ---------- 277 )mydelimiter"); 278 279 // *not ok* 280 m.def("foo", &foo, R"mydelimiter(The foo function 281 282 Parameters 283 ---------- 284 )mydelimiter"); 285 286By default, pybind11 automatically generates and prepends a signature to the docstring of a function 287registered with ``module::def()`` and ``class_::def()``. Sometimes this 288behavior is not desirable, because you want to provide your own signature or remove 289the docstring completely to exclude the function from the Sphinx documentation. 290The class ``options`` allows you to selectively suppress auto-generated signatures: 291 292.. code-block:: cpp 293 294 PYBIND11_MODULE(example, m) { 295 py::options options; 296 options.disable_function_signatures(); 297 298 m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers"); 299 } 300 301Note that changes to the settings affect only function bindings created during the 302lifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function, 303the default settings are restored to prevent unwanted side effects. 304 305.. [#f4] http://www.sphinx-doc.org 306.. [#f5] http://github.com/pybind/python_example 307