misc.rst revision 12391:ceeca8b41e4b
1Miscellaneous 2############# 3 4.. _macro_notes: 5 6General notes regarding convenience macros 7========================================== 8 9pybind11 provides a few convenience macros such as 10:func:`PYBIND11_MAKE_OPAQUE` and :func:`PYBIND11_DECLARE_HOLDER_TYPE`, and 11``PYBIND11_OVERLOAD_*``. Since these are "just" macros that are evaluated 12in the preprocessor (which has no concept of types), they *will* get confused 13by commas in a template argument such as ``PYBIND11_OVERLOAD(MyReturnValue<T1, 14T2>, myFunc)``. In this case, the preprocessor assumes that the comma indicates 15the beginning of the next parameter. Use a ``typedef`` to bind the template to 16another name and use it in the macro to avoid this problem. 17 18.. _gil: 19 20Global Interpreter Lock (GIL) 21============================= 22 23When calling a C++ function from Python, the GIL is always held. 24The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be 25used to acquire and release the global interpreter lock in the body of a C++ 26function call. In this way, long-running C++ code can be parallelized using 27multiple Python threads. Taking :ref:`overriding_virtuals` as an example, this 28could be realized as follows (important changes highlighted): 29 30.. code-block:: cpp 31 :emphasize-lines: 8,9,31,32 32 33 class PyAnimal : public Animal { 34 public: 35 /* Inherit the constructors */ 36 using Animal::Animal; 37 38 /* Trampoline (need one for each virtual function) */ 39 std::string go(int n_times) { 40 /* Acquire GIL before calling Python code */ 41 py::gil_scoped_acquire acquire; 42 43 PYBIND11_OVERLOAD_PURE( 44 std::string, /* Return type */ 45 Animal, /* Parent class */ 46 go, /* Name of function */ 47 n_times /* Argument(s) */ 48 ); 49 } 50 }; 51 52 PYBIND11_MODULE(example, m) { 53 py::class_<Animal, PyAnimal> animal(m, "Animal"); 54 animal 55 .def(py::init<>()) 56 .def("go", &Animal::go); 57 58 py::class_<Dog>(m, "Dog", animal) 59 .def(py::init<>()); 60 61 m.def("call_go", [](Animal *animal) -> std::string { 62 /* Release GIL before calling into (potentially long-running) C++ code */ 63 py::gil_scoped_release release; 64 return call_go(animal); 65 }); 66 } 67 68The ``call_go`` wrapper can also be simplified using the `call_guard` policy 69(see :ref:`call_policies`) which yields the same result: 70 71.. code-block:: cpp 72 73 m.def("call_go", &call_go, py::call_guard<py::gil_scoped_release>()); 74 75 76Binding sequence data types, iterators, the slicing protocol, etc. 77================================================================== 78 79Please refer to the supplemental example for details. 80 81.. seealso:: 82 83 The file :file:`tests/test_sequences_and_iterators.cpp` contains a 84 complete example that shows how to bind a sequence data type, including 85 length queries (``__len__``), iterators (``__iter__``), the slicing 86 protocol and other kinds of useful operations. 87 88 89Partitioning code over multiple extension modules 90================================================= 91 92It's straightforward to split binding code over multiple extension modules, 93while referencing types that are declared elsewhere. Everything "just" works 94without any special precautions. One exception to this rule occurs when 95extending a type declared in another extension module. Recall the basic example 96from Section :ref:`inheritance`. 97 98.. code-block:: cpp 99 100 py::class_<Pet> pet(m, "Pet"); 101 pet.def(py::init<const std::string &>()) 102 .def_readwrite("name", &Pet::name); 103 104 py::class_<Dog>(m, "Dog", pet /* <- specify parent */) 105 .def(py::init<const std::string &>()) 106 .def("bark", &Dog::bark); 107 108Suppose now that ``Pet`` bindings are defined in a module named ``basic``, 109whereas the ``Dog`` bindings are defined somewhere else. The challenge is of 110course that the variable ``pet`` is not available anymore though it is needed 111to indicate the inheritance relationship to the constructor of ``class_<Dog>``. 112However, it can be acquired as follows: 113 114.. code-block:: cpp 115 116 py::object pet = (py::object) py::module::import("basic").attr("Pet"); 117 118 py::class_<Dog>(m, "Dog", pet) 119 .def(py::init<const std::string &>()) 120 .def("bark", &Dog::bark); 121 122Alternatively, you can specify the base class as a template parameter option to 123``class_``, which performs an automated lookup of the corresponding Python 124type. Like the above code, however, this also requires invoking the ``import`` 125function once to ensure that the pybind11 binding code of the module ``basic`` 126has been executed: 127 128.. code-block:: cpp 129 130 py::module::import("basic"); 131 132 py::class_<Dog, Pet>(m, "Dog") 133 .def(py::init<const std::string &>()) 134 .def("bark", &Dog::bark); 135 136Naturally, both methods will fail when there are cyclic dependencies. 137 138Note that pybind11 code compiled with hidden-by-default symbol visibility (e.g. 139via the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is 140required proper pybind11 functionality, can interfere with the ability to 141access types defined in another extension module. Working around this requires 142manually exporting types that are accessed by multiple extension modules; 143pybind11 provides a macro to do just this: 144 145.. code-block:: cpp 146 147 class PYBIND11_EXPORT Dog : public Animal { 148 ... 149 }; 150 151Note also that it is possible (although would rarely be required) to share arbitrary 152C++ objects between extension modules at runtime. Internal library data is shared 153between modules using capsule machinery [#f6]_ which can be also utilized for 154storing, modifying and accessing user-defined data. Note that an extension module 155will "see" other extensions' data if and only if they were built with the same 156pybind11 version. Consider the following example: 157 158.. code-block:: cpp 159 160 auto data = (MyData *) py::get_shared_data("mydata"); 161 if (!data) 162 data = (MyData *) py::set_shared_data("mydata", new MyData(42)); 163 164If the above snippet was used in several separately compiled extension modules, 165the first one to be imported would create a ``MyData`` instance and associate 166a ``"mydata"`` key with a pointer to it. Extensions that are imported later 167would be then able to access the data behind the same pointer. 168 169.. [#f6] https://docs.python.org/3/extending/extending.html#using-capsules 170 171Module Destructors 172================== 173 174pybind11 does not provide an explicit mechanism to invoke cleanup code at 175module destruction time. In rare cases where such functionality is required, it 176is possible to emulate it using Python capsules or weak references with a 177destruction callback. 178 179.. code-block:: cpp 180 181 auto cleanup_callback = []() { 182 // perform cleanup here -- this function is called with the GIL held 183 }; 184 185 m.add_object("_cleanup", py::capsule(cleanup_callback)); 186 187This approach has the potential downside that instances of classes exposed 188within the module may still be alive when the cleanup callback is invoked 189(whether this is acceptable will generally depend on the application). 190 191Alternatively, the capsule may also be stashed within a type object, which 192ensures that it not called before all instances of that type have been 193collected: 194 195.. code-block:: cpp 196 197 auto cleanup_callback = []() { /* ... */ }; 198 m.attr("BaseClass").attr("_cleanup") = py::capsule(cleanup_callback); 199 200Both approaches also expose a potentially dangerous ``_cleanup`` attribute in 201Python, which may be undesirable from an API standpoint (a premature explicit 202call from Python might lead to undefined behavior). Yet another approach that 203avoids this issue involves weak reference with a cleanup callback: 204 205.. code-block:: cpp 206 207 // Register a callback function that is invoked when the BaseClass object is colelcted 208 py::cpp_function cleanup_callback( 209 [](py::handle weakref) { 210 // perform cleanup here -- this function is called with the GIL held 211 212 weakref.dec_ref(); // release weak reference 213 } 214 ); 215 216 // Create a weak reference with a cleanup callback and initially leak it 217 (void) py::weakref(m.attr("BaseClass"), cleanup_callback).release(); 218 219 220Generating documentation using Sphinx 221===================================== 222 223Sphinx [#f4]_ has the ability to inspect the signatures and documentation 224strings in pybind11-based extension modules to automatically generate beautiful 225documentation in a variety formats. The python_example repository [#f5]_ contains a 226simple example repository which uses this approach. 227 228There are two potential gotchas when using this approach: first, make sure that 229the resulting strings do not contain any :kbd:`TAB` characters, which break the 230docstring parsing routines. You may want to use C++11 raw string literals, 231which are convenient for multi-line comments. Conveniently, any excess 232indentation will be automatically be removed by Sphinx. However, for this to 233work, it is important that all lines are indented consistently, i.e.: 234 235.. code-block:: cpp 236 237 // ok 238 m.def("foo", &foo, R"mydelimiter( 239 The foo function 240 241 Parameters 242 ---------- 243 )mydelimiter"); 244 245 // *not ok* 246 m.def("foo", &foo, R"mydelimiter(The foo function 247 248 Parameters 249 ---------- 250 )mydelimiter"); 251 252By default, pybind11 automatically generates and prepends a signature to the docstring of a function 253registered with ``module::def()`` and ``class_::def()``. Sometimes this 254behavior is not desirable, because you want to provide your own signature or remove 255the docstring completely to exclude the function from the Sphinx documentation. 256The class ``options`` allows you to selectively suppress auto-generated signatures: 257 258.. code-block:: cpp 259 260 PYBIND11_MODULE(example, m) { 261 py::options options; 262 options.disable_function_signatures(); 263 264 m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers"); 265 } 266 267Note that changes to the settings affect only function bindings created during the 268lifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function, 269the default settings are restored to prevent unwanted side effects. 270 271.. [#f4] http://www.sphinx-doc.org 272.. [#f5] http://github.com/pybind/python_example 273