misc.rst revision 14299
111986Sandreas.sandberg@arm.comMiscellaneous
211986Sandreas.sandberg@arm.com#############
311986Sandreas.sandberg@arm.com
411986Sandreas.sandberg@arm.com.. _macro_notes:
511986Sandreas.sandberg@arm.com
611986Sandreas.sandberg@arm.comGeneral notes regarding convenience macros
711986Sandreas.sandberg@arm.com==========================================
811986Sandreas.sandberg@arm.com
911986Sandreas.sandberg@arm.compybind11 provides a few convenience macros such as
1014299Sbbruce@ucdavis.edu:func:`PYBIND11_DECLARE_HOLDER_TYPE` and ``PYBIND11_OVERLOAD_*``. Since these
1114299Sbbruce@ucdavis.eduare "just" macros that are evaluated in the preprocessor (which has no concept
1214299Sbbruce@ucdavis.eduof types), they *will* get confused by commas in a template argument; for
1314299Sbbruce@ucdavis.eduexample, consider:
1414299Sbbruce@ucdavis.edu
1514299Sbbruce@ucdavis.edu.. code-block:: cpp
1614299Sbbruce@ucdavis.edu
1714299Sbbruce@ucdavis.edu    PYBIND11_OVERLOAD(MyReturnType<T1, T2>, Class<T3, T4>, func)
1814299Sbbruce@ucdavis.edu
1914299Sbbruce@ucdavis.eduThe limitation of the C preprocessor interprets this as five arguments (with new
2014299Sbbruce@ucdavis.eduarguments beginning after each comma) rather than three.  To get around this,
2114299Sbbruce@ucdavis.eduthere are two alternatives: you can use a type alias, or you can wrap the type
2214299Sbbruce@ucdavis.eduusing the ``PYBIND11_TYPE`` macro:
2314299Sbbruce@ucdavis.edu
2414299Sbbruce@ucdavis.edu.. code-block:: cpp
2514299Sbbruce@ucdavis.edu
2614299Sbbruce@ucdavis.edu    // Version 1: using a type alias
2714299Sbbruce@ucdavis.edu    using ReturnType = MyReturnType<T1, T2>;
2814299Sbbruce@ucdavis.edu    using ClassType = Class<T3, T4>;
2914299Sbbruce@ucdavis.edu    PYBIND11_OVERLOAD(ReturnType, ClassType, func);
3014299Sbbruce@ucdavis.edu
3114299Sbbruce@ucdavis.edu    // Version 2: using the PYBIND11_TYPE macro:
3214299Sbbruce@ucdavis.edu    PYBIND11_OVERLOAD(PYBIND11_TYPE(MyReturnType<T1, T2>),
3314299Sbbruce@ucdavis.edu                      PYBIND11_TYPE(Class<T3, T4>), func)
3414299Sbbruce@ucdavis.edu
3514299Sbbruce@ucdavis.eduThe ``PYBIND11_MAKE_OPAQUE`` macro does *not* require the above workarounds.
3611986Sandreas.sandberg@arm.com
3712391Sjason@lowepower.com.. _gil:
3811986Sandreas.sandberg@arm.com
3911986Sandreas.sandberg@arm.comGlobal Interpreter Lock (GIL)
4011986Sandreas.sandberg@arm.com=============================
4111986Sandreas.sandberg@arm.com
4212037Sandreas.sandberg@arm.comWhen calling a C++ function from Python, the GIL is always held.
4311986Sandreas.sandberg@arm.comThe classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be
4411986Sandreas.sandberg@arm.comused to acquire and release the global interpreter lock in the body of a C++
4511986Sandreas.sandberg@arm.comfunction call. In this way, long-running C++ code can be parallelized using
4611986Sandreas.sandberg@arm.commultiple Python threads. Taking :ref:`overriding_virtuals` as an example, this
4711986Sandreas.sandberg@arm.comcould be realized as follows (important changes highlighted):
4811986Sandreas.sandberg@arm.com
4911986Sandreas.sandberg@arm.com.. code-block:: cpp
5012391Sjason@lowepower.com    :emphasize-lines: 8,9,31,32
5111986Sandreas.sandberg@arm.com
5211986Sandreas.sandberg@arm.com    class PyAnimal : public Animal {
5311986Sandreas.sandberg@arm.com    public:
5411986Sandreas.sandberg@arm.com        /* Inherit the constructors */
5511986Sandreas.sandberg@arm.com        using Animal::Animal;
5611986Sandreas.sandberg@arm.com
5711986Sandreas.sandberg@arm.com        /* Trampoline (need one for each virtual function) */
5811986Sandreas.sandberg@arm.com        std::string go(int n_times) {
5911986Sandreas.sandberg@arm.com            /* Acquire GIL before calling Python code */
6011986Sandreas.sandberg@arm.com            py::gil_scoped_acquire acquire;
6111986Sandreas.sandberg@arm.com
6211986Sandreas.sandberg@arm.com            PYBIND11_OVERLOAD_PURE(
6311986Sandreas.sandberg@arm.com                std::string, /* Return type */
6411986Sandreas.sandberg@arm.com                Animal,      /* Parent class */
6511986Sandreas.sandberg@arm.com                go,          /* Name of function */
6611986Sandreas.sandberg@arm.com                n_times      /* Argument(s) */
6711986Sandreas.sandberg@arm.com            );
6811986Sandreas.sandberg@arm.com        }
6911986Sandreas.sandberg@arm.com    };
7011986Sandreas.sandberg@arm.com
7112391Sjason@lowepower.com    PYBIND11_MODULE(example, m) {
7211986Sandreas.sandberg@arm.com        py::class_<Animal, PyAnimal> animal(m, "Animal");
7311986Sandreas.sandberg@arm.com        animal
7411986Sandreas.sandberg@arm.com            .def(py::init<>())
7511986Sandreas.sandberg@arm.com            .def("go", &Animal::go);
7611986Sandreas.sandberg@arm.com
7711986Sandreas.sandberg@arm.com        py::class_<Dog>(m, "Dog", animal)
7811986Sandreas.sandberg@arm.com            .def(py::init<>());
7911986Sandreas.sandberg@arm.com
8011986Sandreas.sandberg@arm.com        m.def("call_go", [](Animal *animal) -> std::string {
8111986Sandreas.sandberg@arm.com            /* Release GIL before calling into (potentially long-running) C++ code */
8211986Sandreas.sandberg@arm.com            py::gil_scoped_release release;
8311986Sandreas.sandberg@arm.com            return call_go(animal);
8411986Sandreas.sandberg@arm.com        });
8512391Sjason@lowepower.com    }
8611986Sandreas.sandberg@arm.com
8712391Sjason@lowepower.comThe ``call_go`` wrapper can also be simplified using the `call_guard` policy
8812391Sjason@lowepower.com(see :ref:`call_policies`) which yields the same result:
8912391Sjason@lowepower.com
9012391Sjason@lowepower.com.. code-block:: cpp
9112391Sjason@lowepower.com
9212391Sjason@lowepower.com    m.def("call_go", &call_go, py::call_guard<py::gil_scoped_release>());
9311986Sandreas.sandberg@arm.com
9411986Sandreas.sandberg@arm.com
9511986Sandreas.sandberg@arm.comBinding sequence data types, iterators, the slicing protocol, etc.
9611986Sandreas.sandberg@arm.com==================================================================
9711986Sandreas.sandberg@arm.com
9811986Sandreas.sandberg@arm.comPlease refer to the supplemental example for details.
9911986Sandreas.sandberg@arm.com
10011986Sandreas.sandberg@arm.com.. seealso::
10111986Sandreas.sandberg@arm.com
10211986Sandreas.sandberg@arm.com    The file :file:`tests/test_sequences_and_iterators.cpp` contains a
10311986Sandreas.sandberg@arm.com    complete example that shows how to bind a sequence data type, including
10411986Sandreas.sandberg@arm.com    length queries (``__len__``), iterators (``__iter__``), the slicing
10511986Sandreas.sandberg@arm.com    protocol and other kinds of useful operations.
10611986Sandreas.sandberg@arm.com
10711986Sandreas.sandberg@arm.com
10811986Sandreas.sandberg@arm.comPartitioning code over multiple extension modules
10911986Sandreas.sandberg@arm.com=================================================
11011986Sandreas.sandberg@arm.com
11111986Sandreas.sandberg@arm.comIt's straightforward to split binding code over multiple extension modules,
11211986Sandreas.sandberg@arm.comwhile referencing types that are declared elsewhere. Everything "just" works
11311986Sandreas.sandberg@arm.comwithout any special precautions. One exception to this rule occurs when
11411986Sandreas.sandberg@arm.comextending a type declared in another extension module. Recall the basic example
11511986Sandreas.sandberg@arm.comfrom Section :ref:`inheritance`.
11611986Sandreas.sandberg@arm.com
11711986Sandreas.sandberg@arm.com.. code-block:: cpp
11811986Sandreas.sandberg@arm.com
11911986Sandreas.sandberg@arm.com    py::class_<Pet> pet(m, "Pet");
12011986Sandreas.sandberg@arm.com    pet.def(py::init<const std::string &>())
12111986Sandreas.sandberg@arm.com       .def_readwrite("name", &Pet::name);
12211986Sandreas.sandberg@arm.com
12311986Sandreas.sandberg@arm.com    py::class_<Dog>(m, "Dog", pet /* <- specify parent */)
12411986Sandreas.sandberg@arm.com        .def(py::init<const std::string &>())
12511986Sandreas.sandberg@arm.com        .def("bark", &Dog::bark);
12611986Sandreas.sandberg@arm.com
12711986Sandreas.sandberg@arm.comSuppose now that ``Pet`` bindings are defined in a module named ``basic``,
12811986Sandreas.sandberg@arm.comwhereas the ``Dog`` bindings are defined somewhere else. The challenge is of
12911986Sandreas.sandberg@arm.comcourse that the variable ``pet`` is not available anymore though it is needed
13011986Sandreas.sandberg@arm.comto indicate the inheritance relationship to the constructor of ``class_<Dog>``.
13111986Sandreas.sandberg@arm.comHowever, it can be acquired as follows:
13211986Sandreas.sandberg@arm.com
13311986Sandreas.sandberg@arm.com.. code-block:: cpp
13411986Sandreas.sandberg@arm.com
13511986Sandreas.sandberg@arm.com    py::object pet = (py::object) py::module::import("basic").attr("Pet");
13611986Sandreas.sandberg@arm.com
13711986Sandreas.sandberg@arm.com    py::class_<Dog>(m, "Dog", pet)
13811986Sandreas.sandberg@arm.com        .def(py::init<const std::string &>())
13911986Sandreas.sandberg@arm.com        .def("bark", &Dog::bark);
14011986Sandreas.sandberg@arm.com
14111986Sandreas.sandberg@arm.comAlternatively, you can specify the base class as a template parameter option to
14211986Sandreas.sandberg@arm.com``class_``, which performs an automated lookup of the corresponding Python
14311986Sandreas.sandberg@arm.comtype. Like the above code, however, this also requires invoking the ``import``
14411986Sandreas.sandberg@arm.comfunction once to ensure that the pybind11 binding code of the module ``basic``
14511986Sandreas.sandberg@arm.comhas been executed:
14611986Sandreas.sandberg@arm.com
14711986Sandreas.sandberg@arm.com.. code-block:: cpp
14811986Sandreas.sandberg@arm.com
14911986Sandreas.sandberg@arm.com    py::module::import("basic");
15011986Sandreas.sandberg@arm.com
15111986Sandreas.sandberg@arm.com    py::class_<Dog, Pet>(m, "Dog")
15211986Sandreas.sandberg@arm.com        .def(py::init<const std::string &>())
15311986Sandreas.sandberg@arm.com        .def("bark", &Dog::bark);
15411986Sandreas.sandberg@arm.com
15511986Sandreas.sandberg@arm.comNaturally, both methods will fail when there are cyclic dependencies.
15611986Sandreas.sandberg@arm.com
15712391Sjason@lowepower.comNote that pybind11 code compiled with hidden-by-default symbol visibility (e.g.
15812391Sjason@lowepower.comvia the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is
15914299Sbbruce@ucdavis.edurequired for proper pybind11 functionality, can interfere with the ability to
16012391Sjason@lowepower.comaccess types defined in another extension module.  Working around this requires
16112391Sjason@lowepower.commanually exporting types that are accessed by multiple extension modules;
16212391Sjason@lowepower.compybind11 provides a macro to do just this:
16311986Sandreas.sandberg@arm.com
16411986Sandreas.sandberg@arm.com.. code-block:: cpp
16511986Sandreas.sandberg@arm.com
16612391Sjason@lowepower.com    class PYBIND11_EXPORT Dog : public Animal {
16711986Sandreas.sandberg@arm.com        ...
16811986Sandreas.sandberg@arm.com    };
16911986Sandreas.sandberg@arm.com
17011986Sandreas.sandberg@arm.comNote also that it is possible (although would rarely be required) to share arbitrary
17111986Sandreas.sandberg@arm.comC++ objects between extension modules at runtime. Internal library data is shared
17211986Sandreas.sandberg@arm.combetween modules using capsule machinery [#f6]_ which can be also utilized for
17311986Sandreas.sandberg@arm.comstoring, modifying and accessing user-defined data. Note that an extension module
17411986Sandreas.sandberg@arm.comwill "see" other extensions' data if and only if they were built with the same
17511986Sandreas.sandberg@arm.compybind11 version. Consider the following example:
17611986Sandreas.sandberg@arm.com
17711986Sandreas.sandberg@arm.com.. code-block:: cpp
17811986Sandreas.sandberg@arm.com
17911986Sandreas.sandberg@arm.com    auto data = (MyData *) py::get_shared_data("mydata");
18011986Sandreas.sandberg@arm.com    if (!data)
18111986Sandreas.sandberg@arm.com        data = (MyData *) py::set_shared_data("mydata", new MyData(42));
18211986Sandreas.sandberg@arm.com
18311986Sandreas.sandberg@arm.comIf the above snippet was used in several separately compiled extension modules,
18411986Sandreas.sandberg@arm.comthe first one to be imported would create a ``MyData`` instance and associate
18511986Sandreas.sandberg@arm.coma ``"mydata"`` key with a pointer to it. Extensions that are imported later
18611986Sandreas.sandberg@arm.comwould be then able to access the data behind the same pointer.
18711986Sandreas.sandberg@arm.com
18811986Sandreas.sandberg@arm.com.. [#f6] https://docs.python.org/3/extending/extending.html#using-capsules
18911986Sandreas.sandberg@arm.com
19012037Sandreas.sandberg@arm.comModule Destructors
19112037Sandreas.sandberg@arm.com==================
19212037Sandreas.sandberg@arm.com
19312037Sandreas.sandberg@arm.compybind11 does not provide an explicit mechanism to invoke cleanup code at
19412037Sandreas.sandberg@arm.commodule destruction time. In rare cases where such functionality is required, it
19512391Sjason@lowepower.comis possible to emulate it using Python capsules or weak references with a
19612391Sjason@lowepower.comdestruction callback.
19712037Sandreas.sandberg@arm.com
19812037Sandreas.sandberg@arm.com.. code-block:: cpp
19912037Sandreas.sandberg@arm.com
20012037Sandreas.sandberg@arm.com    auto cleanup_callback = []() {
20112037Sandreas.sandberg@arm.com        // perform cleanup here -- this function is called with the GIL held
20212037Sandreas.sandberg@arm.com    };
20312037Sandreas.sandberg@arm.com
20412037Sandreas.sandberg@arm.com    m.add_object("_cleanup", py::capsule(cleanup_callback));
20511986Sandreas.sandberg@arm.com
20612391Sjason@lowepower.comThis approach has the potential downside that instances of classes exposed
20712391Sjason@lowepower.comwithin the module may still be alive when the cleanup callback is invoked
20812391Sjason@lowepower.com(whether this is acceptable will generally depend on the application).
20912391Sjason@lowepower.com
21012391Sjason@lowepower.comAlternatively, the capsule may also be stashed within a type object, which
21112391Sjason@lowepower.comensures that it not called before all instances of that type have been
21212391Sjason@lowepower.comcollected:
21312391Sjason@lowepower.com
21412391Sjason@lowepower.com.. code-block:: cpp
21512391Sjason@lowepower.com
21612391Sjason@lowepower.com    auto cleanup_callback = []() { /* ... */ };
21712391Sjason@lowepower.com    m.attr("BaseClass").attr("_cleanup") = py::capsule(cleanup_callback);
21812391Sjason@lowepower.com
21912391Sjason@lowepower.comBoth approaches also expose a potentially dangerous ``_cleanup`` attribute in
22012391Sjason@lowepower.comPython, which may be undesirable from an API standpoint (a premature explicit
22112391Sjason@lowepower.comcall from Python might lead to undefined behavior). Yet another approach that 
22212391Sjason@lowepower.comavoids this issue involves weak reference with a cleanup callback:
22312391Sjason@lowepower.com
22412391Sjason@lowepower.com.. code-block:: cpp
22512391Sjason@lowepower.com
22612391Sjason@lowepower.com    // Register a callback function that is invoked when the BaseClass object is colelcted
22712391Sjason@lowepower.com    py::cpp_function cleanup_callback(
22812391Sjason@lowepower.com        [](py::handle weakref) {
22912391Sjason@lowepower.com            // perform cleanup here -- this function is called with the GIL held
23012391Sjason@lowepower.com
23112391Sjason@lowepower.com            weakref.dec_ref(); // release weak reference
23212391Sjason@lowepower.com        }
23312391Sjason@lowepower.com    );
23412391Sjason@lowepower.com
23512391Sjason@lowepower.com    // Create a weak reference with a cleanup callback and initially leak it
23612391Sjason@lowepower.com    (void) py::weakref(m.attr("BaseClass"), cleanup_callback).release();
23712391Sjason@lowepower.com
23814299Sbbruce@ucdavis.edu.. note::
23914299Sbbruce@ucdavis.edu
24014299Sbbruce@ucdavis.edu    PyPy (at least version 5.9) does not garbage collect objects when the
24114299Sbbruce@ucdavis.edu    interpreter exits. An alternative approach (which also works on CPython) is to use
24214299Sbbruce@ucdavis.edu    the :py:mod:`atexit` module [#f7]_, for example:
24314299Sbbruce@ucdavis.edu
24414299Sbbruce@ucdavis.edu    .. code-block:: cpp
24514299Sbbruce@ucdavis.edu
24614299Sbbruce@ucdavis.edu        auto atexit = py::module::import("atexit");
24714299Sbbruce@ucdavis.edu        atexit.attr("register")(py::cpp_function([]() {
24814299Sbbruce@ucdavis.edu            // perform cleanup here -- this function is called with the GIL held
24914299Sbbruce@ucdavis.edu        }));
25014299Sbbruce@ucdavis.edu
25114299Sbbruce@ucdavis.edu    .. [#f7] https://docs.python.org/3/library/atexit.html
25214299Sbbruce@ucdavis.edu
25312391Sjason@lowepower.com
25411986Sandreas.sandberg@arm.comGenerating documentation using Sphinx
25511986Sandreas.sandberg@arm.com=====================================
25611986Sandreas.sandberg@arm.com
25711986Sandreas.sandberg@arm.comSphinx [#f4]_ has the ability to inspect the signatures and documentation
25811986Sandreas.sandberg@arm.comstrings in pybind11-based extension modules to automatically generate beautiful
25911986Sandreas.sandberg@arm.comdocumentation in a variety formats. The python_example repository [#f5]_ contains a
26011986Sandreas.sandberg@arm.comsimple example repository which uses this approach.
26111986Sandreas.sandberg@arm.com
26211986Sandreas.sandberg@arm.comThere are two potential gotchas when using this approach: first, make sure that
26311986Sandreas.sandberg@arm.comthe resulting strings do not contain any :kbd:`TAB` characters, which break the
26411986Sandreas.sandberg@arm.comdocstring parsing routines. You may want to use C++11 raw string literals,
26511986Sandreas.sandberg@arm.comwhich are convenient for multi-line comments. Conveniently, any excess
26611986Sandreas.sandberg@arm.comindentation will be automatically be removed by Sphinx. However, for this to
26711986Sandreas.sandberg@arm.comwork, it is important that all lines are indented consistently, i.e.:
26811986Sandreas.sandberg@arm.com
26911986Sandreas.sandberg@arm.com.. code-block:: cpp
27011986Sandreas.sandberg@arm.com
27111986Sandreas.sandberg@arm.com    // ok
27211986Sandreas.sandberg@arm.com    m.def("foo", &foo, R"mydelimiter(
27311986Sandreas.sandberg@arm.com        The foo function
27411986Sandreas.sandberg@arm.com
27511986Sandreas.sandberg@arm.com        Parameters
27611986Sandreas.sandberg@arm.com        ----------
27711986Sandreas.sandberg@arm.com    )mydelimiter");
27811986Sandreas.sandberg@arm.com
27911986Sandreas.sandberg@arm.com    // *not ok*
28011986Sandreas.sandberg@arm.com    m.def("foo", &foo, R"mydelimiter(The foo function
28111986Sandreas.sandberg@arm.com
28211986Sandreas.sandberg@arm.com        Parameters
28311986Sandreas.sandberg@arm.com        ----------
28411986Sandreas.sandberg@arm.com    )mydelimiter");
28511986Sandreas.sandberg@arm.com
28611986Sandreas.sandberg@arm.comBy default, pybind11 automatically generates and prepends a signature to the docstring of a function 
28711986Sandreas.sandberg@arm.comregistered with ``module::def()`` and ``class_::def()``. Sometimes this
28811986Sandreas.sandberg@arm.combehavior is not desirable, because you want to provide your own signature or remove 
28911986Sandreas.sandberg@arm.comthe docstring completely to exclude the function from the Sphinx documentation.
29011986Sandreas.sandberg@arm.comThe class ``options`` allows you to selectively suppress auto-generated signatures:
29111986Sandreas.sandberg@arm.com
29211986Sandreas.sandberg@arm.com.. code-block:: cpp
29311986Sandreas.sandberg@arm.com
29412391Sjason@lowepower.com    PYBIND11_MODULE(example, m) {
29511986Sandreas.sandberg@arm.com        py::options options;
29611986Sandreas.sandberg@arm.com        options.disable_function_signatures();
29712391Sjason@lowepower.com
29811986Sandreas.sandberg@arm.com        m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers");
29911986Sandreas.sandberg@arm.com    }
30011986Sandreas.sandberg@arm.com
30111986Sandreas.sandberg@arm.comNote that changes to the settings affect only function bindings created during the 
30211986Sandreas.sandberg@arm.comlifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function, 
30311986Sandreas.sandberg@arm.comthe default settings are restored to prevent unwanted side effects.
30411986Sandreas.sandberg@arm.com
30511986Sandreas.sandberg@arm.com.. [#f4] http://www.sphinx-doc.org
30611986Sandreas.sandberg@arm.com.. [#f5] http://github.com/pybind/python_example
307