stl.rst revision 12391
111986Sandreas.sandberg@arm.comSTL containers 211986Sandreas.sandberg@arm.com############## 311986Sandreas.sandberg@arm.com 411986Sandreas.sandberg@arm.comAutomatic conversion 511986Sandreas.sandberg@arm.com==================== 611986Sandreas.sandberg@arm.com 711986Sandreas.sandberg@arm.comWhen including the additional header file :file:`pybind11/stl.h`, conversions 812037Sandreas.sandberg@arm.combetween ``std::vector<>``/``std::list<>``/``std::array<>``, 912037Sandreas.sandberg@arm.com``std::set<>``/``std::unordered_set<>``, and 1012037Sandreas.sandberg@arm.com``std::map<>``/``std::unordered_map<>`` and the Python ``list``, ``set`` and 1112037Sandreas.sandberg@arm.com``dict`` data structures are automatically enabled. The types ``std::pair<>`` 1212037Sandreas.sandberg@arm.comand ``std::tuple<>`` are already supported out of the box with just the core 1312037Sandreas.sandberg@arm.com:file:`pybind11/pybind11.h` header. 1411986Sandreas.sandberg@arm.com 1511986Sandreas.sandberg@arm.comThe major downside of these implicit conversions is that containers must be 1611986Sandreas.sandberg@arm.comconverted (i.e. copied) on every Python->C++ and C++->Python transition, which 1711986Sandreas.sandberg@arm.comcan have implications on the program semantics and performance. Please read the 1811986Sandreas.sandberg@arm.comnext sections for more details and alternative approaches that avoid this. 1911986Sandreas.sandberg@arm.com 2011986Sandreas.sandberg@arm.com.. note:: 2111986Sandreas.sandberg@arm.com 2211986Sandreas.sandberg@arm.com Arbitrary nesting of any of these types is possible. 2311986Sandreas.sandberg@arm.com 2411986Sandreas.sandberg@arm.com.. seealso:: 2511986Sandreas.sandberg@arm.com 2612391Sjason@lowepower.com The file :file:`tests/test_stl.cpp` contains a complete 2711986Sandreas.sandberg@arm.com example that demonstrates how to pass STL data types in more detail. 2811986Sandreas.sandberg@arm.com 2912391Sjason@lowepower.com.. _cpp17_container_casters: 3012391Sjason@lowepower.com 3112391Sjason@lowepower.comC++17 library containers 3212391Sjason@lowepower.com======================== 3312391Sjason@lowepower.com 3412391Sjason@lowepower.comThe :file:`pybind11/stl.h` header also includes support for ``std::optional<>`` 3512391Sjason@lowepower.comand ``std::variant<>``. These require a C++17 compiler and standard library. 3612391Sjason@lowepower.comIn C++14 mode, ``std::experimental::optional<>`` is supported if available. 3712391Sjason@lowepower.com 3812391Sjason@lowepower.comVarious versions of these containers also exist for C++11 (e.g. in Boost). 3912391Sjason@lowepower.compybind11 provides an easy way to specialize the ``type_caster`` for such 4012391Sjason@lowepower.comtypes: 4112391Sjason@lowepower.com 4212391Sjason@lowepower.com.. code-block:: cpp 4312391Sjason@lowepower.com 4412391Sjason@lowepower.com // `boost::optional` as an example -- can be any `std::optional`-like container 4512391Sjason@lowepower.com namespace pybind11 { namespace detail { 4612391Sjason@lowepower.com template <typename T> 4712391Sjason@lowepower.com struct type_caster<boost::optional<T>> : optional_caster<boost::optional<T>> {}; 4812391Sjason@lowepower.com }} 4912391Sjason@lowepower.com 5012391Sjason@lowepower.comThe above should be placed in a header file and included in all translation units 5112391Sjason@lowepower.comwhere automatic conversion is needed. Similarly, a specialization can be provided 5212391Sjason@lowepower.comfor custom variant types: 5312391Sjason@lowepower.com 5412391Sjason@lowepower.com.. code-block:: cpp 5512391Sjason@lowepower.com 5612391Sjason@lowepower.com // `boost::variant` as an example -- can be any `std::variant`-like container 5712391Sjason@lowepower.com namespace pybind11 { namespace detail { 5812391Sjason@lowepower.com template <typename... Ts> 5912391Sjason@lowepower.com struct type_caster<boost::variant<Ts...>> : variant_caster<boost::variant<Ts...>> {}; 6012391Sjason@lowepower.com 6112391Sjason@lowepower.com // Specifies the function used to visit the variant -- `apply_visitor` instead of `visit` 6212391Sjason@lowepower.com template <> 6312391Sjason@lowepower.com struct visit_helper<boost::variant> { 6412391Sjason@lowepower.com template <typename... Args> 6512391Sjason@lowepower.com static auto call(Args &&...args) -> decltype(boost::apply_visitor(args...)) { 6612391Sjason@lowepower.com return boost::apply_visitor(args...); 6712391Sjason@lowepower.com } 6812391Sjason@lowepower.com }; 6912391Sjason@lowepower.com }} // namespace pybind11::detail 7012391Sjason@lowepower.com 7112391Sjason@lowepower.comThe ``visit_helper`` specialization is not required if your ``name::variant`` provides 7212391Sjason@lowepower.coma ``name::visit()`` function. For any other function name, the specialization must be 7312391Sjason@lowepower.comincluded to tell pybind11 how to visit the variant. 7412391Sjason@lowepower.com 7512391Sjason@lowepower.com.. note:: 7612391Sjason@lowepower.com 7712391Sjason@lowepower.com pybind11 only supports the modern implementation of ``boost::variant`` 7812391Sjason@lowepower.com which makes use of variadic templates. This requires Boost 1.56 or newer. 7912391Sjason@lowepower.com Additionally, on Windows, MSVC 2017 is required because ``boost::variant`` 8012391Sjason@lowepower.com falls back to the old non-variadic implementation on MSVC 2015. 8112391Sjason@lowepower.com 8211986Sandreas.sandberg@arm.com.. _opaque: 8311986Sandreas.sandberg@arm.com 8411986Sandreas.sandberg@arm.comMaking opaque types 8511986Sandreas.sandberg@arm.com=================== 8611986Sandreas.sandberg@arm.com 8711986Sandreas.sandberg@arm.compybind11 heavily relies on a template matching mechanism to convert parameters 8811986Sandreas.sandberg@arm.comand return values that are constructed from STL data types such as vectors, 8911986Sandreas.sandberg@arm.comlinked lists, hash tables, etc. This even works in a recursive manner, for 9011986Sandreas.sandberg@arm.cominstance to deal with lists of hash maps of pairs of elementary and custom 9111986Sandreas.sandberg@arm.comtypes, etc. 9211986Sandreas.sandberg@arm.com 9311986Sandreas.sandberg@arm.comHowever, a fundamental limitation of this approach is that internal conversions 9411986Sandreas.sandberg@arm.combetween Python and C++ types involve a copy operation that prevents 9511986Sandreas.sandberg@arm.compass-by-reference semantics. What does this mean? 9611986Sandreas.sandberg@arm.com 9711986Sandreas.sandberg@arm.comSuppose we bind the following function 9811986Sandreas.sandberg@arm.com 9911986Sandreas.sandberg@arm.com.. code-block:: cpp 10011986Sandreas.sandberg@arm.com 10111986Sandreas.sandberg@arm.com void append_1(std::vector<int> &v) { 10211986Sandreas.sandberg@arm.com v.push_back(1); 10311986Sandreas.sandberg@arm.com } 10411986Sandreas.sandberg@arm.com 10511986Sandreas.sandberg@arm.comand call it from Python, the following happens: 10611986Sandreas.sandberg@arm.com 10711986Sandreas.sandberg@arm.com.. code-block:: pycon 10811986Sandreas.sandberg@arm.com 10911986Sandreas.sandberg@arm.com >>> v = [5, 6] 11011986Sandreas.sandberg@arm.com >>> append_1(v) 11111986Sandreas.sandberg@arm.com >>> print(v) 11211986Sandreas.sandberg@arm.com [5, 6] 11311986Sandreas.sandberg@arm.com 11411986Sandreas.sandberg@arm.comAs you can see, when passing STL data structures by reference, modifications 11511986Sandreas.sandberg@arm.comare not propagated back the Python side. A similar situation arises when 11611986Sandreas.sandberg@arm.comexposing STL data structures using the ``def_readwrite`` or ``def_readonly`` 11711986Sandreas.sandberg@arm.comfunctions: 11811986Sandreas.sandberg@arm.com 11911986Sandreas.sandberg@arm.com.. code-block:: cpp 12011986Sandreas.sandberg@arm.com 12111986Sandreas.sandberg@arm.com /* ... definition ... */ 12211986Sandreas.sandberg@arm.com 12311986Sandreas.sandberg@arm.com class MyClass { 12411986Sandreas.sandberg@arm.com std::vector<int> contents; 12511986Sandreas.sandberg@arm.com }; 12611986Sandreas.sandberg@arm.com 12711986Sandreas.sandberg@arm.com /* ... binding code ... */ 12811986Sandreas.sandberg@arm.com 12911986Sandreas.sandberg@arm.com py::class_<MyClass>(m, "MyClass") 13012037Sandreas.sandberg@arm.com .def(py::init<>()) 13111986Sandreas.sandberg@arm.com .def_readwrite("contents", &MyClass::contents); 13211986Sandreas.sandberg@arm.com 13311986Sandreas.sandberg@arm.comIn this case, properties can be read and written in their entirety. However, an 13411986Sandreas.sandberg@arm.com``append`` operation involving such a list type has no effect: 13511986Sandreas.sandberg@arm.com 13611986Sandreas.sandberg@arm.com.. code-block:: pycon 13711986Sandreas.sandberg@arm.com 13811986Sandreas.sandberg@arm.com >>> m = MyClass() 13911986Sandreas.sandberg@arm.com >>> m.contents = [5, 6] 14011986Sandreas.sandberg@arm.com >>> print(m.contents) 14111986Sandreas.sandberg@arm.com [5, 6] 14211986Sandreas.sandberg@arm.com >>> m.contents.append(7) 14311986Sandreas.sandberg@arm.com >>> print(m.contents) 14411986Sandreas.sandberg@arm.com [5, 6] 14511986Sandreas.sandberg@arm.com 14611986Sandreas.sandberg@arm.comFinally, the involved copy operations can be costly when dealing with very 14711986Sandreas.sandberg@arm.comlarge lists. To deal with all of the above situations, pybind11 provides a 14811986Sandreas.sandberg@arm.commacro named ``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based 14911986Sandreas.sandberg@arm.comconversion machinery of types, thus rendering them *opaque*. The contents of 15011986Sandreas.sandberg@arm.comopaque objects are never inspected or extracted, hence they *can* be passed by 15111986Sandreas.sandberg@arm.comreference. For instance, to turn ``std::vector<int>`` into an opaque type, add 15211986Sandreas.sandberg@arm.comthe declaration 15311986Sandreas.sandberg@arm.com 15411986Sandreas.sandberg@arm.com.. code-block:: cpp 15511986Sandreas.sandberg@arm.com 15611986Sandreas.sandberg@arm.com PYBIND11_MAKE_OPAQUE(std::vector<int>); 15711986Sandreas.sandberg@arm.com 15811986Sandreas.sandberg@arm.combefore any binding code (e.g. invocations to ``class_::def()``, etc.). This 15911986Sandreas.sandberg@arm.commacro must be specified at the top level (and outside of any namespaces), since 16011986Sandreas.sandberg@arm.comit instantiates a partial template overload. If your binding code consists of 16112391Sjason@lowepower.commultiple compilation units, it must be present in every file (typically via a 16212391Sjason@lowepower.comcommon header) preceding any usage of ``std::vector<int>``. Opaque types must 16312391Sjason@lowepower.comalso have a corresponding ``class_`` declaration to associate them with a name 16412391Sjason@lowepower.comin Python, and to define a set of available operations, e.g.: 16511986Sandreas.sandberg@arm.com 16611986Sandreas.sandberg@arm.com.. code-block:: cpp 16711986Sandreas.sandberg@arm.com 16811986Sandreas.sandberg@arm.com py::class_<std::vector<int>>(m, "IntVector") 16911986Sandreas.sandberg@arm.com .def(py::init<>()) 17011986Sandreas.sandberg@arm.com .def("clear", &std::vector<int>::clear) 17111986Sandreas.sandberg@arm.com .def("pop_back", &std::vector<int>::pop_back) 17211986Sandreas.sandberg@arm.com .def("__len__", [](const std::vector<int> &v) { return v.size(); }) 17311986Sandreas.sandberg@arm.com .def("__iter__", [](std::vector<int> &v) { 17411986Sandreas.sandberg@arm.com return py::make_iterator(v.begin(), v.end()); 17511986Sandreas.sandberg@arm.com }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */ 17611986Sandreas.sandberg@arm.com // .... 17711986Sandreas.sandberg@arm.com 17812391Sjason@lowepower.comPlease take a look at the :ref:`macro_notes` before using the 17912391Sjason@lowepower.com``PYBIND11_MAKE_OPAQUE`` macro. 18012391Sjason@lowepower.com 18112391Sjason@lowepower.com.. seealso:: 18212391Sjason@lowepower.com 18312391Sjason@lowepower.com The file :file:`tests/test_opaque_types.cpp` contains a complete 18412391Sjason@lowepower.com example that demonstrates how to create and expose opaque types using 18512391Sjason@lowepower.com pybind11 in more detail. 18612391Sjason@lowepower.com 18712391Sjason@lowepower.com.. _stl_bind: 18812391Sjason@lowepower.com 18912391Sjason@lowepower.comBinding STL containers 19012391Sjason@lowepower.com====================== 19112391Sjason@lowepower.com 19211986Sandreas.sandberg@arm.comThe ability to expose STL containers as native Python objects is a fairly 19311986Sandreas.sandberg@arm.comcommon request, hence pybind11 also provides an optional header file named 19411986Sandreas.sandberg@arm.com:file:`pybind11/stl_bind.h` that does exactly this. The mapped containers try 19511986Sandreas.sandberg@arm.comto match the behavior of their native Python counterparts as much as possible. 19611986Sandreas.sandberg@arm.com 19711986Sandreas.sandberg@arm.comThe following example showcases usage of :file:`pybind11/stl_bind.h`: 19811986Sandreas.sandberg@arm.com 19911986Sandreas.sandberg@arm.com.. code-block:: cpp 20011986Sandreas.sandberg@arm.com 20111986Sandreas.sandberg@arm.com // Don't forget this 20211986Sandreas.sandberg@arm.com #include <pybind11/stl_bind.h> 20311986Sandreas.sandberg@arm.com 20411986Sandreas.sandberg@arm.com PYBIND11_MAKE_OPAQUE(std::vector<int>); 20511986Sandreas.sandberg@arm.com PYBIND11_MAKE_OPAQUE(std::map<std::string, double>); 20611986Sandreas.sandberg@arm.com 20711986Sandreas.sandberg@arm.com // ... 20811986Sandreas.sandberg@arm.com 20911986Sandreas.sandberg@arm.com // later in binding code: 21011986Sandreas.sandberg@arm.com py::bind_vector<std::vector<int>>(m, "VectorInt"); 21111986Sandreas.sandberg@arm.com py::bind_map<std::map<std::string, double>>(m, "MapStringDouble"); 21211986Sandreas.sandberg@arm.com 21312391Sjason@lowepower.comWhen binding STL containers pybind11 considers the types of the container's 21412391Sjason@lowepower.comelements to decide whether the container should be confined to the local module 21512391Sjason@lowepower.com(via the :ref:`module_local` feature). If the container element types are 21612391Sjason@lowepower.comanything other than already-bound custom types bound without 21712391Sjason@lowepower.com``py::module_local()`` the container binding will have ``py::module_local()`` 21812391Sjason@lowepower.comapplied. This includes converting types such as numeric types, strings, Eigen 21912391Sjason@lowepower.comtypes; and types that have not yet been bound at the time of the stl container 22012391Sjason@lowepower.combinding. This module-local binding is designed to avoid potential conflicts 22112391Sjason@lowepower.combetween module bindings (for example, from two separate modules each attempting 22212391Sjason@lowepower.comto bind ``std::vector<int>`` as a python type). 22312391Sjason@lowepower.com 22412391Sjason@lowepower.comIt is possible to override this behavior to force a definition to be either 22512391Sjason@lowepower.commodule-local or global. To do so, you can pass the attributes 22612391Sjason@lowepower.com``py::module_local()`` (to make the binding module-local) or 22712391Sjason@lowepower.com``py::module_local(false)`` (to make the binding global) into the 22812391Sjason@lowepower.com``py::bind_vector`` or ``py::bind_map`` arguments: 22912391Sjason@lowepower.com 23012391Sjason@lowepower.com.. code-block:: cpp 23112391Sjason@lowepower.com 23212391Sjason@lowepower.com py::bind_vector<std::vector<int>>(m, "VectorInt", py::module_local(false)); 23312391Sjason@lowepower.com 23412391Sjason@lowepower.comNote, however, that such a global binding would make it impossible to load this 23512391Sjason@lowepower.commodule at the same time as any other pybind module that also attempts to bind 23612391Sjason@lowepower.comthe same container type (``std::vector<int>`` in the above example). 23712391Sjason@lowepower.com 23812391Sjason@lowepower.comSee :ref:`module_local` for more details on module-local bindings. 23911986Sandreas.sandberg@arm.com 24011986Sandreas.sandberg@arm.com.. seealso:: 24111986Sandreas.sandberg@arm.com 24211986Sandreas.sandberg@arm.com The file :file:`tests/test_stl_binders.cpp` shows how to use the 24311986Sandreas.sandberg@arm.com convenience STL container wrappers. 244