faq.rst revision 12391
111986Sandreas.sandberg@arm.comFrequently asked questions 211986Sandreas.sandberg@arm.com########################## 311986Sandreas.sandberg@arm.com 411986Sandreas.sandberg@arm.com"ImportError: dynamic module does not define init function" 511986Sandreas.sandberg@arm.com=========================================================== 611986Sandreas.sandberg@arm.com 712391Sjason@lowepower.comYou are likely using an incompatible version of Python (for instance, the 812391Sjason@lowepower.comextension library was compiled against Python 2, while the interpreter is 912391Sjason@lowepower.comrunning on top of some version of Python 3, or vice versa). 1011986Sandreas.sandberg@arm.com 1111986Sandreas.sandberg@arm.com"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" 1211986Sandreas.sandberg@arm.com======================================================================== 1311986Sandreas.sandberg@arm.com 1412391Sjason@lowepower.comSee the first answer. 1511986Sandreas.sandberg@arm.com 1611986Sandreas.sandberg@arm.com"SystemError: dynamic module not initialized properly" 1711986Sandreas.sandberg@arm.com====================================================== 1811986Sandreas.sandberg@arm.com 1912391Sjason@lowepower.comSee the first answer. 2011986Sandreas.sandberg@arm.com 2111986Sandreas.sandberg@arm.comThe Python interpreter immediately crashes when importing my module 2211986Sandreas.sandberg@arm.com=================================================================== 2311986Sandreas.sandberg@arm.com 2412391Sjason@lowepower.comSee the first answer. 2511986Sandreas.sandberg@arm.com 2611986Sandreas.sandberg@arm.comCMake doesn't detect the right Python version 2711986Sandreas.sandberg@arm.com============================================= 2811986Sandreas.sandberg@arm.com 2911986Sandreas.sandberg@arm.comThe CMake-based build system will try to automatically detect the installed 3011986Sandreas.sandberg@arm.comversion of Python and link against that. When this fails, or when there are 3111986Sandreas.sandberg@arm.commultiple versions of Python and it finds the wrong one, delete 3211986Sandreas.sandberg@arm.com``CMakeCache.txt`` and then invoke CMake as follows: 3311986Sandreas.sandberg@arm.com 3411986Sandreas.sandberg@arm.com.. code-block:: bash 3511986Sandreas.sandberg@arm.com 3611986Sandreas.sandberg@arm.com cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> . 3711986Sandreas.sandberg@arm.com 3811986Sandreas.sandberg@arm.comLimitations involving reference arguments 3911986Sandreas.sandberg@arm.com========================================= 4011986Sandreas.sandberg@arm.com 4111986Sandreas.sandberg@arm.comIn C++, it's fairly common to pass arguments using mutable references or 4211986Sandreas.sandberg@arm.commutable pointers, which allows both read and write access to the value 4311986Sandreas.sandberg@arm.comsupplied by the caller. This is sometimes done for efficiency reasons, or to 4411986Sandreas.sandberg@arm.comrealize functions that have multiple return values. Here are two very basic 4511986Sandreas.sandberg@arm.comexamples: 4611986Sandreas.sandberg@arm.com 4711986Sandreas.sandberg@arm.com.. code-block:: cpp 4811986Sandreas.sandberg@arm.com 4911986Sandreas.sandberg@arm.com void increment(int &i) { i++; } 5011986Sandreas.sandberg@arm.com void increment_ptr(int *i) { (*i)++; } 5111986Sandreas.sandberg@arm.com 5211986Sandreas.sandberg@arm.comIn Python, all arguments are passed by reference, so there is no general 5311986Sandreas.sandberg@arm.comissue in binding such code from Python. 5411986Sandreas.sandberg@arm.com 5511986Sandreas.sandberg@arm.comHowever, certain basic Python types (like ``str``, ``int``, ``bool``, 5611986Sandreas.sandberg@arm.com``float``, etc.) are **immutable**. This means that the following attempt 5711986Sandreas.sandberg@arm.comto port the function to Python doesn't have the same effect on the value 5811986Sandreas.sandberg@arm.comprovided by the caller -- in fact, it does nothing at all. 5911986Sandreas.sandberg@arm.com 6011986Sandreas.sandberg@arm.com.. code-block:: python 6111986Sandreas.sandberg@arm.com 6211986Sandreas.sandberg@arm.com def increment(i): 6311986Sandreas.sandberg@arm.com i += 1 # nope.. 6411986Sandreas.sandberg@arm.com 6511986Sandreas.sandberg@arm.compybind11 is also affected by such language-level conventions, which means that 6611986Sandreas.sandberg@arm.combinding ``increment`` or ``increment_ptr`` will also create Python functions 6711986Sandreas.sandberg@arm.comthat don't modify their arguments. 6811986Sandreas.sandberg@arm.com 6911986Sandreas.sandberg@arm.comAlthough inconvenient, one workaround is to encapsulate the immutable types in 7011986Sandreas.sandberg@arm.coma custom type that does allow modifications. 7111986Sandreas.sandberg@arm.com 7211986Sandreas.sandberg@arm.comAn other alternative involves binding a small wrapper lambda function that 7311986Sandreas.sandberg@arm.comreturns a tuple with all output arguments (see the remainder of the 7411986Sandreas.sandberg@arm.comdocumentation for examples on binding lambda functions). An example: 7511986Sandreas.sandberg@arm.com 7611986Sandreas.sandberg@arm.com.. code-block:: cpp 7711986Sandreas.sandberg@arm.com 7811986Sandreas.sandberg@arm.com int foo(int &i) { i++; return 123; } 7911986Sandreas.sandberg@arm.com 8011986Sandreas.sandberg@arm.comand the binding code 8111986Sandreas.sandberg@arm.com 8211986Sandreas.sandberg@arm.com.. code-block:: cpp 8311986Sandreas.sandberg@arm.com 8411986Sandreas.sandberg@arm.com m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); 8511986Sandreas.sandberg@arm.com 8611986Sandreas.sandberg@arm.com 8711986Sandreas.sandberg@arm.comHow can I reduce the build time? 8811986Sandreas.sandberg@arm.com================================ 8911986Sandreas.sandberg@arm.com 9011986Sandreas.sandberg@arm.comIt's good practice to split binding code over multiple files, as in the 9111986Sandreas.sandberg@arm.comfollowing example: 9211986Sandreas.sandberg@arm.com 9311986Sandreas.sandberg@arm.com:file:`example.cpp`: 9411986Sandreas.sandberg@arm.com 9511986Sandreas.sandberg@arm.com.. code-block:: cpp 9611986Sandreas.sandberg@arm.com 9711986Sandreas.sandberg@arm.com void init_ex1(py::module &); 9811986Sandreas.sandberg@arm.com void init_ex2(py::module &); 9911986Sandreas.sandberg@arm.com /* ... */ 10011986Sandreas.sandberg@arm.com 10112391Sjason@lowepower.com PYBIND11_MODULE(example, m) { 10211986Sandreas.sandberg@arm.com init_ex1(m); 10311986Sandreas.sandberg@arm.com init_ex2(m); 10411986Sandreas.sandberg@arm.com /* ... */ 10511986Sandreas.sandberg@arm.com } 10611986Sandreas.sandberg@arm.com 10711986Sandreas.sandberg@arm.com:file:`ex1.cpp`: 10811986Sandreas.sandberg@arm.com 10911986Sandreas.sandberg@arm.com.. code-block:: cpp 11011986Sandreas.sandberg@arm.com 11111986Sandreas.sandberg@arm.com void init_ex1(py::module &m) { 11211986Sandreas.sandberg@arm.com m.def("add", [](int a, int b) { return a + b; }); 11311986Sandreas.sandberg@arm.com } 11411986Sandreas.sandberg@arm.com 11511986Sandreas.sandberg@arm.com:file:`ex2.cpp`: 11611986Sandreas.sandberg@arm.com 11711986Sandreas.sandberg@arm.com.. code-block:: cpp 11811986Sandreas.sandberg@arm.com 11911986Sandreas.sandberg@arm.com void init_ex1(py::module &m) { 12011986Sandreas.sandberg@arm.com m.def("sub", [](int a, int b) { return a - b; }); 12111986Sandreas.sandberg@arm.com } 12211986Sandreas.sandberg@arm.com 12311986Sandreas.sandberg@arm.com:command:`python`: 12411986Sandreas.sandberg@arm.com 12511986Sandreas.sandberg@arm.com.. code-block:: pycon 12611986Sandreas.sandberg@arm.com 12711986Sandreas.sandberg@arm.com >>> import example 12811986Sandreas.sandberg@arm.com >>> example.add(1, 2) 12911986Sandreas.sandberg@arm.com 3 13011986Sandreas.sandberg@arm.com >>> example.sub(1, 1) 13111986Sandreas.sandberg@arm.com 0 13211986Sandreas.sandberg@arm.com 13311986Sandreas.sandberg@arm.comAs shown above, the various ``init_ex`` functions should be contained in 13411986Sandreas.sandberg@arm.comseparate files that can be compiled independently from one another, and then 13511986Sandreas.sandberg@arm.comlinked together into the same final shared object. Following this approach 13611986Sandreas.sandberg@arm.comwill: 13711986Sandreas.sandberg@arm.com 13811986Sandreas.sandberg@arm.com1. reduce memory requirements per compilation unit. 13911986Sandreas.sandberg@arm.com 14011986Sandreas.sandberg@arm.com2. enable parallel builds (if desired). 14111986Sandreas.sandberg@arm.com 14211986Sandreas.sandberg@arm.com3. allow for faster incremental builds. For instance, when a single class 14311986Sandreas.sandberg@arm.com definition is changed, only a subset of the binding code will generally need 14411986Sandreas.sandberg@arm.com to be recompiled. 14511986Sandreas.sandberg@arm.com 14611986Sandreas.sandberg@arm.com"recursive template instantiation exceeded maximum depth of 256" 14711986Sandreas.sandberg@arm.com================================================================ 14811986Sandreas.sandberg@arm.com 14911986Sandreas.sandberg@arm.comIf you receive an error about excessive recursive template evaluation, try 15011986Sandreas.sandberg@arm.comspecifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The 15111986Sandreas.sandberg@arm.comculprit is generally the generation of function signatures at compile time 15211986Sandreas.sandberg@arm.comusing C++14 template metaprogramming. 15311986Sandreas.sandberg@arm.com 15412391Sjason@lowepower.com.. _`faq:hidden_visibility`: 15512391Sjason@lowepower.com 15612391Sjason@lowepower.com"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]" 15712391Sjason@lowepower.com============================================================================================================ 15812391Sjason@lowepower.com 15912391Sjason@lowepower.comThis error typically indicates that you are compiling without the required 16012391Sjason@lowepower.com``-fvisibility`` flag. pybind11 code internally forces hidden visibility on 16112391Sjason@lowepower.comall internal code, but if non-hidden (and thus *exported*) code attempts to 16212391Sjason@lowepower.cominclude a pybind type (for example, ``py::object`` or ``py::list``) you can run 16312391Sjason@lowepower.cominto this warning. 16412391Sjason@lowepower.com 16512391Sjason@lowepower.comTo avoid it, make sure you are specifying ``-fvisibility=hidden`` when 16612391Sjason@lowepower.comcompiling pybind code. 16712391Sjason@lowepower.com 16812391Sjason@lowepower.comAs to why ``-fvisibility=hidden`` is necessary, because pybind modules could 16912391Sjason@lowepower.comhave been compiled under different versions of pybind itself, it is also 17012391Sjason@lowepower.comimportant that the symbols defined in one module do not clash with the 17112391Sjason@lowepower.compotentially-incompatible symbols defined in another. While Python extension 17212391Sjason@lowepower.commodules are usually loaded with localized symbols (under POSIX systems 17312391Sjason@lowepower.comtypically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default 17412391Sjason@lowepower.comcan be changed, but even if it isn't it is not always enough to guarantee 17512391Sjason@lowepower.comcomplete independence of the symbols involved when not using 17612391Sjason@lowepower.com``-fvisibility=hidden``. 17712391Sjason@lowepower.com 17812391Sjason@lowepower.comAdditionally, ``-fvisiblity=hidden`` can deliver considerably binary size 17912391Sjason@lowepower.comsavings. (See the following section for more details). 18012391Sjason@lowepower.com 18111986Sandreas.sandberg@arm.com 18211986Sandreas.sandberg@arm.com.. _`faq:symhidden`: 18311986Sandreas.sandberg@arm.com 18411986Sandreas.sandberg@arm.comHow can I create smaller binaries? 18511986Sandreas.sandberg@arm.com================================== 18611986Sandreas.sandberg@arm.com 18711986Sandreas.sandberg@arm.comTo do its job, pybind11 extensively relies on a programming technique known as 18811986Sandreas.sandberg@arm.com*template metaprogramming*, which is a way of performing computation at compile 18911986Sandreas.sandberg@arm.comtime using type information. Template metaprogamming usually instantiates code 19011986Sandreas.sandberg@arm.cominvolving significant numbers of deeply nested types that are either completely 19111986Sandreas.sandberg@arm.comremoved or reduced to just a few instructions during the compiler's optimization 19211986Sandreas.sandberg@arm.comphase. However, due to the nested nature of these types, the resulting symbol 19311986Sandreas.sandberg@arm.comnames in the compiled extension library can be extremely long. For instance, 19411986Sandreas.sandberg@arm.comthe included test suite contains the following symbol: 19511986Sandreas.sandberg@arm.com 19611986Sandreas.sandberg@arm.com.. only:: html 19711986Sandreas.sandberg@arm.com 19811986Sandreas.sandberg@arm.com .. code-block:: none 19911986Sandreas.sandberg@arm.com 20011986Sandreas.sandberg@arm.com __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 20111986Sandreas.sandberg@arm.com 20211986Sandreas.sandberg@arm.com.. only:: not html 20311986Sandreas.sandberg@arm.com 20411986Sandreas.sandberg@arm.com .. code-block:: cpp 20511986Sandreas.sandberg@arm.com 20611986Sandreas.sandberg@arm.com __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 20711986Sandreas.sandberg@arm.com 20811986Sandreas.sandberg@arm.comwhich is the mangled form of the following function type: 20911986Sandreas.sandberg@arm.com 21011986Sandreas.sandberg@arm.com.. code-block:: cpp 21111986Sandreas.sandberg@arm.com 21211986Sandreas.sandberg@arm.com pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) 21311986Sandreas.sandberg@arm.com 21411986Sandreas.sandberg@arm.comThe memory needed to store just the mangled name of this function (196 bytes) 21511986Sandreas.sandberg@arm.comis larger than the actual piece of code (111 bytes) it represents! On the other 21611986Sandreas.sandberg@arm.comhand, it's silly to even give this function a name -- after all, it's just a 21711986Sandreas.sandberg@arm.comtiny cog in a bigger piece of machinery that is not exposed to the outside 21811986Sandreas.sandberg@arm.comworld. So we'll generally only want to export symbols for those functions which 21911986Sandreas.sandberg@arm.comare actually called from the outside. 22011986Sandreas.sandberg@arm.com 22111986Sandreas.sandberg@arm.comThis can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC 22212391Sjason@lowepower.comand Clang, which sets the default symbol visibility to *hidden*, which has a 22312391Sjason@lowepower.comtremendous impact on the final binary size of the resulting extension library. 22412391Sjason@lowepower.com(On Visual Studio, symbols are already hidden by default, so nothing needs to 22512391Sjason@lowepower.combe done there.) 22612391Sjason@lowepower.com 22712391Sjason@lowepower.comIn addition to decreasing binary size, ``-fvisibility=hidden`` also avoids 22812391Sjason@lowepower.compotential serious issues when loading multiple modules and is required for 22912391Sjason@lowepower.comproper pybind operation. See the previous FAQ entry for more details. 23011986Sandreas.sandberg@arm.com 23111986Sandreas.sandberg@arm.comAnother aspect that can require a fair bit of code are function signature 23211986Sandreas.sandberg@arm.comdescriptions. pybind11 automatically generates human-readable function 23311986Sandreas.sandberg@arm.comsignatures for docstrings, e.g.: 23411986Sandreas.sandberg@arm.com 23511986Sandreas.sandberg@arm.com.. code-block:: none 23611986Sandreas.sandberg@arm.com 23711986Sandreas.sandberg@arm.com | __init__(...) 23811986Sandreas.sandberg@arm.com | __init__(*args, **kwargs) 23911986Sandreas.sandberg@arm.com | Overloaded function. 24011986Sandreas.sandberg@arm.com | 24111986Sandreas.sandberg@arm.com | 1. __init__(example.Example1) -> NoneType 24211986Sandreas.sandberg@arm.com | 24311986Sandreas.sandberg@arm.com | Docstring for overload #1 goes here 24411986Sandreas.sandberg@arm.com | 24511986Sandreas.sandberg@arm.com | 2. __init__(example.Example1, int) -> NoneType 24611986Sandreas.sandberg@arm.com | 24711986Sandreas.sandberg@arm.com | Docstring for overload #2 goes here 24811986Sandreas.sandberg@arm.com | 24911986Sandreas.sandberg@arm.com | 3. __init__(example.Example1, example.Example1) -> NoneType 25011986Sandreas.sandberg@arm.com | 25111986Sandreas.sandberg@arm.com | Docstring for overload #3 goes here 25211986Sandreas.sandberg@arm.com 25311986Sandreas.sandberg@arm.com 25411986Sandreas.sandberg@arm.comIn C++11 mode, these are generated at run time using string concatenation, 25511986Sandreas.sandberg@arm.comwhich can amount to 10-20% of the size of the resulting binary. If you can, 25611986Sandreas.sandberg@arm.comenable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which 25711986Sandreas.sandberg@arm.comcase signatures are efficiently pre-generated at compile time. Unfortunately, 25811986Sandreas.sandberg@arm.comVisual Studio's C++14 support (``constexpr``) is not good enough as of April 25911986Sandreas.sandberg@arm.com2016, so it always uses the more expensive run-time approach. 26011986Sandreas.sandberg@arm.com 26111986Sandreas.sandberg@arm.comWorking with ancient Visual Studio 2009 builds on Windows 26211986Sandreas.sandberg@arm.com========================================================= 26311986Sandreas.sandberg@arm.com 26411986Sandreas.sandberg@arm.comThe official Windows distributions of Python are compiled using truly 26511986Sandreas.sandberg@arm.comancient versions of Visual Studio that lack good C++11 support. Some users 26611986Sandreas.sandberg@arm.comimplicitly assume that it would be impossible to load a plugin built with 26711986Sandreas.sandberg@arm.comVisual Studio 2015 into a Python distribution that was compiled using Visual 26811986Sandreas.sandberg@arm.comStudio 2009. However, no such issue exists: it's perfectly legitimate to 26911986Sandreas.sandberg@arm.cominterface DLLs that are built with different compilers and/or C libraries. 27011986Sandreas.sandberg@arm.comCommon gotchas to watch out for involve not ``free()``-ing memory region 27111986Sandreas.sandberg@arm.comthat that were ``malloc()``-ed in another shared library, using data 27211986Sandreas.sandberg@arm.comstructures with incompatible ABIs, and so on. pybind11 is very careful not 27311986Sandreas.sandberg@arm.comto make these types of mistakes. 274