faq.rst revision 11986
111986Sandreas.sandberg@arm.comFrequently asked questions 211986Sandreas.sandberg@arm.com########################## 311986Sandreas.sandberg@arm.com 411986Sandreas.sandberg@arm.com"ImportError: dynamic module does not define init function" 511986Sandreas.sandberg@arm.com=========================================================== 611986Sandreas.sandberg@arm.com 711986Sandreas.sandberg@arm.com1. Make sure that the name specified in ``pybind::module`` and 811986Sandreas.sandberg@arm.com ``PYBIND11_PLUGIN`` is consistent and identical to the filename of the 911986Sandreas.sandberg@arm.com extension library. The latter should not contain any extra prefixes (e.g. 1011986Sandreas.sandberg@arm.com ``test.so`` instead of ``libtest.so``). 1111986Sandreas.sandberg@arm.com 1211986Sandreas.sandberg@arm.com2. If the above did not fix your issue, then you are likely using an 1311986Sandreas.sandberg@arm.com incompatible version of Python (for instance, the extension library was 1411986Sandreas.sandberg@arm.com compiled against Python 2, while the interpreter is running on top of some 1511986Sandreas.sandberg@arm.com version of Python 3, or vice versa) 1611986Sandreas.sandberg@arm.com 1711986Sandreas.sandberg@arm.com"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" 1811986Sandreas.sandberg@arm.com======================================================================== 1911986Sandreas.sandberg@arm.com 2011986Sandreas.sandberg@arm.comSee item 2 of the first answer. 2111986Sandreas.sandberg@arm.com 2211986Sandreas.sandberg@arm.com"SystemError: dynamic module not initialized properly" 2311986Sandreas.sandberg@arm.com====================================================== 2411986Sandreas.sandberg@arm.com 2511986Sandreas.sandberg@arm.comSee item 2 of the first answer. 2611986Sandreas.sandberg@arm.com 2711986Sandreas.sandberg@arm.comThe Python interpreter immediately crashes when importing my module 2811986Sandreas.sandberg@arm.com=================================================================== 2911986Sandreas.sandberg@arm.com 3011986Sandreas.sandberg@arm.comSee item 2 of the first answer. 3111986Sandreas.sandberg@arm.com 3211986Sandreas.sandberg@arm.comCMake doesn't detect the right Python version 3311986Sandreas.sandberg@arm.com============================================= 3411986Sandreas.sandberg@arm.com 3511986Sandreas.sandberg@arm.comThe CMake-based build system will try to automatically detect the installed 3611986Sandreas.sandberg@arm.comversion of Python and link against that. When this fails, or when there are 3711986Sandreas.sandberg@arm.commultiple versions of Python and it finds the wrong one, delete 3811986Sandreas.sandberg@arm.com``CMakeCache.txt`` and then invoke CMake as follows: 3911986Sandreas.sandberg@arm.com 4011986Sandreas.sandberg@arm.com.. code-block:: bash 4111986Sandreas.sandberg@arm.com 4211986Sandreas.sandberg@arm.com cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> . 4311986Sandreas.sandberg@arm.com 4411986Sandreas.sandberg@arm.comLimitations involving reference arguments 4511986Sandreas.sandberg@arm.com========================================= 4611986Sandreas.sandberg@arm.com 4711986Sandreas.sandberg@arm.comIn C++, it's fairly common to pass arguments using mutable references or 4811986Sandreas.sandberg@arm.commutable pointers, which allows both read and write access to the value 4911986Sandreas.sandberg@arm.comsupplied by the caller. This is sometimes done for efficiency reasons, or to 5011986Sandreas.sandberg@arm.comrealize functions that have multiple return values. Here are two very basic 5111986Sandreas.sandberg@arm.comexamples: 5211986Sandreas.sandberg@arm.com 5311986Sandreas.sandberg@arm.com.. code-block:: cpp 5411986Sandreas.sandberg@arm.com 5511986Sandreas.sandberg@arm.com void increment(int &i) { i++; } 5611986Sandreas.sandberg@arm.com void increment_ptr(int *i) { (*i)++; } 5711986Sandreas.sandberg@arm.com 5811986Sandreas.sandberg@arm.comIn Python, all arguments are passed by reference, so there is no general 5911986Sandreas.sandberg@arm.comissue in binding such code from Python. 6011986Sandreas.sandberg@arm.com 6111986Sandreas.sandberg@arm.comHowever, certain basic Python types (like ``str``, ``int``, ``bool``, 6211986Sandreas.sandberg@arm.com``float``, etc.) are **immutable**. This means that the following attempt 6311986Sandreas.sandberg@arm.comto port the function to Python doesn't have the same effect on the value 6411986Sandreas.sandberg@arm.comprovided by the caller -- in fact, it does nothing at all. 6511986Sandreas.sandberg@arm.com 6611986Sandreas.sandberg@arm.com.. code-block:: python 6711986Sandreas.sandberg@arm.com 6811986Sandreas.sandberg@arm.com def increment(i): 6911986Sandreas.sandberg@arm.com i += 1 # nope.. 7011986Sandreas.sandberg@arm.com 7111986Sandreas.sandberg@arm.compybind11 is also affected by such language-level conventions, which means that 7211986Sandreas.sandberg@arm.combinding ``increment`` or ``increment_ptr`` will also create Python functions 7311986Sandreas.sandberg@arm.comthat don't modify their arguments. 7411986Sandreas.sandberg@arm.com 7511986Sandreas.sandberg@arm.comAlthough inconvenient, one workaround is to encapsulate the immutable types in 7611986Sandreas.sandberg@arm.coma custom type that does allow modifications. 7711986Sandreas.sandberg@arm.com 7811986Sandreas.sandberg@arm.comAn other alternative involves binding a small wrapper lambda function that 7911986Sandreas.sandberg@arm.comreturns a tuple with all output arguments (see the remainder of the 8011986Sandreas.sandberg@arm.comdocumentation for examples on binding lambda functions). An example: 8111986Sandreas.sandberg@arm.com 8211986Sandreas.sandberg@arm.com.. code-block:: cpp 8311986Sandreas.sandberg@arm.com 8411986Sandreas.sandberg@arm.com int foo(int &i) { i++; return 123; } 8511986Sandreas.sandberg@arm.com 8611986Sandreas.sandberg@arm.comand the binding code 8711986Sandreas.sandberg@arm.com 8811986Sandreas.sandberg@arm.com.. code-block:: cpp 8911986Sandreas.sandberg@arm.com 9011986Sandreas.sandberg@arm.com m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); 9111986Sandreas.sandberg@arm.com 9211986Sandreas.sandberg@arm.com 9311986Sandreas.sandberg@arm.comHow can I reduce the build time? 9411986Sandreas.sandberg@arm.com================================ 9511986Sandreas.sandberg@arm.com 9611986Sandreas.sandberg@arm.comIt's good practice to split binding code over multiple files, as in the 9711986Sandreas.sandberg@arm.comfollowing example: 9811986Sandreas.sandberg@arm.com 9911986Sandreas.sandberg@arm.com:file:`example.cpp`: 10011986Sandreas.sandberg@arm.com 10111986Sandreas.sandberg@arm.com.. code-block:: cpp 10211986Sandreas.sandberg@arm.com 10311986Sandreas.sandberg@arm.com void init_ex1(py::module &); 10411986Sandreas.sandberg@arm.com void init_ex2(py::module &); 10511986Sandreas.sandberg@arm.com /* ... */ 10611986Sandreas.sandberg@arm.com 10711986Sandreas.sandberg@arm.com PYBIND11_PLUGIN(example) { 10811986Sandreas.sandberg@arm.com py::module m("example", "pybind example plugin"); 10911986Sandreas.sandberg@arm.com 11011986Sandreas.sandberg@arm.com init_ex1(m); 11111986Sandreas.sandberg@arm.com init_ex2(m); 11211986Sandreas.sandberg@arm.com /* ... */ 11311986Sandreas.sandberg@arm.com 11411986Sandreas.sandberg@arm.com return m.ptr(); 11511986Sandreas.sandberg@arm.com } 11611986Sandreas.sandberg@arm.com 11711986Sandreas.sandberg@arm.com:file:`ex1.cpp`: 11811986Sandreas.sandberg@arm.com 11911986Sandreas.sandberg@arm.com.. code-block:: cpp 12011986Sandreas.sandberg@arm.com 12111986Sandreas.sandberg@arm.com void init_ex1(py::module &m) { 12211986Sandreas.sandberg@arm.com m.def("add", [](int a, int b) { return a + b; }); 12311986Sandreas.sandberg@arm.com } 12411986Sandreas.sandberg@arm.com 12511986Sandreas.sandberg@arm.com:file:`ex2.cpp`: 12611986Sandreas.sandberg@arm.com 12711986Sandreas.sandberg@arm.com.. code-block:: cpp 12811986Sandreas.sandberg@arm.com 12911986Sandreas.sandberg@arm.com void init_ex1(py::module &m) { 13011986Sandreas.sandberg@arm.com m.def("sub", [](int a, int b) { return a - b; }); 13111986Sandreas.sandberg@arm.com } 13211986Sandreas.sandberg@arm.com 13311986Sandreas.sandberg@arm.com:command:`python`: 13411986Sandreas.sandberg@arm.com 13511986Sandreas.sandberg@arm.com.. code-block:: pycon 13611986Sandreas.sandberg@arm.com 13711986Sandreas.sandberg@arm.com >>> import example 13811986Sandreas.sandberg@arm.com >>> example.add(1, 2) 13911986Sandreas.sandberg@arm.com 3 14011986Sandreas.sandberg@arm.com >>> example.sub(1, 1) 14111986Sandreas.sandberg@arm.com 0 14211986Sandreas.sandberg@arm.com 14311986Sandreas.sandberg@arm.comAs shown above, the various ``init_ex`` functions should be contained in 14411986Sandreas.sandberg@arm.comseparate files that can be compiled independently from one another, and then 14511986Sandreas.sandberg@arm.comlinked together into the same final shared object. Following this approach 14611986Sandreas.sandberg@arm.comwill: 14711986Sandreas.sandberg@arm.com 14811986Sandreas.sandberg@arm.com1. reduce memory requirements per compilation unit. 14911986Sandreas.sandberg@arm.com 15011986Sandreas.sandberg@arm.com2. enable parallel builds (if desired). 15111986Sandreas.sandberg@arm.com 15211986Sandreas.sandberg@arm.com3. allow for faster incremental builds. For instance, when a single class 15311986Sandreas.sandberg@arm.com definition is changed, only a subset of the binding code will generally need 15411986Sandreas.sandberg@arm.com to be recompiled. 15511986Sandreas.sandberg@arm.com 15611986Sandreas.sandberg@arm.com"recursive template instantiation exceeded maximum depth of 256" 15711986Sandreas.sandberg@arm.com================================================================ 15811986Sandreas.sandberg@arm.com 15911986Sandreas.sandberg@arm.comIf you receive an error about excessive recursive template evaluation, try 16011986Sandreas.sandberg@arm.comspecifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The 16111986Sandreas.sandberg@arm.comculprit is generally the generation of function signatures at compile time 16211986Sandreas.sandberg@arm.comusing C++14 template metaprogramming. 16311986Sandreas.sandberg@arm.com 16411986Sandreas.sandberg@arm.com 16511986Sandreas.sandberg@arm.com.. _`faq:symhidden`: 16611986Sandreas.sandberg@arm.com 16711986Sandreas.sandberg@arm.comHow can I create smaller binaries? 16811986Sandreas.sandberg@arm.com================================== 16911986Sandreas.sandberg@arm.com 17011986Sandreas.sandberg@arm.comTo do its job, pybind11 extensively relies on a programming technique known as 17111986Sandreas.sandberg@arm.com*template metaprogramming*, which is a way of performing computation at compile 17211986Sandreas.sandberg@arm.comtime using type information. Template metaprogamming usually instantiates code 17311986Sandreas.sandberg@arm.cominvolving significant numbers of deeply nested types that are either completely 17411986Sandreas.sandberg@arm.comremoved or reduced to just a few instructions during the compiler's optimization 17511986Sandreas.sandberg@arm.comphase. However, due to the nested nature of these types, the resulting symbol 17611986Sandreas.sandberg@arm.comnames in the compiled extension library can be extremely long. For instance, 17711986Sandreas.sandberg@arm.comthe included test suite contains the following symbol: 17811986Sandreas.sandberg@arm.com 17911986Sandreas.sandberg@arm.com.. only:: html 18011986Sandreas.sandberg@arm.com 18111986Sandreas.sandberg@arm.com .. code-block:: none 18211986Sandreas.sandberg@arm.com 18311986Sandreas.sandberg@arm.com __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 18411986Sandreas.sandberg@arm.com 18511986Sandreas.sandberg@arm.com.. only:: not html 18611986Sandreas.sandberg@arm.com 18711986Sandreas.sandberg@arm.com .. code-block:: cpp 18811986Sandreas.sandberg@arm.com 18911986Sandreas.sandberg@arm.com __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 19011986Sandreas.sandberg@arm.com 19111986Sandreas.sandberg@arm.comwhich is the mangled form of the following function type: 19211986Sandreas.sandberg@arm.com 19311986Sandreas.sandberg@arm.com.. code-block:: cpp 19411986Sandreas.sandberg@arm.com 19511986Sandreas.sandberg@arm.com pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) 19611986Sandreas.sandberg@arm.com 19711986Sandreas.sandberg@arm.comThe memory needed to store just the mangled name of this function (196 bytes) 19811986Sandreas.sandberg@arm.comis larger than the actual piece of code (111 bytes) it represents! On the other 19911986Sandreas.sandberg@arm.comhand, it's silly to even give this function a name -- after all, it's just a 20011986Sandreas.sandberg@arm.comtiny cog in a bigger piece of machinery that is not exposed to the outside 20111986Sandreas.sandberg@arm.comworld. So we'll generally only want to export symbols for those functions which 20211986Sandreas.sandberg@arm.comare actually called from the outside. 20311986Sandreas.sandberg@arm.com 20411986Sandreas.sandberg@arm.comThis can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC 20511986Sandreas.sandberg@arm.comand Clang, which sets the default symbol visibility to *hidden*. It's best to 20611986Sandreas.sandberg@arm.comdo this only for release builds, since the symbol names can be helpful in 20711986Sandreas.sandberg@arm.comdebugging sessions. On Visual Studio, symbols are already hidden by default, so 20811986Sandreas.sandberg@arm.comnothing needs to be done there. Needless to say, this has a tremendous impact 20911986Sandreas.sandberg@arm.comon the final binary size of the resulting extension library. 21011986Sandreas.sandberg@arm.com 21111986Sandreas.sandberg@arm.comAnother aspect that can require a fair bit of code are function signature 21211986Sandreas.sandberg@arm.comdescriptions. pybind11 automatically generates human-readable function 21311986Sandreas.sandberg@arm.comsignatures for docstrings, e.g.: 21411986Sandreas.sandberg@arm.com 21511986Sandreas.sandberg@arm.com.. code-block:: none 21611986Sandreas.sandberg@arm.com 21711986Sandreas.sandberg@arm.com | __init__(...) 21811986Sandreas.sandberg@arm.com | __init__(*args, **kwargs) 21911986Sandreas.sandberg@arm.com | Overloaded function. 22011986Sandreas.sandberg@arm.com | 22111986Sandreas.sandberg@arm.com | 1. __init__(example.Example1) -> NoneType 22211986Sandreas.sandberg@arm.com | 22311986Sandreas.sandberg@arm.com | Docstring for overload #1 goes here 22411986Sandreas.sandberg@arm.com | 22511986Sandreas.sandberg@arm.com | 2. __init__(example.Example1, int) -> NoneType 22611986Sandreas.sandberg@arm.com | 22711986Sandreas.sandberg@arm.com | Docstring for overload #2 goes here 22811986Sandreas.sandberg@arm.com | 22911986Sandreas.sandberg@arm.com | 3. __init__(example.Example1, example.Example1) -> NoneType 23011986Sandreas.sandberg@arm.com | 23111986Sandreas.sandberg@arm.com | Docstring for overload #3 goes here 23211986Sandreas.sandberg@arm.com 23311986Sandreas.sandberg@arm.com 23411986Sandreas.sandberg@arm.comIn C++11 mode, these are generated at run time using string concatenation, 23511986Sandreas.sandberg@arm.comwhich can amount to 10-20% of the size of the resulting binary. If you can, 23611986Sandreas.sandberg@arm.comenable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which 23711986Sandreas.sandberg@arm.comcase signatures are efficiently pre-generated at compile time. Unfortunately, 23811986Sandreas.sandberg@arm.comVisual Studio's C++14 support (``constexpr``) is not good enough as of April 23911986Sandreas.sandberg@arm.com2016, so it always uses the more expensive run-time approach. 24011986Sandreas.sandberg@arm.com 24111986Sandreas.sandberg@arm.comWorking with ancient Visual Studio 2009 builds on Windows 24211986Sandreas.sandberg@arm.com========================================================= 24311986Sandreas.sandberg@arm.com 24411986Sandreas.sandberg@arm.comThe official Windows distributions of Python are compiled using truly 24511986Sandreas.sandberg@arm.comancient versions of Visual Studio that lack good C++11 support. Some users 24611986Sandreas.sandberg@arm.comimplicitly assume that it would be impossible to load a plugin built with 24711986Sandreas.sandberg@arm.comVisual Studio 2015 into a Python distribution that was compiled using Visual 24811986Sandreas.sandberg@arm.comStudio 2009. However, no such issue exists: it's perfectly legitimate to 24911986Sandreas.sandberg@arm.cominterface DLLs that are built with different compilers and/or C libraries. 25011986Sandreas.sandberg@arm.comCommon gotchas to watch out for involve not ``free()``-ing memory region 25111986Sandreas.sandberg@arm.comthat that were ``malloc()``-ed in another shared library, using data 25211986Sandreas.sandberg@arm.comstructures with incompatible ABIs, and so on. pybind11 is very careful not 25311986Sandreas.sandberg@arm.comto make these types of mistakes. 254