faq.rst revision 14299
111986Sandreas.sandberg@arm.comFrequently asked questions 211986Sandreas.sandberg@arm.com########################## 311986Sandreas.sandberg@arm.com 411986Sandreas.sandberg@arm.com"ImportError: dynamic module does not define init function" 511986Sandreas.sandberg@arm.com=========================================================== 611986Sandreas.sandberg@arm.com 714299Sbbruce@ucdavis.edu1. Make sure that the name specified in PYBIND11_MODULE is identical to the 814299Sbbruce@ucdavis.edufilename of the extension library (without prefixes such as .so) 914299Sbbruce@ucdavis.edu 1014299Sbbruce@ucdavis.edu2. If the above did not fix the issue, you are likely using an incompatible 1114299Sbbruce@ucdavis.eduversion of Python (for instance, the extension library was compiled against 1214299Sbbruce@ucdavis.eduPython 2, while the interpreter is running on top of some version of Python 1314299Sbbruce@ucdavis.edu3, or vice versa). 1411986Sandreas.sandberg@arm.com 1511986Sandreas.sandberg@arm.com"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" 1611986Sandreas.sandberg@arm.com======================================================================== 1711986Sandreas.sandberg@arm.com 1812391Sjason@lowepower.comSee the first answer. 1911986Sandreas.sandberg@arm.com 2011986Sandreas.sandberg@arm.com"SystemError: dynamic module not initialized properly" 2111986Sandreas.sandberg@arm.com====================================================== 2211986Sandreas.sandberg@arm.com 2312391Sjason@lowepower.comSee the first answer. 2411986Sandreas.sandberg@arm.com 2511986Sandreas.sandberg@arm.comThe Python interpreter immediately crashes when importing my module 2611986Sandreas.sandberg@arm.com=================================================================== 2711986Sandreas.sandberg@arm.com 2812391Sjason@lowepower.comSee the first answer. 2911986Sandreas.sandberg@arm.com 3011986Sandreas.sandberg@arm.comCMake doesn't detect the right Python version 3111986Sandreas.sandberg@arm.com============================================= 3211986Sandreas.sandberg@arm.com 3311986Sandreas.sandberg@arm.comThe CMake-based build system will try to automatically detect the installed 3411986Sandreas.sandberg@arm.comversion of Python and link against that. When this fails, or when there are 3511986Sandreas.sandberg@arm.commultiple versions of Python and it finds the wrong one, delete 3611986Sandreas.sandberg@arm.com``CMakeCache.txt`` and then invoke CMake as follows: 3711986Sandreas.sandberg@arm.com 3811986Sandreas.sandberg@arm.com.. code-block:: bash 3911986Sandreas.sandberg@arm.com 4011986Sandreas.sandberg@arm.com cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> . 4111986Sandreas.sandberg@arm.com 4214299Sbbruce@ucdavis.edu.. _faq_reference_arguments: 4314299Sbbruce@ucdavis.edu 4411986Sandreas.sandberg@arm.comLimitations involving reference arguments 4511986Sandreas.sandberg@arm.com========================================= 4611986Sandreas.sandberg@arm.com 4711986Sandreas.sandberg@arm.comIn C++, it's fairly common to pass arguments using mutable references or 4811986Sandreas.sandberg@arm.commutable pointers, which allows both read and write access to the value 4911986Sandreas.sandberg@arm.comsupplied by the caller. This is sometimes done for efficiency reasons, or to 5011986Sandreas.sandberg@arm.comrealize functions that have multiple return values. Here are two very basic 5111986Sandreas.sandberg@arm.comexamples: 5211986Sandreas.sandberg@arm.com 5311986Sandreas.sandberg@arm.com.. code-block:: cpp 5411986Sandreas.sandberg@arm.com 5511986Sandreas.sandberg@arm.com void increment(int &i) { i++; } 5611986Sandreas.sandberg@arm.com void increment_ptr(int *i) { (*i)++; } 5711986Sandreas.sandberg@arm.com 5811986Sandreas.sandberg@arm.comIn Python, all arguments are passed by reference, so there is no general 5911986Sandreas.sandberg@arm.comissue in binding such code from Python. 6011986Sandreas.sandberg@arm.com 6111986Sandreas.sandberg@arm.comHowever, certain basic Python types (like ``str``, ``int``, ``bool``, 6211986Sandreas.sandberg@arm.com``float``, etc.) are **immutable**. This means that the following attempt 6311986Sandreas.sandberg@arm.comto port the function to Python doesn't have the same effect on the value 6411986Sandreas.sandberg@arm.comprovided by the caller -- in fact, it does nothing at all. 6511986Sandreas.sandberg@arm.com 6611986Sandreas.sandberg@arm.com.. code-block:: python 6711986Sandreas.sandberg@arm.com 6811986Sandreas.sandberg@arm.com def increment(i): 6911986Sandreas.sandberg@arm.com i += 1 # nope.. 7011986Sandreas.sandberg@arm.com 7111986Sandreas.sandberg@arm.compybind11 is also affected by such language-level conventions, which means that 7211986Sandreas.sandberg@arm.combinding ``increment`` or ``increment_ptr`` will also create Python functions 7311986Sandreas.sandberg@arm.comthat don't modify their arguments. 7411986Sandreas.sandberg@arm.com 7511986Sandreas.sandberg@arm.comAlthough inconvenient, one workaround is to encapsulate the immutable types in 7611986Sandreas.sandberg@arm.coma custom type that does allow modifications. 7711986Sandreas.sandberg@arm.com 7811986Sandreas.sandberg@arm.comAn other alternative involves binding a small wrapper lambda function that 7911986Sandreas.sandberg@arm.comreturns a tuple with all output arguments (see the remainder of the 8011986Sandreas.sandberg@arm.comdocumentation for examples on binding lambda functions). An example: 8111986Sandreas.sandberg@arm.com 8211986Sandreas.sandberg@arm.com.. code-block:: cpp 8311986Sandreas.sandberg@arm.com 8411986Sandreas.sandberg@arm.com int foo(int &i) { i++; return 123; } 8511986Sandreas.sandberg@arm.com 8611986Sandreas.sandberg@arm.comand the binding code 8711986Sandreas.sandberg@arm.com 8811986Sandreas.sandberg@arm.com.. code-block:: cpp 8911986Sandreas.sandberg@arm.com 9011986Sandreas.sandberg@arm.com m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); 9111986Sandreas.sandberg@arm.com 9211986Sandreas.sandberg@arm.com 9311986Sandreas.sandberg@arm.comHow can I reduce the build time? 9411986Sandreas.sandberg@arm.com================================ 9511986Sandreas.sandberg@arm.com 9611986Sandreas.sandberg@arm.comIt's good practice to split binding code over multiple files, as in the 9711986Sandreas.sandberg@arm.comfollowing example: 9811986Sandreas.sandberg@arm.com 9911986Sandreas.sandberg@arm.com:file:`example.cpp`: 10011986Sandreas.sandberg@arm.com 10111986Sandreas.sandberg@arm.com.. code-block:: cpp 10211986Sandreas.sandberg@arm.com 10311986Sandreas.sandberg@arm.com void init_ex1(py::module &); 10411986Sandreas.sandberg@arm.com void init_ex2(py::module &); 10511986Sandreas.sandberg@arm.com /* ... */ 10611986Sandreas.sandberg@arm.com 10712391Sjason@lowepower.com PYBIND11_MODULE(example, m) { 10811986Sandreas.sandberg@arm.com init_ex1(m); 10911986Sandreas.sandberg@arm.com init_ex2(m); 11011986Sandreas.sandberg@arm.com /* ... */ 11111986Sandreas.sandberg@arm.com } 11211986Sandreas.sandberg@arm.com 11311986Sandreas.sandberg@arm.com:file:`ex1.cpp`: 11411986Sandreas.sandberg@arm.com 11511986Sandreas.sandberg@arm.com.. code-block:: cpp 11611986Sandreas.sandberg@arm.com 11711986Sandreas.sandberg@arm.com void init_ex1(py::module &m) { 11811986Sandreas.sandberg@arm.com m.def("add", [](int a, int b) { return a + b; }); 11911986Sandreas.sandberg@arm.com } 12011986Sandreas.sandberg@arm.com 12111986Sandreas.sandberg@arm.com:file:`ex2.cpp`: 12211986Sandreas.sandberg@arm.com 12311986Sandreas.sandberg@arm.com.. code-block:: cpp 12411986Sandreas.sandberg@arm.com 12514299Sbbruce@ucdavis.edu void init_ex2(py::module &m) { 12611986Sandreas.sandberg@arm.com m.def("sub", [](int a, int b) { return a - b; }); 12711986Sandreas.sandberg@arm.com } 12811986Sandreas.sandberg@arm.com 12911986Sandreas.sandberg@arm.com:command:`python`: 13011986Sandreas.sandberg@arm.com 13111986Sandreas.sandberg@arm.com.. code-block:: pycon 13211986Sandreas.sandberg@arm.com 13311986Sandreas.sandberg@arm.com >>> import example 13411986Sandreas.sandberg@arm.com >>> example.add(1, 2) 13511986Sandreas.sandberg@arm.com 3 13611986Sandreas.sandberg@arm.com >>> example.sub(1, 1) 13711986Sandreas.sandberg@arm.com 0 13811986Sandreas.sandberg@arm.com 13911986Sandreas.sandberg@arm.comAs shown above, the various ``init_ex`` functions should be contained in 14011986Sandreas.sandberg@arm.comseparate files that can be compiled independently from one another, and then 14111986Sandreas.sandberg@arm.comlinked together into the same final shared object. Following this approach 14211986Sandreas.sandberg@arm.comwill: 14311986Sandreas.sandberg@arm.com 14411986Sandreas.sandberg@arm.com1. reduce memory requirements per compilation unit. 14511986Sandreas.sandberg@arm.com 14611986Sandreas.sandberg@arm.com2. enable parallel builds (if desired). 14711986Sandreas.sandberg@arm.com 14811986Sandreas.sandberg@arm.com3. allow for faster incremental builds. For instance, when a single class 14911986Sandreas.sandberg@arm.com definition is changed, only a subset of the binding code will generally need 15011986Sandreas.sandberg@arm.com to be recompiled. 15111986Sandreas.sandberg@arm.com 15211986Sandreas.sandberg@arm.com"recursive template instantiation exceeded maximum depth of 256" 15311986Sandreas.sandberg@arm.com================================================================ 15411986Sandreas.sandberg@arm.com 15511986Sandreas.sandberg@arm.comIf you receive an error about excessive recursive template evaluation, try 15611986Sandreas.sandberg@arm.comspecifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The 15711986Sandreas.sandberg@arm.comculprit is generally the generation of function signatures at compile time 15811986Sandreas.sandberg@arm.comusing C++14 template metaprogramming. 15911986Sandreas.sandberg@arm.com 16012391Sjason@lowepower.com.. _`faq:hidden_visibility`: 16112391Sjason@lowepower.com 16212391Sjason@lowepower.com"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]" 16312391Sjason@lowepower.com============================================================================================================ 16412391Sjason@lowepower.com 16512391Sjason@lowepower.comThis error typically indicates that you are compiling without the required 16612391Sjason@lowepower.com``-fvisibility`` flag. pybind11 code internally forces hidden visibility on 16712391Sjason@lowepower.comall internal code, but if non-hidden (and thus *exported*) code attempts to 16812391Sjason@lowepower.cominclude a pybind type (for example, ``py::object`` or ``py::list``) you can run 16912391Sjason@lowepower.cominto this warning. 17012391Sjason@lowepower.com 17112391Sjason@lowepower.comTo avoid it, make sure you are specifying ``-fvisibility=hidden`` when 17212391Sjason@lowepower.comcompiling pybind code. 17312391Sjason@lowepower.com 17412391Sjason@lowepower.comAs to why ``-fvisibility=hidden`` is necessary, because pybind modules could 17512391Sjason@lowepower.comhave been compiled under different versions of pybind itself, it is also 17612391Sjason@lowepower.comimportant that the symbols defined in one module do not clash with the 17712391Sjason@lowepower.compotentially-incompatible symbols defined in another. While Python extension 17812391Sjason@lowepower.commodules are usually loaded with localized symbols (under POSIX systems 17912391Sjason@lowepower.comtypically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default 18012391Sjason@lowepower.comcan be changed, but even if it isn't it is not always enough to guarantee 18112391Sjason@lowepower.comcomplete independence of the symbols involved when not using 18212391Sjason@lowepower.com``-fvisibility=hidden``. 18312391Sjason@lowepower.com 18412391Sjason@lowepower.comAdditionally, ``-fvisiblity=hidden`` can deliver considerably binary size 18512391Sjason@lowepower.comsavings. (See the following section for more details). 18612391Sjason@lowepower.com 18711986Sandreas.sandberg@arm.com 18811986Sandreas.sandberg@arm.com.. _`faq:symhidden`: 18911986Sandreas.sandberg@arm.com 19011986Sandreas.sandberg@arm.comHow can I create smaller binaries? 19111986Sandreas.sandberg@arm.com================================== 19211986Sandreas.sandberg@arm.com 19311986Sandreas.sandberg@arm.comTo do its job, pybind11 extensively relies on a programming technique known as 19411986Sandreas.sandberg@arm.com*template metaprogramming*, which is a way of performing computation at compile 19511986Sandreas.sandberg@arm.comtime using type information. Template metaprogamming usually instantiates code 19611986Sandreas.sandberg@arm.cominvolving significant numbers of deeply nested types that are either completely 19711986Sandreas.sandberg@arm.comremoved or reduced to just a few instructions during the compiler's optimization 19811986Sandreas.sandberg@arm.comphase. However, due to the nested nature of these types, the resulting symbol 19911986Sandreas.sandberg@arm.comnames in the compiled extension library can be extremely long. For instance, 20011986Sandreas.sandberg@arm.comthe included test suite contains the following symbol: 20111986Sandreas.sandberg@arm.com 20211986Sandreas.sandberg@arm.com.. only:: html 20311986Sandreas.sandberg@arm.com 20411986Sandreas.sandberg@arm.com .. code-block:: none 20511986Sandreas.sandberg@arm.com 20611986Sandreas.sandberg@arm.com __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 20711986Sandreas.sandberg@arm.com 20811986Sandreas.sandberg@arm.com.. only:: not html 20911986Sandreas.sandberg@arm.com 21011986Sandreas.sandberg@arm.com .. code-block:: cpp 21111986Sandreas.sandberg@arm.com 21211986Sandreas.sandberg@arm.com __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 21311986Sandreas.sandberg@arm.com 21411986Sandreas.sandberg@arm.comwhich is the mangled form of the following function type: 21511986Sandreas.sandberg@arm.com 21611986Sandreas.sandberg@arm.com.. code-block:: cpp 21711986Sandreas.sandberg@arm.com 21811986Sandreas.sandberg@arm.com pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) 21911986Sandreas.sandberg@arm.com 22011986Sandreas.sandberg@arm.comThe memory needed to store just the mangled name of this function (196 bytes) 22111986Sandreas.sandberg@arm.comis larger than the actual piece of code (111 bytes) it represents! On the other 22211986Sandreas.sandberg@arm.comhand, it's silly to even give this function a name -- after all, it's just a 22311986Sandreas.sandberg@arm.comtiny cog in a bigger piece of machinery that is not exposed to the outside 22411986Sandreas.sandberg@arm.comworld. So we'll generally only want to export symbols for those functions which 22511986Sandreas.sandberg@arm.comare actually called from the outside. 22611986Sandreas.sandberg@arm.com 22711986Sandreas.sandberg@arm.comThis can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC 22812391Sjason@lowepower.comand Clang, which sets the default symbol visibility to *hidden*, which has a 22912391Sjason@lowepower.comtremendous impact on the final binary size of the resulting extension library. 23012391Sjason@lowepower.com(On Visual Studio, symbols are already hidden by default, so nothing needs to 23112391Sjason@lowepower.combe done there.) 23212391Sjason@lowepower.com 23312391Sjason@lowepower.comIn addition to decreasing binary size, ``-fvisibility=hidden`` also avoids 23412391Sjason@lowepower.compotential serious issues when loading multiple modules and is required for 23512391Sjason@lowepower.comproper pybind operation. See the previous FAQ entry for more details. 23611986Sandreas.sandberg@arm.com 23714299Sbbruce@ucdavis.eduWorking with ancient Visual Studio 2008 builds on Windows 23811986Sandreas.sandberg@arm.com========================================================= 23911986Sandreas.sandberg@arm.com 24011986Sandreas.sandberg@arm.comThe official Windows distributions of Python are compiled using truly 24111986Sandreas.sandberg@arm.comancient versions of Visual Studio that lack good C++11 support. Some users 24211986Sandreas.sandberg@arm.comimplicitly assume that it would be impossible to load a plugin built with 24311986Sandreas.sandberg@arm.comVisual Studio 2015 into a Python distribution that was compiled using Visual 24414299Sbbruce@ucdavis.eduStudio 2008. However, no such issue exists: it's perfectly legitimate to 24511986Sandreas.sandberg@arm.cominterface DLLs that are built with different compilers and/or C libraries. 24611986Sandreas.sandberg@arm.comCommon gotchas to watch out for involve not ``free()``-ing memory region 24711986Sandreas.sandberg@arm.comthat that were ``malloc()``-ed in another shared library, using data 24811986Sandreas.sandberg@arm.comstructures with incompatible ABIs, and so on. pybind11 is very careful not 24911986Sandreas.sandberg@arm.comto make these types of mistakes. 25014299Sbbruce@ucdavis.edu 25114299Sbbruce@ucdavis.eduInconsistent detection of Python version in CMake and pybind11 25214299Sbbruce@ucdavis.edu============================================================== 25314299Sbbruce@ucdavis.edu 25414299Sbbruce@ucdavis.eduThe functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)`` provided by CMake 25514299Sbbruce@ucdavis.edufor Python version detection are not used by pybind11 due to unreliability and limitations that make 25614299Sbbruce@ucdavis.eduthem unsuitable for pybind11's needs. Instead pybind provides its own, more reliable Python detection 25714299Sbbruce@ucdavis.eduCMake code. Conflicts can arise, however, when using pybind11 in a project that *also* uses the CMake 25814299Sbbruce@ucdavis.eduPython detection in a system with several Python versions installed. 25914299Sbbruce@ucdavis.edu 26014299Sbbruce@ucdavis.eduThis difference may cause inconsistencies and errors if *both* mechanisms are used in the same project. Consider the following 26114299Sbbruce@ucdavis.eduCmake code executed in a system with Python 2.7 and 3.x installed: 26214299Sbbruce@ucdavis.edu 26314299Sbbruce@ucdavis.edu.. code-block:: cmake 26414299Sbbruce@ucdavis.edu 26514299Sbbruce@ucdavis.edu find_package(PythonInterp) 26614299Sbbruce@ucdavis.edu find_package(PythonLibs) 26714299Sbbruce@ucdavis.edu find_package(pybind11) 26814299Sbbruce@ucdavis.edu 26914299Sbbruce@ucdavis.eduIt will detect Python 2.7 and pybind11 will pick it as well. 27014299Sbbruce@ucdavis.edu 27114299Sbbruce@ucdavis.eduIn contrast this code: 27214299Sbbruce@ucdavis.edu 27314299Sbbruce@ucdavis.edu.. code-block:: cmake 27414299Sbbruce@ucdavis.edu 27514299Sbbruce@ucdavis.edu find_package(pybind11) 27614299Sbbruce@ucdavis.edu find_package(PythonInterp) 27714299Sbbruce@ucdavis.edu find_package(PythonLibs) 27814299Sbbruce@ucdavis.edu 27914299Sbbruce@ucdavis.eduwill detect Python 3.x for pybind11 and may crash on ``find_package(PythonLibs)`` afterwards. 28014299Sbbruce@ucdavis.edu 28114299Sbbruce@ucdavis.eduIt is advised to avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)`` from CMake and rely 28214299Sbbruce@ucdavis.eduon pybind11 in detecting Python version. If this is not possible CMake machinery should be called *before* including pybind11. 28314299Sbbruce@ucdavis.edu 28414299Sbbruce@ucdavis.eduHow to cite this project? 28514299Sbbruce@ucdavis.edu========================= 28614299Sbbruce@ucdavis.edu 28714299Sbbruce@ucdavis.eduWe suggest the following BibTeX template to cite pybind11 in scientific 28814299Sbbruce@ucdavis.edudiscourse: 28914299Sbbruce@ucdavis.edu 29014299Sbbruce@ucdavis.edu.. code-block:: bash 29114299Sbbruce@ucdavis.edu 29214299Sbbruce@ucdavis.edu @misc{pybind11, 29314299Sbbruce@ucdavis.edu author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan}, 29414299Sbbruce@ucdavis.edu year = {2017}, 29514299Sbbruce@ucdavis.edu note = {https://github.com/pybind/pybind11}, 29614299Sbbruce@ucdavis.edu title = {pybind11 -- Seamless operability between C++11 and Python} 29714299Sbbruce@ucdavis.edu } 298