faq.rst revision 11986:c12e4625ab56
11758SN/AFrequently asked questions 21762SN/A########################## 31758SN/A 41758SN/A"ImportError: dynamic module does not define init function" 51758SN/A=========================================================== 61758SN/A 71758SN/A1. Make sure that the name specified in ``pybind::module`` and 81758SN/A ``PYBIND11_PLUGIN`` is consistent and identical to the filename of the 91758SN/A extension library. The latter should not contain any extra prefixes (e.g. 101758SN/A ``test.so`` instead of ``libtest.so``). 111758SN/A 121758SN/A2. If the above did not fix your issue, then you are likely using an 131758SN/A incompatible version of Python (for instance, the extension library was 141758SN/A compiled against Python 2, while the interpreter is running on top of some 151758SN/A version of Python 3, or vice versa) 161758SN/A 171758SN/A"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" 181758SN/A======================================================================== 191758SN/A 201758SN/ASee item 2 of the first answer. 211758SN/A 221758SN/A"SystemError: dynamic module not initialized properly" 231758SN/A====================================================== 241758SN/A 251758SN/ASee item 2 of the first answer. 261758SN/A 272665Ssaidi@eecs.umich.eduThe Python interpreter immediately crashes when importing my module 282665Ssaidi@eecs.umich.edu=================================================================== 292665Ssaidi@eecs.umich.edu 301758SN/ASee item 2 of the first answer. 312SN/A 322171SN/ACMake doesn't detect the right Python version 33732SN/A============================================= 34732SN/A 35732SN/AThe CMake-based build system will try to automatically detect the installed 36732SN/Aversion of Python and link against that. When this fails, or when there are 37732SN/Amultiple versions of Python and it finds the wrong one, delete 381858SN/A``CMakeCache.txt`` and then invoke CMake as follows: 391717SN/A 402190SN/A.. code-block:: bash 4156SN/A 421070SN/A cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> . 43676SN/A 4456SN/ALimitations involving reference arguments 452SN/A========================================= 461858SN/A 472SN/AIn C++, it's fairly common to pass arguments using mutable references or 481147SN/Amutable pointers, which allows both read and write access to the value 491147SN/Asupplied by the caller. This is sometimes done for efficiency reasons, or to 502SN/Arealize functions that have multiple return values. Here are two very basic 512SN/Aexamples: 522SN/A 532SN/A.. code-block:: cpp 542SN/A 552190SN/A void increment(int &i) { i++; } 562SN/A void increment_ptr(int *i) { (*i)++; } 572190SN/A 58190SN/AIn Python, all arguments are passed by reference, so there is no general 592190SN/Aissue in binding such code from Python. 602190SN/A 612114SN/AHowever, certain basic Python types (like ``str``, ``int``, ``bool``, 622234SN/A``float``, etc.) are **immutable**. This means that the following attempt 632190SN/Ato port the function to Python doesn't have the same effect on the value 642SN/Aprovided by the caller -- in fact, it does nothing at all. 652SN/A 662SN/A.. code-block:: python 672SN/A 682SN/A def increment(i): 692SN/A i += 1 # nope.. 702SN/A 712190SN/Apybind11 is also affected by such language-level conventions, which means that 722SN/Abinding ``increment`` or ``increment_ptr`` will also create Python functions 732190SN/Athat don't modify their arguments. 742190SN/A 752190SN/AAlthough inconvenient, one workaround is to encapsulate the immutable types in 762SN/Aa custom type that does allow modifications. 772190SN/A 782190SN/AAn other alternative involves binding a small wrapper lambda function that 792190SN/Areturns a tuple with all output arguments (see the remainder of the 802SN/Adocumentation for examples on binding lambda functions). An example: 812SN/A 822SN/A.. code-block:: cpp 831133SN/A 84716SN/A int foo(int &i) { i++; return 123; } 851133SN/A 86716SN/Aand the binding code 87716SN/A 88716SN/A.. code-block:: cpp 89716SN/A 90716SN/A m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); 91716SN/A 921133SN/A 93716SN/AHow can I reduce the build time? 942159SN/A================================ 95716SN/A 96716SN/AIt's good practice to split binding code over multiple files, as in the 972159SN/Afollowing example: 98716SN/A 99716SN/A:file:`example.cpp`: 1002159SN/A 101716SN/A.. code-block:: cpp 102716SN/A 103716SN/A void init_ex1(py::module &); 104716SN/A void init_ex2(py::module &); 105716SN/A /* ... */ 106716SN/A 107716SN/A PYBIND11_PLUGIN(example) { 1081133SN/A py::module m("example", "pybind example plugin"); 109716SN/A 110716SN/A init_ex1(m); 111716SN/A init_ex2(m); 112716SN/A /* ... */ 113716SN/A 114716SN/A return m.ptr(); 115716SN/A } 116716SN/A 117716SN/A:file:`ex1.cpp`: 118716SN/A 119716SN/A.. code-block:: cpp 120716SN/A 1212159SN/A void init_ex1(py::module &m) { 1222159SN/A m.def("add", [](int a, int b) { return a + b; }); 1232159SN/A } 1242147SN/A 125716SN/A:file:`ex2.cpp`: 1262159SN/A 127716SN/A.. code-block:: cpp 128716SN/A 129716SN/A void init_ex1(py::module &m) { 130716SN/A m.def("sub", [](int a, int b) { return a - b; }); 1311133SN/A } 132716SN/A 1331133SN/A:command:`python`: 134716SN/A 135716SN/A.. code-block:: pycon 136739SN/A 137739SN/A >>> import example 1382190SN/A >>> example.add(1, 2) 1392455SN/A 3 140716SN/A >>> example.sub(1, 1) 141716SN/A 0 1422SN/A 1432190SN/AAs shown above, the various ``init_ex`` functions should be contained in 1442SN/Aseparate files that can be compiled independently from one another, and then 1451136SN/Alinked together into the same final shared object. Following this approach 1462147SN/Awill: 1472SN/A 1482159SN/A1. reduce memory requirements per compilation unit. 1492SN/A 1502SN/A2. enable parallel builds (if desired). 1512190SN/A 152216SN/A3. allow for faster incremental builds. For instance, when a single class 1531133SN/A definition is changed, only a subset of the binding code will generally need 1542SN/A to be recompiled. 1552SN/A 1562SN/A"recursive template instantiation exceeded maximum depth of 256" 1572154SN/A================================================================ 1582SN/A 1592SN/AIf you receive an error about excessive recursive template evaluation, try 1602245SN/Aspecifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The 1612245SN/Aculprit is generally the generation of function signatures at compile time 1622245SN/Ausing C++14 template metaprogramming. 1632245SN/A 1642245SN/A 1652245SN/A.. _`faq:symhidden`: 1662245SN/A 1672245SN/AHow can I create smaller binaries? 1682245SN/A================================== 1692245SN/A 1702245SN/ATo do its job, pybind11 extensively relies on a programming technique known as 1712245SN/A*template metaprogramming*, which is a way of performing computation at compile 1722159SN/Atime using type information. Template metaprogamming usually instantiates code 1732159SN/Ainvolving significant numbers of deeply nested types that are either completely 1742159SN/Aremoved or reduced to just a few instructions during the compiler's optimization 1752SN/Aphase. However, due to the nested nature of these types, the resulting symbol 1762SN/Anames in the compiled extension library can be extremely long. For instance, 1772SN/Athe included test suite contains the following symbol: 1782SN/A 1792SN/A.. only:: html 1802SN/A 1812SN/A .. code-block:: none 1822SN/A 1832SN/A __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 1842SN/A 1852SN/A.. only:: not html 1862SN/A 1872SN/A .. code-block:: cpp 1882SN/A 1892SN/A __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 1902SN/A 1912SN/Awhich is the mangled form of the following function type: 1922SN/A 1932SN/A.. code-block:: cpp 1942SN/A 1952SN/A pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) 1962SN/A 1972SN/AThe memory needed to store just the mangled name of this function (196 bytes) 1982SN/Ais larger than the actual piece of code (111 bytes) it represents! On the other 1992SN/Ahand, it's silly to even give this function a name -- after all, it's just a 2002SN/Atiny cog in a bigger piece of machinery that is not exposed to the outside 2012SN/Aworld. So we'll generally only want to export symbols for those functions which 2022SN/Aare actually called from the outside. 2032SN/A 2042SN/AThis can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC 2052SN/Aand Clang, which sets the default symbol visibility to *hidden*. It's best to 2062SN/Ado this only for release builds, since the symbol names can be helpful in 2072SN/Adebugging sessions. On Visual Studio, symbols are already hidden by default, so 2082SN/Anothing needs to be done there. Needless to say, this has a tremendous impact 2092SN/Aon the final binary size of the resulting extension library. 2102SN/A 2112SN/AAnother aspect that can require a fair bit of code are function signature 2122SN/Adescriptions. pybind11 automatically generates human-readable function 2132SN/Asignatures for docstrings, e.g.: 2142SN/A 2152SN/A.. code-block:: none 2162SN/A 2172SN/A | __init__(...) 2182SN/A | __init__(*args, **kwargs) 2192SN/A | Overloaded function. 2202SN/A | 2212SN/A | 1. __init__(example.Example1) -> NoneType 2222SN/A | 2232SN/A | Docstring for overload #1 goes here 2242SN/A | 225597SN/A | 2. __init__(example.Example1, int) -> NoneType 226597SN/A | 2272190SN/A | Docstring for overload #2 goes here 228597SN/A | 229597SN/A | 3. __init__(example.Example1, example.Example1) -> NoneType 2302SN/A | 2312SN/A | Docstring for overload #3 goes here 2322SN/A 2332SN/A 2342SN/AIn C++11 mode, these are generated at run time using string concatenation, 2352SN/Awhich can amount to 10-20% of the size of the resulting binary. If you can, 2362SN/Aenable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which 2372SN/Acase signatures are efficiently pre-generated at compile time. Unfortunately, 2382SN/AVisual Studio's C++14 support (``constexpr``) is not good enough as of April 2392SN/A2016, so it always uses the more expensive run-time approach. 2402SN/A 2412SN/AWorking with ancient Visual Studio 2009 builds on Windows 2422SN/A========================================================= 2432SN/A 2442190SN/AThe official Windows distributions of Python are compiled using truly 2452SN/Aancient versions of Visual Studio that lack good C++11 support. Some users 2462SN/Aimplicitly assume that it would be impossible to load a plugin built with 2472SN/AVisual Studio 2015 into a Python distribution that was compiled using Visual 2482SN/AStudio 2009. However, no such issue exists: it's perfectly legitimate to 2492SN/Ainterface DLLs that are built with different compilers and/or C libraries. 2502SN/ACommon gotchas to watch out for involve not ``free()``-ing memory region 2512SN/Athat that were ``malloc()``-ed in another shared library, using data 2522SN/Astructures with incompatible ABIs, and so on. pybind11 is very careful not 2532SN/Ato make these types of mistakes. 2542SN/A