faq.rst revision 12391:ceeca8b41e4b
1Frequently asked questions 2########################## 3 4"ImportError: dynamic module does not define init function" 5=========================================================== 6 7You are likely using an incompatible version of Python (for instance, the 8extension library was compiled against Python 2, while the interpreter is 9running on top of some version of Python 3, or vice versa). 10 11"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" 12======================================================================== 13 14See the first answer. 15 16"SystemError: dynamic module not initialized properly" 17====================================================== 18 19See the first answer. 20 21The Python interpreter immediately crashes when importing my module 22=================================================================== 23 24See the first answer. 25 26CMake doesn't detect the right Python version 27============================================= 28 29The CMake-based build system will try to automatically detect the installed 30version of Python and link against that. When this fails, or when there are 31multiple versions of Python and it finds the wrong one, delete 32``CMakeCache.txt`` and then invoke CMake as follows: 33 34.. code-block:: bash 35 36 cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> . 37 38Limitations involving reference arguments 39========================================= 40 41In C++, it's fairly common to pass arguments using mutable references or 42mutable pointers, which allows both read and write access to the value 43supplied by the caller. This is sometimes done for efficiency reasons, or to 44realize functions that have multiple return values. Here are two very basic 45examples: 46 47.. code-block:: cpp 48 49 void increment(int &i) { i++; } 50 void increment_ptr(int *i) { (*i)++; } 51 52In Python, all arguments are passed by reference, so there is no general 53issue in binding such code from Python. 54 55However, certain basic Python types (like ``str``, ``int``, ``bool``, 56``float``, etc.) are **immutable**. This means that the following attempt 57to port the function to Python doesn't have the same effect on the value 58provided by the caller -- in fact, it does nothing at all. 59 60.. code-block:: python 61 62 def increment(i): 63 i += 1 # nope.. 64 65pybind11 is also affected by such language-level conventions, which means that 66binding ``increment`` or ``increment_ptr`` will also create Python functions 67that don't modify their arguments. 68 69Although inconvenient, one workaround is to encapsulate the immutable types in 70a custom type that does allow modifications. 71 72An other alternative involves binding a small wrapper lambda function that 73returns a tuple with all output arguments (see the remainder of the 74documentation for examples on binding lambda functions). An example: 75 76.. code-block:: cpp 77 78 int foo(int &i) { i++; return 123; } 79 80and the binding code 81 82.. code-block:: cpp 83 84 m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); 85 86 87How can I reduce the build time? 88================================ 89 90It's good practice to split binding code over multiple files, as in the 91following example: 92 93:file:`example.cpp`: 94 95.. code-block:: cpp 96 97 void init_ex1(py::module &); 98 void init_ex2(py::module &); 99 /* ... */ 100 101 PYBIND11_MODULE(example, m) { 102 init_ex1(m); 103 init_ex2(m); 104 /* ... */ 105 } 106 107:file:`ex1.cpp`: 108 109.. code-block:: cpp 110 111 void init_ex1(py::module &m) { 112 m.def("add", [](int a, int b) { return a + b; }); 113 } 114 115:file:`ex2.cpp`: 116 117.. code-block:: cpp 118 119 void init_ex1(py::module &m) { 120 m.def("sub", [](int a, int b) { return a - b; }); 121 } 122 123:command:`python`: 124 125.. code-block:: pycon 126 127 >>> import example 128 >>> example.add(1, 2) 129 3 130 >>> example.sub(1, 1) 131 0 132 133As shown above, the various ``init_ex`` functions should be contained in 134separate files that can be compiled independently from one another, and then 135linked together into the same final shared object. Following this approach 136will: 137 1381. reduce memory requirements per compilation unit. 139 1402. enable parallel builds (if desired). 141 1423. allow for faster incremental builds. For instance, when a single class 143 definition is changed, only a subset of the binding code will generally need 144 to be recompiled. 145 146"recursive template instantiation exceeded maximum depth of 256" 147================================================================ 148 149If you receive an error about excessive recursive template evaluation, try 150specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The 151culprit is generally the generation of function signatures at compile time 152using C++14 template metaprogramming. 153 154.. _`faq:hidden_visibility`: 155 156"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]" 157============================================================================================================ 158 159This error typically indicates that you are compiling without the required 160``-fvisibility`` flag. pybind11 code internally forces hidden visibility on 161all internal code, but if non-hidden (and thus *exported*) code attempts to 162include a pybind type (for example, ``py::object`` or ``py::list``) you can run 163into this warning. 164 165To avoid it, make sure you are specifying ``-fvisibility=hidden`` when 166compiling pybind code. 167 168As to why ``-fvisibility=hidden`` is necessary, because pybind modules could 169have been compiled under different versions of pybind itself, it is also 170important that the symbols defined in one module do not clash with the 171potentially-incompatible symbols defined in another. While Python extension 172modules are usually loaded with localized symbols (under POSIX systems 173typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default 174can be changed, but even if it isn't it is not always enough to guarantee 175complete independence of the symbols involved when not using 176``-fvisibility=hidden``. 177 178Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size 179savings. (See the following section for more details). 180 181 182.. _`faq:symhidden`: 183 184How can I create smaller binaries? 185================================== 186 187To do its job, pybind11 extensively relies on a programming technique known as 188*template metaprogramming*, which is a way of performing computation at compile 189time using type information. Template metaprogamming usually instantiates code 190involving significant numbers of deeply nested types that are either completely 191removed or reduced to just a few instructions during the compiler's optimization 192phase. However, due to the nested nature of these types, the resulting symbol 193names in the compiled extension library can be extremely long. For instance, 194the included test suite contains the following symbol: 195 196.. only:: html 197 198 .. code-block:: none 199 200 __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 201 202.. only:: not html 203 204 .. code-block:: cpp 205 206 __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ 207 208which is the mangled form of the following function type: 209 210.. code-block:: cpp 211 212 pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) 213 214The memory needed to store just the mangled name of this function (196 bytes) 215is larger than the actual piece of code (111 bytes) it represents! On the other 216hand, it's silly to even give this function a name -- after all, it's just a 217tiny cog in a bigger piece of machinery that is not exposed to the outside 218world. So we'll generally only want to export symbols for those functions which 219are actually called from the outside. 220 221This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC 222and Clang, which sets the default symbol visibility to *hidden*, which has a 223tremendous impact on the final binary size of the resulting extension library. 224(On Visual Studio, symbols are already hidden by default, so nothing needs to 225be done there.) 226 227In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids 228potential serious issues when loading multiple modules and is required for 229proper pybind operation. See the previous FAQ entry for more details. 230 231Another aspect that can require a fair bit of code are function signature 232descriptions. pybind11 automatically generates human-readable function 233signatures for docstrings, e.g.: 234 235.. code-block:: none 236 237 | __init__(...) 238 | __init__(*args, **kwargs) 239 | Overloaded function. 240 | 241 | 1. __init__(example.Example1) -> NoneType 242 | 243 | Docstring for overload #1 goes here 244 | 245 | 2. __init__(example.Example1, int) -> NoneType 246 | 247 | Docstring for overload #2 goes here 248 | 249 | 3. __init__(example.Example1, example.Example1) -> NoneType 250 | 251 | Docstring for overload #3 goes here 252 253 254In C++11 mode, these are generated at run time using string concatenation, 255which can amount to 10-20% of the size of the resulting binary. If you can, 256enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which 257case signatures are efficiently pre-generated at compile time. Unfortunately, 258Visual Studio's C++14 support (``constexpr``) is not good enough as of April 2592016, so it always uses the more expensive run-time approach. 260 261Working with ancient Visual Studio 2009 builds on Windows 262========================================================= 263 264The official Windows distributions of Python are compiled using truly 265ancient versions of Visual Studio that lack good C++11 support. Some users 266implicitly assume that it would be impossible to load a plugin built with 267Visual Studio 2015 into a Python distribution that was compiled using Visual 268Studio 2009. However, no such issue exists: it's perfectly legitimate to 269interface DLLs that are built with different compilers and/or C libraries. 270Common gotchas to watch out for involve not ``free()``-ing memory region 271that that were ``malloc()``-ed in another shared library, using data 272structures with incompatible ABIs, and so on. pybind11 is very careful not 273to make these types of mistakes. 274