faq.rst revision 11986:c12e4625ab56
1Frequently asked questions
2##########################
3
4"ImportError: dynamic module does not define init function"
5===========================================================
6
71. Make sure that the name specified in ``pybind::module`` and
8   ``PYBIND11_PLUGIN`` is consistent and identical to the filename of the
9   extension library. The latter should not contain any extra prefixes (e.g.
10   ``test.so`` instead of ``libtest.so``).
11
122. If the above did not fix your issue, then you are likely using an
13   incompatible version of Python (for instance, the extension library was
14   compiled against Python 2, while the interpreter is running on top of some
15   version of Python 3, or vice versa)
16
17"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
18========================================================================
19
20See item 2 of the first answer.
21
22"SystemError: dynamic module not initialized properly"
23======================================================
24
25See item 2 of the first answer.
26
27The Python interpreter immediately crashes when importing my module
28===================================================================
29
30See item 2 of the first answer.
31
32CMake doesn't detect the right Python version
33=============================================
34
35The CMake-based build system will try to automatically detect the installed
36version of Python and link against that. When this fails, or when there are
37multiple versions of Python and it finds the wrong one, delete
38``CMakeCache.txt`` and then invoke CMake as follows:
39
40.. code-block:: bash
41
42    cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> .
43
44Limitations involving reference arguments
45=========================================
46
47In C++, it's fairly common to pass arguments using mutable references or
48mutable pointers, which allows both read and write access to the value
49supplied by the caller. This is sometimes done for efficiency reasons, or to
50realize functions that have multiple return values. Here are two very basic
51examples:
52
53.. code-block:: cpp
54
55    void increment(int &i) { i++; }
56    void increment_ptr(int *i) { (*i)++; }
57
58In Python, all arguments are passed by reference, so there is no general
59issue in binding such code from Python.
60
61However, certain basic Python types (like ``str``, ``int``, ``bool``,
62``float``, etc.) are **immutable**. This means that the following attempt
63to port the function to Python doesn't have the same effect on the value
64provided by the caller -- in fact, it does nothing at all.
65
66.. code-block:: python
67
68    def increment(i):
69        i += 1 # nope..
70
71pybind11 is also affected by such language-level conventions, which means that
72binding ``increment`` or ``increment_ptr`` will also create Python functions
73that don't modify their arguments.
74
75Although inconvenient, one workaround is to encapsulate the immutable types in
76a custom type that does allow modifications.
77
78An other alternative involves binding a small wrapper lambda function that
79returns a tuple with all output arguments (see the remainder of the
80documentation for examples on binding lambda functions). An example:
81
82.. code-block:: cpp
83
84    int foo(int &i) { i++; return 123; }
85
86and the binding code
87
88.. code-block:: cpp
89
90   m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
91
92
93How can I reduce the build time?
94================================
95
96It's good practice to split binding code over multiple files, as in the
97following example:
98
99:file:`example.cpp`:
100
101.. code-block:: cpp
102
103    void init_ex1(py::module &);
104    void init_ex2(py::module &);
105    /* ... */
106
107    PYBIND11_PLUGIN(example) {
108        py::module m("example", "pybind example plugin");
109
110        init_ex1(m);
111        init_ex2(m);
112        /* ... */
113
114        return m.ptr();
115    }
116
117:file:`ex1.cpp`:
118
119.. code-block:: cpp
120
121    void init_ex1(py::module &m) {
122        m.def("add", [](int a, int b) { return a + b; });
123    }
124
125:file:`ex2.cpp`:
126
127.. code-block:: cpp
128
129    void init_ex1(py::module &m) {
130        m.def("sub", [](int a, int b) { return a - b; });
131    }
132
133:command:`python`:
134
135.. code-block:: pycon
136
137    >>> import example
138    >>> example.add(1, 2)
139    3
140    >>> example.sub(1, 1)
141    0
142
143As shown above, the various ``init_ex`` functions should be contained in
144separate files that can be compiled independently from one another, and then
145linked together into the same final shared object.  Following this approach
146will:
147
1481. reduce memory requirements per compilation unit.
149
1502. enable parallel builds (if desired).
151
1523. allow for faster incremental builds. For instance, when a single class
153   definition is changed, only a subset of the binding code will generally need
154   to be recompiled.
155
156"recursive template instantiation exceeded maximum depth of 256"
157================================================================
158
159If you receive an error about excessive recursive template evaluation, try
160specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
161culprit is generally the generation of function signatures at compile time
162using C++14 template metaprogramming.
163
164
165.. _`faq:symhidden`:
166
167How can I create smaller binaries?
168==================================
169
170To do its job, pybind11 extensively relies on a programming technique known as
171*template metaprogramming*, which is a way of performing computation at compile
172time using type information. Template metaprogamming usually instantiates code
173involving significant numbers of deeply nested types that are either completely
174removed or reduced to just a few instructions during the compiler's optimization
175phase. However, due to the nested nature of these types, the resulting symbol
176names in the compiled extension library can be extremely long. For instance,
177the included test suite contains the following symbol:
178
179.. only:: html
180
181    .. code-block:: none
182
183        _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
184
185.. only:: not html
186
187    .. code-block:: cpp
188
189        __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
190
191which is the mangled form of the following function type:
192
193.. code-block:: cpp
194
195    pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
196
197The memory needed to store just the mangled name of this function (196 bytes)
198is larger than the actual piece of code (111 bytes) it represents! On the other
199hand, it's silly to even give this function a name -- after all, it's just a
200tiny cog in a bigger piece of machinery that is not exposed to the outside
201world. So we'll generally only want to export symbols for those functions which
202are actually called from the outside.
203
204This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
205and Clang, which sets the default symbol visibility to *hidden*. It's best to
206do this only for release builds, since the symbol names can be helpful in
207debugging sessions. On Visual Studio, symbols are already hidden by default, so
208nothing needs to be done there. Needless to say, this has a tremendous impact
209on the final binary size of the resulting extension library.
210
211Another aspect that can require a fair bit of code are function signature
212descriptions. pybind11 automatically generates human-readable function
213signatures for docstrings, e.g.:
214
215.. code-block:: none
216
217     |  __init__(...)
218     |      __init__(*args, **kwargs)
219     |      Overloaded function.
220     |
221     |      1. __init__(example.Example1) -> NoneType
222     |
223     |      Docstring for overload #1 goes here
224     |
225     |      2. __init__(example.Example1, int) -> NoneType
226     |
227     |      Docstring for overload #2 goes here
228     |
229     |      3. __init__(example.Example1, example.Example1) -> NoneType
230     |
231     |      Docstring for overload #3 goes here
232
233
234In C++11 mode, these are generated at run time using string concatenation,
235which can amount to 10-20% of the size of the resulting binary. If you can,
236enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which
237case signatures are efficiently pre-generated at compile time. Unfortunately,
238Visual Studio's C++14 support (``constexpr``) is not good enough as of April
2392016, so it always uses the more expensive run-time approach.
240
241Working with ancient Visual Studio 2009 builds on Windows
242=========================================================
243
244The official Windows distributions of Python are compiled using truly
245ancient versions of Visual Studio that lack good C++11 support. Some users
246implicitly assume that it would be impossible to load a plugin built with
247Visual Studio 2015 into a Python distribution that was compiled using Visual
248Studio 2009. However, no such issue exists: it's perfectly legitimate to
249interface DLLs that are built with different compilers and/or C libraries.
250Common gotchas to watch out for involve not ``free()``-ing memory region
251that that were ``malloc()``-ed in another shared library, using data
252structures with incompatible ABIs, and so on. pybind11 is very careful not
253to make these types of mistakes.
254