faq.rst revision 12391:ceeca8b41e4b
1Frequently asked questions
2##########################
3
4"ImportError: dynamic module does not define init function"
5===========================================================
6
7You are likely using an incompatible version of Python (for instance, the
8extension library was compiled against Python 2, while the interpreter is
9running on top of some version of Python 3, or vice versa).
10
11"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
12========================================================================
13
14See the first answer.
15
16"SystemError: dynamic module not initialized properly"
17======================================================
18
19See the first answer.
20
21The Python interpreter immediately crashes when importing my module
22===================================================================
23
24See the first answer.
25
26CMake doesn't detect the right Python version
27=============================================
28
29The CMake-based build system will try to automatically detect the installed
30version of Python and link against that. When this fails, or when there are
31multiple versions of Python and it finds the wrong one, delete
32``CMakeCache.txt`` and then invoke CMake as follows:
33
34.. code-block:: bash
35
36    cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> .
37
38Limitations involving reference arguments
39=========================================
40
41In C++, it's fairly common to pass arguments using mutable references or
42mutable pointers, which allows both read and write access to the value
43supplied by the caller. This is sometimes done for efficiency reasons, or to
44realize functions that have multiple return values. Here are two very basic
45examples:
46
47.. code-block:: cpp
48
49    void increment(int &i) { i++; }
50    void increment_ptr(int *i) { (*i)++; }
51
52In Python, all arguments are passed by reference, so there is no general
53issue in binding such code from Python.
54
55However, certain basic Python types (like ``str``, ``int``, ``bool``,
56``float``, etc.) are **immutable**. This means that the following attempt
57to port the function to Python doesn't have the same effect on the value
58provided by the caller -- in fact, it does nothing at all.
59
60.. code-block:: python
61
62    def increment(i):
63        i += 1 # nope..
64
65pybind11 is also affected by such language-level conventions, which means that
66binding ``increment`` or ``increment_ptr`` will also create Python functions
67that don't modify their arguments.
68
69Although inconvenient, one workaround is to encapsulate the immutable types in
70a custom type that does allow modifications.
71
72An other alternative involves binding a small wrapper lambda function that
73returns a tuple with all output arguments (see the remainder of the
74documentation for examples on binding lambda functions). An example:
75
76.. code-block:: cpp
77
78    int foo(int &i) { i++; return 123; }
79
80and the binding code
81
82.. code-block:: cpp
83
84   m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
85
86
87How can I reduce the build time?
88================================
89
90It's good practice to split binding code over multiple files, as in the
91following example:
92
93:file:`example.cpp`:
94
95.. code-block:: cpp
96
97    void init_ex1(py::module &);
98    void init_ex2(py::module &);
99    /* ... */
100
101    PYBIND11_MODULE(example, m) {
102        init_ex1(m);
103        init_ex2(m);
104        /* ... */
105    }
106
107:file:`ex1.cpp`:
108
109.. code-block:: cpp
110
111    void init_ex1(py::module &m) {
112        m.def("add", [](int a, int b) { return a + b; });
113    }
114
115:file:`ex2.cpp`:
116
117.. code-block:: cpp
118
119    void init_ex1(py::module &m) {
120        m.def("sub", [](int a, int b) { return a - b; });
121    }
122
123:command:`python`:
124
125.. code-block:: pycon
126
127    >>> import example
128    >>> example.add(1, 2)
129    3
130    >>> example.sub(1, 1)
131    0
132
133As shown above, the various ``init_ex`` functions should be contained in
134separate files that can be compiled independently from one another, and then
135linked together into the same final shared object.  Following this approach
136will:
137
1381. reduce memory requirements per compilation unit.
139
1402. enable parallel builds (if desired).
141
1423. allow for faster incremental builds. For instance, when a single class
143   definition is changed, only a subset of the binding code will generally need
144   to be recompiled.
145
146"recursive template instantiation exceeded maximum depth of 256"
147================================================================
148
149If you receive an error about excessive recursive template evaluation, try
150specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
151culprit is generally the generation of function signatures at compile time
152using C++14 template metaprogramming.
153
154.. _`faq:hidden_visibility`:
155
156"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
157============================================================================================================
158
159This error typically indicates that you are compiling without the required
160``-fvisibility`` flag.  pybind11 code internally forces hidden visibility on
161all internal code, but if non-hidden (and thus *exported*) code attempts to
162include a pybind type (for example, ``py::object`` or ``py::list``) you can run
163into this warning.
164
165To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
166compiling pybind code.
167
168As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
169have been compiled under different versions of pybind itself, it is also
170important that the symbols defined in one module do not clash with the
171potentially-incompatible symbols defined in another.  While Python extension
172modules are usually loaded with localized symbols (under POSIX systems
173typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
174can be changed, but even if it isn't it is not always enough to guarantee
175complete independence of the symbols involved when not using
176``-fvisibility=hidden``.
177
178Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size
179savings.  (See the following section for more details).
180
181
182.. _`faq:symhidden`:
183
184How can I create smaller binaries?
185==================================
186
187To do its job, pybind11 extensively relies on a programming technique known as
188*template metaprogramming*, which is a way of performing computation at compile
189time using type information. Template metaprogamming usually instantiates code
190involving significant numbers of deeply nested types that are either completely
191removed or reduced to just a few instructions during the compiler's optimization
192phase. However, due to the nested nature of these types, the resulting symbol
193names in the compiled extension library can be extremely long. For instance,
194the included test suite contains the following symbol:
195
196.. only:: html
197
198    .. code-block:: none
199
200        _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
201
202.. only:: not html
203
204    .. code-block:: cpp
205
206        __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
207
208which is the mangled form of the following function type:
209
210.. code-block:: cpp
211
212    pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
213
214The memory needed to store just the mangled name of this function (196 bytes)
215is larger than the actual piece of code (111 bytes) it represents! On the other
216hand, it's silly to even give this function a name -- after all, it's just a
217tiny cog in a bigger piece of machinery that is not exposed to the outside
218world. So we'll generally only want to export symbols for those functions which
219are actually called from the outside.
220
221This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
222and Clang, which sets the default symbol visibility to *hidden*, which has a
223tremendous impact on the final binary size of the resulting extension library.
224(On Visual Studio, symbols are already hidden by default, so nothing needs to
225be done there.)
226
227In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
228potential serious issues when loading multiple modules and is required for
229proper pybind operation.  See the previous FAQ entry for more details.
230
231Another aspect that can require a fair bit of code are function signature
232descriptions. pybind11 automatically generates human-readable function
233signatures for docstrings, e.g.:
234
235.. code-block:: none
236
237     |  __init__(...)
238     |      __init__(*args, **kwargs)
239     |      Overloaded function.
240     |
241     |      1. __init__(example.Example1) -> NoneType
242     |
243     |      Docstring for overload #1 goes here
244     |
245     |      2. __init__(example.Example1, int) -> NoneType
246     |
247     |      Docstring for overload #2 goes here
248     |
249     |      3. __init__(example.Example1, example.Example1) -> NoneType
250     |
251     |      Docstring for overload #3 goes here
252
253
254In C++11 mode, these are generated at run time using string concatenation,
255which can amount to 10-20% of the size of the resulting binary. If you can,
256enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which
257case signatures are efficiently pre-generated at compile time. Unfortunately,
258Visual Studio's C++14 support (``constexpr``) is not good enough as of April
2592016, so it always uses the more expensive run-time approach.
260
261Working with ancient Visual Studio 2009 builds on Windows
262=========================================================
263
264The official Windows distributions of Python are compiled using truly
265ancient versions of Visual Studio that lack good C++11 support. Some users
266implicitly assume that it would be impossible to load a plugin built with
267Visual Studio 2015 into a Python distribution that was compiled using Visual
268Studio 2009. However, no such issue exists: it's perfectly legitimate to
269interface DLLs that are built with different compilers and/or C libraries.
270Common gotchas to watch out for involve not ``free()``-ing memory region
271that that were ``malloc()``-ed in another shared library, using data
272structures with incompatible ABIs, and so on. pybind11 is very careful not
273to make these types of mistakes.
274