faq.rst revision 14299:2fbea9df56d2
1Frequently asked questions
2##########################
3
4"ImportError: dynamic module does not define init function"
5===========================================================
6
71. Make sure that the name specified in PYBIND11_MODULE is identical to the
8filename of the extension library (without prefixes such as .so)
9
102. If the above did not fix the issue, you are likely using an incompatible
11version of Python (for instance, the extension library was compiled against
12Python 2, while the interpreter is running on top of some version of Python
133, or vice versa).
14
15"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
16========================================================================
17
18See the first answer.
19
20"SystemError: dynamic module not initialized properly"
21======================================================
22
23See the first answer.
24
25The Python interpreter immediately crashes when importing my module
26===================================================================
27
28See the first answer.
29
30CMake doesn't detect the right Python version
31=============================================
32
33The CMake-based build system will try to automatically detect the installed
34version of Python and link against that. When this fails, or when there are
35multiple versions of Python and it finds the wrong one, delete
36``CMakeCache.txt`` and then invoke CMake as follows:
37
38.. code-block:: bash
39
40    cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> .
41
42.. _faq_reference_arguments:
43
44Limitations involving reference arguments
45=========================================
46
47In C++, it's fairly common to pass arguments using mutable references or
48mutable pointers, which allows both read and write access to the value
49supplied by the caller. This is sometimes done for efficiency reasons, or to
50realize functions that have multiple return values. Here are two very basic
51examples:
52
53.. code-block:: cpp
54
55    void increment(int &i) { i++; }
56    void increment_ptr(int *i) { (*i)++; }
57
58In Python, all arguments are passed by reference, so there is no general
59issue in binding such code from Python.
60
61However, certain basic Python types (like ``str``, ``int``, ``bool``,
62``float``, etc.) are **immutable**. This means that the following attempt
63to port the function to Python doesn't have the same effect on the value
64provided by the caller -- in fact, it does nothing at all.
65
66.. code-block:: python
67
68    def increment(i):
69        i += 1 # nope..
70
71pybind11 is also affected by such language-level conventions, which means that
72binding ``increment`` or ``increment_ptr`` will also create Python functions
73that don't modify their arguments.
74
75Although inconvenient, one workaround is to encapsulate the immutable types in
76a custom type that does allow modifications.
77
78An other alternative involves binding a small wrapper lambda function that
79returns a tuple with all output arguments (see the remainder of the
80documentation for examples on binding lambda functions). An example:
81
82.. code-block:: cpp
83
84    int foo(int &i) { i++; return 123; }
85
86and the binding code
87
88.. code-block:: cpp
89
90   m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
91
92
93How can I reduce the build time?
94================================
95
96It's good practice to split binding code over multiple files, as in the
97following example:
98
99:file:`example.cpp`:
100
101.. code-block:: cpp
102
103    void init_ex1(py::module &);
104    void init_ex2(py::module &);
105    /* ... */
106
107    PYBIND11_MODULE(example, m) {
108        init_ex1(m);
109        init_ex2(m);
110        /* ... */
111    }
112
113:file:`ex1.cpp`:
114
115.. code-block:: cpp
116
117    void init_ex1(py::module &m) {
118        m.def("add", [](int a, int b) { return a + b; });
119    }
120
121:file:`ex2.cpp`:
122
123.. code-block:: cpp
124
125    void init_ex2(py::module &m) {
126        m.def("sub", [](int a, int b) { return a - b; });
127    }
128
129:command:`python`:
130
131.. code-block:: pycon
132
133    >>> import example
134    >>> example.add(1, 2)
135    3
136    >>> example.sub(1, 1)
137    0
138
139As shown above, the various ``init_ex`` functions should be contained in
140separate files that can be compiled independently from one another, and then
141linked together into the same final shared object.  Following this approach
142will:
143
1441. reduce memory requirements per compilation unit.
145
1462. enable parallel builds (if desired).
147
1483. allow for faster incremental builds. For instance, when a single class
149   definition is changed, only a subset of the binding code will generally need
150   to be recompiled.
151
152"recursive template instantiation exceeded maximum depth of 256"
153================================================================
154
155If you receive an error about excessive recursive template evaluation, try
156specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
157culprit is generally the generation of function signatures at compile time
158using C++14 template metaprogramming.
159
160.. _`faq:hidden_visibility`:
161
162"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
163============================================================================================================
164
165This error typically indicates that you are compiling without the required
166``-fvisibility`` flag.  pybind11 code internally forces hidden visibility on
167all internal code, but if non-hidden (and thus *exported*) code attempts to
168include a pybind type (for example, ``py::object`` or ``py::list``) you can run
169into this warning.
170
171To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
172compiling pybind code.
173
174As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
175have been compiled under different versions of pybind itself, it is also
176important that the symbols defined in one module do not clash with the
177potentially-incompatible symbols defined in another.  While Python extension
178modules are usually loaded with localized symbols (under POSIX systems
179typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
180can be changed, but even if it isn't it is not always enough to guarantee
181complete independence of the symbols involved when not using
182``-fvisibility=hidden``.
183
184Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size
185savings.  (See the following section for more details).
186
187
188.. _`faq:symhidden`:
189
190How can I create smaller binaries?
191==================================
192
193To do its job, pybind11 extensively relies on a programming technique known as
194*template metaprogramming*, which is a way of performing computation at compile
195time using type information. Template metaprogamming usually instantiates code
196involving significant numbers of deeply nested types that are either completely
197removed or reduced to just a few instructions during the compiler's optimization
198phase. However, due to the nested nature of these types, the resulting symbol
199names in the compiled extension library can be extremely long. For instance,
200the included test suite contains the following symbol:
201
202.. only:: html
203
204    .. code-block:: none
205
206        _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
207
208.. only:: not html
209
210    .. code-block:: cpp
211
212        __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
213
214which is the mangled form of the following function type:
215
216.. code-block:: cpp
217
218    pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
219
220The memory needed to store just the mangled name of this function (196 bytes)
221is larger than the actual piece of code (111 bytes) it represents! On the other
222hand, it's silly to even give this function a name -- after all, it's just a
223tiny cog in a bigger piece of machinery that is not exposed to the outside
224world. So we'll generally only want to export symbols for those functions which
225are actually called from the outside.
226
227This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
228and Clang, which sets the default symbol visibility to *hidden*, which has a
229tremendous impact on the final binary size of the resulting extension library.
230(On Visual Studio, symbols are already hidden by default, so nothing needs to
231be done there.)
232
233In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
234potential serious issues when loading multiple modules and is required for
235proper pybind operation.  See the previous FAQ entry for more details.
236
237Working with ancient Visual Studio 2008 builds on Windows
238=========================================================
239
240The official Windows distributions of Python are compiled using truly
241ancient versions of Visual Studio that lack good C++11 support. Some users
242implicitly assume that it would be impossible to load a plugin built with
243Visual Studio 2015 into a Python distribution that was compiled using Visual
244Studio 2008. However, no such issue exists: it's perfectly legitimate to
245interface DLLs that are built with different compilers and/or C libraries.
246Common gotchas to watch out for involve not ``free()``-ing memory region
247that that were ``malloc()``-ed in another shared library, using data
248structures with incompatible ABIs, and so on. pybind11 is very careful not
249to make these types of mistakes.
250
251Inconsistent detection of Python version in CMake and pybind11
252==============================================================
253
254The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)`` provided by CMake
255for Python version detection are not used by pybind11 due to unreliability and limitations that make
256them unsuitable for pybind11's needs. Instead pybind provides its own, more reliable Python detection
257CMake code. Conflicts can arise, however, when using pybind11 in a project that *also* uses the CMake
258Python detection in a system with several Python versions installed.
259
260This difference may cause inconsistencies and errors if *both* mechanisms are used in the same project. Consider the following
261Cmake code executed in a system with Python 2.7 and 3.x installed:
262
263.. code-block:: cmake
264
265    find_package(PythonInterp)
266    find_package(PythonLibs)
267    find_package(pybind11)
268
269It will detect Python 2.7 and pybind11 will pick it as well.
270
271In contrast this code:
272
273.. code-block:: cmake
274
275    find_package(pybind11)
276    find_package(PythonInterp)
277    find_package(PythonLibs)
278
279will detect Python 3.x for pybind11 and may crash on ``find_package(PythonLibs)`` afterwards.
280
281It is advised to avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)`` from CMake and rely
282on pybind11 in detecting Python version. If this is not possible CMake machinery should be called *before* including pybind11.
283
284How to cite this project?
285=========================
286
287We suggest the following BibTeX template to cite pybind11 in scientific
288discourse:
289
290.. code-block:: bash
291
292    @misc{pybind11,
293       author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan},
294       year = {2017},
295       note = {https://github.com/pybind/pybind11},
296       title = {pybind11 -- Seamless operability between C++11 and Python}
297    }
298