compiling.rst revision 12391
1.. _compiling: 2 3Build systems 4############# 5 6Building with setuptools 7======================== 8 9For projects on PyPI, building with setuptools is the way to go. Sylvain Corlay 10has kindly provided an example project which shows how to set up everything, 11including automatic generation of documentation using Sphinx. Please refer to 12the [python_example]_ repository. 13 14.. [python_example] https://github.com/pybind/python_example 15 16Building with cppimport 17======================== 18 19[cppimport]_ is a small Python import hook that determines whether there is a C++ 20source file whose name matches the requested module. If there is, the file is 21compiled as a Python extension using pybind11 and placed in the same folder as 22the C++ source file. Python is then able to find the module and load it. 23 24.. [cppimport] https://github.com/tbenthompson/cppimport 25 26.. _cmake: 27 28Building with CMake 29=================== 30 31For C++ codebases that have an existing CMake-based build system, a Python 32extension module can be created with just a few lines of code: 33 34.. code-block:: cmake 35 36 cmake_minimum_required(VERSION 2.8.12) 37 project(example) 38 39 add_subdirectory(pybind11) 40 pybind11_add_module(example example.cpp) 41 42This assumes that the pybind11 repository is located in a subdirectory named 43:file:`pybind11` and that the code is located in a file named :file:`example.cpp`. 44The CMake command ``add_subdirectory`` will import the pybind11 project which 45provides the ``pybind11_add_module`` function. It will take care of all the 46details needed to build a Python extension module on any platform. 47 48A working sample project, including a way to invoke CMake from :file:`setup.py` for 49PyPI integration, can be found in the [cmake_example]_ repository. 50 51.. [cmake_example] https://github.com/pybind/cmake_example 52 53pybind11_add_module 54------------------- 55 56To ease the creation of Python extension modules, pybind11 provides a CMake 57function with the following signature: 58 59.. code-block:: cmake 60 61 pybind11_add_module(<name> [MODULE | SHARED] [EXCLUDE_FROM_ALL] 62 [NO_EXTRAS] [THIN_LTO] source1 [source2 ...]) 63 64This function behaves very much like CMake's builtin ``add_library`` (in fact, 65it's a wrapper function around that command). It will add a library target 66called ``<name>`` to be built from the listed source files. In addition, it 67will take care of all the Python-specific compiler and linker flags as well 68as the OS- and Python-version-specific file extension. The produced target 69``<name>`` can be further manipulated with regular CMake commands. 70 71``MODULE`` or ``SHARED`` may be given to specify the type of library. If no 72type is given, ``MODULE`` is used by default which ensures the creation of a 73Python-exclusive module. Specifying ``SHARED`` will create a more traditional 74dynamic library which can also be linked from elsewhere. ``EXCLUDE_FROM_ALL`` 75removes this target from the default build (see CMake docs for details). 76 77Since pybind11 is a template library, ``pybind11_add_module`` adds compiler 78flags to ensure high quality code generation without bloat arising from long 79symbol names and duplication of code in different translation units. It 80sets default visibility to *hidden*, which is required for some pybind11 81features and functionality when attempting to load multiple pybind11 modules 82compiled under different pybind11 versions. It also adds additional flags 83enabling LTO (Link Time Optimization) and strip unneeded symbols. See the 84:ref:`FAQ entry <faq:symhidden>` for a more detailed explanation. These 85latter optimizations are never applied in ``Debug`` mode. If ``NO_EXTRAS`` is 86given, they will always be disabled, even in ``Release`` mode. However, this 87will result in code bloat and is generally not recommended. 88 89As stated above, LTO is enabled by default. Some newer compilers also support 90different flavors of LTO such as `ThinLTO`_. Setting ``THIN_LTO`` will cause 91the function to prefer this flavor if available. The function falls back to 92regular LTO if ``-flto=thin`` is not available. 93 94.. _ThinLTO: http://clang.llvm.org/docs/ThinLTO.html 95 96Configuration variables 97----------------------- 98 99By default, pybind11 will compile modules with the C++14 standard, if available 100on the target compiler, falling back to C++11 if C++14 support is not 101available. Note, however, that this default is subject to change: future 102pybind11 releases are expected to migrate to newer C++ standards as they become 103available. To override this, the standard flag can be given explicitly in 104``PYBIND11_CPP_STANDARD``: 105 106.. code-block:: cmake 107 108 # Use just one of these: 109 # GCC/clang: 110 set(PYBIND11_CPP_STANDARD -std=c++11) 111 set(PYBIND11_CPP_STANDARD -std=c++14) 112 set(PYBIND11_CPP_STANDARD -std=c++1z) # Experimental C++17 support 113 # MSVC: 114 set(PYBIND11_CPP_STANDARD /std:c++14) 115 set(PYBIND11_CPP_STANDARD /std:c++latest) # Enables some MSVC C++17 features 116 117 add_subdirectory(pybind11) # or find_package(pybind11) 118 119Note that this and all other configuration variables must be set **before** the 120call to ``add_subdirectory`` or ``find_package``. The variables can also be set 121when calling CMake from the command line using the ``-D<variable>=<value>`` flag. 122 123The target Python version can be selected by setting ``PYBIND11_PYTHON_VERSION`` 124or an exact Python installation can be specified with ``PYTHON_EXECUTABLE``. 125For example: 126 127.. code-block:: bash 128 129 cmake -DPYBIND11_PYTHON_VERSION=3.6 .. 130 # or 131 cmake -DPYTHON_EXECUTABLE=path/to/python .. 132 133find_package vs. add_subdirectory 134--------------------------------- 135 136For CMake-based projects that don't include the pybind11 repository internally, 137an external installation can be detected through ``find_package(pybind11)``. 138See the `Config file`_ docstring for details of relevant CMake variables. 139 140.. code-block:: cmake 141 142 cmake_minimum_required(VERSION 2.8.12) 143 project(example) 144 145 find_package(pybind11 REQUIRED) 146 pybind11_add_module(example example.cpp) 147 148Once detected, the aforementioned ``pybind11_add_module`` can be employed as 149before. The function usage and configuration variables are identical no matter 150if pybind11 is added as a subdirectory or found as an installed package. You 151can refer to the same [cmake_example]_ repository for a full sample project 152-- just swap out ``add_subdirectory`` for ``find_package``. 153 154.. _Config file: https://github.com/pybind/pybind11/blob/master/tools/pybind11Config.cmake.in 155 156Advanced: interface library target 157---------------------------------- 158 159When using a version of CMake greater than 3.0, pybind11 can additionally 160be used as a special *interface library* . The target ``pybind11::module`` 161is available with pybind11 headers, Python headers and libraries as needed, 162and C++ compile definitions attached. This target is suitable for linking 163to an independently constructed (through ``add_library``, not 164``pybind11_add_module``) target in the consuming project. 165 166.. code-block:: cmake 167 168 cmake_minimum_required(VERSION 3.0) 169 project(example) 170 171 find_package(pybind11 REQUIRED) # or add_subdirectory(pybind11) 172 173 add_library(example MODULE main.cpp) 174 target_link_libraries(example PRIVATE pybind11::module) 175 set_target_properties(example PROPERTIES PREFIX "${PYTHON_MODULE_PREFIX}" 176 SUFFIX "${PYTHON_MODULE_EXTENSION}") 177 178.. warning:: 179 180 Since pybind11 is a metatemplate library, it is crucial that certain 181 compiler flags are provided to ensure high quality code generation. In 182 contrast to the ``pybind11_add_module()`` command, the CMake interface 183 library only provides the *minimal* set of parameters to ensure that the 184 code using pybind11 compiles, but it does **not** pass these extra compiler 185 flags (i.e. this is up to you). 186 187 These include Link Time Optimization (``-flto`` on GCC/Clang/ICPC, ``/GL`` 188 and ``/LTCG`` on Visual Studio) and .OBJ files with many sections on Visual 189 Studio (``/bigobj``). The :ref:`FAQ <faq:symhidden>` contains an 190 explanation on why these are needed. 191 192Embedding the Python interpreter 193-------------------------------- 194 195In addition to extension modules, pybind11 also supports embedding Python into 196a C++ executable or library. In CMake, simply link with the ``pybind11::embed`` 197target. It provides everything needed to get the interpreter running. The Python 198headers and libraries are attached to the target. Unlike ``pybind11::module``, 199there is no need to manually set any additional properties here. For more 200information about usage in C++, see :doc:`/advanced/embedding`. 201 202.. code-block:: cmake 203 204 cmake_minimum_required(VERSION 3.0) 205 project(example) 206 207 find_package(pybind11 REQUIRED) # or add_subdirectory(pybind11) 208 209 add_executable(example main.cpp) 210 target_link_libraries(example PRIVATE pybind11::embed) 211 212.. _building_manually: 213 214Building manually 215================= 216 217pybind11 is a header-only library, hence it is not necessary to link against 218any special libraries and there are no intermediate (magic) translation steps. 219 220On Linux, you can compile an example such as the one given in 221:ref:`simple_example` using the following command: 222 223.. code-block:: bash 224 225 $ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix` 226 227The flags given here assume that you're using Python 3. For Python 2, just 228change the executable appropriately (to ``python`` or ``python2``). 229 230The ``python3 -m pybind11 --includes`` command fetches the include paths for 231both pybind11 and Python headers. This assumes that pybind11 has been installed 232using ``pip`` or ``conda``. If it hasn't, you can also manually specify 233``-I <path-to-pybind11>/include`` together with the Python includes path 234``python3-config --includes``. 235 236Note that Python 2.7 modules don't use a special suffix, so you should simply 237use ``example.so`` instead of ``example`python3-config --extension-suffix```. 238Besides, the ``--extension-suffix`` option may or may not be available, depending 239on the distribution; in the latter case, the module extension can be manually 240set to ``.so``. 241 242On Mac OS: the build command is almost the same but it also requires passing 243the ``-undefined dynamic_lookup`` flag so as to ignore missing symbols when 244building the module: 245 246.. code-block:: bash 247 248 $ c++ -O3 -Wall -shared -std=c++11 -undefined dynamic_lookup `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix` 249 250In general, it is advisable to include several additional build parameters 251that can considerably reduce the size of the created binary. Refer to section 252:ref:`cmake` for a detailed example of a suitable cross-platform CMake-based 253build system that works on all platforms including Windows. 254 255.. note:: 256 257 On Linux and macOS, it's better to (intentionally) not link against 258 ``libpython``. The symbols will be resolved when the extension library 259 is loaded into a Python binary. This is preferable because you might 260 have several different installations of a given Python version (e.g. the 261 system-provided Python, and one that ships with a piece of commercial 262 software). In this way, the plugin will work with both versions, instead 263 of possibly importing a second Python library into a process that already 264 contains one (which will lead to a segfault). 265 266Generating binding code automatically 267===================================== 268 269The ``Binder`` project is a tool for automatic generation of pybind11 binding 270code by introspecting existing C++ codebases using LLVM/Clang. See the 271[binder]_ documentation for details. 272 273.. [binder] http://cppbinder.readthedocs.io/en/latest/about.html 274