embedding.rst revision 12391
112391Sjason@lowepower.com.. _embedding: 212391Sjason@lowepower.com 312391Sjason@lowepower.comEmbedding the interpreter 412391Sjason@lowepower.com######################### 512391Sjason@lowepower.com 612391Sjason@lowepower.comWhile pybind11 is mainly focused on extending Python using C++, it's also 712391Sjason@lowepower.compossible to do the reverse: embed the Python interpreter into a C++ program. 812391Sjason@lowepower.comAll of the other documentation pages still apply here, so refer to them for 912391Sjason@lowepower.comgeneral pybind11 usage. This section will cover a few extra things required 1012391Sjason@lowepower.comfor embedding. 1112391Sjason@lowepower.com 1212391Sjason@lowepower.comGetting started 1312391Sjason@lowepower.com=============== 1412391Sjason@lowepower.com 1512391Sjason@lowepower.comA basic executable with an embedded interpreter can be created with just a few 1612391Sjason@lowepower.comlines of CMake and the ``pybind11::embed`` target, as shown below. For more 1712391Sjason@lowepower.cominformation, see :doc:`/compiling`. 1812391Sjason@lowepower.com 1912391Sjason@lowepower.com.. code-block:: cmake 2012391Sjason@lowepower.com 2112391Sjason@lowepower.com cmake_minimum_required(VERSION 3.0) 2212391Sjason@lowepower.com project(example) 2312391Sjason@lowepower.com 2412391Sjason@lowepower.com find_package(pybind11 REQUIRED) # or `add_subdirectory(pybind11)` 2512391Sjason@lowepower.com 2612391Sjason@lowepower.com add_executable(example main.cpp) 2712391Sjason@lowepower.com target_link_libraries(example PRIVATE pybind11::embed) 2812391Sjason@lowepower.com 2912391Sjason@lowepower.comThe essential structure of the ``main.cpp`` file looks like this: 3012391Sjason@lowepower.com 3112391Sjason@lowepower.com.. code-block:: cpp 3212391Sjason@lowepower.com 3312391Sjason@lowepower.com #include <pybind11/embed.h> // everything needed for embedding 3412391Sjason@lowepower.com namespace py = pybind11; 3512391Sjason@lowepower.com 3612391Sjason@lowepower.com int main() { 3712391Sjason@lowepower.com py::scoped_interpreter guard{}; // start the interpreter and keep it alive 3812391Sjason@lowepower.com 3912391Sjason@lowepower.com py::print("Hello, World!"); // use the Python API 4012391Sjason@lowepower.com } 4112391Sjason@lowepower.com 4212391Sjason@lowepower.comThe interpreter must be initialized before using any Python API, which includes 4312391Sjason@lowepower.comall the functions and classes in pybind11. The RAII guard class `scoped_interpreter` 4412391Sjason@lowepower.comtakes care of the interpreter lifetime. After the guard is destroyed, the interpreter 4512391Sjason@lowepower.comshuts down and clears its memory. No Python functions can be called after this. 4612391Sjason@lowepower.com 4712391Sjason@lowepower.comExecuting Python code 4812391Sjason@lowepower.com===================== 4912391Sjason@lowepower.com 5012391Sjason@lowepower.comThere are a few different ways to run Python code. One option is to use `eval`, 5112391Sjason@lowepower.com`exec` or `eval_file`, as explained in :ref:`eval`. Here is a quick example in 5212391Sjason@lowepower.comthe context of an executable with an embedded interpreter: 5312391Sjason@lowepower.com 5412391Sjason@lowepower.com.. code-block:: cpp 5512391Sjason@lowepower.com 5612391Sjason@lowepower.com #include <pybind11/embed.h> 5712391Sjason@lowepower.com namespace py = pybind11; 5812391Sjason@lowepower.com 5912391Sjason@lowepower.com int main() { 6012391Sjason@lowepower.com py::scoped_interpreter guard{}; 6112391Sjason@lowepower.com 6212391Sjason@lowepower.com py::exec(R"( 6312391Sjason@lowepower.com kwargs = dict(name="World", number=42) 6412391Sjason@lowepower.com message = "Hello, {name}! The answer is {number}".format(**kwargs) 6512391Sjason@lowepower.com print(message) 6612391Sjason@lowepower.com )"); 6712391Sjason@lowepower.com } 6812391Sjason@lowepower.com 6912391Sjason@lowepower.comAlternatively, similar results can be achieved using pybind11's API (see 7012391Sjason@lowepower.com:doc:`/advanced/pycpp/index` for more details). 7112391Sjason@lowepower.com 7212391Sjason@lowepower.com.. code-block:: cpp 7312391Sjason@lowepower.com 7412391Sjason@lowepower.com #include <pybind11/embed.h> 7512391Sjason@lowepower.com namespace py = pybind11; 7612391Sjason@lowepower.com using namespace py::literals; 7712391Sjason@lowepower.com 7812391Sjason@lowepower.com int main() { 7912391Sjason@lowepower.com py::scoped_interpreter guard{}; 8012391Sjason@lowepower.com 8112391Sjason@lowepower.com auto kwargs = py::dict("name"_a="World", "number"_a=42); 8212391Sjason@lowepower.com auto message = "Hello, {name}! The answer is {number}"_s.format(**kwargs); 8312391Sjason@lowepower.com py::print(message); 8412391Sjason@lowepower.com } 8512391Sjason@lowepower.com 8612391Sjason@lowepower.comThe two approaches can also be combined: 8712391Sjason@lowepower.com 8812391Sjason@lowepower.com.. code-block:: cpp 8912391Sjason@lowepower.com 9012391Sjason@lowepower.com #include <pybind11/embed.h> 9112391Sjason@lowepower.com #include <iostream> 9212391Sjason@lowepower.com 9312391Sjason@lowepower.com namespace py = pybind11; 9412391Sjason@lowepower.com using namespace py::literals; 9512391Sjason@lowepower.com 9612391Sjason@lowepower.com int main() { 9712391Sjason@lowepower.com py::scoped_interpreter guard{}; 9812391Sjason@lowepower.com 9912391Sjason@lowepower.com auto locals = py::dict("name"_a="World", "number"_a=42); 10012391Sjason@lowepower.com py::exec(R"( 10112391Sjason@lowepower.com message = "Hello, {name}! The answer is {number}".format(**locals()) 10212391Sjason@lowepower.com )", py::globals(), locals); 10312391Sjason@lowepower.com 10412391Sjason@lowepower.com auto message = locals["message"].cast<std::string>(); 10512391Sjason@lowepower.com std::cout << message; 10612391Sjason@lowepower.com } 10712391Sjason@lowepower.com 10812391Sjason@lowepower.comImporting modules 10912391Sjason@lowepower.com================= 11012391Sjason@lowepower.com 11112391Sjason@lowepower.comPython modules can be imported using `module::import()`: 11212391Sjason@lowepower.com 11312391Sjason@lowepower.com.. code-block:: cpp 11412391Sjason@lowepower.com 11512391Sjason@lowepower.com py::module sys = py::module::import("sys"); 11612391Sjason@lowepower.com py::print(sys.attr("path")); 11712391Sjason@lowepower.com 11812391Sjason@lowepower.comFor convenience, the current working directory is included in ``sys.path`` when 11912391Sjason@lowepower.comembedding the interpreter. This makes it easy to import local Python files: 12012391Sjason@lowepower.com 12112391Sjason@lowepower.com.. code-block:: python 12212391Sjason@lowepower.com 12312391Sjason@lowepower.com """calc.py located in the working directory""" 12412391Sjason@lowepower.com 12512391Sjason@lowepower.com def add(i, j): 12612391Sjason@lowepower.com return i + j 12712391Sjason@lowepower.com 12812391Sjason@lowepower.com 12912391Sjason@lowepower.com.. code-block:: cpp 13012391Sjason@lowepower.com 13112391Sjason@lowepower.com py::module calc = py::module::import("calc"); 13212391Sjason@lowepower.com py::object result = calc.attr("add")(1, 2); 13312391Sjason@lowepower.com int n = result.cast<int>(); 13412391Sjason@lowepower.com assert(n == 3); 13512391Sjason@lowepower.com 13612391Sjason@lowepower.comModules can be reloaded using `module::reload()` if the source is modified e.g. 13712391Sjason@lowepower.comby an external process. This can be useful in scenarios where the application 13812391Sjason@lowepower.comimports a user defined data processing script which needs to be updated after 13912391Sjason@lowepower.comchanges by the user. Note that this function does not reload modules recursively. 14012391Sjason@lowepower.com 14112391Sjason@lowepower.com.. _embedding_modules: 14212391Sjason@lowepower.com 14312391Sjason@lowepower.comAdding embedded modules 14412391Sjason@lowepower.com======================= 14512391Sjason@lowepower.com 14612391Sjason@lowepower.comEmbedded binary modules can be added using the `PYBIND11_EMBEDDED_MODULE` macro. 14712391Sjason@lowepower.comNote that the definition must be placed at global scope. They can be imported 14812391Sjason@lowepower.comlike any other module. 14912391Sjason@lowepower.com 15012391Sjason@lowepower.com.. code-block:: cpp 15112391Sjason@lowepower.com 15212391Sjason@lowepower.com #include <pybind11/embed.h> 15312391Sjason@lowepower.com namespace py = pybind11; 15412391Sjason@lowepower.com 15512391Sjason@lowepower.com PYBIND11_EMBEDDED_MODULE(fast_calc, m) { 15612391Sjason@lowepower.com // `m` is a `py::module` which is used to bind functions and classes 15712391Sjason@lowepower.com m.def("add", [](int i, int j) { 15812391Sjason@lowepower.com return i + j; 15912391Sjason@lowepower.com }); 16012391Sjason@lowepower.com } 16112391Sjason@lowepower.com 16212391Sjason@lowepower.com int main() { 16312391Sjason@lowepower.com py::scoped_interpreter guard{}; 16412391Sjason@lowepower.com 16512391Sjason@lowepower.com auto fast_calc = py::module::import("fast_calc"); 16612391Sjason@lowepower.com auto result = fast_calc.attr("add")(1, 2).cast<int>(); 16712391Sjason@lowepower.com assert(result == 3); 16812391Sjason@lowepower.com } 16912391Sjason@lowepower.com 17012391Sjason@lowepower.comUnlike extension modules where only a single binary module can be created, on 17112391Sjason@lowepower.comthe embedded side an unlimited number of modules can be added using multiple 17212391Sjason@lowepower.com`PYBIND11_EMBEDDED_MODULE` definitions (as long as they have unique names). 17312391Sjason@lowepower.com 17412391Sjason@lowepower.comThese modules are added to Python's list of builtins, so they can also be 17512391Sjason@lowepower.comimported in pure Python files loaded by the interpreter. Everything interacts 17612391Sjason@lowepower.comnaturally: 17712391Sjason@lowepower.com 17812391Sjason@lowepower.com.. code-block:: python 17912391Sjason@lowepower.com 18012391Sjason@lowepower.com """py_module.py located in the working directory""" 18112391Sjason@lowepower.com import cpp_module 18212391Sjason@lowepower.com 18312391Sjason@lowepower.com a = cpp_module.a 18412391Sjason@lowepower.com b = a + 1 18512391Sjason@lowepower.com 18612391Sjason@lowepower.com 18712391Sjason@lowepower.com.. code-block:: cpp 18812391Sjason@lowepower.com 18912391Sjason@lowepower.com #include <pybind11/embed.h> 19012391Sjason@lowepower.com namespace py = pybind11; 19112391Sjason@lowepower.com 19212391Sjason@lowepower.com PYBIND11_EMBEDDED_MODULE(cpp_module, m) { 19312391Sjason@lowepower.com m.attr("a") = 1; 19412391Sjason@lowepower.com } 19512391Sjason@lowepower.com 19612391Sjason@lowepower.com int main() { 19712391Sjason@lowepower.com py::scoped_interpreter guard{}; 19812391Sjason@lowepower.com 19912391Sjason@lowepower.com auto py_module = py::module::import("py_module"); 20012391Sjason@lowepower.com 20112391Sjason@lowepower.com auto locals = py::dict("fmt"_a="{} + {} = {}", **py_module.attr("__dict__")); 20212391Sjason@lowepower.com assert(locals["a"].cast<int>() == 1); 20312391Sjason@lowepower.com assert(locals["b"].cast<int>() == 2); 20412391Sjason@lowepower.com 20512391Sjason@lowepower.com py::exec(R"( 20612391Sjason@lowepower.com c = a + b 20712391Sjason@lowepower.com message = fmt.format(a, b, c) 20812391Sjason@lowepower.com )", py::globals(), locals); 20912391Sjason@lowepower.com 21012391Sjason@lowepower.com assert(locals["c"].cast<int>() == 3); 21112391Sjason@lowepower.com assert(locals["message"].cast<std::string>() == "1 + 2 = 3"); 21212391Sjason@lowepower.com } 21312391Sjason@lowepower.com 21412391Sjason@lowepower.com 21512391Sjason@lowepower.comInterpreter lifetime 21612391Sjason@lowepower.com==================== 21712391Sjason@lowepower.com 21812391Sjason@lowepower.comThe Python interpreter shuts down when `scoped_interpreter` is destroyed. After 21912391Sjason@lowepower.comthis, creating a new instance will restart the interpreter. Alternatively, the 22012391Sjason@lowepower.com`initialize_interpreter` / `finalize_interpreter` pair of functions can be used 22112391Sjason@lowepower.comto directly set the state at any time. 22212391Sjason@lowepower.com 22312391Sjason@lowepower.comModules created with pybind11 can be safely re-initialized after the interpreter 22412391Sjason@lowepower.comhas been restarted. However, this may not apply to third-party extension modules. 22512391Sjason@lowepower.comThe issue is that Python itself cannot completely unload extension modules and 22612391Sjason@lowepower.comthere are several caveats with regard to interpreter restarting. In short, not 22712391Sjason@lowepower.comall memory may be freed, either due to Python reference cycles or user-created 22812391Sjason@lowepower.comglobal data. All the details can be found in the CPython documentation. 22912391Sjason@lowepower.com 23012391Sjason@lowepower.com.. warning:: 23112391Sjason@lowepower.com 23212391Sjason@lowepower.com Creating two concurrent `scoped_interpreter` guards is a fatal error. So is 23312391Sjason@lowepower.com calling `initialize_interpreter` for a second time after the interpreter 23412391Sjason@lowepower.com has already been initialized. 23512391Sjason@lowepower.com 23612391Sjason@lowepower.com Do not use the raw CPython API functions ``Py_Initialize`` and 23712391Sjason@lowepower.com ``Py_Finalize`` as these do not properly handle the lifetime of 23812391Sjason@lowepower.com pybind11's internal data. 23912391Sjason@lowepower.com 24012391Sjason@lowepower.com 24112391Sjason@lowepower.comSub-interpreter support 24212391Sjason@lowepower.com======================= 24312391Sjason@lowepower.com 24412391Sjason@lowepower.comCreating multiple copies of `scoped_interpreter` is not possible because it 24512391Sjason@lowepower.comrepresents the main Python interpreter. Sub-interpreters are something different 24612391Sjason@lowepower.comand they do permit the existence of multiple interpreters. This is an advanced 24712391Sjason@lowepower.comfeature of the CPython API and should be handled with care. pybind11 does not 24812391Sjason@lowepower.comcurrently offer a C++ interface for sub-interpreters, so refer to the CPython 24912391Sjason@lowepower.comdocumentation for all the details regarding this feature. 25012391Sjason@lowepower.com 25112391Sjason@lowepower.comWe'll just mention a couple of caveats the sub-interpreters support in pybind11: 25212391Sjason@lowepower.com 25312391Sjason@lowepower.com 1. Sub-interpreters will not receive independent copies of embedded modules. 25412391Sjason@lowepower.com Instead, these are shared and modifications in one interpreter may be 25512391Sjason@lowepower.com reflected in another. 25612391Sjason@lowepower.com 25712391Sjason@lowepower.com 2. Managing multiple threads, multiple interpreters and the GIL can be 25812391Sjason@lowepower.com challenging and there are several caveats here, even within the pure 25912391Sjason@lowepower.com CPython API (please refer to the Python docs for details). As for 26012391Sjason@lowepower.com pybind11, keep in mind that `gil_scoped_release` and `gil_scoped_acquire` 26112391Sjason@lowepower.com do not take sub-interpreters into account. 262