1.. _embedding:
2
3Embedding the interpreter
4#########################
5
6While pybind11 is mainly focused on extending Python using C++, it's also
7possible to do the reverse: embed the Python interpreter into a C++ program.
8All of the other documentation pages still apply here, so refer to them for
9general pybind11 usage. This section will cover a few extra things required
10for embedding.
11
12Getting started
13===============
14
15A basic executable with an embedded interpreter can be created with just a few
16lines of CMake and the ``pybind11::embed`` target, as shown below. For more
17information, see :doc:`/compiling`.
18
19.. code-block:: cmake
20
21    cmake_minimum_required(VERSION 3.0)
22    project(example)
23
24    find_package(pybind11 REQUIRED)  # or `add_subdirectory(pybind11)`
25
26    add_executable(example main.cpp)
27    target_link_libraries(example PRIVATE pybind11::embed)
28
29The essential structure of the ``main.cpp`` file looks like this:
30
31.. code-block:: cpp
32
33    #include <pybind11/embed.h> // everything needed for embedding
34    namespace py = pybind11;
35
36    int main() {
37        py::scoped_interpreter guard{}; // start the interpreter and keep it alive
38
39        py::print("Hello, World!"); // use the Python API
40    }
41
42The interpreter must be initialized before using any Python API, which includes
43all the functions and classes in pybind11. The RAII guard class `scoped_interpreter`
44takes care of the interpreter lifetime. After the guard is destroyed, the interpreter
45shuts down and clears its memory. No Python functions can be called after this.
46
47Executing Python code
48=====================
49
50There are a few different ways to run Python code. One option is to use `eval`,
51`exec` or `eval_file`, as explained in :ref:`eval`. Here is a quick example in
52the context of an executable with an embedded interpreter:
53
54.. code-block:: cpp
55
56    #include <pybind11/embed.h>
57    namespace py = pybind11;
58
59    int main() {
60        py::scoped_interpreter guard{};
61
62        py::exec(R"(
63            kwargs = dict(name="World", number=42)
64            message = "Hello, {name}! The answer is {number}".format(**kwargs)
65            print(message)
66        )");
67    }
68
69Alternatively, similar results can be achieved using pybind11's API (see
70:doc:`/advanced/pycpp/index` for more details).
71
72.. code-block:: cpp
73
74    #include <pybind11/embed.h>
75    namespace py = pybind11;
76    using namespace py::literals;
77
78    int main() {
79        py::scoped_interpreter guard{};
80
81        auto kwargs = py::dict("name"_a="World", "number"_a=42);
82        auto message = "Hello, {name}! The answer is {number}"_s.format(**kwargs);
83        py::print(message);
84    }
85
86The two approaches can also be combined:
87
88.. code-block:: cpp
89
90    #include <pybind11/embed.h>
91    #include <iostream>
92
93    namespace py = pybind11;
94    using namespace py::literals;
95
96    int main() {
97        py::scoped_interpreter guard{};
98
99        auto locals = py::dict("name"_a="World", "number"_a=42);
100        py::exec(R"(
101            message = "Hello, {name}! The answer is {number}".format(**locals())
102        )", py::globals(), locals);
103
104        auto message = locals["message"].cast<std::string>();
105        std::cout << message;
106    }
107
108Importing modules
109=================
110
111Python modules can be imported using `module::import()`:
112
113.. code-block:: cpp
114
115    py::module sys = py::module::import("sys");
116    py::print(sys.attr("path"));
117
118For convenience, the current working directory is included in ``sys.path`` when
119embedding the interpreter. This makes it easy to import local Python files:
120
121.. code-block:: python
122
123    """calc.py located in the working directory"""
124
125    def add(i, j):
126        return i + j
127
128
129.. code-block:: cpp
130
131    py::module calc = py::module::import("calc");
132    py::object result = calc.attr("add")(1, 2);
133    int n = result.cast<int>();
134    assert(n == 3);
135
136Modules can be reloaded using `module::reload()` if the source is modified e.g.
137by an external process. This can be useful in scenarios where the application
138imports a user defined data processing script which needs to be updated after
139changes by the user. Note that this function does not reload modules recursively.
140
141.. _embedding_modules:
142
143Adding embedded modules
144=======================
145
146Embedded binary modules can be added using the `PYBIND11_EMBEDDED_MODULE` macro.
147Note that the definition must be placed at global scope. They can be imported
148like any other module.
149
150.. code-block:: cpp
151
152    #include <pybind11/embed.h>
153    namespace py = pybind11;
154
155    PYBIND11_EMBEDDED_MODULE(fast_calc, m) {
156        // `m` is a `py::module` which is used to bind functions and classes
157        m.def("add", [](int i, int j) {
158            return i + j;
159        });
160    }
161
162    int main() {
163        py::scoped_interpreter guard{};
164
165        auto fast_calc = py::module::import("fast_calc");
166        auto result = fast_calc.attr("add")(1, 2).cast<int>();
167        assert(result == 3);
168    }
169
170Unlike extension modules where only a single binary module can be created, on
171the embedded side an unlimited number of modules can be added using multiple
172`PYBIND11_EMBEDDED_MODULE` definitions (as long as they have unique names).
173
174These modules are added to Python's list of builtins, so they can also be
175imported in pure Python files loaded by the interpreter. Everything interacts
176naturally:
177
178.. code-block:: python
179
180    """py_module.py located in the working directory"""
181    import cpp_module
182
183    a = cpp_module.a
184    b = a + 1
185
186
187.. code-block:: cpp
188
189    #include <pybind11/embed.h>
190    namespace py = pybind11;
191
192    PYBIND11_EMBEDDED_MODULE(cpp_module, m) {
193        m.attr("a") = 1;
194    }
195
196    int main() {
197        py::scoped_interpreter guard{};
198
199        auto py_module = py::module::import("py_module");
200
201        auto locals = py::dict("fmt"_a="{} + {} = {}", **py_module.attr("__dict__"));
202        assert(locals["a"].cast<int>() == 1);
203        assert(locals["b"].cast<int>() == 2);
204
205        py::exec(R"(
206            c = a + b
207            message = fmt.format(a, b, c)
208        )", py::globals(), locals);
209
210        assert(locals["c"].cast<int>() == 3);
211        assert(locals["message"].cast<std::string>() == "1 + 2 = 3");
212    }
213
214
215Interpreter lifetime
216====================
217
218The Python interpreter shuts down when `scoped_interpreter` is destroyed. After
219this, creating a new instance will restart the interpreter. Alternatively, the
220`initialize_interpreter` / `finalize_interpreter` pair of functions can be used
221to directly set the state at any time.
222
223Modules created with pybind11 can be safely re-initialized after the interpreter
224has been restarted. However, this may not apply to third-party extension modules.
225The issue is that Python itself cannot completely unload extension modules and
226there are several caveats with regard to interpreter restarting. In short, not
227all memory may be freed, either due to Python reference cycles or user-created
228global data. All the details can be found in the CPython documentation.
229
230.. warning::
231
232    Creating two concurrent `scoped_interpreter` guards is a fatal error. So is
233    calling `initialize_interpreter` for a second time after the interpreter
234    has already been initialized.
235
236    Do not use the raw CPython API functions ``Py_Initialize`` and
237    ``Py_Finalize`` as these do not properly handle the lifetime of
238    pybind11's internal data.
239
240
241Sub-interpreter support
242=======================
243
244Creating multiple copies of `scoped_interpreter` is not possible because it
245represents the main Python interpreter. Sub-interpreters are something different
246and they do permit the existence of multiple interpreters. This is an advanced
247feature of the CPython API and should be handled with care. pybind11 does not
248currently offer a C++ interface for sub-interpreters, so refer to the CPython
249documentation for all the details regarding this feature.
250
251We'll just mention a couple of caveats the sub-interpreters support in pybind11:
252
253 1. Sub-interpreters will not receive independent copies of embedded modules.
254    Instead, these are shared and modifications in one interpreter may be
255    reflected in another.
256
257 2. Managing multiple threads, multiple interpreters and the GIL can be
258    challenging and there are several caveats here, even within the pure
259    CPython API (please refer to the Python docs for details). As for
260    pybind11, keep in mind that `gil_scoped_release` and `gil_scoped_acquire`
261    do not take sub-interpreters into account.
262