functions.rst revision 11986:c12e4625ab56
1Functions 2######### 3 4Before proceeding with this section, make sure that you are already familiar 5with the basics of binding functions and classes, as explained in :doc:`/basics` 6and :doc:`/classes`. The following guide is applicable to both free and member 7functions, i.e. *methods* in Python. 8 9Return value policies 10===================== 11 12Python and C++ use fundamentally different ways of managing the memory and 13lifetime of objects managed by them. This can lead to issues when creating 14bindings for functions that return a non-trivial type. Just by looking at the 15type information, it is not clear whether Python should take charge of the 16returned value and eventually free its resources, or if this is handled on the 17C++ side. For this reason, pybind11 provides a several `return value policy` 18annotations that can be passed to the :func:`module::def` and 19:func:`class_::def` functions. The default policy is 20:enum:`return_value_policy::automatic`. 21 22Return value policies are tricky, and it's very important to get them right. 23Just to illustrate what can go wrong, consider the following simple example: 24 25.. code-block:: cpp 26 27 /* Function declaration */ 28 Data *get_data() { return _data; /* (pointer to a static data structure) */ } 29 ... 30 31 /* Binding code */ 32 m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python 33 34What's going on here? When ``get_data()`` is called from Python, the return 35value (a native C++ type) must be wrapped to turn it into a usable Python type. 36In this case, the default return value policy (:enum:`return_value_policy::automatic`) 37causes pybind11 to assume ownership of the static ``_data`` instance. 38 39When Python's garbage collector eventually deletes the Python 40wrapper, pybind11 will also attempt to delete the C++ instance (via ``operator 41delete()``) due to the implied ownership. At this point, the entire application 42will come crashing down, though errors could also be more subtle and involve 43silent data corruption. 44 45In the above example, the policy :enum:`return_value_policy::reference` should have 46been specified so that the global data instance is only *referenced* without any 47implied transfer of ownership, i.e.: 48 49.. code-block:: cpp 50 51 m.def("get_data", &get_data, return_value_policy::reference); 52 53On the other hand, this is not the right policy for many other situations, 54where ignoring ownership could lead to resource leaks. 55As a developer using pybind11, it's important to be familiar with the different 56return value policies, including which situation calls for which one of them. 57The following table provides an overview of available policies: 58 59.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| 60 61+--------------------------------------------------+----------------------------------------------------------------------------+ 62| Return value policy | Description | 63+==================================================+============================================================================+ 64| :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | 65| | ownership. Python will call the destructor and delete operator when the | 66| | object's reference count reaches zero. Undefined behavior ensues when the | 67| | C++ side does the same, or when the data was not dynamically allocated. | 68+--------------------------------------------------+----------------------------------------------------------------------------+ 69| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | 70| | This policy is comparably safe because the lifetimes of the two instances | 71| | are decoupled. | 72+--------------------------------------------------+----------------------------------------------------------------------------+ 73| :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance | 74| | that will be owned by Python. This policy is comparably safe because the | 75| | lifetimes of the two instances (move source and destination) are decoupled.| 76+--------------------------------------------------+----------------------------------------------------------------------------+ 77| :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is | 78| | responsible for managing the object's lifetime and deallocating it when | 79| | it is no longer used. Warning: undefined behavior will ensue when the C++ | 80| | side deletes an object that is still referenced and used by Python. | 81+--------------------------------------------------+----------------------------------------------------------------------------+ 82| :enum:`return_value_policy::reference_internal` | Indicates that the lifetime of the return value is tied to the lifetime | 83| | of a parent object, namely the implicit ``this``, or ``self`` argument of | 84| | the called method or property. Internally, this policy works just like | 85| | :enum:`return_value_policy::reference` but additionally applies a | 86| | ``keep_alive<0, 1>`` *call policy* (described in the next section) that | 87| | prevents the parent object from being garbage collected as long as the | 88| | return value is referenced by Python. This is the default policy for | 89| | property getters created via ``def_property``, ``def_readwrite``, etc. | 90+--------------------------------------------------+----------------------------------------------------------------------------+ 91| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy | 92| | :enum:`return_value_policy::take_ownership` when the return value is a | 93| | pointer. Otherwise, it uses :enum:`return_value::move` or | 94| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. | 95| | See above for a description of what all of these different policies do. | 96+--------------------------------------------------+----------------------------------------------------------------------------+ 97| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | 98| | return value is a pointer. This is the default conversion policy for | 99| | function arguments when calling Python functions manually from C++ code | 100| | (i.e. via handle::operator()). You probably won't need to use this. | 101+--------------------------------------------------+----------------------------------------------------------------------------+ 102 103Return value policies can also be applied to properties: 104 105.. code-block:: cpp 106 107 class_<MyClass>(m, "MyClass") 108 .def_property("data", &MyClass::getData, &MyClass::setData, 109 py::return_value_policy::copy); 110 111Technically, the code above applies the policy to both the getter and the 112setter function, however, the setter doesn't really care about *return* 113value policies which makes this a convenient terse syntax. Alternatively, 114targeted arguments can be passed through the :class:`cpp_function` constructor: 115 116.. code-block:: cpp 117 118 class_<MyClass>(m, "MyClass") 119 .def_property("data" 120 py::cpp_function(&MyClass::getData, py::return_value_policy::copy), 121 py::cpp_function(&MyClass::setData) 122 ); 123 124.. warning:: 125 126 Code with invalid return value policies might access unitialized memory or 127 free data structures multiple times, which can lead to hard-to-debug 128 non-determinism and segmentation faults, hence it is worth spending the 129 time to understand all the different options in the table above. 130 131.. note:: 132 133 One important aspect of the above policies is that they only apply to 134 instances which pybind11 has *not* seen before, in which case the policy 135 clarifies essential questions about the return value's lifetime and 136 ownership. When pybind11 knows the instance already (as identified by its 137 type and address in memory), it will return the existing Python object 138 wrapper rather than creating a new copy. 139 140.. note:: 141 142 The next section on :ref:`call_policies` discusses *call policies* that can be 143 specified *in addition* to a return value policy from the list above. Call 144 policies indicate reference relationships that can involve both return values 145 and parameters of functions. 146 147.. note:: 148 149 As an alternative to elaborate call policies and lifetime management logic, 150 consider using smart pointers (see the section on :ref:`smart_pointers` for 151 details). Smart pointers can tell whether an object is still referenced from 152 C++ or Python, which generally eliminates the kinds of inconsistencies that 153 can lead to crashes or undefined behavior. For functions returning smart 154 pointers, it is not necessary to specify a return value policy. 155 156.. _call_policies: 157 158Additional call policies 159======================== 160 161In addition to the above return value policies, further `call policies` can be 162specified to indicate dependencies between parameters. There is currently just 163one policy named ``keep_alive<Nurse, Patient>``, which indicates that the 164argument with index ``Patient`` should be kept alive at least until the 165argument with index ``Nurse`` is freed by the garbage collector. Argument 166indices start at one, while zero refers to the return value. For methods, index 167``1`` refers to the implicit ``this`` pointer, while regular arguments begin at 168index ``2``. Arbitrarily many call policies can be specified. When a ``Nurse`` 169with value ``None`` is detected at runtime, the call policy does nothing. 170 171This feature internally relies on the ability to create a *weak reference* to 172the nurse object, which is permitted by all classes exposed via pybind11. When 173the nurse object does not support weak references, an exception will be thrown. 174 175Consider the following example: here, the binding code for a list append 176operation ties the lifetime of the newly added element to the underlying 177container: 178 179.. code-block:: cpp 180 181 py::class_<List>(m, "List") 182 .def("append", &List::append, py::keep_alive<1, 2>()); 183 184.. note:: 185 186 ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse, 187 Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient == 188 0) policies from Boost.Python. 189 190.. seealso:: 191 192 The file :file:`tests/test_keep_alive.cpp` contains a complete example 193 that demonstrates using :class:`keep_alive` in more detail. 194 195.. _python_objects_as_args: 196 197Python objects as arguments 198=========================== 199 200pybind11 exposes all major Python types using thin C++ wrapper classes. These 201wrapper classes can also be used as parameters of functions in bindings, which 202makes it possible to directly work with native Python types on the C++ side. 203For instance, the following statement iterates over a Python ``dict``: 204 205.. code-block:: cpp 206 207 void print_dict(py::dict dict) { 208 /* Easily interact with Python types */ 209 for (auto item : dict) 210 std::cout << "key=" << item.first << ", " 211 << "value=" << item.second << std::endl; 212 } 213 214It can be exported: 215 216.. code-block:: cpp 217 218 m.def("print_dict", &print_dict); 219 220And used in Python as usual: 221 222.. code-block:: pycon 223 224 >>> print_dict({'foo': 123, 'bar': 'hello'}) 225 key=foo, value=123 226 key=bar, value=hello 227 228For more information on using Python objects in C++, see :doc:`/advanced/pycpp/index`. 229 230Accepting \*args and \*\*kwargs 231=============================== 232 233Python provides a useful mechanism to define functions that accept arbitrary 234numbers of arguments and keyword arguments: 235 236.. code-block:: python 237 238 def generic(*args, **kwargs): 239 ... # do something with args and kwargs 240 241Such functions can also be created using pybind11: 242 243.. code-block:: cpp 244 245 void generic(py::args args, py::kwargs kwargs) { 246 /// .. do something with args 247 if (kwargs) 248 /// .. do something with kwargs 249 } 250 251 /// Binding code 252 m.def("generic", &generic); 253 254The class ``py::args`` derives from ``py::tuple`` and ``py::kwargs`` derives 255from ``py::dict``. Note that the ``kwargs`` argument is invalid if no keyword 256arguments were actually provided. Please refer to the other examples for 257details on how to iterate over these, and on how to cast their entries into 258C++ objects. A demonstration is also available in 259``tests/test_kwargs_and_defaults.cpp``. 260 261.. warning:: 262 263 Unlike Python, pybind11 does not allow combining normal parameters with the 264 ``args`` / ``kwargs`` special parameters. 265 266Default arguments revisited 267=========================== 268 269The section on :ref:`default_args` previously discussed basic usage of default 270arguments using pybind11. One noteworthy aspect of their implementation is that 271default arguments are converted to Python objects right at declaration time. 272Consider the following example: 273 274.. code-block:: cpp 275 276 py::class_<MyClass>("MyClass") 277 .def("myFunction", py::arg("arg") = SomeType(123)); 278 279In this case, pybind11 must already be set up to deal with values of the type 280``SomeType`` (via a prior instantiation of ``py::class_<SomeType>``), or an 281exception will be thrown. 282 283Another aspect worth highlighting is that the "preview" of the default argument 284in the function signature is generated using the object's ``__repr__`` method. 285If not available, the signature may not be very helpful, e.g.: 286 287.. code-block:: pycon 288 289 FUNCTIONS 290 ... 291 | myFunction(...) 292 | Signature : (MyClass, arg : SomeType = <SomeType object at 0x101b7b080>) -> NoneType 293 ... 294 295The first way of addressing this is by defining ``SomeType.__repr__``. 296Alternatively, it is possible to specify the human-readable preview of the 297default argument manually using the ``arg_v`` notation: 298 299.. code-block:: cpp 300 301 py::class_<MyClass>("MyClass") 302 .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)")); 303 304Sometimes it may be necessary to pass a null pointer value as a default 305argument. In this case, remember to cast it to the underlying type in question, 306like so: 307 308.. code-block:: cpp 309 310 py::class_<MyClass>("MyClass") 311 .def("myFunction", py::arg("arg") = (SomeType *) nullptr); 312