functions.rst revision 12037:d28054ac6ec9
16791SN/AFunctions 26791SN/A######### 36791SN/A 46791SN/ABefore proceeding with this section, make sure that you are already familiar 56791SN/Awith the basics of binding functions and classes, as explained in :doc:`/basics` 66791SN/Aand :doc:`/classes`. The following guide is applicable to both free and member 76791SN/Afunctions, i.e. *methods* in Python. 86791SN/A 96791SN/A.. _return_value_policies: 106791SN/A 116791SN/AReturn value policies 126791SN/A===================== 136791SN/A 146791SN/APython and C++ use fundamentally different ways of managing the memory and 156791SN/Alifetime of objects managed by them. This can lead to issues when creating 166791SN/Abindings for functions that return a non-trivial type. Just by looking at the 176791SN/Atype information, it is not clear whether Python should take charge of the 186791SN/Areturned value and eventually free its resources, or if this is handled on the 196791SN/AC++ side. For this reason, pybind11 provides a several *return value policy* 206791SN/Aannotations that can be passed to the :func:`module::def` and 216791SN/A:func:`class_::def` functions. The default policy is 226791SN/A:enum:`return_value_policy::automatic`. 236791SN/A 246791SN/AReturn value policies are tricky, and it's very important to get them right. 256791SN/AJust to illustrate what can go wrong, consider the following simple example: 266791SN/A 276791SN/A.. code-block:: cpp 286791SN/A 2910301Snilay@cs.wisc.edu /* Function declaration */ 306791SN/A Data *get_data() { return _data; /* (pointer to a static data structure) */ } 317055SN/A ... 327055SN/A 336791SN/A /* Binding code */ 347039SN/A m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python 357039SN/A 367039SN/AWhat's going on here? When ``get_data()`` is called from Python, the return 377039SN/Avalue (a native C++ type) must be wrapped to turn it into a usable Python type. 387039SN/AIn this case, the default return value policy (:enum:`return_value_policy::automatic`) 397039SN/Acauses pybind11 to assume ownership of the static ``_data`` instance. 406791SN/A 416797SN/AWhen Python's garbage collector eventually deletes the Python 426791SN/Awrapper, pybind11 will also attempt to delete the C++ instance (via ``operator 436791SN/Adelete()``) due to the implied ownership. At this point, the entire application 446791SN/Awill come crashing down, though errors could also be more subtle and involve 456791SN/Asilent data corruption. 466791SN/A 477054SN/AIn the above example, the policy :enum:`return_value_policy::reference` should have 486791SN/Abeen specified so that the global data instance is only *referenced* without any 497039SN/Aimplied transfer of ownership, i.e.: 507039SN/A 517039SN/A.. code-block:: cpp 527039SN/A 536791SN/A m.def("get_data", &get_data, return_value_policy::reference); 547039SN/A 557039SN/AOn the other hand, this is not the right policy for many other situations, 567039SN/Awhere ignoring ownership could lead to resource leaks. 577039SN/AAs a developer using pybind11, it's important to be familiar with the different 586791SN/Areturn value policies, including which situation calls for which one of them. 597039SN/AThe following table provides an overview of available policies: 607039SN/A 616791SN/A.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| 627039SN/A 637039SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 647455SN/A| Return value policy | Description | 657455SN/A+==================================================+============================================================================+ 667455SN/A| :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | 677455SN/A| | ownership. Python will call the destructor and delete operator when the | 687455SN/A| | object's reference count reaches zero. Undefined behavior ensues when the | 697455SN/A| | C++ side does the same, or when the data was not dynamically allocated. | 707455SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 717455SN/A| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | 727039SN/A| | This policy is comparably safe because the lifetimes of the two instances | 737039SN/A| | are decoupled. | 747455SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 757039SN/A| :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance | 767455SN/A| | that will be owned by Python. This policy is comparably safe because the | 777455SN/A| | lifetimes of the two instances (move source and destination) are decoupled.| 787455SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 797455SN/A| :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is | 807455SN/A| | responsible for managing the object's lifetime and deallocating it when | 817039SN/A| | it is no longer used. Warning: undefined behavior will ensue when the C++ | 827039SN/A| | side deletes an object that is still referenced and used by Python. | 837039SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 847039SN/A| :enum:`return_value_policy::reference_internal` | Indicates that the lifetime of the return value is tied to the lifetime | 857039SN/A| | of a parent object, namely the implicit ``this``, or ``self`` argument of | 867039SN/A| | the called method or property. Internally, this policy works just like | 877039SN/A| | :enum:`return_value_policy::reference` but additionally applies a | 887039SN/A| | ``keep_alive<0, 1>`` *call policy* (described in the next section) that | 897039SN/A| | prevents the parent object from being garbage collected as long as the | 907039SN/A| | return value is referenced by Python. This is the default policy for | 917039SN/A| | property getters created via ``def_property``, ``def_readwrite``, etc. | 927039SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 937039SN/A| :enum:`return_value_policy::automatic` | **Default policy.** This policy falls back to the policy | 947039SN/A| | :enum:`return_value_policy::take_ownership` when the return value is a | 957039SN/A| | pointer. Otherwise, it uses :enum:`return_value_policy::move` or | 967039SN/A| | :enum:`return_value_policy::copy` for rvalue and lvalue references, | 977455SN/A| | respectively. See above for a description of what all of these different | 987455SN/A| | policies do. | 996797SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 1006797SN/A| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | 1017039SN/A| | return value is a pointer. This is the default conversion policy for | 1026797SN/A| | function arguments when calling Python functions manually from C++ code | 1037039SN/A| | (i.e. via handle::operator()). You probably won't need to use this. | 1047039SN/A+--------------------------------------------------+----------------------------------------------------------------------------+ 1057039SN/A 1067039SN/AReturn value policies can also be applied to properties: 1077039SN/A 1087039SN/A.. code-block:: cpp 1096791SN/A 1107039SN/A class_<MyClass>(m, "MyClass") 1117039SN/A .def_property("data", &MyClass::getData, &MyClass::setData, 1127039SN/A py::return_value_policy::copy); 1137455SN/A 1146791SN/ATechnically, the code above applies the policy to both the getter and the 1156791SN/Asetter function, however, the setter doesn't really care about *return* 1166791SN/Avalue policies which makes this a convenient terse syntax. Alternatively, 1177039SN/Atargeted arguments can be passed through the :class:`cpp_function` constructor: 1187039SN/A 1197039SN/A.. code-block:: cpp 1206791SN/A 1217039SN/A class_<MyClass>(m, "MyClass") 1227455SN/A .def_property("data" 1237455SN/A py::cpp_function(&MyClass::getData, py::return_value_policy::copy), 1247455SN/A py::cpp_function(&MyClass::setData) 1257039SN/A ); 1267039SN/A 1277455SN/A.. warning:: 1287455SN/A 1297455SN/A Code with invalid return value policies might access unitialized memory or 1307455SN/A free data structures multiple times, which can lead to hard-to-debug 1317455SN/A non-determinism and segmentation faults, hence it is worth spending the 1327039SN/A time to understand all the different options in the table above. 1337039SN/A 1347039SN/A.. note:: 1357039SN/A 1367455SN/A One important aspect of the above policies is that they only apply to 1377455SN/A instances which pybind11 has *not* seen before, in which case the policy 1386791SN/A clarifies essential questions about the return value's lifetime and 1396791SN/A ownership. When pybind11 knows the instance already (as identified by its 1407039SN/A type and address in memory), it will return the existing Python object 1417039SN/A wrapper rather than creating a new copy. 1426791SN/A 1437039SN/A.. note:: 1447455SN/A 1457455SN/A The next section on :ref:`call_policies` discusses *call policies* that can be 1467455SN/A specified *in addition* to a return value policy from the list above. Call 1477039SN/A policies indicate reference relationships that can involve both return values 1486791SN/A and parameters of functions. 1496791SN/A 1507039SN/A.. note:: 1517039SN/A 1526791SN/A As an alternative to elaborate call policies and lifetime management logic, 1537039SN/A consider using smart pointers (see the section on :ref:`smart_pointers` for 1547455SN/A details). Smart pointers can tell whether an object is still referenced from 1557455SN/A C++ or Python, which generally eliminates the kinds of inconsistencies that 1567455SN/A can lead to crashes or undefined behavior. For functions returning smart 1577039SN/A pointers, it is not necessary to specify a return value policy. 1587039SN/A 1597039SN/A.. _call_policies: 1607039SN/A 1617039SN/AAdditional call policies 1627039SN/A======================== 1636791SN/A 1646791SN/AIn addition to the above return value policies, further *call policies* can be 1657039SN/Aspecified to indicate dependencies between parameters. In general, call policies 1667039SN/Aare required when the C++ object is any kind of container and another object is being 1676791SN/Aadded to the container. 1687039SN/A 1697455SN/AThere is currently just 1707455SN/Aone policy named ``keep_alive<Nurse, Patient>``, which indicates that the 1717455SN/Aargument with index ``Patient`` should be kept alive at least until the 1727039SN/Aargument with index ``Nurse`` is freed by the garbage collector. Argument 1737455SN/Aindices start at one, while zero refers to the return value. For methods, index 1747039SN/A``1`` refers to the implicit ``this`` pointer, while regular arguments begin at 1757455SN/Aindex ``2``. Arbitrarily many call policies can be specified. When a ``Nurse`` 1767455SN/Awith value ``None`` is detected at runtime, the call policy does nothing. 1777455SN/A 1787455SN/AThis feature internally relies on the ability to create a *weak reference* to 1797455SN/Athe nurse object, which is permitted by all classes exposed via pybind11. When 1806791SN/Athe nurse object does not support weak references, an exception will be thrown. 1816791SN/A 1827039SN/AConsider the following example: here, the binding code for a list append 1837039SN/Aoperation ties the lifetime of the newly added element to the underlying 1846791SN/Acontainer: 1857039SN/A 1866797SN/A.. code-block:: cpp 1877039SN/A 1887455SN/A py::class_<List>(m, "List") 1896791SN/A .def("append", &List::append, py::keep_alive<1, 2>()); 1906791SN/A 1917039SN/A.. note:: 1927039SN/A 1936791SN/A ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse, 1947455SN/A Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient == 1957455SN/A 0) policies from Boost.Python. 1967455SN/A 1977039SN/A.. seealso:: 1987455SN/A 1997455SN/A The file :file:`tests/test_keep_alive.cpp` contains a complete example 2007455SN/A that demonstrates using :class:`keep_alive` in more detail. 2016791SN/A 2026791SN/A.. _python_objects_as_args: 2037039SN/A 2047039SN/APython objects as arguments 2056791SN/A=========================== 2067455SN/A 2077455SN/Apybind11 exposes all major Python types using thin C++ wrapper classes. These 2087455SN/Awrapper classes can also be used as parameters of functions in bindings, which 2097039SN/Amakes it possible to directly work with native Python types on the C++ side. 2107455SN/AFor instance, the following statement iterates over a Python ``dict``: 2117455SN/A 2127455SN/A.. code-block:: cpp 2136791SN/A 2146791SN/A void print_dict(py::dict dict) { 2157039SN/A /* Easily interact with Python types */ 2167039SN/A for (auto item : dict) 2176797SN/A std::cout << "key=" << std::string(py::str(item.first)) << ", " 2186797SN/A << "value=" << std::string(py::str(item.second)) << std::endl; 2196791SN/A } 220 221It can be exported: 222 223.. code-block:: cpp 224 225 m.def("print_dict", &print_dict); 226 227And used in Python as usual: 228 229.. code-block:: pycon 230 231 >>> print_dict({'foo': 123, 'bar': 'hello'}) 232 key=foo, value=123 233 key=bar, value=hello 234 235For more information on using Python objects in C++, see :doc:`/advanced/pycpp/index`. 236 237Accepting \*args and \*\*kwargs 238=============================== 239 240Python provides a useful mechanism to define functions that accept arbitrary 241numbers of arguments and keyword arguments: 242 243.. code-block:: python 244 245 def generic(*args, **kwargs): 246 ... # do something with args and kwargs 247 248Such functions can also be created using pybind11: 249 250.. code-block:: cpp 251 252 void generic(py::args args, py::kwargs kwargs) { 253 /// .. do something with args 254 if (kwargs) 255 /// .. do something with kwargs 256 } 257 258 /// Binding code 259 m.def("generic", &generic); 260 261The class ``py::args`` derives from ``py::tuple`` and ``py::kwargs`` derives 262from ``py::dict``. 263 264You may also use just one or the other, and may combine these with other 265arguments as long as the ``py::args`` and ``py::kwargs`` arguments are the last 266arguments accepted by the function. 267 268Please refer to the other examples for details on how to iterate over these, 269and on how to cast their entries into C++ objects. A demonstration is also 270available in ``tests/test_kwargs_and_defaults.cpp``. 271 272.. note:: 273 274 When combining \*args or \*\*kwargs with :ref:`keyword_args` you should 275 *not* include ``py::arg`` tags for the ``py::args`` and ``py::kwargs`` 276 arguments. 277 278Default arguments revisited 279=========================== 280 281The section on :ref:`default_args` previously discussed basic usage of default 282arguments using pybind11. One noteworthy aspect of their implementation is that 283default arguments are converted to Python objects right at declaration time. 284Consider the following example: 285 286.. code-block:: cpp 287 288 py::class_<MyClass>("MyClass") 289 .def("myFunction", py::arg("arg") = SomeType(123)); 290 291In this case, pybind11 must already be set up to deal with values of the type 292``SomeType`` (via a prior instantiation of ``py::class_<SomeType>``), or an 293exception will be thrown. 294 295Another aspect worth highlighting is that the "preview" of the default argument 296in the function signature is generated using the object's ``__repr__`` method. 297If not available, the signature may not be very helpful, e.g.: 298 299.. code-block:: pycon 300 301 FUNCTIONS 302 ... 303 | myFunction(...) 304 | Signature : (MyClass, arg : SomeType = <SomeType object at 0x101b7b080>) -> NoneType 305 ... 306 307The first way of addressing this is by defining ``SomeType.__repr__``. 308Alternatively, it is possible to specify the human-readable preview of the 309default argument manually using the ``arg_v`` notation: 310 311.. code-block:: cpp 312 313 py::class_<MyClass>("MyClass") 314 .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)")); 315 316Sometimes it may be necessary to pass a null pointer value as a default 317argument. In this case, remember to cast it to the underlying type in question, 318like so: 319 320.. code-block:: cpp 321 322 py::class_<MyClass>("MyClass") 323 .def("myFunction", py::arg("arg") = (SomeType *) nullptr); 324 325.. _nonconverting_arguments: 326 327Non-converting arguments 328======================== 329 330Certain argument types may support conversion from one type to another. Some 331examples of conversions are: 332 333* :ref:`implicit_conversions` declared using ``py::implicitly_convertible<A,B>()`` 334* Calling a method accepting a double with an integer argument 335* Calling a ``std::complex<float>`` argument with a non-complex python type 336 (for example, with a float). (Requires the optional ``pybind11/complex.h`` 337 header). 338* Calling a function taking an Eigen matrix reference with a numpy array of the 339 wrong type or of an incompatible data layout. (Requires the optional 340 ``pybind11/eigen.h`` header). 341 342This behaviour is sometimes undesirable: the binding code may prefer to raise 343an error rather than convert the argument. This behaviour can be obtained 344through ``py::arg`` by calling the ``.noconvert()`` method of the ``py::arg`` 345object, such as: 346 347.. code-block:: cpp 348 349 m.def("floats_only", [](double f) { return 0.5 * f; }, py::arg("f").noconvert()); 350 m.def("floats_preferred", [](double f) { return 0.5 * f; }, py::arg("f")); 351 352Attempting the call the second function (the one without ``.noconvert()``) with 353an integer will succeed, but attempting to call the ``.noconvert()`` version 354will fail with a ``TypeError``: 355 356.. code-block:: pycon 357 358 >>> floats_preferred(4) 359 2.0 360 >>> floats_only(4) 361 Traceback (most recent call last): 362 File "<stdin>", line 1, in <module> 363 TypeError: floats_only(): incompatible function arguments. The following argument types are supported: 364 1. (f: float) -> float 365 366 Invoked with: 4 367 368You may, of course, combine this with the :var:`_a` shorthand notation (see 369:ref:`keyword_args`) and/or :ref:`default_args`. It is also permitted to omit 370the argument name by using the ``py::arg()`` constructor without an argument 371name, i.e. by specifying ``py::arg().noconvert()``. 372 373.. note:: 374 375 When specifying ``py::arg`` options it is necessary to provide the same 376 number of options as the bound function has arguments. Thus if you want to 377 enable no-convert behaviour for just one of several arguments, you will 378 need to specify a ``py::arg()`` annotation for each argument with the 379 no-convert argument modified to ``py::arg().noconvert()``. 380 381Overload resolution order 382========================= 383 384When a function or method with multiple overloads is called from Python, 385pybind11 determines which overload to call in two passes. The first pass 386attempts to call each overload without allowing argument conversion (as if 387every argument had been specified as ``py::arg().noconvert()`` as decribed 388above). 389 390If no overload succeeds in the no-conversion first pass, a second pass is 391attempted in which argument conversion is allowed (except where prohibited via 392an explicit ``py::arg().noconvert()`` attribute in the function definition). 393 394If the second pass also fails a ``TypeError`` is raised. 395 396Within each pass, overloads are tried in the order they were registered with 397pybind11. 398 399What this means in practice is that pybind11 will prefer any overload that does 400not require conversion of arguments to an overload that does, but otherwise prefers 401earlier-defined overloads to later-defined ones. 402 403.. note:: 404 405 pybind11 does *not* further prioritize based on the number/pattern of 406 overloaded arguments. That is, pybind11 does not prioritize a function 407 requiring one conversion over one requiring three, but only prioritizes 408 overloads requiring no conversion at all to overloads that require 409 conversion of at least one argument. 410