strings.rst revision 14299
112037Sandreas.sandberg@arm.comStrings, bytes and Unicode conversions 212037Sandreas.sandberg@arm.com###################################### 312037Sandreas.sandberg@arm.com 412037Sandreas.sandberg@arm.com.. note:: 512037Sandreas.sandberg@arm.com 612391Sjason@lowepower.com This section discusses string handling in terms of Python 3 strings. For 712391Sjason@lowepower.com Python 2.7, replace all occurrences of ``str`` with ``unicode`` and 812391Sjason@lowepower.com ``bytes`` with ``str``. Python 2.7 users may find it best to use ``from 912391Sjason@lowepower.com __future__ import unicode_literals`` to avoid unintentionally using ``str`` 1012391Sjason@lowepower.com instead of ``unicode``. 1112037Sandreas.sandberg@arm.com 1212037Sandreas.sandberg@arm.comPassing Python strings to C++ 1312037Sandreas.sandberg@arm.com============================= 1412037Sandreas.sandberg@arm.com 1512391Sjason@lowepower.comWhen a Python ``str`` is passed from Python to a C++ function that accepts 1612391Sjason@lowepower.com``std::string`` or ``char *`` as arguments, pybind11 will encode the Python 1712391Sjason@lowepower.comstring to UTF-8. All Python ``str`` can be encoded in UTF-8, so this operation 1812391Sjason@lowepower.comdoes not fail. 1912037Sandreas.sandberg@arm.com 2012391Sjason@lowepower.comThe C++ language is encoding agnostic. It is the responsibility of the 2112391Sjason@lowepower.comprogrammer to track encodings. It's often easiest to simply `use UTF-8 2212391Sjason@lowepower.comeverywhere <http://utf8everywhere.org/>`_. 2312037Sandreas.sandberg@arm.com 2412037Sandreas.sandberg@arm.com.. code-block:: c++ 2512037Sandreas.sandberg@arm.com 2612037Sandreas.sandberg@arm.com m.def("utf8_test", 2712037Sandreas.sandberg@arm.com [](const std::string &s) { 2812037Sandreas.sandberg@arm.com cout << "utf-8 is icing on the cake.\n"; 2912037Sandreas.sandberg@arm.com cout << s; 3012037Sandreas.sandberg@arm.com } 3112037Sandreas.sandberg@arm.com ); 3212037Sandreas.sandberg@arm.com m.def("utf8_charptr", 3312037Sandreas.sandberg@arm.com [](const char *s) { 3412037Sandreas.sandberg@arm.com cout << "My favorite food is\n"; 3512037Sandreas.sandberg@arm.com cout << s; 3612037Sandreas.sandberg@arm.com } 3712037Sandreas.sandberg@arm.com ); 3812037Sandreas.sandberg@arm.com 3912037Sandreas.sandberg@arm.com.. code-block:: python 4012037Sandreas.sandberg@arm.com 4112037Sandreas.sandberg@arm.com >>> utf8_test('') 4212037Sandreas.sandberg@arm.com utf-8 is icing on the cake. 4312037Sandreas.sandberg@arm.com 4412037Sandreas.sandberg@arm.com 4512037Sandreas.sandberg@arm.com >>> utf8_charptr('') 4612037Sandreas.sandberg@arm.com My favorite food is 4712037Sandreas.sandberg@arm.com 4812037Sandreas.sandberg@arm.com 4912037Sandreas.sandberg@arm.com.. note:: 5012037Sandreas.sandberg@arm.com 5112391Sjason@lowepower.com Some terminal emulators do not support UTF-8 or emoji fonts and may not 5212391Sjason@lowepower.com display the example above correctly. 5312037Sandreas.sandberg@arm.com 5412391Sjason@lowepower.comThe results are the same whether the C++ function accepts arguments by value or 5512391Sjason@lowepower.comreference, and whether or not ``const`` is used. 5612037Sandreas.sandberg@arm.com 5712037Sandreas.sandberg@arm.comPassing bytes to C++ 5812037Sandreas.sandberg@arm.com-------------------- 5912037Sandreas.sandberg@arm.com 6012391Sjason@lowepower.comA Python ``bytes`` object will be passed to C++ functions that accept 6114299Sbbruce@ucdavis.edu``std::string`` or ``char*`` *without* conversion. On Python 3, in order to 6214299Sbbruce@ucdavis.edumake a function *only* accept ``bytes`` (and not ``str``), declare it as taking 6314299Sbbruce@ucdavis.edua ``py::bytes`` argument. 6412037Sandreas.sandberg@arm.com 6512037Sandreas.sandberg@arm.com 6612037Sandreas.sandberg@arm.comReturning C++ strings to Python 6712037Sandreas.sandberg@arm.com=============================== 6812037Sandreas.sandberg@arm.com 6912391Sjason@lowepower.comWhen a C++ function returns a ``std::string`` or ``char*`` to a Python caller, 7012391Sjason@lowepower.com**pybind11 will assume that the string is valid UTF-8** and will decode it to a 7112391Sjason@lowepower.comnative Python ``str``, using the same API as Python uses to perform 7212391Sjason@lowepower.com``bytes.decode('utf-8')``. If this implicit conversion fails, pybind11 will 7312391Sjason@lowepower.comraise a ``UnicodeDecodeError``. 7412037Sandreas.sandberg@arm.com 7512037Sandreas.sandberg@arm.com.. code-block:: c++ 7612037Sandreas.sandberg@arm.com 7712037Sandreas.sandberg@arm.com m.def("std_string_return", 7812037Sandreas.sandberg@arm.com []() { 7912037Sandreas.sandberg@arm.com return std::string("This string needs to be UTF-8 encoded"); 8012037Sandreas.sandberg@arm.com } 8112037Sandreas.sandberg@arm.com ); 8212037Sandreas.sandberg@arm.com 8312037Sandreas.sandberg@arm.com.. code-block:: python 8412037Sandreas.sandberg@arm.com 8512037Sandreas.sandberg@arm.com >>> isinstance(example.std_string_return(), str) 8612037Sandreas.sandberg@arm.com True 8712037Sandreas.sandberg@arm.com 8812037Sandreas.sandberg@arm.com 8912391Sjason@lowepower.comBecause UTF-8 is inclusive of pure ASCII, there is never any issue with 9012391Sjason@lowepower.comreturning a pure ASCII string to Python. If there is any possibility that the 9112391Sjason@lowepower.comstring is not pure ASCII, it is necessary to ensure the encoding is valid 9212391Sjason@lowepower.comUTF-8. 9312037Sandreas.sandberg@arm.com 9412037Sandreas.sandberg@arm.com.. warning:: 9512037Sandreas.sandberg@arm.com 9612391Sjason@lowepower.com Implicit conversion assumes that a returned ``char *`` is null-terminated. 9712391Sjason@lowepower.com If there is no null terminator a buffer overrun will occur. 9812037Sandreas.sandberg@arm.com 9912037Sandreas.sandberg@arm.comExplicit conversions 10012037Sandreas.sandberg@arm.com-------------------- 10112037Sandreas.sandberg@arm.com 10212391Sjason@lowepower.comIf some C++ code constructs a ``std::string`` that is not a UTF-8 string, one 10312391Sjason@lowepower.comcan perform a explicit conversion and return a ``py::str`` object. Explicit 10412391Sjason@lowepower.comconversion has the same overhead as implicit conversion. 10512037Sandreas.sandberg@arm.com 10612037Sandreas.sandberg@arm.com.. code-block:: c++ 10712037Sandreas.sandberg@arm.com 10812037Sandreas.sandberg@arm.com // This uses the Python C API to convert Latin-1 to Unicode 10912037Sandreas.sandberg@arm.com m.def("str_output", 11012037Sandreas.sandberg@arm.com []() { 11112037Sandreas.sandberg@arm.com std::string s = "Send your r\xe9sum\xe9 to Alice in HR"; // Latin-1 11212037Sandreas.sandberg@arm.com py::str py_s = PyUnicode_DecodeLatin1(s.data(), s.length()); 11312037Sandreas.sandberg@arm.com return py_s; 11412037Sandreas.sandberg@arm.com } 11512037Sandreas.sandberg@arm.com ); 11612037Sandreas.sandberg@arm.com 11712037Sandreas.sandberg@arm.com.. code-block:: python 11812037Sandreas.sandberg@arm.com 11912037Sandreas.sandberg@arm.com >>> str_output() 12012037Sandreas.sandberg@arm.com 'Send your résumé to Alice in HR' 12112037Sandreas.sandberg@arm.com 12212391Sjason@lowepower.comThe `Python C API 12312391Sjason@lowepower.com<https://docs.python.org/3/c-api/unicode.html#built-in-codecs>`_ provides 12412391Sjason@lowepower.comseveral built-in codecs. 12512037Sandreas.sandberg@arm.com 12612037Sandreas.sandberg@arm.com 12712391Sjason@lowepower.comOne could also use a third party encoding library such as libiconv to transcode 12812391Sjason@lowepower.comto UTF-8. 12912037Sandreas.sandberg@arm.com 13012037Sandreas.sandberg@arm.comReturn C++ strings without conversion 13112037Sandreas.sandberg@arm.com------------------------------------- 13212037Sandreas.sandberg@arm.com 13312391Sjason@lowepower.comIf the data in a C++ ``std::string`` does not represent text and should be 13412391Sjason@lowepower.comreturned to Python as ``bytes``, then one can return the data as a 13512391Sjason@lowepower.com``py::bytes`` object. 13612037Sandreas.sandberg@arm.com 13712037Sandreas.sandberg@arm.com.. code-block:: c++ 13812037Sandreas.sandberg@arm.com 13912037Sandreas.sandberg@arm.com m.def("return_bytes", 14012037Sandreas.sandberg@arm.com []() { 14112037Sandreas.sandberg@arm.com std::string s("\xba\xd0\xba\xd0"); // Not valid UTF-8 14212037Sandreas.sandberg@arm.com return py::bytes(s); // Return the data without transcoding 14312037Sandreas.sandberg@arm.com } 14412037Sandreas.sandberg@arm.com ); 14512037Sandreas.sandberg@arm.com 14612037Sandreas.sandberg@arm.com.. code-block:: python 14712037Sandreas.sandberg@arm.com 14812037Sandreas.sandberg@arm.com >>> example.return_bytes() 14912037Sandreas.sandberg@arm.com b'\xba\xd0\xba\xd0' 15012037Sandreas.sandberg@arm.com 15112037Sandreas.sandberg@arm.com 15212391Sjason@lowepower.comNote the asymmetry: pybind11 will convert ``bytes`` to ``std::string`` without 15312391Sjason@lowepower.comencoding, but cannot convert ``std::string`` back to ``bytes`` implicitly. 15412037Sandreas.sandberg@arm.com 15512037Sandreas.sandberg@arm.com.. code-block:: c++ 15612037Sandreas.sandberg@arm.com 15712037Sandreas.sandberg@arm.com m.def("asymmetry", 15812037Sandreas.sandberg@arm.com [](std::string s) { // Accepts str or bytes from Python 15912037Sandreas.sandberg@arm.com return s; // Looks harmless, but implicitly converts to str 16012037Sandreas.sandberg@arm.com } 16112391Sjason@lowepower.com ); 16212037Sandreas.sandberg@arm.com 16312037Sandreas.sandberg@arm.com.. code-block:: python 16412037Sandreas.sandberg@arm.com 16512037Sandreas.sandberg@arm.com >>> isinstance(example.asymmetry(b"have some bytes"), str) 16612037Sandreas.sandberg@arm.com True 16712037Sandreas.sandberg@arm.com 16812037Sandreas.sandberg@arm.com >>> example.asymmetry(b"\xba\xd0\xba\xd0") # invalid utf-8 as bytes 16912037Sandreas.sandberg@arm.com UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte 17012037Sandreas.sandberg@arm.com 17112037Sandreas.sandberg@arm.com 17212037Sandreas.sandberg@arm.comWide character strings 17312037Sandreas.sandberg@arm.com====================== 17412037Sandreas.sandberg@arm.com 17512391Sjason@lowepower.comWhen a Python ``str`` is passed to a C++ function expecting ``std::wstring``, 17612391Sjason@lowepower.com``wchar_t*``, ``std::u16string`` or ``std::u32string``, the ``str`` will be 17712391Sjason@lowepower.comencoded to UTF-16 or UTF-32 depending on how the C++ compiler implements each 17812391Sjason@lowepower.comtype, in the platform's native endianness. When strings of these types are 17912391Sjason@lowepower.comreturned, they are assumed to contain valid UTF-16 or UTF-32, and will be 18012391Sjason@lowepower.comdecoded to Python ``str``. 18112037Sandreas.sandberg@arm.com 18212037Sandreas.sandberg@arm.com.. code-block:: c++ 18312037Sandreas.sandberg@arm.com 18412037Sandreas.sandberg@arm.com #define UNICODE 18512037Sandreas.sandberg@arm.com #include <windows.h> 18612037Sandreas.sandberg@arm.com 18712037Sandreas.sandberg@arm.com m.def("set_window_text", 18812037Sandreas.sandberg@arm.com [](HWND hwnd, std::wstring s) { 18912037Sandreas.sandberg@arm.com // Call SetWindowText with null-terminated UTF-16 string 19012037Sandreas.sandberg@arm.com ::SetWindowText(hwnd, s.c_str()); 19112037Sandreas.sandberg@arm.com } 19212037Sandreas.sandberg@arm.com ); 19312037Sandreas.sandberg@arm.com m.def("get_window_text", 19412037Sandreas.sandberg@arm.com [](HWND hwnd) { 19512037Sandreas.sandberg@arm.com const int buffer_size = ::GetWindowTextLength(hwnd) + 1; 19612037Sandreas.sandberg@arm.com auto buffer = std::make_unique< wchar_t[] >(buffer_size); 19712037Sandreas.sandberg@arm.com 19812037Sandreas.sandberg@arm.com ::GetWindowText(hwnd, buffer.data(), buffer_size); 19912037Sandreas.sandberg@arm.com 20012037Sandreas.sandberg@arm.com std::wstring text(buffer.get()); 20112037Sandreas.sandberg@arm.com 20212037Sandreas.sandberg@arm.com // wstring will be converted to Python str 20312037Sandreas.sandberg@arm.com return text; 20412037Sandreas.sandberg@arm.com } 20512037Sandreas.sandberg@arm.com ); 20612037Sandreas.sandberg@arm.com 20712037Sandreas.sandberg@arm.com.. warning:: 20812037Sandreas.sandberg@arm.com 20912391Sjason@lowepower.com Wide character strings may not work as described on Python 2.7 or Python 21012391Sjason@lowepower.com 3.3 compiled with ``--enable-unicode=ucs2``. 21112037Sandreas.sandberg@arm.com 21212391Sjason@lowepower.comStrings in multibyte encodings such as Shift-JIS must transcoded to a 21312391Sjason@lowepower.comUTF-8/16/32 before being returned to Python. 21412037Sandreas.sandberg@arm.com 21512037Sandreas.sandberg@arm.com 21612037Sandreas.sandberg@arm.comCharacter literals 21712037Sandreas.sandberg@arm.com================== 21812037Sandreas.sandberg@arm.com 21912391Sjason@lowepower.comC++ functions that accept character literals as input will receive the first 22012391Sjason@lowepower.comcharacter of a Python ``str`` as their input. If the string is longer than one 22112391Sjason@lowepower.comUnicode character, trailing characters will be ignored. 22212037Sandreas.sandberg@arm.com 22312391Sjason@lowepower.comWhen a character literal is returned from C++ (such as a ``char`` or a 22412391Sjason@lowepower.com``wchar_t``), it will be converted to a ``str`` that represents the single 22512391Sjason@lowepower.comcharacter. 22612037Sandreas.sandberg@arm.com 22712037Sandreas.sandberg@arm.com.. code-block:: c++ 22812037Sandreas.sandberg@arm.com 22912037Sandreas.sandberg@arm.com m.def("pass_char", [](char c) { return c; }); 23012037Sandreas.sandberg@arm.com m.def("pass_wchar", [](wchar_t w) { return w; }); 23112037Sandreas.sandberg@arm.com 23212037Sandreas.sandberg@arm.com.. code-block:: python 23312391Sjason@lowepower.com 23412037Sandreas.sandberg@arm.com >>> example.pass_char('A') 23512037Sandreas.sandberg@arm.com 'A' 23612037Sandreas.sandberg@arm.com 23712391Sjason@lowepower.comWhile C++ will cast integers to character types (``char c = 0x65;``), pybind11 23812391Sjason@lowepower.comdoes not convert Python integers to characters implicitly. The Python function 23912391Sjason@lowepower.com``chr()`` can be used to convert integers to characters. 24012037Sandreas.sandberg@arm.com 24112037Sandreas.sandberg@arm.com.. code-block:: python 24212391Sjason@lowepower.com 24312037Sandreas.sandberg@arm.com >>> example.pass_char(0x65) 24412037Sandreas.sandberg@arm.com TypeError 24512037Sandreas.sandberg@arm.com 24612037Sandreas.sandberg@arm.com >>> example.pass_char(chr(0x65)) 24712037Sandreas.sandberg@arm.com 'A' 24812037Sandreas.sandberg@arm.com 24912391Sjason@lowepower.comIf the desire is to work with an 8-bit integer, use ``int8_t`` or ``uint8_t`` 25012391Sjason@lowepower.comas the argument type. 25112037Sandreas.sandberg@arm.com 25212037Sandreas.sandberg@arm.comGrapheme clusters 25312037Sandreas.sandberg@arm.com----------------- 25412037Sandreas.sandberg@arm.com 25512391Sjason@lowepower.comA single grapheme may be represented by two or more Unicode characters. For 25612391Sjason@lowepower.comexample 'é' is usually represented as U+00E9 but can also be expressed as the 25712391Sjason@lowepower.comcombining character sequence U+0065 U+0301 (that is, the letter 'e' followed by 25812391Sjason@lowepower.coma combining acute accent). The combining character will be lost if the 25912391Sjason@lowepower.comtwo-character sequence is passed as an argument, even though it renders as a 26012391Sjason@lowepower.comsingle grapheme. 26112037Sandreas.sandberg@arm.com 26212037Sandreas.sandberg@arm.com.. code-block:: python 26312037Sandreas.sandberg@arm.com 26412037Sandreas.sandberg@arm.com >>> example.pass_wchar('é') 26512037Sandreas.sandberg@arm.com 'é' 26612037Sandreas.sandberg@arm.com 26712037Sandreas.sandberg@arm.com >>> combining_e_acute = 'e' + '\u0301' 26812037Sandreas.sandberg@arm.com 26912037Sandreas.sandberg@arm.com >>> combining_e_acute 27012037Sandreas.sandberg@arm.com 'é' 27112037Sandreas.sandberg@arm.com 27212037Sandreas.sandberg@arm.com >>> combining_e_acute == 'é' 27312037Sandreas.sandberg@arm.com False 27412037Sandreas.sandberg@arm.com 27512037Sandreas.sandberg@arm.com >>> example.pass_wchar(combining_e_acute) 27612037Sandreas.sandberg@arm.com 'e' 27712037Sandreas.sandberg@arm.com 27812391Sjason@lowepower.comNormalizing combining characters before passing the character literal to C++ 27912391Sjason@lowepower.commay resolve *some* of these issues: 28012037Sandreas.sandberg@arm.com 28112037Sandreas.sandberg@arm.com.. code-block:: python 28212037Sandreas.sandberg@arm.com 28312037Sandreas.sandberg@arm.com >>> example.pass_wchar(unicodedata.normalize('NFC', combining_e_acute)) 28412037Sandreas.sandberg@arm.com 'é' 28512037Sandreas.sandberg@arm.com 28612391Sjason@lowepower.comIn some languages (Thai for example), there are `graphemes that cannot be 28712391Sjason@lowepower.comexpressed as a single Unicode code point 28812391Sjason@lowepower.com<http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries>`_, so there is 28912391Sjason@lowepower.comno way to capture them in a C++ character type. 29012037Sandreas.sandberg@arm.com 29112037Sandreas.sandberg@arm.com 29212391Sjason@lowepower.comC++17 string views 29312391Sjason@lowepower.com================== 29412391Sjason@lowepower.com 29512391Sjason@lowepower.comC++17 string views are automatically supported when compiling in C++17 mode. 29612391Sjason@lowepower.comThey follow the same rules for encoding and decoding as the corresponding STL 29712391Sjason@lowepower.comstring type (for example, a ``std::u16string_view`` argument will be passed 29812391Sjason@lowepower.comUTF-16-encoded data, and a returned ``std::string_view`` will be decoded as 29912391Sjason@lowepower.comUTF-8). 30012391Sjason@lowepower.com 30112037Sandreas.sandberg@arm.comReferences 30212037Sandreas.sandberg@arm.com========== 30312037Sandreas.sandberg@arm.com 30412037Sandreas.sandberg@arm.com* `The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) <https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/>`_ 30512391Sjason@lowepower.com* `C++ - Using STL Strings at Win32 API Boundaries <https://msdn.microsoft.com/en-ca/magazine/mt238407.aspx>`_ 306