strings.rst revision 12037
1Strings, bytes and Unicode conversions 2###################################### 3 4.. note:: 5 6 This section discusses string handling in terms of Python 3 strings. For Python 2.7, replace all occurrences of ``str`` with ``unicode`` and ``bytes`` with ``str``. Python 2.7 users may find it best to use ``from __future__ import unicode_literals`` to avoid unintentionally using ``str`` instead of ``unicode``. 7 8Passing Python strings to C++ 9============================= 10 11When a Python ``str`` is passed from Python to a C++ function that accepts ``std::string`` or ``char *`` as arguments, pybind11 will encode the Python string to UTF-8. All Python ``str`` can be encoded in UTF-8, so this operation does not fail. 12 13The C++ language is encoding agnostic. It is the responsibility of the programmer to track encodings. It's often easiest to simply `use UTF-8 everywhere <http://utf8everywhere.org/>`_. 14 15.. code-block:: c++ 16 17 m.def("utf8_test", 18 [](const std::string &s) { 19 cout << "utf-8 is icing on the cake.\n"; 20 cout << s; 21 } 22 ); 23 m.def("utf8_charptr", 24 [](const char *s) { 25 cout << "My favorite food is\n"; 26 cout << s; 27 } 28 ); 29 30.. code-block:: python 31 32 >>> utf8_test('') 33 utf-8 is icing on the cake. 34 35 36 >>> utf8_charptr('') 37 My favorite food is 38 39 40.. note:: 41 42 Some terminal emulators do not support UTF-8 or emoji fonts and may not display the example above correctly. 43 44The results are the same whether the C++ function accepts arguments by value or reference, and whether or not ``const`` is used. 45 46Passing bytes to C++ 47-------------------- 48 49A Python ``bytes`` object will be passed to C++ functions that accept ``std::string`` or ``char*`` *without* conversion. 50 51 52Returning C++ strings to Python 53=============================== 54 55When a C++ function returns a ``std::string`` or ``char*`` to a Python caller, **pybind11 will assume that the string is valid UTF-8** and will decode it to a native Python ``str``, using the same API as Python uses to perform ``bytes.decode('utf-8')``. If this implicit conversion fails, pybind11 will raise a ``UnicodeDecodeError``. 56 57.. code-block:: c++ 58 59 m.def("std_string_return", 60 []() { 61 return std::string("This string needs to be UTF-8 encoded"); 62 } 63 ); 64 65.. code-block:: python 66 67 >>> isinstance(example.std_string_return(), str) 68 True 69 70 71Because UTF-8 is inclusive of pure ASCII, there is never any issue with returning a pure ASCII string to Python. If there is any possibility that the string is not pure ASCII, it is necessary to ensure the encoding is valid UTF-8. 72 73.. warning:: 74 75 Implicit conversion assumes that a returned ``char *`` is null-terminated. If there is no null terminator a buffer overrun will occur. 76 77Explicit conversions 78-------------------- 79 80If some C++ code constructs a ``std::string`` that is not a UTF-8 string, one can perform a explicit conversion and return a ``py::str`` object. Explicit conversion has the same overhead as implicit conversion. 81 82.. code-block:: c++ 83 84 // This uses the Python C API to convert Latin-1 to Unicode 85 m.def("str_output", 86 []() { 87 std::string s = "Send your r\xe9sum\xe9 to Alice in HR"; // Latin-1 88 py::str py_s = PyUnicode_DecodeLatin1(s.data(), s.length()); 89 return py_s; 90 } 91 ); 92 93.. code-block:: python 94 95 >>> str_output() 96 'Send your résumé to Alice in HR' 97 98The `Python C API <https://docs.python.org/3/c-api/unicode.html#built-in-codecs>`_ provides several built-in codecs. 99 100 101One could also use a third party encoding library such as libiconv to transcode to UTF-8. 102 103Return C++ strings without conversion 104------------------------------------- 105 106If the data in a C++ ``std::string`` does not represent text and should be returned to Python as ``bytes``, then one can return the data as a ``py::bytes`` object. 107 108.. code-block:: c++ 109 110 m.def("return_bytes", 111 []() { 112 std::string s("\xba\xd0\xba\xd0"); // Not valid UTF-8 113 return py::bytes(s); // Return the data without transcoding 114 } 115 ); 116 117.. code-block:: python 118 119 >>> example.return_bytes() 120 b'\xba\xd0\xba\xd0' 121 122 123Note the asymmetry: pybind11 will convert ``bytes`` to ``std::string`` without encoding, but cannot convert ``std::string`` back to ``bytes`` implicitly. 124 125.. code-block:: c++ 126 127 m.def("asymmetry", 128 [](std::string s) { // Accepts str or bytes from Python 129 return s; // Looks harmless, but implicitly converts to str 130 } 131 ); 132 133.. code-block:: python 134 135 >>> isinstance(example.asymmetry(b"have some bytes"), str) 136 True 137 138 >>> example.asymmetry(b"\xba\xd0\xba\xd0") # invalid utf-8 as bytes 139 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte 140 141 142Wide character strings 143====================== 144 145When a Python ``str`` is passed to a C++ function expecting ``std::wstring``, ``wchar_t*``, ``std::u16string`` or ``std::u32string``, the ``str`` will be encoded to UTF-16 or UTF-32 depending on how the C++ compiler implements each type, in the platform's endian. When strings of these types are returned, they are assumed to contain valid UTF-16 or UTF-32, and will be decoded to Python ``str``. 146 147.. code-block:: c++ 148 149 #define UNICODE 150 #include <windows.h> 151 152 m.def("set_window_text", 153 [](HWND hwnd, std::wstring s) { 154 // Call SetWindowText with null-terminated UTF-16 string 155 ::SetWindowText(hwnd, s.c_str()); 156 } 157 ); 158 m.def("get_window_text", 159 [](HWND hwnd) { 160 const int buffer_size = ::GetWindowTextLength(hwnd) + 1; 161 auto buffer = std::make_unique< wchar_t[] >(buffer_size); 162 163 ::GetWindowText(hwnd, buffer.data(), buffer_size); 164 165 std::wstring text(buffer.get()); 166 167 // wstring will be converted to Python str 168 return text; 169 } 170 ); 171 172.. warning:: 173 174 Wide character strings may not work as described on Python 2.7 or Python 3.3 compiled with ``--enable-unicode=ucs2``. 175 176Strings in multibyte encodings such as Shift-JIS must transcoded to a UTF-8/16/32 before being returned to Python. 177 178 179Character literals 180================== 181 182C++ functions that accept character literals as input will receive the first character of a Python ``str`` as their input. If the string is longer than one Unicode character, trailing characters will be ignored. 183 184When a character literal is returned from C++ (such as a ``char`` or a ``wchar_t``), it will be converted to a ``str`` that represents the single character. 185 186.. code-block:: c++ 187 188 m.def("pass_char", [](char c) { return c; }); 189 m.def("pass_wchar", [](wchar_t w) { return w; }); 190 191.. code-block:: python 192 193 >>> example.pass_char('A') 194 'A' 195 196While C++ will cast integers to character types (``char c = 0x65;``), pybind11 does not convert Python integers to characters implicitly. The Python function ``chr()`` can be used to convert integers to characters. 197 198.. code-block:: python 199 200 >>> example.pass_char(0x65) 201 TypeError 202 203 >>> example.pass_char(chr(0x65)) 204 'A' 205 206If the desire is to work with an 8-bit integer, use ``int8_t`` or ``uint8_t`` as the argument type. 207 208Grapheme clusters 209----------------- 210 211A single grapheme may be represented by two or more Unicode characters. For example 'é' is usually represented as U+00E9 but can also be expressed as the combining character sequence U+0065 U+0301 (that is, the letter 'e' followed by a combining acute accent). The combining character will be lost if the two-character sequence is passed as an argument, even though it renders as a single grapheme. 212 213.. code-block:: python 214 215 >>> example.pass_wchar('é') 216 'é' 217 218 >>> combining_e_acute = 'e' + '\u0301' 219 220 >>> combining_e_acute 221 'é' 222 223 >>> combining_e_acute == 'é' 224 False 225 226 >>> example.pass_wchar(combining_e_acute) 227 'e' 228 229Normalizing combining characters before passing the character literal to C++ may resolve *some* of these issues: 230 231.. code-block:: python 232 233 >>> example.pass_wchar(unicodedata.normalize('NFC', combining_e_acute)) 234 'é' 235 236In some languages (Thai for example), there are `graphemes that cannot be expressed as a single Unicode code point <http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries>`_, so there is no way to capture them in a C++ character type. 237 238 239References 240========== 241 242* `The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) <https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/>`_ 243* `C++ - Using STL Strings at Win32 API Boundaries <https://msdn.microsoft.com/en-ca/magazine/mt238407.aspx>`_