1Strings, bytes and Unicode conversions
2######################################
3
4.. note::
5
6    This section discusses string handling in terms of Python 3 strings. For
7    Python 2.7, replace all occurrences of ``str`` with ``unicode`` and
8    ``bytes`` with ``str``.  Python 2.7 users may find it best to use ``from
9    __future__ import unicode_literals`` to avoid unintentionally using ``str``
10    instead of ``unicode``.
11
12Passing Python strings to C++
13=============================
14
15When a Python ``str`` is passed from Python to a C++ function that accepts
16``std::string`` or ``char *`` as arguments, pybind11 will encode the Python
17string to UTF-8. All Python ``str`` can be encoded in UTF-8, so this operation
18does not fail.
19
20The C++ language is encoding agnostic. It is the responsibility of the
21programmer to track encodings. It's often easiest to simply `use UTF-8
22everywhere <http://utf8everywhere.org/>`_.
23
24.. code-block:: c++
25
26    m.def("utf8_test",
27        [](const std::string &s) {
28            cout << "utf-8 is icing on the cake.\n";
29            cout << s;
30        }
31    );
32    m.def("utf8_charptr",
33        [](const char *s) {
34            cout << "My favorite food is\n";
35            cout << s;
36        }
37    );
38
39.. code-block:: python
40
41    >>> utf8_test('��')
42    utf-8 is icing on the cake.
43    ��
44
45    >>> utf8_charptr('��')
46    My favorite food is
47    ��
48
49.. note::
50
51    Some terminal emulators do not support UTF-8 or emoji fonts and may not
52    display the example above correctly.
53
54The results are the same whether the C++ function accepts arguments by value or
55reference, and whether or not ``const`` is used.
56
57Passing bytes to C++
58--------------------
59
60A Python ``bytes`` object will be passed to C++ functions that accept
61``std::string`` or ``char*`` *without* conversion.  On Python 3, in order to
62make a function *only* accept ``bytes`` (and not ``str``), declare it as taking
63a ``py::bytes`` argument.
64
65
66Returning C++ strings to Python
67===============================
68
69When a C++ function returns a ``std::string`` or ``char*`` to a Python caller,
70**pybind11 will assume that the string is valid UTF-8** and will decode it to a
71native Python ``str``, using the same API as Python uses to perform
72``bytes.decode('utf-8')``. If this implicit conversion fails, pybind11 will
73raise a ``UnicodeDecodeError``.
74
75.. code-block:: c++
76
77    m.def("std_string_return",
78        []() {
79            return std::string("This string needs to be UTF-8 encoded");
80        }
81    );
82
83.. code-block:: python
84
85    >>> isinstance(example.std_string_return(), str)
86    True
87
88
89Because UTF-8 is inclusive of pure ASCII, there is never any issue with
90returning a pure ASCII string to Python. If there is any possibility that the
91string is not pure ASCII, it is necessary to ensure the encoding is valid
92UTF-8.
93
94.. warning::
95
96    Implicit conversion assumes that a returned ``char *`` is null-terminated.
97    If there is no null terminator a buffer overrun will occur.
98
99Explicit conversions
100--------------------
101
102If some C++ code constructs a ``std::string`` that is not a UTF-8 string, one
103can perform a explicit conversion and return a ``py::str`` object. Explicit
104conversion has the same overhead as implicit conversion.
105
106.. code-block:: c++
107
108    // This uses the Python C API to convert Latin-1 to Unicode
109    m.def("str_output",
110        []() {
111            std::string s = "Send your r\xe9sum\xe9 to Alice in HR"; // Latin-1
112            py::str py_s = PyUnicode_DecodeLatin1(s.data(), s.length());
113            return py_s;
114        }
115    );
116
117.. code-block:: python
118
119    >>> str_output()
120    'Send your résumé to Alice in HR'
121
122The `Python C API
123<https://docs.python.org/3/c-api/unicode.html#built-in-codecs>`_ provides
124several built-in codecs.
125
126
127One could also use a third party encoding library such as libiconv to transcode
128to UTF-8.
129
130Return C++ strings without conversion
131-------------------------------------
132
133If the data in a C++ ``std::string`` does not represent text and should be
134returned to Python as ``bytes``, then one can return the data as a
135``py::bytes`` object.
136
137.. code-block:: c++
138
139    m.def("return_bytes",
140        []() {
141            std::string s("\xba\xd0\xba\xd0");  // Not valid UTF-8
142            return py::bytes(s);  // Return the data without transcoding
143        }
144    );
145
146.. code-block:: python
147
148    >>> example.return_bytes()
149    b'\xba\xd0\xba\xd0'
150
151
152Note the asymmetry: pybind11 will convert ``bytes`` to ``std::string`` without
153encoding, but cannot convert ``std::string`` back to ``bytes`` implicitly.
154
155.. code-block:: c++
156
157    m.def("asymmetry",
158        [](std::string s) {  // Accepts str or bytes from Python
159            return s;  // Looks harmless, but implicitly converts to str
160        }
161    );
162
163.. code-block:: python
164
165    >>> isinstance(example.asymmetry(b"have some bytes"), str)
166    True
167
168    >>> example.asymmetry(b"\xba\xd0\xba\xd0")  # invalid utf-8 as bytes
169    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte
170
171
172Wide character strings
173======================
174
175When a Python ``str`` is passed to a C++ function expecting ``std::wstring``,
176``wchar_t*``, ``std::u16string`` or ``std::u32string``, the ``str`` will be
177encoded to UTF-16 or UTF-32 depending on how the C++ compiler implements each
178type, in the platform's native endianness. When strings of these types are
179returned, they are assumed to contain valid UTF-16 or UTF-32, and will be
180decoded to Python ``str``.
181
182.. code-block:: c++
183
184    #define UNICODE
185    #include <windows.h>
186
187    m.def("set_window_text",
188        [](HWND hwnd, std::wstring s) {
189            // Call SetWindowText with null-terminated UTF-16 string
190            ::SetWindowText(hwnd, s.c_str());
191        }
192    );
193    m.def("get_window_text",
194        [](HWND hwnd) {
195            const int buffer_size = ::GetWindowTextLength(hwnd) + 1;
196            auto buffer = std::make_unique< wchar_t[] >(buffer_size);
197
198            ::GetWindowText(hwnd, buffer.data(), buffer_size);
199
200            std::wstring text(buffer.get());
201
202            // wstring will be converted to Python str
203            return text;
204        }
205    );
206
207.. warning::
208
209    Wide character strings may not work as described on Python 2.7 or Python
210    3.3 compiled with ``--enable-unicode=ucs2``.
211
212Strings in multibyte encodings such as Shift-JIS must transcoded to a
213UTF-8/16/32 before being returned to Python.
214
215
216Character literals
217==================
218
219C++ functions that accept character literals as input will receive the first
220character of a Python ``str`` as their input. If the string is longer than one
221Unicode character, trailing characters will be ignored.
222
223When a character literal is returned from C++ (such as a ``char`` or a
224``wchar_t``), it will be converted to a ``str`` that represents the single
225character.
226
227.. code-block:: c++
228
229    m.def("pass_char", [](char c) { return c; });
230    m.def("pass_wchar", [](wchar_t w) { return w; });
231
232.. code-block:: python
233
234    >>> example.pass_char('A')
235    'A'
236
237While C++ will cast integers to character types (``char c = 0x65;``), pybind11
238does not convert Python integers to characters implicitly. The Python function
239``chr()`` can be used to convert integers to characters.
240
241.. code-block:: python
242
243    >>> example.pass_char(0x65)
244    TypeError
245
246    >>> example.pass_char(chr(0x65))
247    'A'
248
249If the desire is to work with an 8-bit integer, use ``int8_t`` or ``uint8_t``
250as the argument type.
251
252Grapheme clusters
253-----------------
254
255A single grapheme may be represented by two or more Unicode characters. For
256example 'é' is usually represented as U+00E9 but can also be expressed as the
257combining character sequence U+0065 U+0301 (that is, the letter 'e' followed by
258a combining acute accent). The combining character will be lost if the
259two-character sequence is passed as an argument, even though it renders as a
260single grapheme.
261
262.. code-block:: python
263
264    >>> example.pass_wchar('é')
265    'é'
266
267    >>> combining_e_acute = 'e' + '\u0301'
268
269    >>> combining_e_acute
270    'é'
271
272    >>> combining_e_acute == 'é'
273    False
274
275    >>> example.pass_wchar(combining_e_acute)
276    'e'
277
278Normalizing combining characters before passing the character literal to C++
279may resolve *some* of these issues:
280
281.. code-block:: python
282
283    >>> example.pass_wchar(unicodedata.normalize('NFC', combining_e_acute))
284    'é'
285
286In some languages (Thai for example), there are `graphemes that cannot be
287expressed as a single Unicode code point
288<http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries>`_, so there is
289no way to capture them in a C++ character type.
290
291
292C++17 string views
293==================
294
295C++17 string views are automatically supported when compiling in C++17 mode.
296They follow the same rules for encoding and decoding as the corresponding STL
297string type (for example, a ``std::u16string_view`` argument will be passed
298UTF-16-encoded data, and a returned ``std::string_view`` will be decoded as
299UTF-8).
300
301References
302==========
303
304* `The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) <https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/>`_
305* `C++ - Using STL Strings at Win32 API Boundaries <https://msdn.microsoft.com/en-ca/magazine/mt238407.aspx>`_
306