1Benchmark
2=========
3
4The following is the result of a synthetic benchmark comparing both compilation
5time and module size of pybind11 against Boost.Python. A detailed report about a
6Boost.Python to pybind11 conversion of a real project is available here: [#f1]_.
7
8.. [#f1] http://graylab.jhu.edu/RosettaCon2016/PyRosetta-4.pdf
9
10Setup
11-----
12
13A python script (see the ``docs/benchmark.py`` file) was used to generate a set
14of files with dummy classes whose count increases for each successive benchmark
15(between 1 and 2048 classes in powers of two). Each class has four methods with
16a randomly generated signature with a return value and four arguments. (There
17was no particular reason for this setup other than the desire to generate many
18unique function signatures whose count could be controlled in a simple way.)
19
20Here is an example of the binding code for one class:
21
22.. code-block:: cpp
23
24    ...
25    class cl034 {
26    public:
27        cl279 *fn_000(cl084 *, cl057 *, cl065 *, cl042 *);
28        cl025 *fn_001(cl098 *, cl262 *, cl414 *, cl121 *);
29        cl085 *fn_002(cl445 *, cl297 *, cl145 *, cl421 *);
30        cl470 *fn_003(cl200 *, cl323 *, cl332 *, cl492 *);
31    };
32    ...
33
34    PYBIND11_MODULE(example, m) {
35        ...
36        py::class_<cl034>(m, "cl034")
37            .def("fn_000", &cl034::fn_000)
38            .def("fn_001", &cl034::fn_001)
39            .def("fn_002", &cl034::fn_002)
40            .def("fn_003", &cl034::fn_003)
41        ...
42    }
43
44The Boost.Python version looks almost identical except that a return value
45policy had to be specified as an argument to ``def()``. For both libraries,
46compilation was done with
47
48.. code-block:: bash
49
50    Apple LLVM version 7.0.2 (clang-700.1.81)
51
52and the following compilation flags
53
54.. code-block:: bash
55
56    g++ -Os -shared -rdynamic -undefined dynamic_lookup -fvisibility=hidden -std=c++14
57
58Compilation time
59----------------
60
61The following log-log plot shows how the compilation time grows for an
62increasing number of class and function declarations. pybind11 includes many
63fewer headers, which initially leads to shorter compilation times, but the
64performance is ultimately fairly similar (pybind11 is 19.8 seconds faster for
65the largest largest file with 2048 classes and a total of 8192 methods -- a
66modest **1.2x** speedup relative to Boost.Python, which required 116.35
67seconds).
68
69.. only:: not latex
70
71    .. image:: pybind11_vs_boost_python1.svg
72
73.. only:: latex
74
75    .. image:: pybind11_vs_boost_python1.png
76
77Module size
78-----------
79
80Differences between the two libraries become much more pronounced when
81considering the file size of the generated Python plugin: for the largest file,
82the binary generated by Boost.Python required 16.8 MiB, which was **2.17
83times** / **9.1 megabytes** larger than the output generated by pybind11. For
84very small inputs, Boost.Python has an edge in the plot below -- however, note
85that it stores many definitions in an external library, whose size was not
86included here, hence the comparison is slightly shifted in Boost.Python's
87favor.
88
89.. only:: not latex
90
91    .. image:: pybind11_vs_boost_python2.svg
92
93.. only:: latex
94
95    .. image:: pybind11_vs_boost_python2.png
96
97
98