benchmark.rst revision 11986:c12e4625ab56 
1Benchmark
2=========
3
4The following is the result of a synthetic benchmark comparing both compilation
5time and module size of pybind11 against Boost.Python. A detailed report about a
6Boost.Python to pybind11 conversion of a real project is available here: [#f1]_.
7
8.. [#f1] http://graylab.jhu.edu/RosettaCon2016/PyRosetta-4.pdf
9
10Setup
11-----
12
13A python script (see the ``docs/benchmark.py`` file) was used to generate a set
14of files with dummy classes whose count increases for each successive benchmark
15(between 1 and 2048 classes in powers of two). Each class has four methods with
16a randomly generated signature with a return value and four arguments. (There
17was no particular reason for this setup other than the desire to generate many
18unique function signatures whose count could be controlled in a simple way.)
19
20Here is an example of the binding code for one class:
21
22.. code-block:: cpp
23
24    ...
25    class cl034 {
26    public:
27        cl279 *fn_000(cl084 *, cl057 *, cl065 *, cl042 *);
28        cl025 *fn_001(cl098 *, cl262 *, cl414 *, cl121 *);
29        cl085 *fn_002(cl445 *, cl297 *, cl145 *, cl421 *);
30        cl470 *fn_003(cl200 *, cl323 *, cl332 *, cl492 *);
31    };
32    ...
33
34    PYBIND11_PLUGIN(example) {
35        py::module m("example");
36        ...
37        py::class_<cl034>(m, "cl034")
38            .def("fn_000", &cl034::fn_000)
39            .def("fn_001", &cl034::fn_001)
40            .def("fn_002", &cl034::fn_002)
41            .def("fn_003", &cl034::fn_003)
42        ...
43        return m.ptr();
44    }
45
46The Boost.Python version looks almost identical except that a return value
47policy had to be specified as an argument to ``def()``. For both libraries,
48compilation was done with
49
50.. code-block:: bash
51
52    Apple LLVM version 7.0.2 (clang-700.1.81)
53
54and the following compilation flags
55
56.. code-block:: bash
57
58    g++ -Os -shared -rdynamic -undefined dynamic_lookup -fvisibility=hidden -std=c++14
59
60Compilation time
61----------------
62
63The following log-log plot shows how the compilation time grows for an
64increasing number of class and function declarations. pybind11 includes many
65fewer headers, which initially leads to shorter compilation times, but the
66performance is ultimately fairly similar (pybind11 is 19.8 seconds faster for
67the largest largest file with 2048 classes and a total of 8192 methods -- a
68modest **1.2x** speedup relative to Boost.Python, which required 116.35
69seconds).
70
71.. only:: not latex
72
73    .. image:: pybind11_vs_boost_python1.svg
74
75.. only:: latex
76
77    .. image:: pybind11_vs_boost_python1.png
78
79Module size
80-----------
81
82Differences between the two libraries become much more pronounced when
83considering the file size of the generated Python plugin: for the largest file,
84the binary generated by Boost.Python required 16.8 MiB, which was **2.17
85times** / **9.1 megabytes** larger than the output generated by pybind11. For
86very small inputs, Boost.Python has an edge in the plot below -- however, note
87that it stores many definitions in an external library, whose size was not
88included here, hence the comparison is slightly shifted in Boost.Python's
89favor.
90
91.. only:: not latex
92
93    .. image:: pybind11_vs_boost_python2.svg
94
95.. only:: latex
96
97    .. image:: pybind11_vs_boost_python2.png
98
99
100