benchmark.rst revision 11986
111986Sandreas.sandberg@arm.comBenchmark 211986Sandreas.sandberg@arm.com========= 311986Sandreas.sandberg@arm.com 411986Sandreas.sandberg@arm.comThe following is the result of a synthetic benchmark comparing both compilation 511986Sandreas.sandberg@arm.comtime and module size of pybind11 against Boost.Python. A detailed report about a 611986Sandreas.sandberg@arm.comBoost.Python to pybind11 conversion of a real project is available here: [#f1]_. 711986Sandreas.sandberg@arm.com 811986Sandreas.sandberg@arm.com.. [#f1] http://graylab.jhu.edu/RosettaCon2016/PyRosetta-4.pdf 911986Sandreas.sandberg@arm.com 1011986Sandreas.sandberg@arm.comSetup 1111986Sandreas.sandberg@arm.com----- 1211986Sandreas.sandberg@arm.com 1311986Sandreas.sandberg@arm.comA python script (see the ``docs/benchmark.py`` file) was used to generate a set 1411986Sandreas.sandberg@arm.comof files with dummy classes whose count increases for each successive benchmark 1511986Sandreas.sandberg@arm.com(between 1 and 2048 classes in powers of two). Each class has four methods with 1611986Sandreas.sandberg@arm.coma randomly generated signature with a return value and four arguments. (There 1711986Sandreas.sandberg@arm.comwas no particular reason for this setup other than the desire to generate many 1811986Sandreas.sandberg@arm.comunique function signatures whose count could be controlled in a simple way.) 1911986Sandreas.sandberg@arm.com 2011986Sandreas.sandberg@arm.comHere is an example of the binding code for one class: 2111986Sandreas.sandberg@arm.com 2211986Sandreas.sandberg@arm.com.. code-block:: cpp 2311986Sandreas.sandberg@arm.com 2411986Sandreas.sandberg@arm.com ... 2511986Sandreas.sandberg@arm.com class cl034 { 2611986Sandreas.sandberg@arm.com public: 2711986Sandreas.sandberg@arm.com cl279 *fn_000(cl084 *, cl057 *, cl065 *, cl042 *); 2811986Sandreas.sandberg@arm.com cl025 *fn_001(cl098 *, cl262 *, cl414 *, cl121 *); 2911986Sandreas.sandberg@arm.com cl085 *fn_002(cl445 *, cl297 *, cl145 *, cl421 *); 3011986Sandreas.sandberg@arm.com cl470 *fn_003(cl200 *, cl323 *, cl332 *, cl492 *); 3111986Sandreas.sandberg@arm.com }; 3211986Sandreas.sandberg@arm.com ... 3311986Sandreas.sandberg@arm.com 3411986Sandreas.sandberg@arm.com PYBIND11_PLUGIN(example) { 3511986Sandreas.sandberg@arm.com py::module m("example"); 3611986Sandreas.sandberg@arm.com ... 3711986Sandreas.sandberg@arm.com py::class_<cl034>(m, "cl034") 3811986Sandreas.sandberg@arm.com .def("fn_000", &cl034::fn_000) 3911986Sandreas.sandberg@arm.com .def("fn_001", &cl034::fn_001) 4011986Sandreas.sandberg@arm.com .def("fn_002", &cl034::fn_002) 4111986Sandreas.sandberg@arm.com .def("fn_003", &cl034::fn_003) 4211986Sandreas.sandberg@arm.com ... 4311986Sandreas.sandberg@arm.com return m.ptr(); 4411986Sandreas.sandberg@arm.com } 4511986Sandreas.sandberg@arm.com 4611986Sandreas.sandberg@arm.comThe Boost.Python version looks almost identical except that a return value 4711986Sandreas.sandberg@arm.compolicy had to be specified as an argument to ``def()``. For both libraries, 4811986Sandreas.sandberg@arm.comcompilation was done with 4911986Sandreas.sandberg@arm.com 5011986Sandreas.sandberg@arm.com.. code-block:: bash 5111986Sandreas.sandberg@arm.com 5211986Sandreas.sandberg@arm.com Apple LLVM version 7.0.2 (clang-700.1.81) 5311986Sandreas.sandberg@arm.com 5411986Sandreas.sandberg@arm.comand the following compilation flags 5511986Sandreas.sandberg@arm.com 5611986Sandreas.sandberg@arm.com.. code-block:: bash 5711986Sandreas.sandberg@arm.com 5811986Sandreas.sandberg@arm.com g++ -Os -shared -rdynamic -undefined dynamic_lookup -fvisibility=hidden -std=c++14 5911986Sandreas.sandberg@arm.com 6011986Sandreas.sandberg@arm.comCompilation time 6111986Sandreas.sandberg@arm.com---------------- 6211986Sandreas.sandberg@arm.com 6311986Sandreas.sandberg@arm.comThe following log-log plot shows how the compilation time grows for an 6411986Sandreas.sandberg@arm.comincreasing number of class and function declarations. pybind11 includes many 6511986Sandreas.sandberg@arm.comfewer headers, which initially leads to shorter compilation times, but the 6611986Sandreas.sandberg@arm.comperformance is ultimately fairly similar (pybind11 is 19.8 seconds faster for 6711986Sandreas.sandberg@arm.comthe largest largest file with 2048 classes and a total of 8192 methods -- a 6811986Sandreas.sandberg@arm.commodest **1.2x** speedup relative to Boost.Python, which required 116.35 6911986Sandreas.sandberg@arm.comseconds). 7011986Sandreas.sandberg@arm.com 7111986Sandreas.sandberg@arm.com.. only:: not latex 7211986Sandreas.sandberg@arm.com 7311986Sandreas.sandberg@arm.com .. image:: pybind11_vs_boost_python1.svg 7411986Sandreas.sandberg@arm.com 7511986Sandreas.sandberg@arm.com.. only:: latex 7611986Sandreas.sandberg@arm.com 7711986Sandreas.sandberg@arm.com .. image:: pybind11_vs_boost_python1.png 7811986Sandreas.sandberg@arm.com 7911986Sandreas.sandberg@arm.comModule size 8011986Sandreas.sandberg@arm.com----------- 8111986Sandreas.sandberg@arm.com 8211986Sandreas.sandberg@arm.comDifferences between the two libraries become much more pronounced when 8311986Sandreas.sandberg@arm.comconsidering the file size of the generated Python plugin: for the largest file, 8411986Sandreas.sandberg@arm.comthe binary generated by Boost.Python required 16.8 MiB, which was **2.17 8511986Sandreas.sandberg@arm.comtimes** / **9.1 megabytes** larger than the output generated by pybind11. For 8611986Sandreas.sandberg@arm.comvery small inputs, Boost.Python has an edge in the plot below -- however, note 8711986Sandreas.sandberg@arm.comthat it stores many definitions in an external library, whose size was not 8811986Sandreas.sandberg@arm.comincluded here, hence the comparison is slightly shifted in Boost.Python's 8911986Sandreas.sandberg@arm.comfavor. 9011986Sandreas.sandberg@arm.com 9111986Sandreas.sandberg@arm.com.. only:: not latex 9211986Sandreas.sandberg@arm.com 9311986Sandreas.sandberg@arm.com .. image:: pybind11_vs_boost_python2.svg 9411986Sandreas.sandberg@arm.com 9511986Sandreas.sandberg@arm.com.. only:: latex 9611986Sandreas.sandberg@arm.com 9711986Sandreas.sandberg@arm.com .. image:: pybind11_vs_boost_python2.png 9811986Sandreas.sandberg@arm.com 9911986Sandreas.sandberg@arm.com 100