History log of /gem5/src/python/pybind11/stats.cc
Revision Date Author Comments
# 14209:7efe1c187149 22-Apr-2016 Andreas Sandberg <andreas.sandberg@arm.com>

stats: Add beta support for HDF5 stat dumps

This changeset add support for stat dumps in the HDF5 file
format. HDF5 is a binary data format that represents data in a
file-system-like balanced tree. It has native support for
N-dimensional arrays and binary data (e.g., frame buffers).

It has the following benefits over traditional text stat files:

* Efficient storage of time series (multiple stat dumps)

* Fast lookup of stats

* Plenty of existing tooling (e.g., Python libraries and graphical
viewers)

* File format can be used to store frame buffers together with
normal stats.

Drawbacks:

* Large startup cost (single stat dump larger than text equivalent)

* Stat dumps are slower than text

Known limitations:

* Distributions and histograms aren't supported.

HDF5 stat output can be enabled using the 'h5' URL scheme when
overriding the stat file name on gem5's command line. The following
parameters are supported:

* chunking (unsigned): Number of time steps to pre-allocate
(default: 10)

* desc (bool): Output stat descriptions (default: True)

* formulas (bool): Output derived stats (default: True)

Example gem5 command line:

./build/ARM/gem5.opt \
--stats-file="h5://stats.h5?desc=False;formulas=False" \
configs/example/fs.py

Example Python stat consumer that computes IPC:
import h5py

f = h5py.File('stats.h5', 'r')
group = f['/system/cpu']
for i, c in zip(group['committedInsts'], group['numCycles']):
print i, c, i / c

Change-Id: I351c6cbff2fb7bef9012f47876ba227ed288975b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/8121
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Ciro Santilli <ciro.santilli@arm.com>


# 14205:197360deaa20 26-Jun-2019 Andreas Sandberg <andreas.sandberg@arm.com>

stats: Add support for hierarchical stats

This change makes the stat system aware of the hierarchical nature of
stats. The aim is to achieve the following goals:

* Make the SimObject hierarchy explicit in the stat system (i.e.,
get rid of name() + ".foo"). This makes stat naming less fragile
and makes it possible to implement hierarchical formats like
XML/HDF5/JSON in a clean way.

* Make it more convenient to split stats into a separate
struct/class that can be bound to a SimObject. This makes the
namespace cleaner and makes stat accesses a bit more obvious.

* Make it possible to build groups of stats in C++ that can be used
in subcomponents in a SimObject (similar to what we do for
checkpoint sections). This makes it easier to structure large
components.

* Enable partial stat dumps. Some of our internal users have been
asking for this since a full stat dump can be large.

* Enable better stat access from Python.

This changeset implements solves the first three points by introducing
a class (Stats::Group) that owns statistics belonging to the same
object. SimObjects inherit from Stats::Group since they typically have
statistics.

New-style statistics need to be associated with a parent group at
instantiation time. Instantiation typically sets the name and the
description, other parameters need to be set by overriding
Group::regStats() just like with legacy stats. Simple objects with
scalar stats can typically avoid implementing regStats() altogether
since the stat name and description are both specified in the
constructor.

For convenience reasons, statistics groups can be merged into other
groups. This means that a SimObject can create a stat struct that
inherits from Stats::Group and merge it into the parent group
(SimObject). This can make the code cleaner since statistics tracking
gets grouped into a single object.

Stat visitors have a new API to expose the group structure. The
Output::beginGroup(name) method is called at the beginning of a group
and the Output::endGroup() method is called when all stats, and
sub-groups, have been visited. Flat formats (e.g., the text format)
typically need to maintain a stack to track the full path to a stat.

Legacy, flat, statistics are still supported after applying this
change. These stats don't belong to any group and stat visitors will
not see a Output::beginGroup(name) call before their corresponding
Output::visit() methods are called.

Change-Id: I9025d61dfadeabcc8ecf30813ab2060def455648
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19368
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>


# 11988:665cd5f8b52b 27-Feb-2017 Andreas Sandberg <andreas.sandberg@arm.com>

python: Use PyBind11 instead of SWIG for Python wrappers

Use the PyBind11 wrapping infrastructure instead of SWIG to generate
wrappers for functionality that needs to be exported to Python. This
has several benefits:

* PyBind11 can be redistributed with gem5, which means that we have
full control of the version used. This avoid a large number of
hard-to-debug SWIG issues we have seen in the past.

* PyBind11 doesn't rely on a custom C++ parser, instead it relies on
wrappers being explicitly declared in C++. The leads to slightly
more boiler-plate code in manually created wrappers, but doesn't
doesn't increase the overall code size. A big benefit is that this
avoids strange compilation errors when SWIG doesn't understand
modern language features.

* Unlike SWIG, there is no risk that the wrapper code incorporates
incorrect type casts (this has happened on numerous occasions in
the past) since these will result in compile-time errors.

As a part of this change, the mechanism to define exported methods has
been redesigned slightly. New methods can be exported either by
declaring them in the SimObject declaration and decorating them with
the cxxMethod decorator or by adding an instance of
PyBindMethod/PyBindProperty to the cxx_exports class variable. The
decorator has the added benefit of making it possible to add a
docstring and naming the method's parameters.

The new wrappers have the following known issues:

* Global events can't be memory managed correctly. This was the
case in SWIG as well.

Change-Id: I88c5a95b6cf6c32fa9e1ad31dfc08b2e8199a763
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Andrew Bardsley <andrew.bardsley@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2231
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Reviewed-by: Pierre-Yves PĂ©neau <pierre-yves.peneau@lirmm.fr>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>