stats: Add beta support for HDF5 stat dumps

This changeset add support for stat dumps in the HDF5 file
format. HDF5 is a binary data format that represents data in a
file-system-like balanced tree. It has native support for
N-dimensional arrays and binary data (e.g., frame buffers).

It has the following benefits over traditional text stat files:

* Efficient storage of time series (multiple stat dumps)

* Fast lookup of stats

* Plenty of existing tooling (e.g., Python libraries and graphical
viewers)

* File format can be used to store frame buffers together with
normal stats.

Drawbacks:

* Large startup cost (single stat dump larger than text equivalent)

* Stat dumps are slower than text

Known limitations:

* Distributions and histograms aren't supported.

HDF5 stat output can be enabled using the 'h5' URL scheme when
overriding the stat file name on gem5's command line. The following
parameters are supported:

* chunking (unsigned): Number of time steps to pre-allocate
(default: 10)

* desc (bool): Output stat descriptions (default: True)

* formulas (bool): Output derived stats (default: True)

Example gem5 command line:

./build/ARM/gem5.opt \
--stats-file="h5://stats.h5?desc=False;formulas=False" \
configs/example/fs.py

Example Python stat consumer that computes IPC:
import h5py

f = h5py.File('stats.h5', 'r')
group = f['/system/cpu']
for i, c in zip(group['committedInsts'], group['numCycles']):
print i, c, i / c

Change-Id: I351c6cbff2fb7bef9012f47876ba227ed288975b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/8121
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Ciro Santilli <ciro.santilli@arm.com>

stats: Add support for hierarchical stats

This change makes the stat system aware of the hierarchical nature of
stats. The aim is to achieve the following goals:

* Make the SimObject hierarchy explicit in the stat system (i.e.,
get rid of name() + ".foo"). This makes stat naming less fragile
and makes it possible to implement hierarchical formats like
XML/HDF5/JSON in a clean way.

* Make it more convenient to split stats into a separate
struct/class that can be bound to a SimObject. This makes the
namespace cleaner and makes stat accesses a bit more obvious.

* Make it possible to build groups of stats in C++ that can be used
in subcomponents in a SimObject (similar to what we do for
checkpoint sections). This makes it easier to structure large
components.

* Enable partial stat dumps. Some of our internal users have been
asking for this since a full stat dump can be large.

* Enable better stat access from Python.

This changeset implements solves the first three points by introducing
a class (Stats::Group) that owns statistics belonging to the same
object. SimObjects inherit from Stats::Group since they typically have
statistics.

New-style statistics need to be associated with a parent group at
instantiation time. Instantiation typically sets the name and the
description, other parameters need to be set by overriding
Group::regStats() just like with legacy stats. Simple objects with
scalar stats can typically avoid implementing regStats() altogether
since the stat name and description are both specified in the
constructor.

For convenience reasons, statistics groups can be merged into other
groups. This means that a SimObject can create a stat struct that
inherits from Stats::Group and merge it into the parent group
(SimObject). This can make the code cleaner since statistics tracking
gets grouped into a single object.

Stat visitors have a new API to expose the group structure. The
Output::beginGroup(name) method is called at the beginning of a group
and the Output::endGroup() method is called when all stats, and
sub-groups, have been visited. Flat formats (e.g., the text format)
typically need to maintain a stack to track the full path to a stat.

Legacy, flat, statistics are still supported after applying this
change. These stats don't belong to any group and stat visitors will
not see a Output::beginGroup(name) call before their corresponding
Output::visit() methods are called.

Change-Id: I9025d61dfadeabcc8ecf30813ab2060def455648
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19368
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>

python: Add binding for the new AddrRange c++ constructor

Change-Id: I5b3fb59a11d8587a753759310dd3b2748ac13a0b
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19132
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

base: Extend AddrRange to support more flexible addressing

Previously an AddrRange could express interleaving using a number of
consecutive bits and in additional optionally a second number of
consecutive bits. The two sets of consecutive bits would be xored and
matched against a value to determine if an address is in the
AddrRange. For example:

sel[0] = a[8] ^ a[12]
sel[1] = a[9] ^ a[13]
where sel == intlvMatch

This change extends AddrRange to allow more flexible interleavings
with an abritary number of set of bits which do not need be
consecutive. For example:

sel[0] = a[8] ^ a[11] ^ a[13]
sel[1] = a[15] ^ a[17] ^ a[19]
where sel == intlvMatch

Change-Id: I42220a6d5011a31f0560535762a25bfc823c3ebb
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19130
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>

base,python: Fix to allow multiple --debug-ignore values.

When adding multiple SimObjects to --debug-ignore, either separating the values with
a colon or adding multiple --debug-ignore flags, the previous code only ignored the
last SimObject in the list. This changeset adds and uses new `ObjectMatch::add` and
`Logger::addIgnore` methods to make the functionality of the flag consistent with
its description.

Change-Id: Ib6967a48611ea59a211f81af2a970c4de429b1be
Signed-off-by: Isaac Sánchez Barrera <isaac.sanchez@bsc.es>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17488
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

arch, cpu, dev, gpu, mem, sim, python: start using getPort.

Replace the getMasterPort, getSlavePort, and getEthPort functions
with getPort, and remove extraneous mechanisms that are no longer
necessary.

Change-Id: Iab7e3c02d2f3a0cf33e7e824e18c28646b5bc318
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17040
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Change || to && for MessageBuffers in connectPorts.

The connectPorts function currently checks if *either* of the peers in
a port connection are a MessageBuffer, and if so will ignore the
connection. This CL changes that || into a && so that *both* of the
peers need to be a Ruby types (either a MessageBuffer or Network) for
the connection to be ignored. That makes it easier to contain that
abnormal behavior to those types instead of having it apply even when
other types of port owners are involved.

Unfortunately the number of interesting Ruby types is unbounded, but
these are the types with ports as of today. This mechanism will
hopefully be replacedall together so this should be a temporary issue.

Change-Id: I140498770e5d37eb2abd3d99261d47e111f1c8ab
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17031
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

python: Simplify connectPorts() around EtherObject/EtherDevice.

EtherDevice now inherits EtherObject and shares the same getEthPort
virtual function, so there's no need to treat them separately any more.

Change-Id: Ia6c147fd97fece4a281c296521a7b095f793d32e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17030
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Remove getCode() type workaround

Python 2.7 requires a workaround when wrapping exit objects to
explicitly convert the return of getCode() to int to not confuse
sys.exit. This workaround isn't needed and doesn't work on Python 3
since it doesn't have a separate long integer type.

Change-Id: I57bc3fd8f4699676c046ece8a52baa2796959ffd
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15978
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>

sim: Push the global frequency management code into C++.

That makes it available when python is left out, and makes it available
to c++ code without having to call back into python.

Change-Id: If82e7e8eff526f2b957f84afe046e1d56fed4aa2
Reviewed-on: https://gem5-review.googlesource.com/c/14055
Reviewed-by: Srikant Bharadwaj <srikant.bharadwaj@amd.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Stop conditionally excluding code from pyobject.cc

Now that the Ether* classes are included in all builds, there's no
reason to conditionally compile code in pyobject.cc.

Change-Id: If94602af71774b1f090a3344a633207f4b37d308
Reviewed-on: https://gem5-review.googlesource.com/c/13470
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

misc: Rename misc.(hh|cc) to logging.(hh|cc)

These files aren't a collection of miscellaneous stuff, they're the
definition of the Logger interface, and a few utility macros for
calling into that interface (panic, warn, etc.).

Change-Id: I84267ac3f45896a83c0ef027f8f19c5e9a5667d1
Reviewed-on: https://gem5-review.googlesource.com/6226
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>

python: Make GlobalExitEvent.getCode() return an int

PyBind normally casts integers returned from the C to long in
Python. This is normally fine since long in most cases behaves just
like an int. However, when passing the return value from getcode() to
sys.exit, unexpected behavior ensues. Due to the way the function is
defined, any type other than int (with the exception of None) will be
treated as an error and be equivalent to sys.exit(1).

Since we frequently use the sys.exit(event.getCode()) pattern, we need
to ensure that the function returns an integer. This change adds an
explicit type conversion to a Python integer in the wrapper code.

Change-Id: I73d6b881025064afa2b2e6eb4512fa2a4b0a87da
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jose Marinho <jose.marinho@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/4280
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Joe Gross <joe.gross@amd.com>

python: Fix PyEvent reference counting bug

The current implementation of reference counting for PyEvents only
partially works. The native object is currently kept alive while it is
in the event queue. However, if the Python object goes out of scope,
the Python side of this object is garbage collected which leaves a
"dangling" native object. This results in confusing error messages
where PyBind is unable to find the Python implementation of an event
when it is triggered.

Implement reference counting using the generalized reference counting
API instead.

Change-Id: I4e8e04abc4f61dff238d718065f5371e73b38ab3
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/3222
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Prevent Python wrappers from deleting SimObjects

The PyBind wrappers could potentially delete SimObjects if they don't
have any references. This is not desirable since there could be
pointers to such objects within the C++ world. This problem doesn't
normally occur since Python typically holds a pointer to the root node
as long as the simulator is running.

Prevent SimObject and Param deletion by using a PyBind-prescribed
unique_ptr with a dummy deleter as the pointer wrapper for the Python
world.

Change-Id: Ied14602c9ee69a083a69c5dae1b5fcf8efb4548a
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/3224
Reviewed-by: Gabe Black <gabeblack@google.com>

python: Fix weird memory issue in wrapped AddrRange vectors

There is a weird issue with the PyBind wrapper of
vector<AddrRange>. Assigning new values to a param that is a vector of
AddrRange sometimes results in an out-of-bounds memory access.

We work around this issue by treating AddrRange vectors as opaque
types. This slightly changes the semantics of the wrapper since Python
now manipulates the real object rather than a copy that has been
converted to a list.

Change-Id: Ie027c06e7a7262214b43b19a76b24fe4b20426c5
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Timothy Hayes <timothy.hayes@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/3223
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

misc: Expose the listener loopbackOnly function to python.

Change-Id: Ibb405af54a46a93706a6f476b5314491e84be0c8
Reviewed-on: https://gem5-review.googlesource.com/3081
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Remove SWIG

Remove SWIG-specific Python code.

Change-Id: If1d1b253d84021c9a8f9a64027ea7a94f2336dff
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2922
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>

python: Use PyBind11 instead of SWIG for Python wrappers

Use the PyBind11 wrapping infrastructure instead of SWIG to generate
wrappers for functionality that needs to be exported to Python. This
has several benefits:

* PyBind11 can be redistributed with gem5, which means that we have
full control of the version used. This avoid a large number of
hard-to-debug SWIG issues we have seen in the past.

* PyBind11 doesn't rely on a custom C++ parser, instead it relies on
wrappers being explicitly declared in C++. The leads to slightly
more boiler-plate code in manually created wrappers, but doesn't
doesn't increase the overall code size. A big benefit is that this
avoids strange compilation errors when SWIG doesn't understand
modern language features.

* Unlike SWIG, there is no risk that the wrapper code incorporates
incorrect type casts (this has happened on numerous occasions in
the past) since these will result in compile-time errors.

As a part of this change, the mechanism to define exported methods has
been redesigned slightly. New methods can be exported either by
declaring them in the SimObject declaration and decorating them with
the cxxMethod decorator or by adding an instance of
PyBindMethod/PyBindProperty to the cxx_exports class variable. The
decorator has the added benefit of making it possible to add a
docstring and naming the method's parameters.

The new wrappers have the following known issues:

* Global events can't be memory managed correctly. This was the
case in SWIG as well.

Change-Id: I88c5a95b6cf6c32fa9e1ad31dfc08b2e8199a763
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Andrew Bardsley <andrew.bardsley@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2231
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Reviewed-by: Pierre-Yves Péneau <pierre-yves.peneau@lirmm.fr>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>