python: Don't try to bind a stat group to the NULL simobject.

That is a SimObject like object which is used when a SimObject
parameter is purposefully left empty, vs. being accidentally left
empty through a typo or accidental ommission.

It doesn't have a getCCObject method, and attempting to use it anyway
causes gem5 to crash.

Change-Id: Ie86321fbdbcc41cf88b7009184423acd7b64484b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/21059
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Anthony Gutierrez <anthony.gutierrez@amd.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

stats: Ignore non-Group objects in stat hierarchy

Some objects, such as SystemC modules, are not a subclass of
Stat::Group. Calling the addStatGroup function on them causes errors.
This changes ignores those objects that are not Stat::Group in the stat
hierarchy.

Signed-off-by: Chun-Chen TK Hsu <chunchenhsu@google.com>
Change-Id: I9b62419417b7af7331461fbfaf15e45a4ee2b35f
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/20680
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

stats: Add support for listing available formats

Add a command line option to list available stat formats and their
documentation.

Change-Id: I7f5f2272d9b0176639f59f2efedb9cab2f7da5b9
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19670
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

stats: Add beta support for HDF5 stat dumps

This changeset add support for stat dumps in the HDF5 file
format. HDF5 is a binary data format that represents data in a
file-system-like balanced tree. It has native support for
N-dimensional arrays and binary data (e.g., frame buffers).

It has the following benefits over traditional text stat files:

* Efficient storage of time series (multiple stat dumps)

* Fast lookup of stats

* Plenty of existing tooling (e.g., Python libraries and graphical
viewers)

* File format can be used to store frame buffers together with
normal stats.

Drawbacks:

* Large startup cost (single stat dump larger than text equivalent)

* Stat dumps are slower than text

Known limitations:

* Distributions and histograms aren't supported.

HDF5 stat output can be enabled using the 'h5' URL scheme when
overriding the stat file name on gem5's command line. The following
parameters are supported:

* chunking (unsigned): Number of time steps to pre-allocate
(default: 10)

* desc (bool): Output stat descriptions (default: True)

* formulas (bool): Output derived stats (default: True)

Example gem5 command line:

./build/ARM/gem5.opt \
--stats-file="h5://stats.h5?desc=False;formulas=False" \
configs/example/fs.py

Example Python stat consumer that computes IPC:
import h5py

f = h5py.File('stats.h5', 'r')
group = f['/system/cpu']
for i, c in zip(group['committedInsts'], group['numCycles']):
print i, c, i / c

Change-Id: I351c6cbff2fb7bef9012f47876ba227ed288975b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/8121
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Ciro Santilli <ciro.santilli@arm.com>

stats: Add support for partial stat dumps

Add support for partial stat dumps by passing an optional 'root'
keyword argument to m5.stats.dump(). Specifying root slightly changes
the semantics of the dump command. For legacy reasons, gem5 only
allows one stat dump per tick. This is likely a limitation introduced
as a hack to prevent automatic dumping at the end of simulation from
interfering with explicit dumping from a simulation script. This
restriction does not apply when specifying a root. However, these stat
dumps will still prevent an additional stat dump in the same tick with
an unspecified root.

N.B.: This new API /only/ works for new-style stats that have an
explicit hierarchy. Legacy stats will not be dumped if a root is
specified.

Change-Id: Idc8ff448b9f70a796427b4a5231e7371485130b4
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19369
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

stats: Add support for hierarchical stats

This change makes the stat system aware of the hierarchical nature of
stats. The aim is to achieve the following goals:

* Make the SimObject hierarchy explicit in the stat system (i.e.,
get rid of name() + ".foo"). This makes stat naming less fragile
and makes it possible to implement hierarchical formats like
XML/HDF5/JSON in a clean way.

* Make it more convenient to split stats into a separate
struct/class that can be bound to a SimObject. This makes the
namespace cleaner and makes stat accesses a bit more obvious.

* Make it possible to build groups of stats in C++ that can be used
in subcomponents in a SimObject (similar to what we do for
checkpoint sections). This makes it easier to structure large
components.

* Enable partial stat dumps. Some of our internal users have been
asking for this since a full stat dump can be large.

* Enable better stat access from Python.

This changeset implements solves the first three points by introducing
a class (Stats::Group) that owns statistics belonging to the same
object. SimObjects inherit from Stats::Group since they typically have
statistics.

New-style statistics need to be associated with a parent group at
instantiation time. Instantiation typically sets the name and the
description, other parameters need to be set by overriding
Group::regStats() just like with legacy stats. Simple objects with
scalar stats can typically avoid implementing regStats() altogether
since the stat name and description are both specified in the
constructor.

For convenience reasons, statistics groups can be merged into other
groups. This means that a SimObject can create a stat struct that
inherits from Stats::Group and merge it into the parent group
(SimObject). This can make the code cleaner since statistics tracking
gets grouped into a single object.

Stat visitors have a new API to expose the group structure. The
Output::beginGroup(name) method is called at the beginning of a group
and the Output::endGroup() method is called when all stats, and
sub-groups, have been visited. Flat formats (e.g., the text format)
typically need to maintain a stack to track the full path to a stat.

Legacy, flat, statistics are still supported after applying this
change. These stats don't belong to any group and stat visitors will
not see a Output::beginGroup(name) call before their corresponding
Output::visit() methods are called.

Change-Id: I9025d61dfadeabcc8ecf30813ab2060def455648
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19368
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>

python: Add fallbacks for packages that have been renamed

Python 3 has restructured some packages. Specifically, __builtin__ has
been renamed to builtins and urlparse has been included in urllib.

Change-Id: I81f8f3942471db1043006a36abbad6e5a49e0a43
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15994
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>

python: Switch from using compare to key in list sort

Python 3 has deprecated the use of a comparison function in favour of
a key extraction function.

Change-Id: I4b7eab791ecbdfbf7147f57fdbc7cbe8f1de20dd
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15995
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Add a generalized mechanism to configure stats

Add a mechanism to configure the stat output format using a URL-like
syntax. This makes it possible to specify both an output format
(currently, only text is supported) and override default
parameters.

On the Python-side, this is implemented using a helper function
(m5.stats.addStatVisitor) that adds a visitor to the list of active
stat visitors. The helper function parses a URL-like stat
specification to determine the stat output type. Optional parameters
can be specified to change how stat visitors behave.

For example, to output stats in text format without stat descriptions:

m5.stats.addStatVisitor("text://stats.txt?desc=False")

From the command line:

gem5.opt --stats-file="text://stats.txt?desc=False"

Internally, the stat framework uses the _url_factory decorator
to wrap a Python function with the fn(path, **kwargs) signature in a
function that takes a parsed URL as its only argument. The path and
keyword arguments are automatically derived from the URL in the
wrapper function.

New output formats can be registered in the m5.stats.factories
dictionary. This dictionary contains a mapping between format names
(URL schemes) and factory methods.

To retain backwards compatibility, the code automatically assumes that
the user wants text output if no format has been specified (i.e., when
specifying a plain path).

Change-Id: Ic4dce93ab4ead07ffdf71e55a22ba0ae5a143061
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Ilias Vougioukas <ilias.vougioukas@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>

python: Move native wrappers to the _m5 namespace

Swig wrappers for native objects currently share the _m5.internal name
space with Python code. This is undesirable if we ever want to switch
from Swig to some other framework for native binding (e.g., PyBind11
or Boost::Python). This changeset moves all of such wrappers to the
_m5 namespace, which is now reserved for native code.

Change-Id: I2d2bc12dbc05b57b7c5a75f072e08124413d77f3
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Don't use Swig to cast stats

Call the stat visitor from the stat itself rather than casting stats
in Python. This reduces the number of ways visitors are called.

Change-Id: Ic4d0b7b32e3ab9897b9a34cd22d353f4da62d738
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Joe Gross <joseph.gross@amd.com>

python: Export periodicStatDump

Some configuration scripts need periodic stat dumps. One of the ways
this can be achieved is by using the pariodicStatDump helper
function. This function was previously only exported in the internal
name space. Export it as a normal function in m5.stat instead.

Change-Id: Ic88bf1fd33042a62ab436d5944d8ed778264ac98
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>

config: Add a --without-python option to build process

Add the ability to build libgem5 without embedded Python or the
ability to configure with Python.

This is a prelude to a patch to allow config.ini files to be loaded
into libgem5 using only C++ which would make embedding gem5 within
other simulation systems easier.

This adds a few registration interfaces to things which cross
between Python and C++. Namely: stats dumping and SimObject resolving

stats: better error message for uninitialized statistic

As suggested by Nathan Binkert in 2008:
http://permalink.gmane.org/gmane.comp.emulators.m5.users/2676

stats: Provide a mechanism to get a callback when stats are dumped.

This mechanism is useful for dumping output that is correlated with stats
dumping, but isn't tracked by the gem5 statistics.

stats: track if the stats have been enabled and prevent requesting master id

Track the point in the initialization where statistics have been registered.
After this point registering new masterIds can no longer work as some
SimObjects may have sized stats vectors based on the previous value. If someone
tries to register a masterId after this point the simulator executes fatal().

stats: delete mysql support
we can add it back within python in some future changeset

stats: move code that loops over all stats into python

stats: better expose statistics to python.
Build a python list and dict of all stats and expose flags properly.