stats: Add beta support for HDF5 stat dumps

This changeset add support for stat dumps in the HDF5 file
format. HDF5 is a binary data format that represents data in a
file-system-like balanced tree. It has native support for
N-dimensional arrays and binary data (e.g., frame buffers).

It has the following benefits over traditional text stat files:

* Efficient storage of time series (multiple stat dumps)

* Fast lookup of stats

* Plenty of existing tooling (e.g., Python libraries and graphical
viewers)

* File format can be used to store frame buffers together with
normal stats.

Drawbacks:

* Large startup cost (single stat dump larger than text equivalent)

* Stat dumps are slower than text

Known limitations:

* Distributions and histograms aren't supported.

HDF5 stat output can be enabled using the 'h5' URL scheme when
overriding the stat file name on gem5's command line. The following
parameters are supported:

* chunking (unsigned): Number of time steps to pre-allocate
(default: 10)

* desc (bool): Output stat descriptions (default: True)

* formulas (bool): Output derived stats (default: True)

Example gem5 command line:

./build/ARM/gem5.opt \
--stats-file="h5://stats.h5?desc=False;formulas=False" \
configs/example/fs.py

Example Python stat consumer that computes IPC:
import h5py

f = h5py.File('stats.h5', 'r')
group = f['/system/cpu']
for i, c in zip(group['committedInsts'], group['numCycles']):
print i, c, i / c

Change-Id: I351c6cbff2fb7bef9012f47876ba227ed288975b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/8121
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Ciro Santilli <ciro.santilli@arm.com>

stats: Add support for hierarchical stats

This change makes the stat system aware of the hierarchical nature of
stats. The aim is to achieve the following goals:

* Make the SimObject hierarchy explicit in the stat system (i.e.,
get rid of name() + ".foo"). This makes stat naming less fragile
and makes it possible to implement hierarchical formats like
XML/HDF5/JSON in a clean way.

* Make it more convenient to split stats into a separate
struct/class that can be bound to a SimObject. This makes the
namespace cleaner and makes stat accesses a bit more obvious.

* Make it possible to build groups of stats in C++ that can be used
in subcomponents in a SimObject (similar to what we do for
checkpoint sections). This makes it easier to structure large
components.

* Enable partial stat dumps. Some of our internal users have been
asking for this since a full stat dump can be large.

* Enable better stat access from Python.

This changeset implements solves the first three points by introducing
a class (Stats::Group) that owns statistics belonging to the same
object. SimObjects inherit from Stats::Group since they typically have
statistics.

New-style statistics need to be associated with a parent group at
instantiation time. Instantiation typically sets the name and the
description, other parameters need to be set by overriding
Group::regStats() just like with legacy stats. Simple objects with
scalar stats can typically avoid implementing regStats() altogether
since the stat name and description are both specified in the
constructor.

For convenience reasons, statistics groups can be merged into other
groups. This means that a SimObject can create a stat struct that
inherits from Stats::Group and merge it into the parent group
(SimObject). This can make the code cleaner since statistics tracking
gets grouped into a single object.

Stat visitors have a new API to expose the group structure. The
Output::beginGroup(name) method is called at the beginning of a group
and the Output::endGroup() method is called when all stats, and
sub-groups, have been visited. Flat formats (e.g., the text format)
typically need to maintain a stack to track the full path to a stat.

Legacy, flat, statistics are still supported after applying this
change. These stats don't belong to any group and stat visitors will
not see a Output::beginGroup(name) call before their corresponding
Output::visit() methods are called.

Change-Id: I9025d61dfadeabcc8ecf30813ab2060def455648
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19368
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>

base: Add GTest to SatCounter

Add a GTest to the SatCounter class.

Change-Id: Iaf1b18db9fe8d7fe32e0e40c7947dcd1fd6cc33b
Signed-off-by: Daniel <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17994
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

base: Iterable CircularQueue implementation

The former implementation of CircleBuf is functional but a bit too
tailored to match a use-case. This patches introduces a new iterable
circular queue, which adds some more functionality so it can also be
used for the newer LSQ implementation, where iteration and iterators
are a very desirable feature.

Additional contributors: Gabor Dozsa.

Change-Id: I5cfb95c8abc1f5e566a114acdbf23fc52a38ce5e
Signed-off-by: Giacomo Gabrielli <giacomo.gabrielli@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/13127
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>

base: Change the unit test binary names to use .test, and to be consistent.

Most tests were named *test where * was the base name of the file being
tested, but some were named differently based on, for instance, the
name of the class that file implemented.

This change makes all the test names consistently based off of the file
name they test, and also brings in the new .test convention to make
them easier to read.

Now, if you have a file like fiber.cc you want to test, you'd have a
unit test in a file called fiber.test.cc, and a test called fiber.test
which would generate a binary called fiber.test.opt, fiber.test.debug,
etc.

Change-Id: I61d59016090371a9bae72066e7473a34aecea21f
Reviewed-on: https://gem5-review.googlesource.com/c/14677
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>

base: Rename unit test cc files to be *.test.cc.

This makes the name easier to read, looks ok if the file is named with
underscores between words or not, is easy to grep for, and shouldn't
introduce any ambiguities in the file names.

Change-Id: I34b7bcccea2d87c10c0de417dd5e3ef27c4b5666
Reviewed-on: https://gem5-review.googlesource.com/c/14676
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Gabe Black <gabeblack@google.com>

tests: Convert IniFile unit test to a GTest

Change-Id: I47d6c9cbae21877420a15ffcf8489e3c26959139
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/14615
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

tests: Convert AddrRangeMap unit test to a GTest

Change-Id: Ifeb0b57c0cda77706691286f78325e50edb31c0d
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/13736
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

tests: Convert CircleBuf unit test to a GTest

Change-Id: I028c6b8d8e0ec06cac3d636689ae647f717096cd
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/13735
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

base: Fix isSubset() for addr ranges with interleaving

This change extends isSubset() which checks whether the range is a
subset of an input range to support address ranges with interleaving
and hashing.

Change-Id: I3dc9ceccb189b7c8665de0355f0555fc2c37d872
Reviewed-on: https://gem5-review.googlesource.com/12319
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

base: Add an asymmetrical Coroutine class

This patch is providing gem5 a Coroutine class to be used for
instantiating asymmetrical coroutines. Coroutines are built on top of
gem5 fibers, which makes them ucontext based.

Change-Id: I7bb673a954d4a456997afd45b696933534f3e239
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/11195
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

base: Add a class which encapsulates Fibers.

This class encapsulates the idea of a Fiber in such a way that other
implementations can be substituted in in the future. This
implementation uses the ucontext family of functions.

This change also adds a new unit test which exercises the new class. It
creates three new fibers which accept a sequence of other fibers to
switch to, one after the other. The main test function switches to
the these fibers which switch with each other and occasionally back to
the main fiber. Each time a test fiber is activated, it checks against
a list which shows the correct order for the fibers to run in. When the
main fiber gets control, it makes sure that list has been progressed
through by the correct amount.

Change-Id: I1fc2afa414b51baaa91e350a4ebc791d989f0b8a
Reviewed-on: https://gem5-review.googlesource.com/10935
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>

arm,sparc,x86,base,cpu,sim: Replace the Twin(32|64)_t types with.

Replace them with std::array<>s.

Change-Id: I76624c87a1cd9b21c386a96147a18de92b8a8a34
Reviewed-on: https://gem5-review.googlesource.com/6602
Maintainer: Gabe Black <gabeblack@google.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

tests: Turn fbtest into a gtest and move it to src/base.

Change-Id: I9ca57e24f27e0eb747d1f27262972a8abcd10fc8
Reviewed-on: https://gem5-review.googlesource.com/6342
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

tests: Move the cprintftest unit test into src/base.

That way it will live alongside the code it tests.

Change-Id: I00baad2206870a4619b7cee792a1d4c303dad04d
Reviewed-on: https://gem5-review.googlesource.com/6324
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

tests: Move the trietest unit test into base.

This puts it alongside trie.hh, the header file it tests.

Change-Id: Id8ca0c1d68bdc01807c5ba4b51c0142b1221385d
Reviewed-on: https://gem5-review.googlesource.com/6281
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>

tests: Add an implementation of the Logger interface for use gtests.

On exiting log types (panic and fatal), the message is set to an
ADD_FAILURE_AT macro, and the test is exited by throwing an otherwise
unexpected exception. On non-exiting log types, the message is sent to
the SUCCEEDED macro which currently doesn't output anything.

Change-Id: I1bb569e6cb8308dbc4c3e04eea7a962bd2b1ddd8
Reviewed-on: https://gem5-review.googlesource.com/6264
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>

base: Split out the pixel class in framebuffer.(cc|hh).

These are really two separate things. Also, while it's realitively
straightforward to write a unit test for the pixel conversion code, the
framebuffer object is serializable and brings in more dependencies.

Change-Id: If954caeb0bfedb1002cfb1a7a115a00c90d56d19
Reviewed-on: https://gem5-review.googlesource.com/6341
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

misc: Rename misc.(hh|cc) to logging.(hh|cc)

These files aren't a collection of miscellaneous stuff, they're the
definition of the Logger interface, and a few utility macros for
calling into that interface (panic, warn, etc.).

Change-Id: I84267ac3f45896a83c0ef027f8f19c5e9a5667d1
Reviewed-on: https://gem5-review.googlesource.com/6226
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>

tests: Move the bituniontest to be alongside the bitunion header.

Change-Id: I7c1a49c41672a1108fcf67c5505b0441f90588ef
Reviewed-on: https://gem5-review.googlesource.com/6142
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

base: Introducing utility for writing raw data in png format

Originally it was possible to use a Bitmap writer class for dumping a
framebuffer snapshot in a .bmp file. This patch enables you to choose
another format. In particular it implements the writing of PNG Images
using libpng library. The latter has to be already installed in your
machine, otherwise gem5 will default to the Bitmap format. This
configurable writer has been introduced in the VNC frame dumping mechanism,
which is storing changed frame buffers from the VNC server

Change-Id: Id7e5763c82235f1ce90381c8486b85a7cce734ce
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5181
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

base: Function for mirroring bits in variable length word

This patch introduces a high-speed template function for mirroring the
bits (MSB=>LSB) in a variable length word. The function is achieving
high performances since it is using a look-up table.

Change-Id: Ib0d0480e68d902f25655f74d243de305103eff75
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5261
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

base: Rewrite the CircleBuf to fix bugs and add serialization

The CircleBuf class has at least one bug causing it to overwrite the
wrong elements when wrapping. The current code has a lot of unused
functionality and duplicated code. This changeset replaces the old
implementation with a new version that supports serialization and
arbitrary types in the buffer (not just char).

base: Redesign internal frame buffer handling

Currently, frame buffer handling in gem5 is quite ad hoc. In practice,
we pass around naked pointers to raw pixel data and expect consumers
to convert frame buffers using the (broken) VideoConverter.

This changeset completely redesigns the way we handle frame buffers
internally. In summary, it fixes several color conversion bugs, adds
support for more color formats (e.g., big endian), and makes the code
base easier to follow.

In the new world, gem5 always represents pixel data using the Pixel
struct when pixels need to be passed between different classes (e.g.,
a display controller and the VNC server). Producers of entire frames
(e.g., display controllers) should use the FrameBuffer class to
represent a frame.

Frame producers are expected to create one instance of the FrameBuffer
class in their constructors and register it with its consumers
once. Consumers are expected to check the dimensions of the frame
buffer when they consume it.

Conversion between the external representation and the internal
representation is supported for all common "true color" RGB formats of
up to 32-bit color depth. The external pixel representation is
expected to be between 1 and 4 bytes in either big endian or little
endian. Color channels are assumed to be contiguous ranges of bits
within each pixel word. The external pixel value is scaled to an 8-bit
internal representation using a floating multiplication to map it to
the entire 8-bit range.

base: Use STL C++11 random number generation

This patch changes the random number generator from the in-house
Mersenne twister to an implementation relying entirely on C++11 STL.

The format for the checkpointing of the twister is simplified. As the
functionality was never used this should not matter. Note that this
patch does not actually make use of the checkpointing
functionality. As the random number generator is not thread safe, it
may be sensible to create one generator per thread, system, or even
object. Until this is decided the status quo is maintained in that no
generator state is part of the checkpoint.

base: Remove unused files

A bit of pruning

arch: Resurrect the NOISA build target and rename it NULL

This patch makes it possible to once again build gem5 without any
ISA. The main purpose is to enable work around the interconnect and
memory system without having to build any CPU models or device models.

The regress script is updated to include the NULL ISA target. Currently
no regressions make use of it, but all the testers could (and perhaps
should) transition to it.

loader: add a flattened device tree blob (dtb) object

this adds a dtb_object so the loader can load in the dtb
file for linux/android ARM kernels.

ruby: replace Time with Cycles in Message class
Concomitant changes are being committed as well, including the io operator<<
for the Cycles class.

AddrRange: Simplify Range by removing stream input/output

This patch simplifies the Range class in preparation for the
introduction of a more specific AddrRange class that allows
interleaving/striping.

The only place where the parsing was used was in the unit test.

sim: Remove FastAlloc

While FastAlloc provides a small performance increase (~1.5%) over regular malloc it isn't thread safe.
After removing FastAlloc and using tcmalloc I've seen a performance increase of 12% over libc malloc
when running twolf for ARM.

scons: rename TraceFlags to DebugFlags

stats: delete mysql support
we can add it back within python in some future changeset

stats: move code that loops over all stats into python

VNC: Add VNC server to M5

Delete unused files from src/base directory.

SCons: Support building without an ISA

stats: get rid of the never-really-used event stuff

callback: Make helper functions that create callback objects for you
clean up callback stuff a little bit while we're at it.

CPA: Add new object for gathering critical path annotations.

stats: get rid of the convoluted 'database' code.
Just use the stuff directly and things ought to be more clear

debug: Move debug_break into src/base

tracing: Add help strings for some of the trace flags

atomicio: provide atomic read and write functions.

These functions keep trying to read and write until all data has been
transferred, or an error occurrs. In the case where an end of file
hasn't been reached, but all of the bytes have not been read/written,
try again. On EINTR, try again.

Add in files from merge-bare-iron, get them compiling in FS and SE mode

Traceflags: Add SCons function to created a traceflag instead of having one file with them all.

Base: Rework the way M5 provides and creates random numbers.

only compile fenv.c if we're using fenv

create base/fenv.c to standerdize fenv across platforms. It's a c file and not a cpp file because c99
(which defines fenv) doesn't necessarily extend to c++ and it is a problem with solaris. If really
desired this could wrap the ieeefp interface found in bsd* as well, but I see no need at the moment.

src/arch/alpha/isa/fp.isa:
src/arch/sparc/isa/formats/basic.isa:
use m5_fesetround()/m5_fegetround() istead of fenv interface directly
src/arch/sparc/isa/includes.isa:
use base/fenv instead of fenv directly
src/base/SConscript:
add fenv to sconscript
src/base/fenv.hh:
src/base/random.cc:
m5 implementation to standerdize fenv across platforms.

Completely re-work how the scons framework incorporates swig
and python code into m5 to allow swig an python code to
easily added by any SConscript instead of just the one in
src/python. This provides SwigSource and PySource for
adding new files to m5 (similar to Source for C++). Also
provides SimObject for including files that contain SimObject
information and build the m5.objects __init__.py file.

Instead of creating a new python process to run traceflags.py,
just directly exec the file and generate the flags

Rework the way SCons recurses into subdirectories, making it
automatic. The point is that now a subdirectory can be added
to the build process just by creating a SConscript file in it.
The process has two passes. On the first pass, all subdirs
of the root of the tree are searched for SConsopts files.
These files contain any command line options that ought to be
added for a particular subdirectory. On the second pass,
all subdirs of the src directory are searched for SConscript
files. These files describe how to build any given subdirectory.
I have added a Source() function. Any file (relative to the
directory in which the SConscript resides) passed to that
function is added to the build. Clean up everything to take
advantage of Source().
function is added to the list of files to be built.