stats: Add support for hierarchical stats

This change makes the stat system aware of the hierarchical nature of
stats. The aim is to achieve the following goals:

* Make the SimObject hierarchy explicit in the stat system (i.e.,
get rid of name() + ".foo"). This makes stat naming less fragile
and makes it possible to implement hierarchical formats like
XML/HDF5/JSON in a clean way.

* Make it more convenient to split stats into a separate
struct/class that can be bound to a SimObject. This makes the
namespace cleaner and makes stat accesses a bit more obvious.

* Make it possible to build groups of stats in C++ that can be used
in subcomponents in a SimObject (similar to what we do for
checkpoint sections). This makes it easier to structure large
components.

* Enable partial stat dumps. Some of our internal users have been
asking for this since a full stat dump can be large.

* Enable better stat access from Python.

This changeset implements solves the first three points by introducing
a class (Stats::Group) that owns statistics belonging to the same
object. SimObjects inherit from Stats::Group since they typically have
statistics.

New-style statistics need to be associated with a parent group at
instantiation time. Instantiation typically sets the name and the
description, other parameters need to be set by overriding
Group::regStats() just like with legacy stats. Simple objects with
scalar stats can typically avoid implementing regStats() altogether
since the stat name and description are both specified in the
constructor.

For convenience reasons, statistics groups can be merged into other
groups. This means that a SimObject can create a stat struct that
inherits from Stats::Group and merge it into the parent group
(SimObject). This can make the code cleaner since statistics tracking
gets grouped into a single object.

Stat visitors have a new API to expose the group structure. The
Output::beginGroup(name) method is called at the beginning of a group
and the Output::endGroup() method is called when all stats, and
sub-groups, have been visited. Flat formats (e.g., the text format)
typically need to maintain a stack to track the full path to a stat.

Legacy, flat, statistics are still supported after applying this
change. These stats don't belong to any group and stat visitors will
not see a Output::beginGroup(name) call before their corresponding
Output::visit() methods are called.

Change-Id: I9025d61dfadeabcc8ecf30813ab2060def455648
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19368
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>

misc: Rename misc.(hh|cc) to logging.(hh|cc)

These files aren't a collection of miscellaneous stuff, they're the
definition of the Logger interface, and a few utility macros for
calling into that interface (panic, warn, etc.).

Change-Id: I84267ac3f45896a83c0ef027f8f19c5e9a5667d1
Reviewed-on: https://gem5-review.googlesource.com/6226
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>

style: [patch 1/22] use /r/3648/ to reorganize includes

base: Add total() to Vector2D stat

This patch adds a total() function to the Vector2D
stat type. Similar to other stats such as Scalar or
Vector it is useful to be able to read the total for
a given stat.

base: Add support for changing output directories

This changeset adds support for changing the simulator output
directory. This can be useful when the simulation goes through several
stages (e.g., a warming phase, a simulation phase, and a verification
phase) since it allows the output from each stage to be located in a
different directory. Relocation is done by calling core.setOutputDir()
from Python or simout.setOutputDirectory() from C++.

This change affects several parts of the design of the gem5's output
subsystem. First, files returned by an OutputDirectory instance (e.g.,
simout) are of the type OutputStream instead of a std::ostream. This
allows us to do some more book keeping and control re-opening of files
when the output directory is changed. Second, new subdirectories are
OutputDirectory instances, which should be used to create files in
that sub-directory.

Signed-off-by: Andreas Sandberg <andreas@sandberg.pp.se>
[sascha.bischoff@arm.com: Rebased patches onto a newer gem5 version]
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>

base: Clean up redundant string functions and use C++11

This patch does a bit of housekeeping on the string helper functions
and relies on the C++11 standard library where possible. It also does
away with our custom string hash as an implementation is already part
of the standard library.

stats: Fix flow-control bug in Vector2D printing

stats: add function for adding two histograms
This patch adds a function to the HistStor class for adding two histograms.
This functionality is required for Ruby. It also adds support for printing
histograms in a single line.

base: Fix VectorPrint initialisation

This patch changes how the initialisation of the VectorPrint struct is
done so that gcc 4.4 is happy again.

stats: Fix issue when printing 2D vectors

This patch addresses an issue with the text-based stats output which
resulted in Vector2D stats being printed without subnames in the event
that one of the dimensions was of length 1.

This patch also fixes the total printing for the 2D vector. Previously
totals were printed without explicitly stating that a total was being
printed. This has been rectified in this patch.

stats: Remove printing of SparseHist total

This patch removes the printing of the SparseHist total in the
stats.txt output file. This has been removed as a sparse histogram has
no total, and therefore this was printing out the value of a
non-local, unrelated variable.

stats: allow printing vectors on a single line
This patch adds a new flag to specify if the data values for a given vector
should be printed in one line in the stats.txt file. The default behavior
will be to print the data in multiple lines. It makes changes to print
functions to enforce this behavior.

scons: Enforce gcc >= 4.4 or clang >= 2.9 and c++0x support

This patch checks that the compiler in use is either gcc >= 4.4 or
clang >= 2.9. and enables building with --std=c++0x in all cases. As a
consequence, we can tidy up the hashmap and always have static_assert
available. If anyone wants to use alternative compilers, icc for
example supports c++0x to a similar level and could be added if
needed.

This patch opens up for a more elaborate use of c++0x features that
are present in gcc 4.4 and clang 2.9, e.g. auto typed variables,
variadic templates, rvalues and move semantics, and strongly typed
enums. There will be no going back on this one...

stats: use nan instead of no_value

clang/gcc: Fix compilation issues with clang 3.0 and gcc 4.6

This patch addresses a number of minor issues that cause problems when
compiling with clang >= 3.0 and gcc >= 4.6. Most importantly, it
avoids using the deprecated ext/hash_map and instead uses
unordered_map (and similarly so for the hash_set). To make use of the
new STL containers, g++ and clang has to be invoked with "-std=c++0x",
and this is now added for all gcc versions >= 4.6, and for clang >=
3.0. For gcc >= 4.3 and <= 4.5 and clang <= 3.0 we use the tr1
unordered_map to avoid the deprecation warning.

The addition of c++0x in turn causes a few problems, as the
compiler is more stringent and adds a number of new warnings. Below,
the most important issues are enumerated:

1) the use of namespaces is more strict, e.g. for isnan, and all
headers opening the entire namespace std are now fixed.

2) another other issue caused by the more stringent compiler is the
narrowing of the embedded python, which used to be a char array,
and is now unsigned char since there were values larger than 128.

3) a particularly odd issue that arose with the new c++0x behaviour is
found in range.hh, where the operator< causes gcc to complain about
the template type parsing (the "<" is interpreted as the beginning
of a template argument), and the problem seems to be related to the
begin/end members introduced for the range-type iteration, which is
a new feature in c++11.

As a minor update, this patch also fixes the build flags for the clang
debug target that used to be shared with gcc and incorrectly use
"-ggdb".

stats: fix Vector2d to display stats correctly when y_subname is not specified.

Vector2d stats with no y_subname were not displayed as the VectorPrint subname was not initialized correctly to reflect the empty field.

sim: Enable sampling of run-time for code-sections marked using pseudo insts.

This patch adds a mechanism to collect run time samples for specific portions
of a benchmark, using work_begin and work_end pseudo instructions.It also enhances
the histogram stat to report geometric mean.

Output: Add hierarchical output support and cleanup existing codebase.

Stats: Add a sparse histogram stat object.

stats: move code that loops over all stats into python

stats: add user settable separator string for arrayed stats

Default is '::', so no visible change unless it is overridden

includes: fix up code after sorting

includes: sort all includes

stats: Add a histogram statistic type

Make commenting on close namespace brackets consistent.

Ran all the source files through 'perl -pi' with this script:

s|\s*(};?\s*)?/\*\s*(end\s*)?namespace\s*(\S+)\s*\*/(\s*})?|} // namespace $3|;
s|\s*};?\s*//\s*(end\s*)?namespace\s*(\S+)\s*|} // namespace $2\n|;
s|\s*};?\s*//\s*(\S+)\s*namespace\s*|} // namespace $1\n|;

Also did a little manual editing on some of the arch/*/isa_traits.hh files
and src/SConscript.

stats: unify the two stats distribution type better

stats: cleanup a few small problems in stats

stats: rename print to display so it work in python

Stats: fix dist stat and enable VectorDistStat

stats: fancy is a bad name

stats: clean up the code for printing stats

stats: Move flags into info.hh and use base/flags.hh to manage the flags

stats: Shuffle around info stuff so it can be accessed separately

stats: Rename the info classes to hopefully make things a bit clearer
FooInfoBase became FooInfo
FooInfo became FooInfoProxy

stats: remove simplescalar compatibility for printing

stats: fix initialization bug in distribution text output

stats: cleanup text output stuff and fix mysql output

stats: stick the distribution's fancy parameter into the parameters structure.

stats: reorganize how parameters are stored and accessed.

stats: get rid of the convoluted 'database' code.
Just use the stuff directly and things ought to be more clear

stats: Try to make the names of things more intuitive.
Basically, this means renaming several things called data to info, which
is information about the statistics. Things that are named data now are
actual data stored for the statistic.

stats: fix text printout for distributions

stats: use properly signed types for looping and comparison

style: Bring statistics code in line with the proper style.

Output: Verify output files are open after opening them.

gcc: Add extra parens to quell warnings.
Even though we're not incorrect about operator precedence, let's add
some parens in some particularly confusing places to placate GCC 4.3
so that we don't have to turn the warning off. Agreed that this is a
bit of a pain for those users who get the order of operations correct,
but it is likely to prevent bugs in certain cases.

Get rid of those pesky valgrind warnings, Conditional jump or move depends on uninitialised value(s), in the stats package

Get rid of the Statistics and Statreset ParamContexts, and
expose all of the relevant functionality to python. Clean
up the mysql code while we're at it.

make our code a little more standards compliant
pretty close to compiling w/ suns compiler

briefly:
add dummy return after panic()/fatal()
split out flags by compiler vendor
include cstring and cmath where appropriate
use std namespace for string ops

SConstruct:
Add code to detect compiler and choose cflags based on detected compiler
Fix zlib check to work with suncc
src/SConscript:
split out flags by compiler vendor
src/arch/sparc/isa/decoder.isa:
use correct namespace for sqrt
src/arch/sparc/isa/formats/basic.isa:
add dummy return around panic
src/arch/sparc/isa/formats/integerop.isa:
use correct namespace for stringops
src/arch/sparc/isa/includes.isa:
include cstring and cmath where appropriate
src/arch/sparc/isa_traits.hh:
remove dangling comma
src/arch/sparc/system.cc:
dummy return to make sun cc front end happy
src/arch/sparc/tlb.cc:
src/base/compression/lzss_compression.cc:
use std namespace for string ops
src/arch/sparc/utility.hh:
no reason to say something is unsigned unsigned int
src/base/compression/null_compression.hh:
dummy returns to for suncc front end
src/base/cprintf.hh:
use standard variadic argument syntax instead of gnuc specefic renaming
src/base/hashmap.hh:
don't need to define hash for suncc
src/base/hostinfo.cc:
need stdio.h for sprintf
src/base/loader/object_file.cc:
munmap is in std namespace not null
src/base/misc.hh:
use M5 generic noreturn macros
use standard variadic macro __VA_ARGS__
src/base/pollevent.cc:
we need file.h for file flags
src/base/random.cc:
mess with include files to make suncc happy
src/base/remote_gdb.cc:
malloc memory for function instead of having a non-constant in an array size
src/base/statistics.hh:
use std namespace for floor
src/base/stats/text.cc:
include math.h for rint (cmath won't work)
src/base/time.cc:
use suncc version of ctime_r
src/base/time.hh:
change macro to work with both gcc and suncc
src/base/timebuf.hh:
include cstring from memset and use std::
src/base/trace.hh:
change variadic macros to be normal format
src/cpu/SConscript:
add dummy returns where appropriate
src/cpu/activity.cc:
include cstring for memset
src/cpu/exetrace.hh:
include cstring fro memcpy
src/cpu/simple/base.hh:
add dummy return for panic
src/dev/baddev.cc:
src/dev/pciconfigall.cc:
src/dev/platform.cc:
src/dev/sparc/t1000.cc:
add dummy return where appropriate
src/dev/ide_atareg.h:
make define work for both gnuc and suncc
src/dev/io_device.hh:
add dummy returns where approirate
src/dev/pcidev.hh:
src/mem/cache/cache_impl.hh:
src/mem/cache/miss/blocking_buffer.cc:
src/mem/cache/tags/lru.hh:
src/mem/cache/tags/split.hh:
src/mem/cache/tags/split_lifo.hh:
src/mem/cache/tags/split_lru.hh:
src/mem/dram.cc:
src/mem/packet.cc:
src/mem/port.cc:
include cstring for string ops
src/dev/sparc/mm_disk.cc:
add dummy return where appropriate
include cstring for string ops
src/mem/cache/miss/blocking_buffer.hh:
src/mem/port.hh:
Add dummy return where appropriate
src/mem/cache/tags/iic.cc:
cast hastSets to double for log() call
src/mem/physical.cc:
cast pmemAddr to char* for munmap
src/sim/byteswap.hh:
make define work for suncc and gnuc

Merge iceaxe.:/Volumes/work/research/m5/head
into iceaxe.:/Volumes/work/research/m5/merge

src/cpu/simple/base.cc:
src/kern/kernel_stats.cc:
src/kern/kernel_stats.hh:
src/kern/system_events.cc:
src/kern/system_events.hh:
src/python/m5/objects/System.py:
src/sim/system.cc:
src/sim/system.hh:
hand merge

Updated Authors from bk prs info

New directory structure:
- simulator source now in 'src' subdirectory
- imported files from 'ext' repository
- support building in arbitrary places, including
outside of the source tree. See comment at top
of SConstruct file for more details.
Regression tests are temporarily disabled; that
syetem needs more extensive revisions.

SConstruct:
Update for new directory structure.
Modify to support build trees that are not subdirectories
of the source tree. See comment at top of file for
more details.
Regression tests are temporarily disabled.
src/arch/SConscript:
src/arch/isa_parser.py:
src/python/SConscript:
Update for new directory structure.