stats: Add support for listing available formats

Add a command line option to list available stat formats and their
documentation.

Change-Id: I7f5f2272d9b0176639f59f2efedb9cab2f7da5b9
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19670
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

python: fix tracing after Python 3 refactor

gem5.opt --debug-flags ExecAll

was failing with:

ImportError: No module named defines

on Python 2.

Change-Id: I06959d6e0ccb7c661c7c749af3392632dac779f3
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18108
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

python: Enforce absolute imports for Python 3 compatibility

Change-Id: Ia88d7fd472f7aed9b97df81468211384981bf6c6
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15983
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

python: Make iterator handling Python 3 compatible

Many functions that used to return lists (e.g., dict.items()) now
return iterators and their iterator counterparts (e.g.,
dict.iteritems()) have been removed. Switch calls to the Python 2.7
iterator methods to use the Python 3 equivalent and add explicit list
conversions where necessary.

Change-Id: I0c18114955af8f4932d81fb689a0adb939dafaba
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15992
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Add missing defines import

The _check_tracing helper function in main.py depends on defines to
check if tracing has been enabled at compile time. This module is
imported in main() but not at the module level, which breaks this
function.

Change-Id: I26d65a4320da8618e0e552553695884fd2c880e0
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/16402
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Update use of exec to work with Python 3

Python 3 uses 'exec(code, globals)' instead of 'exec code in
globals'. Switch to the newer syntax since it is supported by Python
2.7. Also, move check_tracing out of main to work around a bug in
Python 2.7.

Change-Id: I6d390160f58783e1b038a572b64cdf3ff09535fa
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15986
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Switch to using open instead of file

Python 3 doesn't support the file(name, mode) syntax which has been
deprecated in favour of open.

Change-Id: I35ef8690d97a5243860a64ff985fd22fa86253f1
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15985
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Make exception handling Python 3 safe

Change-Id: I9c2cdfad20deb1ddfa224320cf93f2105d126652
Reviewed-on: https://gem5-review.googlesource.com/c/15980
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

scons: Switch from the print statement to the print function.

Starting with version 3, scons imposes using the print function instead
of the print statement in code it processes. To get things building
again, this change moves all python code within gem5 to use the
function version. Another change by another author separately made this
same change to the site_tools and site_init.py files.

Change-Id: I2de7dc3b1be756baad6f60574c47c8b7e80ea3b0
Reviewed-on: https://gem5-review.googlesource.com/8761
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>

python: Remove unused readline import

The readline module is imported by main.py but doesn't seem to be
used.

Change-Id: I7888e3b6ad0a0fedc14b0feec2adf0a39883bbf8
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/3421
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

misc: Add the --listener-loopback-only command line option.

This option invokes the Listener::loopbackOnly() static function which
will make the port listeners bind to the loopback device exclusively and
ignore connections on other devices. That prevents external agents like
port scanners from disrupting simulations with spurious connections.

Change-Id: I46b22165046792a6f970826c109bdbce7db25c84
Reviewed-on: https://gem5-review.googlesource.com/3082
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Automatically disable listeners in batch setups

Determine if gem5 is running in a batch environment by checking if
STDIN is wired to a TTY or not. If the simulator is running in a batch
environment, disable all listeners by default. This behavior can be
overridden using the --enable-listeners option.

Change-Id: I404c709135339144216bf08a2769c016c543333c
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sean McGoogan <sean.mcgoogan@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2322
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Add a generalized mechanism to configure stats

Add a mechanism to configure the stat output format using a URL-like
syntax. This makes it possible to specify both an output format
(currently, only text is supported) and override default
parameters.

On the Python-side, this is implemented using a helper function
(m5.stats.addStatVisitor) that adds a visitor to the list of active
stat visitors. The helper function parses a URL-like stat
specification to determine the stat output type. Optional parameters
can be specified to change how stat visitors behave.

For example, to output stats in text format without stat descriptions:

m5.stats.addStatVisitor("text://stats.txt?desc=False")

From the command line:

gem5.opt --stats-file="text://stats.txt?desc=False"

Internally, the stat framework uses the _url_factory decorator
to wrap a Python function with the fn(path, **kwargs) signature in a
function that takes a parsed URL as its only argument. The path and
keyword arguments are automatically derived from the URL in the
wrapper function.

New output formats can be registered in the m5.stats.factories
dictionary. This dictionary contains a mapping between format names
(URL schemes) and factory methods.

To retain backwards compatibility, the code automatically assumes that
the user wants text output if no format has been specified (i.e., when
specifying a plain path).

Change-Id: Ic4dce93ab4ead07ffdf71e55a22ba0ae5a143061
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Ilias Vougioukas <ilias.vougioukas@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>

base: eliminate ipython warning

Change-Id: I3e282baeb969b6bb9534813a2f433d68246c0669
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>

misc: Bail out of DVFS dot if we cannot resolve the domains

This changeset updates the dot output to bail out if it is unable to
resolve the voltage or clock domains (which will cause it to raise an
AttributeError). Additionally, the DVFS dot output is disabled by
default for speed purposes.

Minor fixup for 0aeca8f.

misc: Add secondary dot output for DVFS domains

This patch adds a secondary dot output file which shows the DVFS domains. This
has been done separately for now to avoid cluttering the already existing
diagram. Due to the way that the clock domains are assigned to components in
gem5, this output must be generated after the C++ objects have been
instantiated. This further motivates the need to generate this file separately
to the current dot output, and not to replace it entirely.

configs: add command-line option to stop debug output

This patch adds a --debug-end flag to main.py so that debug output can be
stoped at a specified tick, while allowing the simulation to continue. It is
useful in situations where you would like to produce a trace for a region of
interest while still collecting stats for the entire run. This is in contrast
to the currently existing --debug-break flag, which terminates the simulation
at the tick.

sim: fix redundant --debug-start help string

Just changes the metavar for --debug-start from TIME
to TICK in cset 72046b9b3323 and didn't notice that the
comment "must be in ticks" is now redundant.

sim: don't ignore SIG_TRAP

By ignoring SIG_TRAP, using --debug-break <N> when not connected to
a debugger becomes a no-op. Apparently this was intended to be a
feature, though the rationale is not clear.

If we don't ignore SIG_TRAP, then using --debug-break <N> when not
connected to a debugger causes the simulation process to terminate
at tick N. This is occasionally useful, e.g., if you just want to
collect a trace for a specific window of execution then you can combine
this with --debug-start to do exactly that.

In addition to not ignoring the signal, this patch also updates
the --debug-break help message and deletes a handful of unprotected
calls to Debug::breakpoint() that relied on the prior behavior.

sim: print pid in output header

This information is useful if you have a bunch of simulations running
and want to know which one to kill, but you've neglected to take
advantage of the previous patch and embed the pid in your output path.

misc: quote args in echoed command line

Currently if there are shell special characters in a
command-line argument, you can't copy and paste the
echoed command line onto a shell prompt because the
characters aren't quoted properly. This patch fixes
that problem.

misc: Fix -q (quiet) flag

Check the right flag.

sim: simulate with multiple threads and event queues
This patch adds support for simulating with multiple threads, each of
which operates on an event queue. Each sim object specifies which eventq
is would like to be on. A custom barrier implementation is being added
using which eventqs synchronize.

The patch was tested in two different configurations:
1. ruby_network_test.py: in this simulation L1 cache controllers receive
requests from the cpu. The requests are replied to immediately without
any communication taking place with any other level.
2. twosys-tsunami-simple-atomic: this configuration simulates a client-server
system which are connected by an ethernet link.

We still lack the ability to communicate using message buffers or ports. But
other things like simulation start and end, synchronizing after every quantum
are working.

Committed by: Nilay Vaish

sim: Clarify the difference between tracing and debugging

This patch changes the name the command-line options related to debug
output to all start with "debug" rather than being a mix of that and
"trace". It also makes it clear that the breakpoint time is specified
in ticks and not in cycles.

base: Add support for newer versions of IPython

IPython is used for the interactive gem5 shell if it exists. IPython
made API changes in version 0.11. This patch adds support for IPython
version 0.11 and above.

base: Fix broken IPython argument handling

Prior to this changeset, we used to clear sys.argv before entering the
IPython shell. This caused some versions of IPython to crash because
they assume argv[0] to exist. The correct way of overriding the
arguments passed to IPython is to set the argv keyword argument when
initializing the shell.

DOT: fixed broken code for visualizing configuration using dot

Fixed broken code which visualizes the system configuration by generating a
tree from each component's children, starting from root.
Requires DOT (hence pydot).

sim: Enable sampling of run-time for code-sections marked using pseudo insts.

This patch adds a mechanism to collect run time samples for specific portions
of a benchmark, using work_begin and work_end pseudo instructions.It also enhances
the histogram stat to report geometric mean.

config: support outputing a pickle of the configuration tree

copyright: Add code for finding all copyright blocks and create a COPYING file

The end of the COPYING file was generated with:
% python ./util/find_copyrights.py configs src system tests util

Update -C command line option to spit out COPYING file

Name: Replace M5 with gem5 in a few places it's printed on startup.

Trace: Remove the options trace-help and trace-flags
The options trace-help and trace-flags are no longer required. In there place,
the options debug-help and debug-flags have been provided.

fix some build problems from prior changesets

python: cleanup python code so stuff doesn't automatically happen at startup
this allows things to be overridden at startup (e.g. for tests)

trace: reimplement the DTRACE function so it doesn't use a vector
At the same time, rename the trace flags to debug flags since they
have broader usage than simply tracing. This means that
--trace-flags is now --debug-flags and --trace-help is now --debug-help

main: separate out interact() so it can be used by other functions

SCons: Stop embedding the mercurial revision into the binary.

This causes a lot of rebuilds that could have otherwise possibly been
avoided, and, more annoyingly, a lot of unnecessary rerunning of the
regressions. The benefits of having the revision in the output haven't
materialized, so this change removes it.

Info: Clean up some info files.

Get rid of RELEASE_NOTES since we no longer do releases, update some of the
information in README, and update the date in LICENSE.

python: use ipython in --interactive if it is available

Minor remote GDB cleanup.
Expand the help text on the --remote-gdb-port option so
people know you can use it to disable remote gdb without
reading the source code, and thus don't waste any time
trying to add a separate option to do that.
Clean up some gdb-related cruft I found while looking
for where one would add a gdb disable option, before
I found the comment that told me that I didn't need
to do that.

python: Move more code into m5.util allow SCons to use that code.
Get rid of misc.py and just stick misc things in __init__.py
Move utility functions out of SCons files and into m5.util
Move utility type stuff from m5/__init__.py to m5/util/__init__.py
Remove buildEnv from m5 and allow access only from m5.defines
Rename AddToPath to addToPath while we're moving it to m5.util
Rename read_command to readCommand while we're moving it
Rename compare_versions to compareVersions while we're moving it.

python: pull out common code from main that processes arguments

Trace: fix the --trace-start option

events: Make trace events happen at the right priority.
Also, while we're at it, remember that priorities are in the Event class
and add a disable method to disable tracing.

traceflags: fix --trace-help

SCons: Fix how we get Mercurial revision information since internals keep changing.

python: add fatal() function to the m5 package and use it

python: Try to isolate the stuff that's in the m5.internal package a bit more.

tracing: panic() if people try to use tracing, but TRACING_ON is not set.
Also clean things up so that help strings can more easily be added.
Move the help function into trace.py

output: Change default output directory and files and update tests.

traceflags: Make "All" a valid trace flag.

Sort trace flags before printing them.

eventq: convert all usage of events to use the new API.
For now, there is still a single global event queue, but this is
necessary for making the steps towards a parallelized m5.

pdb: Try to make pdb work better.
I've done a few things here. First, I invoke the script a little bit
differently so that pdb doesn't get confused. Second, I've stored the
actual filename in the module's __file__ so that pdb can find the
source file on your machine.

python: cleanup options parsing stuff so that it properly deals with defaults.
While we're at it, make it possible to run main.py in a somewhat
standalone mode again so that we can test things without compiling.

Make time format in 'started' line same as 'compiled'.
Also make -B output consistent with normal header, and
only include actual build options.

Add -r/-e options to redirect stdout/stderr.
Better than using shell since it automatically uses -d directory
for output files (creating it as needed).

RemoteGDB: add an m5 command line option for setting or disabling remote gdb.

Command line option to print out List of SimObjects and their parameters

main: add .m5/options.py processing. This file is processed before
arguments are parsed so that they can change the default options for
various config parameters.

Add .m5 configuration directory

python: Separate the options parsing stuff. Remove options parsing stuff from
main.py so things are a bit more obvious.

python: Move various utility classes into a new m5.util package so
they're all in the same place. This also involves having just one
jobfile.py and moving it into the utils directory to avoid
duplication. Lots of improvements to the utility as well.

HG: Add compiled hg revision and date to the standard M5 output.

Update copyright dates

main: expose the build information as a simple command line parameter

Move all of the parameters of the Root SimObject so they are
directly configured by python. Move stuff from root.(cc|hh) to
core.(cc|hh) since it really belogs there now.
In the process, simplify how ticks are used in the python code.

Factor code out of main.cc and main.i into a bunch of files
so things are organized in a more sensible manner. Take apart
finalInit and expose the individual functions which are now
called from python. Make checkpointing a bit easier to use.

Fix tracing so it starts right away if --trace-start is not
specified.

Get rid of the Statistics and Statreset ParamContexts, and
expose all of the relevant functionality to python. Clean
up the mysql code while we're at it.

Default to tracing being disabled in C++, it will be turned
on in python. Fix the trace start code so it actually starts
when it is suppsed to. Make the Exec tracing stuff obey the
trace enabled flag.

Merge all of the execution trace configuration stuff into
the traceflags infrastructure. InstExec is now just Exec
and all of the command line options are now trace options.

Rearrange traceflags.py so that the file generation only happens if
the script is invoked as main. This allows us to import traceflags.py
if we just want the list of available flags.
Embed traceflags.py into the zipfile so it can be accessed from the
python side of things. With this, print an error on invalid flags and
add --trace-help option that will print out the list of trace flags
that are compiled in. If a flag is prefixed with a '-', now that flag
will be disabled.

Clean up tracing stuff more, get rid of the trace log since
its not all that useful. Fix a few bugs with python/C++
integration.

Clean up from my last commit to the trace stuff.

Get rid of the Trace ParamContext and give python direct
access to enabling/disabling tracing. Command line is
unchanged except for the removal of --trace-cycle since
it's not so clear what that means.

set __name__ in the root m5 script to __m5_main__ so we can
tell if the script is run from m5 as the m5 script

Expose debugBreakCycle through swig and get rid of
the Debug param context

Create a module called internal where swigged stuff goes.
Rename cc_main to internal.main

add code to operate in lockstep with legion

src/python/m5/main.py:
add option to operate in lockstep with legion

Oops, forgot to assign the option to the param context.

Add in ability to start a trace at a specific cycle.

Started adding a system to output data after every instruction.

src/arch/alpha/regfile.hh:
src/arch/mips/regfile/float_regfile.hh:
src/arch/mips/regfile/int_regfile.hh:
src/arch/mips/regfile/misc_regfile.hh:
src/cpu/exetrace.hh:
Added functions to start to support dumping register values once per cycle.
src/cpu/exetrace.cc:
Added some code to support printing the value of registers after each cycle.
src/python/m5/main.py:
Options to turn on output after every instruction. They are commented out.

Output the command line.

src/python/m5/main.py:
Output the command line being used.

Merge ktlim@zizzer:/bk/newmem
into zamp.eecs.umich.edu:/z/ktlim2/clean/newmem-merge

configs/test/fs.py:
configs/test/test.py:
SCCS merged

Minor updates.

src/python/m5/config.py:
Formatting.
src/python/m5/main.py:
Slightly more useful output when you don't enter in a valid script file.

Merge zizzer:/bk/newmem
into zeep.pool:/z/saidi/work/m5.newmem

src/python/m5/main.py:
merge two help fixes

Fix help message printing. Might need to clean up the handling of the sys.exit() call, as right now it prints out "None" at the end (not sure why).

src/python/m5/main.py:
Fix help message printing.

fix help when no arguments are passed to m5

Add --pdb

src/python/m5/main.py:
Add a command line option to invoke pdb on your script

Fix __file__ for scripts

src/python/m5/main.py:
set __file__ to the script, not the m5 binary.

Fix option parsing.

src/python/m5/main.py:
Don't allow interspersed arguments, it messes things up

Migrate most of main() and and all option parsing to python

configs/test/fs.py:
configs/test/test.py:
update for the new way that m5 deals with options
src/python/SConscript:
Compile AUTHORS, LICENSE, README, and RELEASE_NOTES into the
python stuff.
src/python/m5/__init__.py:
redo the way options work.
Move them all to main.py
src/sim/main.cc:
Migrate more functionality for main() into python.
Namely option parsing
src/python/m5/attrdict.py:
A dictionary object that overrides attribute access to
do item access.
src/python/m5/main.py:
The new location for M5's option parsing, and the main()
routine to set up the simulation.