systemc: Make the verify.py script work when run from different dirs.

The verify.py script ran scons from the CWD, and that would fail if
there wasn't a SConstruct in that directory, ie if it wasn't from the
source of the checkout.

This change makes verify.py use scons' --directory option to run from
where the SConstruct is, or at least the SConstruct which was checked
out alongside that copy of verify.py. That location can be overridden
using the new -C or --scons-dir options.

Change-Id: I9f033d6dd30e0c2992b7f3102c573b34ea9c49e0
Reviewed-on: https://gem5-review.googlesource.com/c/16562
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>

misc: updated shabang for python script

The default python on MacOS doesn’t have an alias to python2.
The official python version supported in gem5 is Python2.7.

This patch updates the shabang according to the version required in gem5.

Change-Id: I9533c0f7858b5b3cab0ef101be1ee5cd718105b0
Reviewed-on: https://gem5-review.googlesource.com/c/15375
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>

systemc: Run the systemc tests directly from their build directories.

We were previously running them from the current directory to start
with, and then having the config script switch to the build directory.
That worked, except when output streams might be opened as part of the
global constructors which would run before the config script.

This change makes us start from the build directory directly, making
the switch in the config script unnecessary and ensuring that no files
leak outside of the build when running tests.

Change-Id: I484168793bfc5abc4e5631fb3468733fb9d829af
Reviewed-on: https://gem5-review.googlesource.com/c/14519
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Run the systemc tests with the CWD in the build directory.

This will prevent some (but not all) output files the tests generate
from ending up outside the build directory. Because some output file
streams are constructed as global objects, their paths are resolved
relative to the CWD when gem5 starts, before the config script has a
chance to change it.

Subsequent changes will make verify.py should make gem5 start with the
correct working directory, cleaning up the remaining leaking files.

Change-Id: I75a1256719dab4c98ab868c209d09b9dcdabb458
Reviewed-on: https://gem5-review.googlesource.com/c/14518
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Stop filtering out warnings about unimplemented features.

These no longer cause any of the tests to break, so there's no reason
to filter them out.

Change-Id: I9c0b25fc42eb3060ac7d6d6a46ded130227c302d
Reviewed-on: https://gem5-review.googlesource.com/c/13396
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Filter out an Info about delta cycles and pseudo timesteps.

In the Accellera implementation, every time a delta cycle is traced a
check is done to see if the user has been told what the pseudo timestep
is. To avoid doing that check over and over, we'll leave that out and
tell the verify.py to ignore that message in the reference output.

Change-Id: I825f05394dccf03e951d29561a11c3cc6d4bcda7
Reviewed-on: https://gem5-review.googlesource.com/c/12969
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py how to verify vcd files.

The reference output skips the first 7 lines which have volatile info
like the current time.

Change-Id: I9c173ff3903982a07349ca6957ab25e07bdf8e54
Reviewed-on: https://gem5-review.googlesource.com/c/12824
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Fix a typo in verify.py.

In the generic (non-console output) output checker, a variable should
have been called self.test, but was called self.text.

Change-Id: I2518d6ff01c51fc195eef2b61e987d1b3104c89b
Reviewed-on: https://gem5-review.googlesource.com/c/12817
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Ignore a gem5 warning about setting the stack size in verify.py.

This warning shouldn't make a test fail, but it's still useful to keep
around.

Change-Id: I9ebdbec804e11445edb82fa824ee0a6bce5943b0
Reviewed-on: https://gem5-review.googlesource.com/c/12812
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Centralize module callbacks and report new warnings.

By centralizing module callbacks, the gem5 module class knows when
different stages of the simulation are happening and can do it's own
extra checks. It also compartmentalizes modules more since the kernel
object doesn't have to reach into them to enumerate ports and exports.

Change-Id: I55887284af9c05150fe9d054f5b6147cad6092a1
Reviewed-on: https://gem5-review.googlesource.com/c/12610
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Update some defaults in verify.py.

Change the defaults in verify.py to be more like how I've been running
it. This is as close of an approximation as I have to how someone else
would want to run it manually. When run as part of a script, it's less
cumbersome to have to add extra arguments.

Change-Id: Ibd7c7168a38aa5c014ab5c1246c9617c7358e4f9
Reviewed-on: https://gem5-review.googlesource.com/c/12602
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Merge stderr and stdout when running tests.

The golden output for tests doesn't distinguish between stderr and
stdout, and by only comparing against stdout we have to throw away
errors which would be good to verify we get right. Also the tests
sometimes send output to stderr for no apparent reason, requiring
manually patching the tests.

This change adds filters for two messages which used to go to stderr
in gem5 but now show up in the diffs, one that just says the simulation
is starting, and the other for warns of unimplemented functionality.
The second warning should be turned on at some point so we make sure
everything the tests touch works and they don't just work by
coincidence, but for now it introduces a lot of noise among otherwise
passing tests.

Change-Id: I3b14f7807af561a79d6e0ca87aff1ab6051be596
Reviewed-on: https://gem5-review.googlesource.com/c/12600
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py to filter messages which start within a line.

It's possible for a test to generate a warning or error in the middle
of a line of output. The previous filter generator function would
create a filter which would only detect those messages which started
at the beginning of a new line.

Change-Id: I40372dc33049df84f3111e4d63a6619db97dcaa3
Reviewed-on: https://gem5-review.googlesource.com/c/12597
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py to ignore two types of errors.

Change-Id: I9e61a034d7f71bc9b1f28cb976ae8b17d6f37612
Reviewed-on: https://gem5-review.googlesource.com/c/12465
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py to ignore a new error.

Change-Id: Id967719803b5b306792c9fe6e6ddd36c36e09a88
Reviewed-on: https://gem5-review.googlesource.com/c/12460
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py to ignore "In file: *" in test output.

Those lines are sensitive to the paths to the test files and are even
redacted in the golden reference output, presumably for that reason.

Change-Id: I9fbd94c1b6d9d4e76397e84a4175d326f27b6e4d
Reviewed-on: https://gem5-review.googlesource.com/c/12451
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py filter out some error messages.

Some of the details of these messages would be annoying to match
exactly, and the error messages in gem5 go to simerr which isn't
being checked.

Change-Id: If80b124dd99987e205ccaf81d313d35df4191252
Reviewed-on: https://gem5-review.googlesource.com/c/12449
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py how to expect failing error codes.

Some tests expect to fail. For those tests (and only those tests) we
need to tell verify.py that it's ok if their exit status isn't 0. Also
if those tests *don't* fail, then that will also be flagged as an
error.

This is done by adding an expected_returncode file into the test's
source directory which holds what the expected return code should be.

Change-Id: I239a28e1d98dd3f76b71028660e492f675a0b3cb
Reviewed-on: https://gem5-review.googlesource.com/c/12446
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Abort verify.py if no tests were selected.

The various phases assume there are at least some tests, and if there
are none they may try to run malformed commands.

Change-Id: I041d35c504da57b830c490651ab1b3c98e0288ca
Reviewed-on: https://gem5-review.googlesource.com/12273
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py run the tests from the directory they expect.

Change-Id: I4c902cd81f7e46f81f601cae0ff2da044ef48f85
Reviewed-on: https://gem5-review.googlesource.com/12272
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py to ignore another deprecation warning.

Change-Id: I4c45431abebafd69f485f9d740da30f4b957f609
Reviewed-on: https://gem5-review.googlesource.com/12207
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Generalize ignoring info messages in the reference output.

Those messages are very implementation specific and don't (generally)
affect correctness. This makes it easier to ignore info messages based
on their number.

This change also makes the output checker ignore a similarly styled
message gem5 generates. We should consider making gem5 not generate
that message and have it generate another message instead which is
specific to gem5. We would need to filter that out too when comparing
results.

Change-Id: I93b9e2d547b6259512db091cfc557d21f86f4a3d
Reviewed-on: https://gem5-review.googlesource.com/12086
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Ignore a warning in the reference test outputs.

The warning is about deprecated sc_module constructors which don't take
an sc_module_name.

Change-Id: I2ef864a5bdac93eb8104a842179ffe45a8335085
Reviewed-on: https://gem5-review.googlesource.com/12068
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Always set USE_SYSTEMC=1 when compiling with verify.py.

It doesn't make sense not to have systemc enabled when building the
systemc regression tests.

Change-Id: I23b81fd4d7cbd0dbf5efbea773d816296d6492be
Reviewed-on: https://gem5-review.googlesource.com/12065
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py ignore an Info: message.

This message is about how to disable a warning about sc_bit being
deprecated.

Change-Id: I1a0b12e2ca5b58328e37605ba3f7e3e1a384e351
Reviewed-on: https://gem5-review.googlesource.com/12064
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py ignore another warning.

Also make warning filters more general and easier to add by writing a
function to generate them.

Change-Id: I8d813ded9ad8a9ccac4e48e5ea80a3c9c23959da
Reviewed-on: https://gem5-review.googlesource.com/12061
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py compare non output log reference files.

There are only a few of these which are vcd files. If there are
reference files which aren't the log and which aren't in the gem5
output directory, mark those tests as failed as well.

Change-Id: I2c880c13d0f90ccf16ac0439dbac68de9223cc90
Reviewed-on: https://gem5-review.googlesource.com/12060
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py delete obsolete diff files.

If diff file exists but the underlying diff has been fixed, delete the
diff file.

Change-Id: Icadc21a61c084198a8a246ab6d00a9b885647cde
Reviewed-on: https://gem5-review.googlesource.com/12056
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make verify.py ignore a warning.

Change-Id: Ia887b7195d84d0ba9e77370d7b928e201ca78731
Reviewed-on: https://gem5-review.googlesource.com/12055
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Teach verify.py to diff files when checking test results.

Currently it just diffs the stdout and ignores other reference files.
It also doesn't filter out noise in the diffs from non test related
simulator messages. These include startup messages, messages when the
simulator finishes executing, and some non-standard warnings, etc.

Change-Id: Idcb19edd893cd8818423c2c5ebb6cbfb278baffa
Reviewed-on: https://gem5-review.googlesource.com/12054
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Add return code verification to verify.py.

This just checks whether gem5 exited of its own accord with a
successful error code, or in other words that it didn't hang or crash.
More checking will need to be added to verify the output against the
golden reference.

Change-Id: I1ddef56aa73b5f700743830bd6212804531c484f
Reviewed-on: https://gem5-review.googlesource.com/12053
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Record the test's return code in the output directory.

verify.py will be able to use that value later to determine whether a
test was successful.

Change-Id: Ib499a4b6f202ee46e66e5de1b61a7e6309df9886
Reviewed-on: https://gem5-review.googlesource.com/12052
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Extend the execute phase of verify.py.

Add -j and --timeout options to the execute phase of verify.py.

The --timeout option is implemented using the timeout utility program
which is assumed to be available on the host system. Python 3.3 added
a timeout argument to the subprocess module which is an alternative
approach, but then we would be dependent on python 3.3.

-j is implemented using the standard multiprocess.pool.ThreadPool
class.

Change-Id: I15b92f2b14de6710e2027a6a19984b2644b2a8df
Reviewed-on: https://gem5-review.googlesource.com/12051
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Make the verify.py --list option print a total count.

Make the --list option of verify.py print a count of how many tests it
printed. Because --list respects the filter options, this is an easy
way to check how many tests have certain properties.

Change-Id: I03fac349a946631c20c8e6b49e0ad8934872898e
Reviewed-on: https://gem5-review.googlesource.com/11288
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Add a filter-file option.

The --filter option is very useful, but can get to be long and complex
and a bit too cumbersome to use from the command line. This change
adds a --filter-file option which is mutually exclusive with --filter
and which reads the filter expression from a file instead of accepting
it on the command line.

Change-Id: I381c92ddf0d9fe62acd20432fa4868e2121405b8
Reviewed-on: https://gem5-review.googlesource.com/11257
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Fix a small bug in verify.py.

The src_dir method of the Test class was using the wrong attribute of
the test from the json file. It should use path, but was using src_dir.

Change-Id: Iaaaf31c31b11b68ecd7dd98807e6b78597cebb99
Reviewed-on: https://gem5-review.googlesource.com/12029
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

systemc: Add a script which automates building and running tests.

The only required option for the script is the path to a build
directory, for instance build/ARM. It uses that when running scons and
when referring to generated files (like the built versions of the
tests). It uses the location of the script itself to find source files,
like the "golden" version of outputs.

This script runs tests in three phases, compilation, running, and
then verification. By default, all three phases are run for all tests,
but which phases run and how they work can be customized by specifying
them explicitly on the command line with the --phase option.

Each time the --phase option shows up on the command line, it should
be followed by the name of the phase being specified. That both
declares that that phase should be run, and also that any subsequent
options apply to that phase, at least until the next --phase option.

Currently, only the "compile" and "execute" phases are implemented, and
only the "compile" phase has been tested at all or has any options
defined. The "compile" phase simply takes all the options it was given
and pass them on to scons. The "verify" phase simply prints a message
which says what it would have verified.

The script can list the available tests with the --list option, as
determined by the json systemc test manifest file which scons can
generate, and which the script can refresh if passed the --update-json
option. You can also specify what "flavor" of build you want to do, ie
opt, debug, fast, etc., with the --flavor option. Only one flavor can
be tested at a time, but that should be the desired behavior most of
the time. If multiple flavors need to be tested, the script can be
invoked multiple times.

Finally, there is a --filter option which accepts a python expression
as a string. That expression is evaluated in the namespace of the
properties of each test in the json file, and if it returns true then
that test is run. For instance, to verify only compile_only tests, you
would use:

systemc/tests/verify.py build/ARM --filter 'compile_only'

To test only compile_only tests named bob, you would run this:

systemc/tests/verify.py build/ARM --filter \
'compile_only && name ="bob"'

Also included is a simple config.py which is the beginning of a config
which will run the systemc regression tests. Right now, all it knows
how to do is run sc_main.

Change-Id: I62666be8b1622d1355153e623b4274a939507e44
Reviewed-on: https://gem5-review.googlesource.com/10975
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>