python: Make the dot writer handle unconnected Port vector elements.

Change-Id: I5234013716cdcce5fc39e239dc3d92cd1f2b7799
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/20699
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>

python: FdtState using interrupt-cells

Change-Id: I37813748f518b442d2b53c2bc4f381edb2e26146
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/20050
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

python: Add support for exporting static class methods from c++

This change adds support for exporting static methods in a c++
SimObject from the coressponsing python wrapper class. This will allow
us to define and use c++ methods without the need to instantiate an
object of the corresponding class.

Change-Id: Iaf24c1aa6f20feb5c91241f46ec8db005a6a0c0c
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19168
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

misc: dot_writer fixup

In large configs the tooltip may be greater then the maximum line
size graphviz supports when parsing the dot file (typically 16k).
Adding '/' causes graphviz to break the string in multiple lines
while parsing and works around this limitation.

Change-Id: I16a0030127de4165080de97f5213309eed9fdeca
Signed-off-by: Tiago Mück <tiago.muck@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19208
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

misc: Added dot_writer for Ruby's network topology

Change-Id: Ic71ca7bc2eb4174d70afa368bc9cc987f3df89e9
Signed-off-by: Tiago Muck <tiago.muck@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17548
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

python: Generalize the dot_writer to handle non Master/Slave roles.

Change-Id: I6d93c28e754c0cacacdd5e8885c45bc861135e94
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18169
Maintainer: Gabe Black <gabeblack@google.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

python: Teach PyBindMethod how to set return_value_policy.

Change-Id: Ia208e43672672556b36f905e8f71dce44b978d22
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17033
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Stop using basestring to test for strings

The base class basestring doesn't exist in Python 3. Use string_types
from six instead.

Change-Id: I7e84903fb7dd4a0af7ae4e9f4ec2e54338f212bb
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15998
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>

python: Add Python 3 workarounds for long

Python 3 doesn't have a separate long type. Make long an alias for int
where needed to maintain compatibility.

Change-Id: I4c0861302bc3a2fa5226b3041803ef975d29b2fd
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15988
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>

python: Enforce absolute imports for Python 3 compatibility

Change-Id: Ia88d7fd472f7aed9b97df81468211384981bf6c6
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15983
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>

python: Add fallbacks for packages that have been renamed

Python 3 has restructured some packages. Specifically, __builtin__ has
been renamed to builtins and urlparse has been included in urllib.

Change-Id: I81f8f3942471db1043006a36abbad6e5a49e0a43
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15994
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>

python: Make iterator handling Python 3 compatible

Many functions that used to return lists (e.g., dict.items()) now
return iterators and their iterator counterparts (e.g.,
dict.iteritems()) have been removed. Switch calls to the Python 2.7
iterator methods to use the Python 3 equivalent and add explicit list
conversions where necessary.

Change-Id: I0c18114955af8f4932d81fb689a0adb939dafaba
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15992
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Add __bool__ helpers in addition to __nonzero__

Python 3 uses __bool__ instead of __nonzero__ when performing a
Boolean comparison.

Change-Id: I85185bbe136ecae67346fa23569e24edd7329222
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15996
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>

config: Make parameter conversion handle integers in other bases.

Python's float() function/type can't handle hexadecimal notation, but
int() can. Since there are also cases where converting to a float and
then back to an int (or long) can cause rounding error, this change
splits toFloat and toInteger apart and makes them call a worker
function which accepts a conversion function which does the work of
converting a numeric string into an actual number.

in the case of toFloat, it still uses the standard float(), and in the
case of toInteger it uses a lambda which wraps int(x, 0).

Change-Id: Ic46cf4ae86b7eba6f55d731d1b25e3f84b8bb64c
Reviewed-on: https://gem5-review.googlesource.com/c/16504
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Replace deprecated repr syntax

Change-Id: I5f9538cf2ca5ee17c51e7c5388d3aef363fcfa54
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15989
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Replace dict.has_key with 'key in dict'

Python 3 has removed dict.has_key in favour of 'key in dict'.

Change-Id: I9852a5f57d672bea815308eb647a0ce45624fad5
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15987
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Replace orderdict with collections.OrderedDict

Python 2.7 and newer has support for ordered dictionaries in the
standard library. Remove this custom class.

Change-Id: I4b720405aa3c4ce8d5c0b401eefe744a85ac3a3e
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/16362
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Update use of exec to work with Python 3

Python 3 uses 'exec(code, globals)' instead of 'exec code in
globals'. Switch to the newer syntax since it is supported by Python
2.7. Also, move check_tracing out of main to work around a bug in
Python 2.7.

Change-Id: I6d390160f58783e1b038a572b64cdf3ff09535fa
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15986
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Switch to using open instead of file

Python 3 doesn't support the file(name, mode) syntax which has been
deprecated in favour of open.

Change-Id: I35ef8690d97a5243860a64ff985fd22fa86253f1
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15985
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Make exception handling Python 3 safe

Change-Id: I9c2cdfad20deb1ddfa224320cf93f2105d126652
Reviewed-on: https://gem5-review.googlesource.com/c/15980
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>

python: Fixup incorrect syntax in PyBind argument handler

Change-Id: Ie81104d89b554795ec1020d5ce4edcf28795eda8
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/11511
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

scons: Switch from the print statement to the print function.

Starting with version 3, scons imposes using the print function instead
of the print statement in code it processes. To get things building
again, this change moves all python code within gem5 to use the
function version. Another change by another author separately made this
same change to the site_tools and site_init.py files.

Change-Id: I2de7dc3b1be756baad6f60574c47c8b7e80ea3b0
Reviewed-on: https://gem5-review.googlesource.com/8761
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>

arm: DT autogeneration - Device Tree generation methods

This patch adds an extra layer to the pyfdt library such that usage
gets easier and device tree nodes can be specified in less code,
without limiting original usage. Note to not import both the pyfdt
and fdthelper in the same namespace (but generally fdthelper is all
you need, because it supplies the same classes even when they are not
extended in any way)

Also, this patch lays out the primary functionality for generating a
device tree, where every SimObject gets an empty generateDeviceTree
method and ArmSystems loop over their children in an effort to merge
all the nodes. Devices are implemented in other patches.

Change-Id: I4d0a0666827287fe42e18447f19acab4dc80cc49
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5962
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

util: Add a "toEnergy" function to the convert module.

Change-Id: I7299af0e2a6ce9bd2272d6ccb898997336e95e51
Reviewed-on: https://gem5-review.googlesource.com/5623
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

util: Simplify/consolidate the python conversion module.

The python conversion module was really repetitive and fragmented,
where some types of conversions use common code, and some use hand
written case statements which did something very similar. Also, some
types like Voltage could only handle V and mV but no other scaling
prefix.

This change restructures the module to centralize a lot of the unit
handling code into toFloat, and makes the various other functions use
it.

Change-Id: Ic8529203cc226c9b551b8535a444e3f2f25ad1eb
Reviewed-on: https://gem5-review.googlesource.com/5621
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

util: Make dot_writer ignore NULL simobjects.

Also centralize the code which iterates over the parameters of a simnode's
children, and remove a copy/paste block of code in the DVFS block of code.

Change-Id: I01305fbff59346010780ee386ba97ad9106b1f5d
Reviewed-on: https://gem5-review.googlesource.com/4849
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>

python: Use PyBind11 instead of SWIG for Python wrappers

Use the PyBind11 wrapping infrastructure instead of SWIG to generate
wrappers for functionality that needs to be exported to Python. This
has several benefits:

* PyBind11 can be redistributed with gem5, which means that we have
full control of the version used. This avoid a large number of
hard-to-debug SWIG issues we have seen in the past.

* PyBind11 doesn't rely on a custom C++ parser, instead it relies on
wrappers being explicitly declared in C++. The leads to slightly
more boiler-plate code in manually created wrappers, but doesn't
doesn't increase the overall code size. A big benefit is that this
avoids strange compilation errors when SWIG doesn't understand
modern language features.

* Unlike SWIG, there is no risk that the wrapper code incorporates
incorrect type casts (this has happened on numerous occasions in
the past) since these will result in compile-time errors.

As a part of this change, the mechanism to define exported methods has
been redesigned slightly. New methods can be exported either by
declaring them in the SimObject declaration and decorating them with
the cxxMethod decorator or by adding an instance of
PyBindMethod/PyBindProperty to the cxx_exports class variable. The
decorator has the added benefit of making it possible to add a
docstring and naming the method's parameters.

The new wrappers have the following known issues:

* Global events can't be memory managed correctly. This was the
case in SWIG as well.

Change-Id: I88c5a95b6cf6c32fa9e1ad31dfc08b2e8199a763
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Andrew Bardsley <andrew.bardsley@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2231
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Reviewed-by: Pierre-Yves Péneau <pierre-yves.peneau@lirmm.fr>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Automatically disable listeners in batch setups

Determine if gem5 is running in a batch environment by checking if
STDIN is wired to a TTY or not. If the simulator is running in a batch
environment, disable all listeners by default. This behavior can be
overridden using the --enable-listeners option.

Change-Id: I404c709135339144216bf08a2769c016c543333c
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sean McGoogan <sean.mcgoogan@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2322
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

misc: Bail out of DVFS dot if we cannot resolve the domains

This changeset updates the dot output to bail out if it is unable to
resolve the voltage or clock domains (which will cause it to raise an
AttributeError). Additionally, the DVFS dot output is disabled by
default for speed purposes.

Minor fixup for 0aeca8f.

misc: Add secondary dot output for DVFS domains

This patch adds a secondary dot output file which shows the DVFS domains. This
has been done separately for now to avoid cluttering the already existing
diagram. Due to the way that the clock domains are assigned to components in
gem5, this output must be generated after the C++ objects have been
instantiated. This further motivates the need to generate this file separately
to the current dot output, and not to replace it entirely.

style: Refactor the style checker as a Python package

Refactor the style checker into a Python module that can be reused by
command line tools that integrate with git. In particular:

* Create a style package in util
* Move style validators from style.py to the style/validators.py.
* Move style verifiers from style.py to the style/verifiers.py.
* Move utility functions (sort_includes, region handling,
file_types) into the style package
* Move generic code from style.py to style/style.py.

Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Steve Reinhardt <steve.reinhardt@amd.com>

style: remove trailing whitespace

Result of running 'hg m5style --skip-all --fix-white -a'.

config: Add Current as a parameter type

This patch adds the Python parameter type Current, which is used for
the DRAM power modelling (to start with). With this addition we avoid
implicit unit assumptions.

mem: Rename Bus to XBar to better reflect its behaviour

This patch changes the name of the Bus classes to XBar to better
reflect the actual timing behaviour. The actual instances in the
config scripts are not renamed, and remain as e.g. iobus or membus.

As part of this renaming, the code has also been clean up slightly,
making use of range-based for loops and tidying up some comments. The
only changes outside the bus/crossbar code is due to the delay
variables in the packet.

misc: Proper type check and import for PortRef

Rewriting the type checking around PortRef, which was interacting strangely
with other Python scripts.

Tested-by: stephan.diestelhorst@arm.com

util: Add ini string as tooltip info in dot output

This patch adds the config ini string as a tooltip that can be
displayed in most browsers rendering the resulting svg. Certain
characters are modified for HTML output.

Tested on chrome and firefox.

util: Add colours to the dot output

This patch is adding a splash of colour to the dot output to make it
easier to distinguish objects of different types. As a bonus, the
pastel-colour palette also makes the output look like a something from
the 21st century.

util: Add class name to dot graph and output to svg

This patch adds the class name to the label, creates some more space
by increasing the rank separation, and additionally outputs the graph
as an editable SVG in addition to the PDF.

power: Add voltage domains to the clock domains

This patch adds the notion of voltage domains, and groups clock
domains that operate under the same voltage (i.e. power supply) into
domains. Each clock domain is required to be associated with a voltage
domain, and the latter requires the voltage to be explicitly set.

A voltage domain is an independently controllable voltage supply being
provided to section of the design. Thus, if you wish to perform
dynamic voltage scaling on a CPU, its clock domain should be
associated with a separate voltage domain.

The current implementation of the voltage domain does not take into
consideration cases where there are derived voltage domains running at
ratio of native voltage domains, as with the case where there can be
on-chip buck/boost (charge pumps) voltage regulation logic.

The regression and configuration scripts are updated with a generic
voltage domain for the system, and one for the CPUs.

base: Add warn() and inform() to m5.utils for use from python

This patch adds two fuctions to m5.util, warn and inform, which mirror those
found in the C++ side of gem5. These are added in addition to the already
existing m5.util.panic and m5.util.fatal which already mirror the C++
functionality. This ensures that warning and information messages generated
by python are in the same format as those generated by C++.

Occurrences of
print "Warning: %s..." % name
have been replaced with
warn("%s...", name)

DOT: improved dot-based system visualization
Revised system visualization to reflect structure and memory hierarchy.
Improved visualization: less congested and cluttered; more colorful.
Nodes reflect components; directed edges reflect dirctional relation, from
a master port to a slave port. Requires pydot.

Regression: Add ANSI colours to highlight test status

This patch adds a very basic pretty-printing of the test status
(passed or failed) to highlight failing tests even more: green for
passed, and red for failed. The printing only uses ANSI it the target
output is a tty and supports ANSI colours. Hence, any regression
scripts that are outputting to files or sending e-mails etc should
still be fine.

slicc: cleanup slicc code and make it less verbose

grammar: better encapsulation of a grammar and parsing
This makes it possible to use the grammar multiple times and use the multiple
instances concurrently. This makes implementing an include statement as part
of a grammar possible.

Misc: Remove the URL from warnings, fatals, panics, etc.

region: add a utility class for keeping track of regions of some range

This is basically like the range_map stuff in src/base (range already
exists in Python). This code is like a set of ranges. I'm using it
to keep track of changed lines in source code, but it could be use to
keep track of memory ranges and holes in memory regions. It could
also be used in memory allocation type stuff. (Though it's not at all
optimized.)

SortedDict: add functions for getting ranges of keys, values, items

scons: show sources and targets when building, and colorize output.

I like the brevity of Ali's recent change, but the ambiguity of
sometimes showing the source and sometimes the target is a little
confusing. This patch makes scons typically list all sources and
all targets for each action, with the common path prefix factored
out for brevity. It's a little more verbose now but also more
informative.

Somehow Ali talked me into adding colors too, which is a whole
'nother story.

Copyright: Add AMD copyright to the param changes I just made.

Params: Add parameter types for IP addresses in various forms.

New parameter forms are:
IP address in the format "a.b.c.d" where a-d are from decimal 0 to 255.
IP address with netmask which is an IP followed by "/n" where n is a netmask
length in bits from decimal 0 to 32 or by "/e.f.g.h" where e-h are from
decimal 0 to 255 and which is all 1 bits followed by all 0 bits when
represented in binary. These can also be specified as an integral IP and
netmask passed in separately.
IP address with port which is an IP followed by ":p" where p is a port index
from decimal 0 to 65535. These can also be specified as an integral IP and
port value passed in separately.

code_formatter: make it easier to insert whitespace
a newline by just doing "code()". indent() and dedent() now take a
"count" parameter to indent/dedent multiple levels.

python: add a sorted dictionary class
It would be nice if python had a tree class that would do this for real,
but since we don't, we'll just keep a sorted list of keys and update
it on demand.

util: clean up attrdict and import multiattrdict into m5.util

slicc: have a central mechanism for creating a code_formatter.
This makes it easier to add global variables like protocol

SmartDict: Make SmartDict an attrdict

merge

ply grammar: Fixup Tokenizer class so you can get lexer arguments

python: Move more code into m5.util allow SCons to use that code.
Get rid of misc.py and just stick misc things in __init__.py
Move utility functions out of SCons files and into m5.util
Move utility type stuff from m5/__init__.py to m5/util/__init__.py
Remove buildEnv from m5 and allow access only from m5.defines
Rename AddToPath to addToPath while we're moving it to m5.util
Rename read_command to readCommand while we're moving it
Rename compare_versions to compareVersions while we're moving it.

multiattrdict: make multilevel nesting work properly

attrdict: add pickle support to attrdict

code_formatter: use __builtin__ which is correct, not __builtins__

code_formatter: Add a python class for writing code generator templates

ply: add a base class called Grammar that encapsulates a ply grammar

orderdict: Use DictMixin and add orderdict to m5.util

attrdict: correct delattr

traceflags: fix --trace-help

jobfile: Add support for dictionaries as jobfile options.
If the same dictionary option is seen in several options, those
dictionaries are composed. If you define the same dictionary key in
multiple options, the system flags an error.
Also, clean up the jobfile code so that it is more debuggable.

python: Add a utility for nested attribute dicts.
Change attrdict so that attributes that begin with an underscore don't
go into the dict.

python: Move various utility classes into a new m5.util package so
they're all in the same place. This also involves having just one
jobfile.py and moving it into the utils directory to avoid
duplication. Lots of improvements to the utility as well.