misc: Added dot_writer for Ruby's network topology

Change-Id: Ic71ca7bc2eb4174d70afa368bc9cc987f3df89e9
Signed-off-by: Tiago Muck <tiago.muck@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17548
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

arch, cpu, dev, gpu, mem, sim, python: start using getPort.

Replace the getMasterPort, getSlavePort, and getEthPort functions
with getPort, and remove extraneous mechanisms that are no longer
necessary.

Change-Id: Iab7e3c02d2f3a0cf33e7e824e18c28646b5bc318
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17040
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Replace orderdict with collections.OrderedDict

Python 2.7 and newer has support for ordered dictionaries in the
standard library. Remove this custom class.

Change-Id: I4b720405aa3c4ce8d5c0b401eefe744a85ac3a3e
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/16362
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

arm: DT autogeneration - Device Tree generation methods

This patch adds an extra layer to the pyfdt library such that usage
gets easier and device tree nodes can be specified in less code,
without limiting original usage. Note to not import both the pyfdt
and fdthelper in the same namespace (but generally fdthelper is all
you need, because it supplies the same classes even when they are not
extended in any way)

Also, this patch lays out the primary functionality for generating a
device tree, where every SimObject gets an empty generateDeviceTree
method and ArmSystems loop over their children in an effort to merge
all the nodes. Devices are implemented in other patches.

Change-Id: I4d0a0666827287fe42e18447f19acab4dc80cc49
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5962
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

ext: DT autogeneration - Add PyFtd to m5 space

This patch adds pyfdt.py to the m5.ext module. This is used in
succeeding patches for generating and editing dtb files and flat
device trees for DT autogeneration.

The file is in the m5_root/src/python/m5/ext directory, as opposed to
the m5_root/ext, because this library is part of the m5 object space
and linking to the m5_root/ext directory from the SConscript file
in src/python can not be done reliably. Linking from the root level
SConscript is also not an option, because it doesn't have the PySource
method defined.

Cloned from: https://github.com/superna9999/pyfdt
Commit: accbcd254584c9295a18878d32999d0c7c156f8e
Version: 0.3

Change-Id: I928bdc912a9507d1f8a3290acf445c7cae496552
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5961
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

scons: Switch from "guards" to "tags" on source files.

Tags are just arbitrary strings which are attached to source files
which mark them as having some property. By default, all source files
have the "gem5 lib" tag added to them which marks them as part of the
gem5 library, the primary component of the gem5 binary but also a
seperable component for use in, for example, system C.

The tags can be completely overridden by setting the "tags" parameter
on Source, etc., functions, and can be augmented by setting "add_tags"
which are tags that will be added, or alternatively additional tags.
It's possible to specify both, in which case the tags will be set to
the union of tags and add_tags. add_tags is supposed to be a way to
add extra tags to the default without actually overriding the default.
Both tags and add_tags can be a list/tuple/etc of tags, or a single
string which will be converted into a set internally.

Other existing tags include:
1. "python" for files that need or are used with python and are
excluded when the --without-python option is set
2. "main" for the file(s) which implement the gem5 binary's main
function.
3. The name of a unit test to group its files together.
4. Tags which group source files for partial linking.

By grouping the "tags" into a single parameter instead of taking all
extra parameters as tags, the extra parameters can, in the future, be
passed to the underlying scons environment. Also, the tags are either
present or not. With guards, they could be present and True, present
and False, or not present at all.

Change-Id: I6d0404211a393968df66f7eddfe019897b6573a2
Reviewed-on: https://gem5-review.googlesource.com/5822
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

python: Use PyBind11 instead of SWIG for Python wrappers

Use the PyBind11 wrapping infrastructure instead of SWIG to generate
wrappers for functionality that needs to be exported to Python. This
has several benefits:

* PyBind11 can be redistributed with gem5, which means that we have
full control of the version used. This avoid a large number of
hard-to-debug SWIG issues we have seen in the past.

* PyBind11 doesn't rely on a custom C++ parser, instead it relies on
wrappers being explicitly declared in C++. The leads to slightly
more boiler-plate code in manually created wrappers, but doesn't
doesn't increase the overall code size. A big benefit is that this
avoids strange compilation errors when SWIG doesn't understand
modern language features.

* Unlike SWIG, there is no risk that the wrapper code incorporates
incorrect type casts (this has happened on numerous occasions in
the past) since these will result in compile-time errors.

As a part of this change, the mechanism to define exported methods has
been redesigned slightly. New methods can be exported either by
declaring them in the SimObject declaration and decorating them with
the cxxMethod decorator or by adding an instance of
PyBindMethod/PyBindProperty to the cxx_exports class variable. The
decorator has the added benefit of making it possible to add a
docstring and naming the method's parameters.

The new wrappers have the following known issues:

* Global events can't be memory managed correctly. This was the
case in SWIG as well.

Change-Id: I88c5a95b6cf6c32fa9e1ad31dfc08b2e8199a763
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Andrew Bardsley <andrew.bardsley@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2231
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Reviewed-by: Pierre-Yves Péneau <pierre-yves.peneau@lirmm.fr>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

python: Move native wrappers to the _m5 namespace

Swig wrappers for native objects currently share the _m5.internal name
space with Python code. This is undesirable if we ever want to switch
from Swig to some other framework for native binding (e.g., PyBind11
or Boost::Python). This changeset moves all of such wrappers to the
_m5 namespace, which is now reserved for native code.

Change-Id: I2d2bc12dbc05b57b7c5a75f072e08124413d77f3
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

misc: Don't build region.py as a PySource

The style refactor change (style: Refactor the style checker as a
Python package) moved region.py from src/python/m5/util/ to
util/style/. The SConscript update accidentally got lost in that
commit. This commit removes region.py from src/python/SConscript.

Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>

config: Add a --without-python option to build process

Add the ability to build libgem5 without embedded Python or the
ability to configure with Python.

This is a prelude to a patch to allow config.ini files to be loaded
into libgem5 using only C++ which would make embedding gem5 within
other simulation systems easier.

This adds a few registration interfaces to things which cross
between Python and C++. Namely: stats dumping and SimObject resolving

sim: Add SWIG interface for Serializable

This changeset adds a SWIG interface for the Serializable class, which
fixes a warning when compiling the SWIG interface for the event
queue. Currently, the only method exported is the name() method.

sim: Move the draining interface into a separate base class

This patch moves the draining interface from SimObject to a separate
class that can be used by any object needing draining. However,
objects not visible to the Python code (i.e., objects not deriving
from SimObject) still depend on their parents informing them when to
drain. This patch also gets rid of the CountedDrainEvent (which isn't
really an event) and replaces it with a DrainManager.

DOT: improved dot-based system visualization
Revised system visualization to reflect structure and memory hierarchy.
Improved visualization: less congested and cluttered; more colorful.
Nodes reflect components; directed edges reflect dirctional relation, from
a master port to a slave port. Requires pydot.

SimObject: add export_method* hooks to export C++ methods to Python

Replace the (broken as of previous changeset) swig_objdecl() method
that allowed/forced you to substitute a whole new C++ struct
definition for SWIG to wrap with a set of export_method* hooks
that let you just declare a set of C++ methods (or other declarations)
that get inserted in the auto-generated struct.

Restore the System get/setMemoryMode methods, and use this mechanism
to specialize SimObject as well, eliminating teh need for sim_object.i.
Needed bits of sim_object.i are moved to the new pyobject.i.
Also sucked a little SimObject specialization into cxx_param_decl()
allowing us to get rid of src/sim/sim_object_params.hh. Now the
generation and wrapping of the base SimObject param struct is more
in line with how derived objects are handled.

stats: better expose statistics to python.
Build a python list and dict of all stats and expose flags properly.

region: add a utility class for keeping track of regions of some range

This is basically like the range_map stuff in src/base (range already
exists in Python). This code is like a set of ranges. I'm using it
to keep track of changed lines in source code, but it could be use to
keep track of memory ranges and holes in memory regions. It could
also be used in memory allocation type stuff. (Though it's not at all
optimized.)

swig: get rid of m5.internal.random module (swig/random.i)
Thanks to swig this was interfering with the standard Python
random module. The only function in that module was seed(),
which erroneously called srand48(). Moved the function to
m5.internal.core, renamed it seedRandom(), and made it call
random_mt.init() instead.

scons: show sources and targets when building, and colorize output.

I like the brevity of Ali's recent change, but the ambiguity of
sometimes showing the source and sometimes the target is a little
confusing. This patch makes scons typically list all sources and
all targets for each action, with the common path prefix factored
out for brevity. It's a little more verbose now but also more
informative.

Somehow Ali talked me into adding colors too, which is a whole
'nother story.

python: get rid of internal.enums package.

Move generated enums into internal.params, which gets
imported into object.params, restoring backward
compatibility for scripts that expect to find them there.

swig: make all generated files go into the m5.internal package

This is necessary because versions of swig older than 1.3.39 fail to
do the right thing and try to do relative imports for everything (even
with the package= option to %module). Instead of putting params in
the m5.internal.params package, put params in the m5.internal package
and make all param modules start with param_. Same thing for
m5.internal.enums.

Also, stop importing all generated params into m5.objects. They are
not necessary and now with everything using relative imports we wound
up with pollution of the namespace (where builtin-range got overridden).

scons: Stop building the big monolithic swigged params module
kill params.i and create a separate .i for each object (param, enums, etc.)

init: don't build files that centralize python and swig code

Instead of putting all object files into m5/object/__init__.py, interrogate
the importer to find out what should be imported.
Instead of creating a single file that lists all of the embedded python
modules, use static object construction to put those objects onto a list.
Do something similar for embedded swig (C++) code.

python: add a sorted dictionary class
It would be nice if python had a tree class that would do this for real,
but since we don't, we'll just keep a sorted list of keys and update
it on demand.

python: Move more code into m5.util allow SCons to use that code.
Get rid of misc.py and just stick misc things in __init__.py
Move utility functions out of SCons files and into m5.util
Move utility type stuff from m5/__init__.py to m5/util/__init__.py
Remove buildEnv from m5 and allow access only from m5.defines
Rename AddToPath to addToPath while we're moving it to m5.util
Rename read_command to readCommand while we're moving it
Rename compare_versions to compareVersions while we're moving it.

code_formatter: Add a python class for writing code generator templates

ply: add a base class called Grammar that encapsulates a ply grammar

orderdict: Use DictMixin and add orderdict to m5.util

python: Try to isolate the stuff that's in the m5.internal package a bit more.

tracing: panic() if people try to use tracing, but TRACING_ON is not set.
Also clean things up so that help strings can more easily be added.
Move the help function into trace.py

libm5: Create a libm5 static library for embedding m5.

This should allow m5 to be more easily embedded into other simulators.
The m5 binary adds a simple main function which then calls into the m5
libarary to start the simulation. In order to make this work
correctly, it was necessary embed python code directly into the
library instead of the zipfile hack. This is because you can't just
append the zipfile to the end of a library the way you can a binary.
As a result, Python files that are part of the m5 simulator are now
compile, marshalled, compressed, and then inserted into the library's
data section with a certain symbol name. Additionally, a new Importer
was needed to allow python to get at the embedded python code.

Small additional changes include:
- Get rid of the PYTHONHOME stuff since I don't think anyone ever used
it, and it just confuses things. Easy enough to add back if I'm wrong.
- Create a few new functions that are key to initializing and running
the simulator: initSignals, initM5Python, m5Main.

The original code for creating libm5 was inspired by a patch Michael
Adler, though the code here was done by me.

Add .m5 configuration directory

python: Separate the options parsing stuff. Remove options parsing stuff from
main.py so things are a bit more obvious.

python: Move various utility classes into a new m5.util package so
they're all in the same place. This also involves having just one
jobfile.py and moving it into the utils directory to avoid
duplication. Lots of improvements to the utility as well.

python: Improve support for python calling back to C++ member functions.
Add support for declaring SimObjects to swig so their members can be wrapped.
Make sim_object.i only contain declarations for SimObject.
Create system.i to contain declarations for System.
Update python code to properly call the C++ given the new changes.

Major changes to how SimObjects are created and initialized. Almost all
creation and initialization now happens in python. Parameter objects
are generated and initialized by python. The .ini file is now solely for
debugging purposes and is not used in construction of the objects in any
way.

Move SimObject python files alongside the C++ and fix
the SConscript files so that only the objects that are
actually available in a given build are compiled in.
Remove a bunch of files that aren't used anymore.

Move full CPU sim object stuff into the encumbered directory

Move the turbolaser python simobject stuff into the
encumbered directory

Completely re-work how the scons framework incorporates swig
and python code into m5 to allow swig an python code to
easily added by any SConscript instead of just the one in
src/python. This provides SwigSource and PySource for
adding new files to m5 (similar to Source for C++). Also
provides SimObject for including files that contain SimObject
information and build the m5.objects __init__.py file.

Add a scons hack to force symlinks to the swig .i files
to be created

Rework the way SCons recurses into subdirectories, making it
automatic. The point is that now a subdirectory can be added
to the build process just by creating a SConscript file in it.
The process has two passes. On the first pass, all subdirs
of the root of the tree are searched for SConsopts files.
These files contain any command line options that ought to be
added for a particular subdirectory. On the second pass,
all subdirs of the src directory are searched for SConscript
files. These files describe how to build any given subdirectory.
I have added a Source() function. Any file (relative to the
directory in which the SConscript resides) passed to that
function is added to the build. Clean up everything to take
advantage of Source().
function is added to the list of files to be built.

Factor code out of main.cc and main.i into a bunch of files
so things are organized in a more sensible manner. Take apart
finalInit and expose the individual functions which are now
called from python. Make checkpointing a bit easier to use.

Automatically generate m5/internal/__init__.py and swig/init.cc
based on the swig modules that we have

Get rid of the Statistics and Statreset ParamContexts, and
expose all of the relevant functionality to python. Clean
up the mysql code while we're at it.

Rearrange traceflags.py so that the file generation only happens if
the script is invoked as main. This allows us to import traceflags.py
if we just want the list of available flags.
Embed traceflags.py into the zipfile so it can be accessed from the
python side of things. With this, print an error on invalid flags and
add --trace-help option that will print out the list of trace flags
that are compiled in. If a flag is prefixed with a '-', now that flag
will be disabled.

Get rid of the Random context and add the support directly to python.
We don't currently use randomness much, so I didn't go too far, but
in the future, we may want to actually expose the random number values
themselves to python. For now, I'll at least let you seed it.
While we're at it, clean up a clearly bad way for generating random
doubles.

Get rid of the Trace ParamContext and give python direct
access to enabling/disabling tracing. Command line is
unchanged except for the removal of --trace-cycle since
it's not so clear what that means.

Expose the C++ event queue to python via the python function
m5.internal.event.create(). It takes a python object and a
Tick and calls process() when the Tick occurs.

Create a wrapper function to more easily add swig stuff to the build

Expose debugBreakCycle through swig and get rid of
the Debug param context

Create a module called internal where swigged stuff goes.
Rename cc_main to internal.main

Migrate most of main() and and all option parsing to python

configs/test/fs.py:
configs/test/test.py:
update for the new way that m5 deals with options
src/python/SConscript:
Compile AUTHORS, LICENSE, README, and RELEASE_NOTES into the
python stuff.
src/python/m5/__init__.py:
redo the way options work.
Move them all to main.py
src/sim/main.cc:
Migrate more functionality for main() into python.
Namely option parsing
src/python/m5/attrdict.py:
A dictionary object that overrides attribute access to
do item access.
src/python/m5/main.py:
The new location for M5's option parsing, and the main()
routine to set up the simulation.

Rename SWIG "main" module to "cc_main" so it's
clear from the Python side that this is the
interface to C++.

src/SConscript:
main_wrap.cc -> cc_main_wrap.cc
src/python/SConscript:
src/python/m5/__init__.py:
src/sim/main.cc:
s/main/cc_main/
src/python/m5/config.py:
s/main/cc_main/
Also directly import cc_main so we don't need
to put the "m5." in front all the time.

Move main control from C++ into Python.
User script now invokes initialization and
simulation loop after building configuration.
These functions are exported from C++ to Python
using SWIG.

SConstruct:
Set up SWIG builder & scanner.
Set up symlinking of source files into build directory
(by not disabling the default behavior).
configs/test/test.py:
Rewrite to use new script-driven interface.
Include a sample option.
src/SConscript:
Set up symlinking of source files into build directory
(by not disabling the default behavior).
Add SWIG-generated main_wrap.cc to source list.
src/arch/SConscript:
Set up symlinking of source files into build directory
(by not disabling the default behavior).
src/arch/alpha/ev5.cc:
src/arch/alpha/isa/decoder.isa:
src/cpu/o3/alpha_cpu_impl.hh:
src/cpu/trace/opt_cpu.cc:
src/cpu/trace/trace_cpu.cc:
src/sim/pseudo_inst.cc:
src/sim/root.cc:
src/sim/serialize.cc:
src/sim/syscall_emul.cc:
SimExit() is now exitSimLoop().
src/cpu/base.cc:
SimExitEvent is now SimLoopExitEvent
src/python/SConscript:
Add SWIG build command for main.i.
Use python/m5 in build dir as source for zip archive...
easy now with file duplication enabled.
src/python/m5/__init__.py:
- Move copyright notice back to C++ so we can print
it right away, even for interactive sessions.
- Get rid of argument parsing code; just provide default
option descriptors for user script to call optparse with.
- Don't clutter m5 namespace by sucking in all of m5.config
and m5.objects.
- Move instantiate() function here from config.py.
src/python/m5/config.py:
- Move instantiate() function to __init__.py.
- Param.Foo deferred type lookups must use m5.objects
namespace now (not m5).
src/python/m5/objects/AlphaConsole.py:
src/python/m5/objects/AlphaFullCPU.py:
src/python/m5/objects/AlphaTLB.py:
src/python/m5/objects/BadDevice.py:
src/python/m5/objects/BaseCPU.py:
src/python/m5/objects/BaseCache.py:
src/python/m5/objects/Bridge.py:
src/python/m5/objects/Bus.py:
src/python/m5/objects/CoherenceProtocol.py:
src/python/m5/objects/Device.py:
src/python/m5/objects/DiskImage.py:
src/python/m5/objects/Ethernet.py:
src/python/m5/objects/Ide.py:
src/python/m5/objects/IntrControl.py:
src/python/m5/objects/MemObject.py:
src/python/m5/objects/MemTest.py:
src/python/m5/objects/Pci.py:
src/python/m5/objects/PhysicalMemory.py:
src/python/m5/objects/Platform.py:
src/python/m5/objects/Process.py:
src/python/m5/objects/Repl.py:
src/python/m5/objects/Root.py:
src/python/m5/objects/SimConsole.py:
src/python/m5/objects/SimpleDisk.py:
src/python/m5/objects/System.py:
src/python/m5/objects/Tsunami.py:
src/python/m5/objects/Uart.py:
Fix up imports (m5 namespace no longer includes m5.config).
src/sim/eventq.cc:
src/sim/eventq.hh:
Support for Python-called simulate() function:
- Use IsExitEvent flag to signal events that want
to exit the simulation loop gracefully (instead of
calling exit() to terminate the process).
- Modify interface to hand exit event object back to
caller so it can be inspected for cause.
src/sim/host.hh:
Add MaxTick constant.
src/sim/main.cc:
Move copyright notice back to C++ so we can print
it right away, even for interactive sessions.
Use PYTHONPATH environment var to set module path
(instead of clunky code injection method).
Move main control from here into Python:
- Separate initialization code and simulation loop
into separate functions callable from Python.
- Make Python interpreter invocation more pure (more
like directly invoking interpreter).
Add -i and -p flags (only options on binary itself;
other options processed by Python).
Import readline package when using interactive mode.
src/sim/sim_events.cc:
SimExitEvent is now SimLoopExitEvent, and uses
IsSimExit flag to terminate loop (instead of
exiting simulator process).
src/sim/sim_events.hh:
SimExitEvent is now SimLoopExitEvent, and uses
IsSimExit flag to terminate loop (instead of
exiting simulator process).
Get rid of a few unused constructors.
src/sim/sim_exit.hh:
SimExit() is now exitSimLoop().
Get rid of unused functions.
Add comments.

Updated Authors from bk prs info

Link in Python interpreter.
Use embedded zip archive to carry Python code instead
of homegrown embedded string/file mechanism.
Do argument parsing in Python instead of C++.

SConstruct:
Add Python interpreter include path & library.
Define two new simple builders which copy &
concatenate files, respectively, for use by
the Python embedded zipfile code.
src/SConscript:
Encapsulate environment creation in a function.
Add code to append Python zip archive to final executable.
Eliminate references to obsolete files.
src/python/SConscript:
Rewrite to generate embedded zip archive of Python code
(replacing old "embedded string" mechanism).
src/python/m5/__init__.py:
Move main arg-parsing loop here (out of C++ main()).
src/python/m5/config.py:
Minor fix (version incompatibility?).
src/sim/main.cc:
Invoke embedded Python interpreter to parse args
and generate config.ini, replacing C++ arg parsing code.

New directory structure:
- simulator source now in 'src' subdirectory
- imported files from 'ext' repository
- support building in arbitrary places, including
outside of the source tree. See comment at top
of SConstruct file for more details.
Regression tests are temporarily disabled; that
syetem needs more extensive revisions.

SConstruct:
Update for new directory structure.
Modify to support build trees that are not subdirectories
of the source tree. See comment at top of file for
more details.
Regression tests are temporarily disabled.
src/arch/SConscript:
src/arch/isa_parser.py:
src/python/SConscript:
Update for new directory structure.