python: Fix native module initialisation on Python 3

The approach we currently use to register our native modules doesn't
work on Python 3. Convert the code to use the Python inittab instead
of the old ad-hoc method.

Change-Id: I961f8a33993c621473732faeaab955a882769a4b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15979
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Gabe Black <gabeblack@google.com>

sim: Prepare C++ side for Python 3

Python 3 uses wide strings instead of ordinary strings for many
APIs. Add the necessary conversions to comply with the new API.

Change-Id: I6f45c9c532537d50d54b542f34eb8fd8cb375874
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15977
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>

config: Add a --without-python option to build process

Add the ability to build libgem5 without embedded Python or the
ability to configure with Python.

This is a prelude to a patch to allow config.ini files to be loaded
into libgem5 using only C++ which would make embedding gem5 within
other simulation systems easier.

This adds a few registration interfaces to things which cross
between Python and C++. Namely: stats dumping and SimObject resolving

libm5: Create a libm5 static library for embedding m5.

This should allow m5 to be more easily embedded into other simulators.
The m5 binary adds a simple main function which then calls into the m5
libarary to start the simulation. In order to make this work
correctly, it was necessary embed python code directly into the
library instead of the zipfile hack. This is because you can't just
append the zipfile to the end of a library the way you can a binary.
As a result, Python files that are part of the m5 simulator are now
compile, marshalled, compressed, and then inserted into the library's
data section with a certain symbol name. Additionally, a new Importer
was needed to allow python to get at the embedded python code.

Small additional changes include:
- Get rid of the PYTHONHOME stuff since I don't think anyone ever used
it, and it just confuses things. Easy enough to add back if I'm wrong.
- Create a few new functions that are key to initializing and running
the simulator: initSignals, initM5Python, m5Main.

The original code for creating libm5 was inspired by a patch Michael
Adler, though the code here was done by me.

String constant const-ness changes to placate g++ 4.2.
Also some bug fixes in MIPS ISA uncovered by g++ warnings
(Python string compares don't work in C++!).

main: return an an exit code of 1 when we exit due to a python exception.
This requires us to not use PyRun_SimpleString, but PyRun_String since the
latter actually returns a result

Move all of the parameters of the Root SimObject so they are
directly configured by python. Move stuff from root.(cc|hh) to
core.(cc|hh) since it really belogs there now.
In the process, simplify how ticks are used in the python code.

include signal.h

Factor code out of main.cc and main.i into a bunch of files
so things are organized in a more sensible manner. Take apart
finalInit and expose the individual functions which are now
called from python. Make checkpointing a bit easier to use.

Get rid of the stand alone ParamContext since all of the
relevant stuff has now been moved to python.

Get rid of the Statistics and Statreset ParamContexts, and
expose all of the relevant functionality to python. Clean
up the mysql code while we're at it.

Pass an exception from a python event through the event queue
back into python so we don't just silently ignore those errors

Clean up tracing stuff more, get rid of the trace log since
its not all that useful. Fix a few bugs with python/C++
integration.

move the swig initialization calls from src/sim/main.cc to
src/python/swig/init.cc so that it's not as easy to forget
about it when you add a new swig module.

Expose debugBreakCycle through swig and get rid of
the Debug param context

Create a module called internal where swigged stuff goes.
Rename cc_main to internal.main

simplify maxtick parsing in both the python and the c++.

configs/common/Simulation.py:
simplify maxtick code a little bit - instead of checking for -1, just set it at MaxTick.
src/python/m5/__init__.py:
make a new m5 param called MaxTick.
src/sim/host.hh:
fix the M5 def. of MaxTick
src/sim/main.cc:
Simplify the MaxTick/num_cycles parsing within main.cc

Construct a correct value of PYTHONHOME from the interpreter
running SCons, make it into a sticky option that can be
overridden at build time, and set it up before the interpreter
is started. Also, fix the code that turns sticky options into
config/*.hh so that it works with types other than bool.

since memoryMode was put into the System (from SimObject), things got broken - this fixes it so that changeToTiming/changeToAtomic works.

src/python/m5/SimObject.py:
now that setMemoryMode is a method in System, need to convert the SimObject * _ccObject into a system ptr to call setMemoryMode.
src/sim/main.cc:
need this conversion now.
src/sim/sim_object.hh:
put the enum back into SimObject.
src/sim/system.hh:
memoryMode is now a part of SimObject, need the ::'s

there are two main thrusts of this changeset.

1) return the periodicity of checkpoints back into the code (i.e. make m5 checkpoint n m meaningful again).
2) to do this, i had to much around with being able to repeatedly schedule and SimLoopExitEvent, which led to changes in how exit simloop events are handled to make this easier.

src/arch/alpha/isa/decoder.isa:
src/mem/cache/cache_impl.hh:
modify arg. order for new calling convention of exitSimLoop.
src/cpu/base.cc:
src/sim/main.cc:
src/sim/pseudo_inst.cc:
src/sim/root.cc:
now, instead of creating a new SimLoopExitEvent, call a wrapper schedExitSimLoop which handles all the default args.
src/sim/sim_events.cc:
src/sim/sim_events.hh:
src/sim/sim_exit.hh:
add the periodicity of checkpointing back into the code.

to facilitate this, there are now two wrappers (instead of just overloading exitSimLoop). exitSimLoop is only for exiting NOW (i.e. at curTick), while schedExitSimLoop schedules and exit event for the future.

Static global object don't work well, if the variables are
accessed during the construction of another static global
object because there are no guarantees on ordering of
construction, so stick the static global into a function
as a static local and return a reference to the variable.
This fixes the exit callback stuff on my Mac.

More Python hacking to deal with config.py split
and resulting recursive import trickiness.

Get the path to load the ini file from. I'm not sure if this fix is needed in other places as well.

src/sim/main.cc:
Get the path to load the ini file from.

Migrate most of main() and and all option parsing to python

configs/test/fs.py:
configs/test/test.py:
update for the new way that m5 deals with options
src/python/SConscript:
Compile AUTHORS, LICENSE, README, and RELEASE_NOTES into the
python stuff.
src/python/m5/__init__.py:
redo the way options work.
Move them all to main.py
src/sim/main.cc:
Migrate more functionality for main() into python.
Namely option parsing
src/python/m5/attrdict.py:
A dictionary object that overrides attribute access to
do item access.
src/python/m5/main.py:
The new location for M5's option parsing, and the main()
routine to set up the simulation.

Take the name of the checkpoint directory in when calling checkpoint() or restoreCheckpoint().

src/sim/main.cc:
src/sim/serialize.cc:
src/sim/serialize.hh:
Take in the directory name when checkpointing.

Rename quiesce to drain to avoid confusion with the pseudo instruction.

src/cpu/simple/timing.cc:
src/cpu/simple/timing.hh:
src/python/m5/__init__.py:
src/python/m5/config.py:
src/sim/main.cc:
src/sim/sim_events.cc:
src/sim/sim_events.hh:
src/sim/sim_object.cc:
src/sim/sim_object.hh:
Rename quiesce to drain.

Add in support for quiescing the system, taking checkpoints, restoring from checkpoints, changing memory modes, and switching CPUs.

Key new functions that can be called on the m5 object at the python interpreter:
doQuiesce(root) - A helper function that quiesces the object passed in and all of its children.
resume(root) - Another helper function that tells the object and all of its children that the quiesce is over.
checkpoint(root) - Takes a checkpoint of the system. Checkpoint directory must be set before hand.
setCheckpointDir(name) - Sets the checkpoint directory.
restoreCheckpoint(root) - Restores the values from the checkpoint located in the checkpoint directory.
changeToAtomic(system) - Changes the system and all of its children to atomic memory mode.
changeToTiming(system) - Changes the system and all of its children to timing memory mode.
switchCpus(list) - Takes in a list of tuples, where each tuple is a pair of (old CPU, new CPU). Quiesces the old CPUs, and then switches over to the new CPUs.

src/SConscript:
Remove serializer, replaced by python code.
src/python/m5/__init__.py:
Updates to support quiescing, checkpointing, changing memory modes, and switching CPUs.
src/python/m5/config.py:
Several functions defined on the SimObject for quiescing, changing timing modes, and switching CPUs
src/sim/main.cc:
Add some extra functions that are exported to python through SWIG.
src/sim/serialize.cc:
Change serialization around a bit. Now it is controlled through Python, so there's no need for SerializeEvents or SerializeParams.

Also add in a new unserializeAll() function that loads a checkpoint and handles unserializing all objects.
src/sim/serialize.hh:
Add unserializeAll function and a setCheckpointName function.
src/sim/sim_events.cc:
Add process() function for CountedQuiesceEvent, which calls exitSimLoop() once its counter reaches 0.
src/sim/sim_events.hh:
Add in a CountedQuiesceEvent, which is used when the system is preparing to quiesce. Any objects that can't be quiesced immediately are given a pointer to a CountedQuiesceEvent. The event has its counter set via Python, and as objects finish quiescing they call process() on the event. Eventually the event causes the simulation to stop once all objects have quiesced.
src/sim/sim_object.cc:
Add a few functions for quiescing, checkpointing, and changing memory modes.
src/sim/sim_object.hh:
Add a state variable to all SimObjects that tracks both the timing mode of the object and the quiesce state of the object. Currently this isn't serialized, and I'm not sure it needs to be so long as the timing mode starts up the same after a checkpoint.

remove profile.cc

Merge zizzer:/bk/newmem
into zeep.eecs.umich.edu:/z/saidi/work/m5.newmem

Merge zizzer:/bk/newmem
into zeep.eecs.umich.edu:/z/saidi/work/m5.newmem

Rename SWIG "main" module to "cc_main" so it's
clear from the Python side that this is the
interface to C++.

src/SConscript:
main_wrap.cc -> cc_main_wrap.cc
src/python/SConscript:
src/python/m5/__init__.py:
src/sim/main.cc:
s/main/cc_main/
src/python/m5/config.py:
s/main/cc_main/
Also directly import cc_main so we don't need
to put the "m5." in front all the time.

Add --outdir option. Didn't call it "-d" since
that's already being used for "detailed cpu".
Needed to add extra function for user script
to pass parsed options back to m5 module.

configs/test/fs.py:
configs/test/test.py:
Call setStandardOptions().
src/python/m5/__init__.py:
Add --outdir option.
Add setStandardOptions() so user script can
pass parsed options back to m5 module.
src/sim/main.cc:
Add SWIG-wrappable function to set output dir.

Move SimObject creation and Port connection loops
into Python.
Add Port and VectorPort objects and support for
specifying port connections via assignment.
The whole C++ ConfigNode hierarchy is gone now, as are
C++ Connector objects.

configs/test/fs.py:
configs/test/test.py:
Rewrite for new port connector syntax.
src/SConscript:
Remove unneeded files:
- mem/connector.*
- sim/config*
src/dev/io_device.hh:
src/mem/bridge.cc:
src/mem/bridge.hh:
src/mem/bus.cc:
src/mem/bus.hh:
src/mem/mem_object.hh:
src/mem/physical.cc:
src/mem/physical.hh:
Allow getPort() to take an optional index to
support vector ports (eventually).
src/python/m5/__init__.py:
Move SimObject construction and port connection
operations into Python (with C++ calls).
src/python/m5/config.py:
Move SimObject construction and port connection
operations into Python (with C++ calls).
Add support for declaring and connecting MemObject
ports in Python.
src/python/m5/objects/Bus.py:
src/python/m5/objects/PhysicalMemory.py:
Add port declaration.
src/sim/builder.cc:
src/sim/builder.hh:
src/sim/serialize.cc:
src/sim/serialize.hh:
ConfigNodes are gone; builder just gets the
name of a .ini file section now.
src/sim/main.cc:
Move SimObject construction and port connection
operations into Python (with C++ calls).
Split remaining initialization operations into two parts,
loadIniFile() and finalInit().
src/sim/param.cc:
src/sim/param.hh:
SimObject resolution done globally in Python now
(not via ConfigNode hierarchy).
src/sim/sim_object.cc:
Remove unneeded #include.

add a comment that should be able to pass --help after config file

allow long opts to m5 and add a help flag back.

Move main control from C++ into Python.
User script now invokes initialization and
simulation loop after building configuration.
These functions are exported from C++ to Python
using SWIG.

SConstruct:
Set up SWIG builder & scanner.
Set up symlinking of source files into build directory
(by not disabling the default behavior).
configs/test/test.py:
Rewrite to use new script-driven interface.
Include a sample option.
src/SConscript:
Set up symlinking of source files into build directory
(by not disabling the default behavior).
Add SWIG-generated main_wrap.cc to source list.
src/arch/SConscript:
Set up symlinking of source files into build directory
(by not disabling the default behavior).
src/arch/alpha/ev5.cc:
src/arch/alpha/isa/decoder.isa:
src/cpu/o3/alpha_cpu_impl.hh:
src/cpu/trace/opt_cpu.cc:
src/cpu/trace/trace_cpu.cc:
src/sim/pseudo_inst.cc:
src/sim/root.cc:
src/sim/serialize.cc:
src/sim/syscall_emul.cc:
SimExit() is now exitSimLoop().
src/cpu/base.cc:
SimExitEvent is now SimLoopExitEvent
src/python/SConscript:
Add SWIG build command for main.i.
Use python/m5 in build dir as source for zip archive...
easy now with file duplication enabled.
src/python/m5/__init__.py:
- Move copyright notice back to C++ so we can print
it right away, even for interactive sessions.
- Get rid of argument parsing code; just provide default
option descriptors for user script to call optparse with.
- Don't clutter m5 namespace by sucking in all of m5.config
and m5.objects.
- Move instantiate() function here from config.py.
src/python/m5/config.py:
- Move instantiate() function to __init__.py.
- Param.Foo deferred type lookups must use m5.objects
namespace now (not m5).
src/python/m5/objects/AlphaConsole.py:
src/python/m5/objects/AlphaFullCPU.py:
src/python/m5/objects/AlphaTLB.py:
src/python/m5/objects/BadDevice.py:
src/python/m5/objects/BaseCPU.py:
src/python/m5/objects/BaseCache.py:
src/python/m5/objects/Bridge.py:
src/python/m5/objects/Bus.py:
src/python/m5/objects/CoherenceProtocol.py:
src/python/m5/objects/Device.py:
src/python/m5/objects/DiskImage.py:
src/python/m5/objects/Ethernet.py:
src/python/m5/objects/Ide.py:
src/python/m5/objects/IntrControl.py:
src/python/m5/objects/MemObject.py:
src/python/m5/objects/MemTest.py:
src/python/m5/objects/Pci.py:
src/python/m5/objects/PhysicalMemory.py:
src/python/m5/objects/Platform.py:
src/python/m5/objects/Process.py:
src/python/m5/objects/Repl.py:
src/python/m5/objects/Root.py:
src/python/m5/objects/SimConsole.py:
src/python/m5/objects/SimpleDisk.py:
src/python/m5/objects/System.py:
src/python/m5/objects/Tsunami.py:
src/python/m5/objects/Uart.py:
Fix up imports (m5 namespace no longer includes m5.config).
src/sim/eventq.cc:
src/sim/eventq.hh:
Support for Python-called simulate() function:
- Use IsExitEvent flag to signal events that want
to exit the simulation loop gracefully (instead of
calling exit() to terminate the process).
- Modify interface to hand exit event object back to
caller so it can be inspected for cause.
src/sim/host.hh:
Add MaxTick constant.
src/sim/main.cc:
Move copyright notice back to C++ so we can print
it right away, even for interactive sessions.
Use PYTHONPATH environment var to set module path
(instead of clunky code injection method).
Move main control from here into Python:
- Separate initialization code and simulation loop
into separate functions callable from Python.
- Make Python interpreter invocation more pure (more
like directly invoking interpreter).
Add -i and -p flags (only options on binary itself;
other options processed by Python).
Import readline package when using interactive mode.
src/sim/sim_events.cc:
SimExitEvent is now SimLoopExitEvent, and uses
IsSimExit flag to terminate loop (instead of
exiting simulator process).
src/sim/sim_events.hh:
SimExitEvent is now SimLoopExitEvent, and uses
IsSimExit flag to terminate loop (instead of
exiting simulator process).
Get rid of a few unused constructors.
src/sim/sim_exit.hh:
SimExit() is now exitSimLoop().
Get rid of unused functions.
Add comments.

Updated Authors from bk prs info

Clean up Python embedding to build on zizzer (where python2.4
is currently in /usr/local instead of /usr).

SConstruct:
Use information from the Python interpreter used to run scons
to find the version & paths for include & library files.
This means that linking with a different interpreter requires
invoking scons with that interpreter... add comment to that effect.
src/sim/main.cc:
Check return codes of Python interpreter calls for errors.
Get rid of include of obsolete header.

Link in Python interpreter.
Use embedded zip archive to carry Python code instead
of homegrown embedded string/file mechanism.
Do argument parsing in Python instead of C++.

SConstruct:
Add Python interpreter include path & library.
Define two new simple builders which copy &
concatenate files, respectively, for use by
the Python embedded zipfile code.
src/SConscript:
Encapsulate environment creation in a function.
Add code to append Python zip archive to final executable.
Eliminate references to obsolete files.
src/python/SConscript:
Rewrite to generate embedded zip archive of Python code
(replacing old "embedded string" mechanism).
src/python/m5/__init__.py:
Move main arg-parsing loop here (out of C++ main()).
src/python/m5/config.py:
Minor fix (version incompatibility?).
src/sim/main.cc:
Invoke embedded Python interpreter to parse args
and generate config.ini, replacing C++ arg parsing code.

New directory structure:
- simulator source now in 'src' subdirectory
- imported files from 'ext' repository
- support building in arbitrary places, including
outside of the source tree. See comment at top
of SConstruct file for more details.
Regression tests are temporarily disabled; that
syetem needs more extensive revisions.

SConstruct:
Update for new directory structure.
Modify to support build trees that are not subdirectories
of the source tree. See comment at top of file for
more details.
Regression tests are temporarily disabled.
src/arch/SConscript:
src/arch/isa_parser.py:
src/python/SConscript:
Update for new directory structure.