mem: Move eraseIfNullEntry to when holder is updated

The entry should only be tested for deletion when holder is updated.

Change-Id: I5a10b6fa876912709b7467860d43c23c60f38568
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19750
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

mem: Encapsulate retry variables of SnoopFilter

Group all variables related to the restoration of a snoop filter
entry due to a crossbar retry.

Change-Id: I4e03edb3afd06563b7a5812959739876709eceeb
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19749
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

mem: Snoop filter support for large systems

Changed SnoopMask to use std::bitset instead of uint64 so we can simulate
larger systems without having to workaround limitations on the number of
ports. No noticeable performance drop was observed after this change.
The size of the bitset is currently set to 256 which should fit most
needs.

Change-Id: I216882300500e2dcb789889756e73a1033271621
Signed-off-by: Tiago Muck <tiago.muck@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18791
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>

style: change C/C++ source permissions to noexec

Several files in the repository were tracked with execute permissions
even though the files are just normal C/C++ files (and the one .isa).

Change-Id: I976b096acab4a1fc74c5699ef1f9b222c1e635c2
Reviewed-on: https://gem5-review.googlesource.com/7241
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>

mem: Handle CMO responses in the snoop filter

Previously responses would either transfer the ownership of the line
or the actual data to the cache that send out the original request.
Cache clean operations are different since they bring neither data nor
ownership. When they are also invalidating the cache that send out the
original request will invalidate any existing copies. This patch
makes the snoop filter handle the cache clean responses accordingly.

Change-Id: I27165cb45b9dc57882526329c62db35f100d23df
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>
Reviewed-by: Sudhanshu Jha <sudhanshu.jha@arm.com>
Reviewed-by: Anouk Van Laer <anouk.vanlaer@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5053
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

mem: Allow CMOs as snooping requests in the snoop filter

The snoop filter performs sanity checks of the type of packets that
are expected to snoop caches above. Cache maintenace operations are
expected to perform a clean and or invalidate on all caches down to
the specified point of reference and therefore could also generate
snoops.

Change-Id: I7f8fef246a85faa87ccd289c28b49686ed7caa08
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>
Reviewed-by: Anouk Van Laer <anouk.vanlaer@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5052
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>

misc: Rename misc.(hh|cc) to logging.(hh|cc)

These files aren't a collection of miscellaneous stuff, they're the
definition of the Logger interface, and a few utility macros for
calling into that interface (panic, warn, etc.).

Change-Id: I84267ac3f45896a83c0ef027f8f19c5e9a5667d1
Reviewed-on: https://gem5-review.googlesource.com/6226
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>

style: [patch 1/22] use /r/3648/ to reorganize includes

mem: Don't use hasSharers in the snoopFilter for memory responses

When the snoopFilter receives a response, it updates its state using
the hasSharers flag (indicates whether there are more than one copies
of the block in the caches above). The hasSharers flag of the packet
was previously populated when the request was traversing and snooping
the caches looking for the block.
1) When the response is coming from the memory-side port, its order
with respect to other responses is not necessarily preserved (e.g., a
request that arrived second to the xbar can get its response first). As
a result the snoopFilter might process responses out of order updating
its residency information using the non valid hasSharers flag which was
populated much earlier.
2) When the response is from an on-chip, the MSHRs preserve a well
defined order and the hasSharers flag should contain valid
information.

This patch changes the snoopFilter by avoiding the hasSharers flag
when the response is from the memory-side port.

Change-Id: Ib2d22a5b7bf3eccac64445127d2ea20ee74bb25b
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>

mem: Make packet debug printing more uniform

Previously DPRINTFs printing information about a packet would use ad hoc
formats. This patch changes all DPRINTFs to use the print function
defined by the packet class, making the packet printing format more
uniform and easier to change.

Change-Id: Idd436a9758d4bf70c86a574d524648b2a2580970
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>

mem: Add support for secure packets in the snoop filter

Secure and non-secure data can coexist in the cache and therefore the
snoop filter should treat differently packets with secure and non
secure accesses. This patch uses the lower bits of the line address to
keep track of whether the packet is addressing secure memory or not.

Change-Id: I54a5e614dad566a5083582bede86c86896f2c2c1
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>

mem: Use FromCache attribute in snoop filter allocation

This patch improves the snoop filter allocation decisions by not only
looking at whether a port is snooping or not, but also if the packet
actually came from a cache. The issue with only looking at isSnooping
is that the CPU ports, for example, are snooping, but not actually
caching. Previously we ended up incorrectly allocating entries in
systems without caches (such as the atomic and timing quick
regressions). Eventually these misguided allocations caused the snoop
filter to panic due to an excessive size.

On the request path we now include the fromCache check on the packet
itself, and for responses we check if we actually have a snoop-filter
entry.

Change-Id: Idd2dbc4f00c7e07d331e9a02658aee30d0350d7e
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>

mem: Fix the snoop filter when there is a downstream addr mapper

The snoop filter handles requests in two steps which preceed and
follow the call to send the packet downstream. An address mapper could
possibly change the address of the packet when it is sent downstream
breaking the snoop filter assumption that the address is unchanged

Change-Id: Ib2db755e9ebef4f2f7c0169a46b1b11185ffbe79
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>

stats: Fixing regStats function for some SimObjects

Fixing an issue with regStats not calling the parent class method
for most SimObjects in Gem5. This causes issues if one adds new
stats in the base class (since they are never initialized properly!).

Change-Id: Iebc5aa66f58816ef4295dc8e48a357558d76a77c
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>

Revert to 74c1e6513bd0 (sim: Thermal support for Linux)

power: Add power states to ClockedObject

Add 4 power states to the ClockedObject, provides necessary access functions
to check and update the power state. Default power state is UNDEFINED, it is
responsibility of the respective simulation model to provide the startup state
and any other logic for state change.

Add number of transition stat.
Add distribution of time spent in clock gated state.
Add power state residency stat.

Add dump call back function to allow stats update of distribution and residency
stats.

mem: Do not rely on the NeedsWritable flag for responses

This patch removes the NeedsWritable flag for all responses, as it is
really only the request that needs a writable response. The response,
on the other hand, should in these cases always provide the line in a
writable state, as indicated by the hasSharers flag not being set.

When we send requests that has NeedsWritable set, the response will
always have the hasSharers flag not set. Additionally, there are cases
where the request did not have NeedsWritable set, and we still get a
writable response with the hasSharers flag not set. This never happens
on snoops, but is used by downstream caches to pass ownership
upstream.

As part of this patch, the affected response types are updated, and
the snoop filter is similarly modified to check only the hasSharers
flag (as it should). A sanity check is also added to the packet class,
asserting that we never look at the NeedsWritable flag for responses.

No regressions are affected.

mem: Make cache terminology easier to understand

This patch changes the name of a bunch of packet flags and MSHR member
functions and variables to make the coherency protocol easier to
understand. In addition the patch adds and updates lots of
descriptions, explicitly spelling out assumptions.

The following name changes are made:

* the packet memInhibit flag is renamed to cacheResponding

* the packet sharedAsserted flag is renamed to hasSharers

* the packet NeedsExclusive attribute is renamed to NeedsWritable

* the packet isSupplyExclusive is renamed responderHadWritable

* the MSHR pendingDirty is renamed to pendingModified

The cache states, Modified, Owned, Exclusive, Shared are also called
out in the cache and MSHR code to make it easier to understand.

mem: Add an option to perform clean writebacks from caches

This patch adds the necessary commands and cache functionality to
allow clean writebacks. This functionality is crucial, especially when
having exclusive (victim) caches. For example, if read-only L1
instruction caches are not sending clean writebacks, there will never
be any spills from the L1 to the L2. At the moment the cache model
defaults to not sending clean writebacks, and this should possibly be
re-evaluated.

The implementation of clean writebacks relies on a new packet command
WritebackClean, which acts much like a Writeback (renamed
WritebackDirty), and also much like a CleanEvict. On eviction of a
clean block the cache either sends a clean evict, or a clean
writeback, and if any copies are still cached upstream the clean
evict/writeback is dropped. Similarly, if a clean evict/writeback
reaches a cache where there are outstanding MSHRs for the block, the
packet is dropped. In the typical case though, the clean writeback
allocates a block in the downstream cache, and marks it writable if
the evicted block was writable.

The patch changes the O3_ARM_v7a L1 cache configuration and the
default L1 caches in config/common/Caches.py

mem: Comment clean-up for the snoop filter

Merely fixing up some style issues and adding more comments.

mem: Avoid adding and then removing empty snoop-filter items

This patch tidies up how we access the snoop filter for snoops, and
avoids adding items only to later remove them.

mem: Add snoop filters to L2 crossbars, and check size

This patch adds a snoop filter to the L2XBar. For now we refrain from
globally adding a snoop filter to the SystemXBar, since the latter is
also used in systems without caches. In scenarios without caches the
snoop filter will not see any writeback/clean evicts from the CPU
ports, despite the fact that they are snooping. To avoid inadvertent
use of the snoop filter in these cases we leave it out for now.

A size check is added to the snoop filter, merely to ensure it does
not grow beyond the total capacity of the caches above it. The size
has to be set manually, and a value of 8 MByte is choosen as suitably
high default.

mem: Store snoop filter lookup result to avoid second lookup

This patch introduces a private member storing the iterator from the
lookupRequest call, such that it can be re-used when the request
eventually finishes. The method previously called updateRequest is
renamed finishRequest to make it more clear that the two functions
must be called together.

mem: Add CleanEvict and Writeback support to snoop filters

This patch adds the functionality to properly track CleanEvicts and
Writebacks in the snoop filter. Previously there were no CleanEvicts, and
Writebacks did not send up snoops to ensure there were no copies in
caches above. Hence a writeback could never erase an entry from the
snoop filter.

When a CleanEvict message reaches a snoop filter, it confirms that the
BLOCK_CACHED flag is not set and resets the bits corresponding to the
CleanEvict address and port it arrived on. If none of the other peer
caches have (or have requested) the block, the snoop filter forwards
the CleanEvict to lower levels of memory. In case of a Writeback
message, the snoop filter checks if the BLOCK_CACHED flag is not set
and only then resets the bits corresponding to the Writeback
address. If any of the other peer caches have (or has requested) the
same block, the snoop filter sets the BLOCK_CACHED flag in the
Writeback before forwarding it to lower levels of memory heirarachy.

mem: Add check for snooping ports in the snoop filter

This patch prevents the snoop filter from creating items for requests
originating from non-snooping ports. The allocation decision is thus
based both on the cacheability of the line, and the snooping status of
the source port. Ultimately we should check if the source of the
packet is caching, since also the CPU ports are snooping (but not
allocating). Thus, at the moment we rely on the snoop filter being
used together with caches.

The patch also transitions to use the Packet::getBlockAddr in
determining the line address.

mem: Add clean evicts to improve snoop filter tracking

This patch adds eviction notices to the caches, to provide accurate
tracking of cache blocks in snoop filters. We add the CleanEvict
message to the memory heirarchy and use both CleanEvicts and
Writebacks with BLOCK_CACHED flags to propagate notice of clean and
dirty evictions respectively, down the memory hierarchy. Note that the
BLOCK_CACHED flag indicates whether there exist any copies of the
evicted block in the caches above the evicting cache.

The purpose of the CleanEvict message is to notify snoop filters of
silent evictions in the relevant caches. The CleanEvict message
behaves much like a Writeback. CleanEvict is a write and a request but
unlike a Writeback, CleanEvict does not have data and does not need
exclusive access to the block. The cache generates the CleanEvict
message on a fill resulting in eviction of a clean block. Before
travelling downwards CleanEvict requests generate zero-time snoop
requests to check if the same block is cached in upper levels of the
memory heirarchy. If the block exists, the cache discards the
CleanEvict message. The snoops check the tags, writeback queue and the
MSHRs of upper level caches in a manner similar to snoops generated
from HardPFReqs. Currently CleanEvicts keep travelling towards main
memory unless they encounter the block corresponding to their address
or reach main memory (since we have no well defined point of
serialisation). Main memory simply discards CleanEvict messages.

We have modified the behavior of Writebacks, such that they generate
snoops to check for the presence of blocks in upper level caches. It
is possible in our current implmentation for a lower level cache to be
writing back a block while a shared copy of the same block exists in
the upper level cache. If the snoops find the same block in upper
level caches, we set the BLOCK_CACHED flag in the Writeback message.

We have also added logic to account for interaction of other message
types with CleanEvicts waiting in the writeback queue. A simple
example is of a response arriving at a cache removing any CleanEvicts
to the same address from the cache's writeback queue.

mem: Snoop into caches on uncacheable accesses

This patch takes a last step in fixing issues related to uncacheable
accesses. We do not separate uncacheable memory from uncacheable
devices, and in cases where it is really memory, there are valid
scenarios where we need to snoop since we do not support cache
maintenance instructions (yet). On snooping an uncacheable access we
thus provide data if possible. In essence this makes uncacheable
accesses IO coherent.

The snoop filter is also queried to steer the snoops, but not updated
since the uncacheable accesses do not allocate a block.

mem: Add access statistics for the snoop filter

Adds a simple access counter for requests and snoops for the snoop filter and
also classifies hits based on whether a single other holder existed or whether
multiple shares held the line.

mem: Simple Snoop Filter

This is a first cut at a simple snoop filter that tracks presence of lines in
the caches "above" it. The snoop filter can be applied at any given cache
hierarchy and will then handle the caches above it appropriately; there is no
need to use this only in the last-level bus.

This design currently has some limitations: missing stats, no notion of clean
evictions (these will not update the underlying snoop filter, because they are
not sent from the evicting cache down), no notion of capacity for the snoop
filter and thus no need for invalidations caused by capacity pressure in the
snoop filter. These are planned to be added on top with future change sets.