cpu,mem: missing override specifier

Change-Id: I731d3ef021596450ac307461f215760a148bb28a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18348
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>

mem-cache: Add match functions to QueueEntry

Having the caller decide the matching logic is error-prone, and
frequently ends up with the secure bit being forgotten. This
change adds matching functions to the QueueEntry to avoid this
problem.

As a side effect the signature of findPending has been changed.

Change-Id: I6e494a821c1e6e841ab103ec69632c0e1b269a08
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17530
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem-cache: Move Target to QueueEntry

WriteQueueEntry's target has 100% functionality overlap with MSHR's,
therefore make it base to MSHR::Target.

Change-Id: I48614e78179d708bd91bbe75a752e5a05146e8eb
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/17534
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem: Delay servicing an MSHR after its allocation

An MSHR is allocated and the computed latency determines when the MSHR
will be ready and can be serviced by the cache. This patch adds a
function that allows changing the time that an MSHR is ready and
adjusts the queue such that other MSHRs can be serviced first if they
are ready.

Change-Id: Ie908191fcb3c2d84d4c6f855c8b1e41ca5881bff
Reviewed-on: https://gem5-review.googlesource.com/c/12906
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem: Determine if an MSHR does a whole-line write

This patch adds support for determining whether the targets in an MSHR
are 1) only writes and 2) whether these writes are effectively a
whole-line write. This patch adds the necessary functions in the MSHR
to allow for write coalescing in the cache.

Change-Id: I2c9a9a83d2d9b506a491ba5b0b9ac1054bdb31b4
Reviewed-on: https://gem5-review.googlesource.com/c/12904
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem: Rename Packet::checkFunctional to trySatisfyFunctional

Packet::checkFunctional also wrote data to/from the packet depending
on if it was read/write, respectively, which the 'check' in the name
would suggest otherwise. This renames it to doFunctional, which is
more suggestive. It also renames any function called checkFunctional
which calls Packet::checkFunctional. These are

- Bridge::BridgeMasterPort::checkFunctional
- calls Packet::checkFunctional
- MSHR::checkFunctional
- calls Packet::checkFunctional
- MSHR::TargetList::checkFunctional
- calls Packet::checkFunctional
- Queue<>::checkFunctional
(of src/mem/cache/queue.hh, not src/cpu/minor/buffers.h)
- Instantiated with Queue<WriteQueueEntry> and Queue<MSHR>
- WriteQueueEntry
- calls Packet::checkFunctional
- WriteQueueEntry::TargetList
- calls Packet::checkFunctional
- MemDelay::checkFunctional
- calls QueuedSlavePort/QueuedMasterPort::checkFunctional
- Packet::checkFunctional
- PacketQueue::checkFunctional
- calls Packet::checkFunctional
- QueuedSlavePort::checkFunctional
- calls PacketQueue::doFunctional
- QueuedMasterPort::checkFunctional
- calls PacketQueue::doFunctional
- SerialLink::SerialLinkMasterPort::checkFunctional
- calls Packet::doFunctional

Change-Id: Ieca2579c020c329040da053ba8e25820801b62c5
Reviewed-on: https://gem5-review.googlesource.com/11810
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

mem-cache: Promote targets that don't require writable

Until now, all deferred targets of an MSHR would be promoted together
as soon as the targets were serviced. Due to the way we handle cache
clean operations we might need to promote only deferred targets that
don't require writable, leaving some targets as deferred. This change
adds support for this selective promotion.

Change-Id: I502e523dc9adbaf394955cbacea8286ab6a9b6bc
Reviewed-on: https://gem5-review.googlesource.com/11017
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

mem-cache: Fix promoting of targets that need writable

There are cases where a request which does not need a writable copy
gets an response upgraded reponse and fills in a writable copy. When
this happens, we promote deferred MSHR targets that were deferred
because they needed a writable copy to service them immediately.

Previously, we would uncoditionally promote deferred targets. Since
the deferred targets might contain a cache invalidation operation, we
have to make sure that any targets following the cache invalidation is
not promoted.

Change-Id: I1f7b28f7d35f84329e065c8f63117db21852365a
Reviewed-on: https://gem5-review.googlesource.com/11016
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

mem-cache: Selectively clear downstream pending

Until now, all deferred targets of an MSHR would be promoted together
as soon as the targets were serviced. When we promote deferred targets
we also clear the downstreamPending flag.

Due to the way we handle cache clean operations we might need to
promote only deferred targets that don't require writable, leaving
some targets as deferred. To allow for partial target promotion, this
change adds support for clearing the downstreamPending only for a
subset of a TargetsList.

Change-Id: Id06953643ba9a975ebacc76ac10215441e264e74
Reviewed-on: https://gem5-review.googlesource.com/11015
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

mem-cache: Fix include directives in the cache related classes

Change-Id: I111b0f662897c43974aadb08da1ed85c7542585c
Reviewed-on: https://gem5-review.googlesource.com/10433
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem-cache: Adopt a more sensible cache class hierarchy

This patch changes what goes into the BaseCache and what goes into the
Cache, to make it easier to add a NoncoherentCache with as much re-use
as possible. A number of redundant members and definitions are also
removed in the process.

This is a modified version of a changeset put together by Andreas
Hansson <andreas.hansson@arm.com>

Change-Id: Ie9dd73c4ec07732e778e7416b712dad8b4bd5d4b
Reviewed-on: https://gem5-review.googlesource.com/10431
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem-cache: Determine if an MSHR has requests from another cache

To decide whether we allocate upon receiving a response we need to
determine if any of the currently serviced requests (non-deferred
targets) is comming from another cache. This change adds support for
tracking this information in the MSHR.

Change-Id: If1db93c12b6af5813b91b9d6b6e5e196d327f038
Reviewed-on: https://gem5-review.googlesource.com/10422
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Nikos Nikoleris <nikos.nikoleris@arm.com>

mem-cache: Allow clean operations when block allocation fails

Block allocation can fail when there is an in-service MSHR that
operates on the victim block. This can happed due to:
* an upgrade operation: a request that needs a writable copy of the
block finds a shared (non-writable) copy of the block in the cache
and has allocates an MSHR for the pending upgrade operation, or
* a clean operation: a clean request finds a dirty copy of the block
and allocates an MSHR for the pending clean operation.
This changes relaxes an assertion to allow for the 2nd case (cache
clean operations).

Change-Id: Ib51482160b5f2b3702ed744b0eac2029d34bc9d4
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/9021
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>

mem: Service only the 1st FromCPU MSHR target on ReadRespWithInv

A response to a ReadReq can either be a ReadResp or a
ReadRespWithInvalidate. As we add targets to an MSHR for a ReadReq we
assume that the response will be a ReadResp. When the response is
invalidating (ReadRespWithInvalidate) servicing more than one targets
can potentially violate the memory ordering. This change fixes the way
we handle a ReadRespWithInvalidate. When a cache receives a
ReadRespWithInvalidate we service only the first FromCPU target and
all the FromSnoop targets from the MSHR target list. The rest of the
FromCPU targets are deferred and serviced by a new request.

Change-Id: I75c30c268851987ee5f8644acb46f440b4eeeec2
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>

mem: Keep track of allocOnFill in the TargetList

Previously the information of whether a response was allocating or not
was a property of the MSHR. This change makes this flag a property of
the TargetList. Differernt TargetLists, e.g. the targets and the
deferred targets lists might have different values. Additionally, the
information about whether each of the target expects an allocating
response is stored inside the TargetList container. This allows for
repopulating the flag in case some of the targets are removed.

Change-Id: If3ec2516992f42a6d9da907009ffe3ab8d0d2021
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>

mem: Add support for repopulating the flags of an MSHR TargetList

This patch adds support for repopulating the flags of an MSHR
TargetList. The added functionality makes it possible to remove
targets from a TargetList without leaving it in an inconsistent state.

Change-Id: I3f7a8e97bfd3e2e49bebad056d11bbfb087aad91
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Stephan Diestelhorst <stephan.diestelhorst@arm.com>

mem: Create a separate class for the cache write buffer

This patch breaks out the cache write buffer into a separate class,
without affecting any stats. The goal of the patch is to avoid
encumbering the much-simpler write queue with the complex MSHR
handling. In a follow on patch this simplification allows us to
implement write combining.

The WriteQueue gets its own class, but shares a common ancestor, the
generic Queue, with the MSHRQueue.

mem, cpu: Add assertions to snoop invalidation logic

This patch adds assertions that enforce that only invalidating snoops
will ever reach into the logic that tracks in-order load completion and
also invalidation of LL/SC (and MONITOR / MWAIT) monitors. Also adds
some comments to MSHR::replaceUpgrades().

mem: Make cache terminology easier to understand

This patch changes the name of a bunch of packet flags and MSHR member
functions and variables to make the coherency protocol easier to
understand. In addition the patch adds and updates lots of
descriptions, explicitly spelling out assumptions.

The following name changes are made:

* the packet memInhibit flag is renamed to cacheResponding

* the packet sharedAsserted flag is renamed to hasSharers

* the packet NeedsExclusive attribute is renamed to NeedsWritable

* the packet isSupplyExclusive is renamed responderHadWritable

* the MSHR pendingDirty is renamed to pendingModified

The cache states, Modified, Owned, Exclusive, Shared are also called
out in the cache and MSHR code to make it easier to understand.

mem: Remove unused cache squash functionality

This patch removes the unused squash function from the MSHR queue, and
the associated (and also unused) threadNum member from the MSHR.

mem: Add cache clusivity

This patch adds a parameter to control the cache clusivity, that is if
the cache is mostly inclusive or exclusive. At the moment there is no
intention to support strict policies, and thus the options are: 1)
mostly inclusive, or 2) mostly exclusive.

The choice of policy guides the behaviuor on a cache fill, and a new
helper function, allocOnFill, is created to encapsulate the decision
making process. For the timing mode, the decision is annotated on the
MSHR on sending out the downstream packet, and in atomic we directly
pass the decision to handleFill. We (ab)use the tempBlock in cases
where we are not allocating on fill, leaving the rest of the cache
unaffected. Simple and effective.

This patch also makes it more explicit that multiple caches are
allowed to consider a block writable (this is the case
also before this patch). That is, for a mostly inclusive cache,
multiple caches upstream may also consider the block exclusive. The
caches considering the block writable/exclusive all appear along the
same path to memory, and from a coherency protocol point of view it
works due to the fact that we always snoop upwards in zero time before
querying any downstream cache.

Note that this patch does not introduce clean writebacks. Thus, for
clean lines we are essentially removing a cache level if it is made
mostly exclusive. For example, lines from the read-only L1 instruction
cache or table-walker cache are always clean, and simply get dropped
rather than being passed to the L2. If the L2 is mostly exclusive and
does not allocate on fill it will thus never hold the line. A follow
on patch adds the clean writebacks.

The patch changes the L2 of the O3_ARM_v7a CPU configuration to be
mostly exclusive (and stats are affected accordingly).

mem: Clarify cache MSHR handling on fill

This patch addresses the upgrading of deferred targets in the MSHR,
and makes it clearer by explicitly calling out what is happening
(deferred targets are promoted if we get exclusivity without asking
for it).

mem: Modernise MSHR iterators to C++11

This patch updates the iterators in the MSHR and MSHR queues to use
C++11 range-based for loops. It also does a bit of additional house
keeping.

mem: Align all MSHR entries to block boundaries

This patch aligns all MSHR queue entries to block boundaries to
simplify checks for matches. Previously there were corner cases that
could lead to existing entries not being identified as matches.

There are, rather alarmingly, a few regressions that change with this
patch.

mem: Clarify cache behaviour for pending dirty responses

This patch adds a bit of clarification around the assumptions made in
the cache when packets are sent out, and dirty responses are
pending. As part of the change, the marking of an MSHR as in service
is simplified slightly, and comments are added to explain what
assumptions are made.

mem: Remove WriteInvalidate support

Prepare for a different implementation following in the next patch

mem: don't inhibit WriteInv's or defer snoops on their MSHRs

WriteInvalidate semantics depend on the unconditional writeback
or they won't complete. Also, there's no point in deferring snoops
on their MSHRs, as they don't get new data at the end of their life
cycle the way other transactions do.

Add comment in the cache about a minor inefficiency re: WriteInvalidate.

mem: have WriteInvalidate obsolete MSHRs

Since WriteInvalidate directly writes into the cache, it can
create tricky timing interleavings with reads and writes to the
same cache line that haven't yet completed. This patch ensures
that these requests, when completed, don't overwrite the newer
data from the WriteInvalidate.

mem: Add support for a security bit in the memory system

This patch adds the basic building blocks required to support e.g. ARM
TrustZone by discerning secure and non-secure memory accesses.

mem: Spring cleaning of MSHR and MSHRQueue

This patch does some minor tidying up of the MSHR and MSHRQueue. The
clean up started as part of some ad-hoc tracing and debugging, but
seems worthwhile enough to go in as a separate patch.

The highlights of the changes are reduced scoping (private) members
where possible, avoiding redundant new/delete, and constructor
initialisation to please static code analyzers.

mem: Adding verbose debug output in the memory system

This patch provides useful printouts throughut the memory system. This
includes pretty-printed cache tags and function call messages
(call-stack like).

MEM: Put memory system document into doxygen

Replace curTick global variable with accessor functions.
This step makes it easy to replace the accessor functions
(which still access a global variable) with ones that access
per-thread curTick values.

cache: coherence protocol enhancements & bug fixes
Allow lower-level caches (e.g., L2 or L3) to pass exclusive
copies to higher levels (e.g., L1). This eliminates a lot
of unnecessary upgrade transactions on read-write sequences
to non-shared data.

Also some cleanup of MSHR coherence handling and multiple
bug fixes.

types: clean up types, especially signed vs unsigned

types: add a type for thread IDs and try to use it everywhere

Fixes to get prefetching working again.
Apparently we broke it with the cache rewrite and never noticed.
Thanks to Bao Yungang <baoyungang@gmail.com> for a significant part
of these changes (and for inspiring me to work on the rest).
Some other overdue cleanup on the prefetch code too.

Cache: Refactor packet forwarding a bit.
Makes adding write-through operations easier.

Rename cache files for brevity and consistency with rest of tree.