memory_system.doxygen revision 13892
1955SN/A# Copyright (c) 2012 ARM Limited
2955SN/A# All rights reserved
31762SN/A#
4955SN/A# The license below extends only to copyright in the software and shall
5955SN/A# not be construed as granting a license to any other intellectual
6955SN/A# property including but not limited to intellectual property relating
7955SN/A# to a hardware implementation of the functionality of the software
8955SN/A# licensed hereunder.  You may use the software subject to the license
9955SN/A# terms below provided that you ensure that this notice is replicated
10955SN/A# unmodified and in its entirety in all distributions of the software,
11955SN/A# modified or unmodified, in source code or in binary form.
12955SN/A#
13955SN/A# Redistribution and use in source and binary forms, with or without
14955SN/A# modification, are permitted provided that the following conditions are
15955SN/A# met: redistributions of source code must retain the above copyright
16955SN/A# notice, this list of conditions and the following disclaimer;
17955SN/A# redistributions in binary form must reproduce the above copyright
18955SN/A# notice, this list of conditions and the following disclaimer in the
19955SN/A# documentation and/or other materials provided with the distribution;
20955SN/A# neither the name of the copyright holders nor the names of its
21955SN/A# contributors may be used to endorse or promote products derived from
22955SN/A# this software without specific prior written permission.
23955SN/A#
24955SN/A# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
25955SN/A# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
26955SN/A# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
27955SN/A# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
282665Ssaidi@eecs.umich.edu# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
294762Snate@binkert.org# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
30955SN/A# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
315522Snate@binkert.org# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
326143Snate@binkert.org# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
334762Snate@binkert.org# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
345522Snate@binkert.org# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
35955SN/A#
365522Snate@binkert.org# Author: Djordje Kovacevic
37955SN/A
385522Snate@binkert.org/*! \page gem5MemorySystem Memory System in gem5
394202Sbinkertn@umich.edu
405742Snate@binkert.org  \tableofcontents
41955SN/A
424381Sbinkertn@umich.edu  The document describes memory subsystem in gem5 with focus on program flow
434381Sbinkertn@umich.edu  during CPU’s simple memory transactions (read or write).
448334Snate@binkert.org
45955SN/A
46955SN/A  \section gem5_MS_MH MODEL HIERARCHY
474202Sbinkertn@umich.edu
48955SN/A  Model that is used in this document consists of two out-of-order (O3)
494382Sbinkertn@umich.edu  ARM v7 CPUs with corresponding L1 data caches and Simple Memory. It is
504382Sbinkertn@umich.edu  created by running gem5 with the following parameters:
514382Sbinkertn@umich.edu
526654Snate@binkert.org  configs/example/fs.py --caches --cpu-type=arm_detailed --num-cpus=2
535517Snate@binkert.org
548614Sgblack@eecs.umich.edu  Gem5 uses Simulation Objects (SimObject) derived objects as basic blocks for
557674Snate@binkert.org  building memory system. They are connected via ports with established
566143Snate@binkert.org  master/slave hierarchy. Data flow is initiated on master port while the
576143Snate@binkert.org  response messages and snoop queries appear on the slave port. The following
586143Snate@binkert.org  figure shows the hierarchy of Simulation Objects used in this document:
598233Snate@binkert.org
608233Snate@binkert.org  \image html "gem5_MS_Fig1.PNG" "Simulation Object hierarchy of the model" width=3cm
618233Snate@binkert.org
628233Snate@binkert.org  \section gem5_CPU CPU
638233Snate@binkert.org
648334Snate@binkert.org  It is not in the scope of this document to describe O3 CPU model in details, so
658334Snate@binkert.org  here are only a few relevant notes about the model:
668233Snate@binkert.org
678233Snate@binkert.org  <b>Read access </b>is initiated by sending message to the port towards DCache
688233Snate@binkert.org  object. If DCache rejects the message (for being blocked or busy) CPU will
698233Snate@binkert.org  flush the pipeline and the access will be re-attempted later on. The access
708233Snate@binkert.org  is completed upon receiving reply message (ReadRep) from DCache.
718233Snate@binkert.org
726143Snate@binkert.org  <b>Write access</b> is initiated by storing the request into store buffer whose
738233Snate@binkert.org  context is emptied and sent to DCache on every tick. DCache may also reject
748233Snate@binkert.org  the request. Write access is completed when write reply (WriteRep) message is
758233Snate@binkert.org  received from DCache.
766143Snate@binkert.org
776143Snate@binkert.org  Load & store buffers  (for read and write access) don’t impose any
786143Snate@binkert.org  restriction on the number of active memory accesses. Therefore, the maximum
796143Snate@binkert.org  number of outstanding CPU’s memory access requests is not limited by CPU
808233Snate@binkert.org  Simulation Object but by underlying memory system model.
818233Snate@binkert.org
828233Snate@binkert.org  <b>Split memory access</b> is implemented.
836143Snate@binkert.org
848233Snate@binkert.org  The message that is sent by CPU contains memory type (Normal, Device, Strongly
858233Snate@binkert.org  Ordered and cachebility) of the accessed region. However, this is not being used
868233Snate@binkert.org  by the rest of the model that takes more simplified approach towards memory types.
878233Snate@binkert.org
886143Snate@binkert.org  \section gem5_DCache DATA CACHE OBJECT
896143Snate@binkert.org
906143Snate@binkert.org  Data Cache object implements a standard cache structure:
914762Snate@binkert.org
926143Snate@binkert.org  \image html "gem5_MS_Fig2.PNG" "DCache Simulation Object" width=3cm
938233Snate@binkert.org
948233Snate@binkert.org  <b>Cached memory reads</b> that match particular cache tag (with Valid & Read
958233Snate@binkert.org  flags) will be completed (by sending ReadResp to CPU) after a configurable time.
968233Snate@binkert.org  Otherwise, the request is forwarded to Miss Status and Handling Register
978233Snate@binkert.org  (MSHR) block.
986143Snate@binkert.org
998233Snate@binkert.org  <b>Cached memory writes</b> that match particular cache tag (with Valid, Read
1008233Snate@binkert.org  & Write flags) will be completed (by sending WriteResp CPU) after the same
1018233Snate@binkert.org  configurable time. Otherwise, the request is forwarded to Miss Status and
1028233Snate@binkert.org  Handling Register(MSHR) block.
1036143Snate@binkert.org
1046143Snate@binkert.org  <b>Uncached memory reads</b> are forwarded to MSHR block.
1056143Snate@binkert.org
1066143Snate@binkert.org  <b>Uncached memory writes</b> are forwarded to WriteBuffer block.
1076143Snate@binkert.org
1086143Snate@binkert.org  <b>Evicted (& dirty) cache lines</b> are forwarded to WriteBuffer block.
1096143Snate@binkert.org
1106143Snate@binkert.org  CPU’s access to Data Cache is blocked if any of the following is true:
1116143Snate@binkert.org
1127065Snate@binkert.org    - MSHR block is full. (The size of MSHR’s buffer is configurable.)
1136143Snate@binkert.org
1148233Snate@binkert.org    - Writeback block is full. (The size of the block’s buffer is
1158233Snate@binkert.org    configurable.)
1168233Snate@binkert.org
1178233Snate@binkert.org    - The number of outstanding memory accesses against the same memory cache line
1188233Snate@binkert.org    has reached configurable threshold value – see MSHR and Write Buffer for details.
1198233Snate@binkert.org
1208233Snate@binkert.org  Data Cache in block state will reject any request from slave port (from CPU)
1218233Snate@binkert.org  regardless of whether it would result in cache hit or miss. Note that
1228233Snate@binkert.org  incoming messages on master port (response messages and snoop requests)
1238233Snate@binkert.org  are never rejected.
1248233Snate@binkert.org
1258233Snate@binkert.org  Cache hit on uncachable memory region (unpredicted behaviour according to
1268233Snate@binkert.org  ARM ARM) will invalidate cache line and fetch data from memory.
1278233Snate@binkert.org
1288233Snate@binkert.org  \subsection gem5_MS_TAndDBlock Tags & Data Block
1298233Snate@binkert.org
1308233Snate@binkert.org  Cache lines (referred as blocks in source code) are organised into sets with
1318233Snate@binkert.org  configurable associativity and size. They have the following status flags:
1328233Snate@binkert.org    - <b>Valid.</b> It holds data. Address tag is valid
1338233Snate@binkert.org    - <b>Read.</b> No read request will be accepted without this flag being set.
1348233Snate@binkert.org      For example, cache line is valid and unreadable when it waits for write flag
1358233Snate@binkert.org      to complete write access.
1368233Snate@binkert.org    - <b>Write.</b> It may accept writes. Cache line with Write flags
1378233Snate@binkert.org      identifies Unique state – no other cache memory holds the copy.
1388233Snate@binkert.org    - <b>Dirty.</b> It needs Writeback when evicted.
1398233Snate@binkert.org
1408233Snate@binkert.org  Read access will hit cache line if address tags match and Valid and Read
1418233Snate@binkert.org  flags are set. Write access will hit cache line if address tags match and
1428233Snate@binkert.org  Valid, Read and Write flags are set.
1438233Snate@binkert.org
1448233Snate@binkert.org  \subsection gem5_MS_Queues MSHR and Write Buffer Queues
1456143Snate@binkert.org
1466143Snate@binkert.org  Miss Status and Handling Register (MSHR) queue holds the list of CPU’s
1476143Snate@binkert.org  outstanding memory requests that require read access to lower memory
1486143Snate@binkert.org  level. They are:
1496143Snate@binkert.org    - Cached Read misses.
1506143Snate@binkert.org    - Cached Write misses.
1516143Snate@binkert.org    - Uncached reads.
1526143Snate@binkert.org
1536143Snate@binkert.org  WriteBuffer queue holds the following memory requests:
1548913Sandreas.hansson@arm.com    - Uncached writes.
1558233Snate@binkert.org    - Writeback from evicted (& dirty) cache lines.
1568233Snate@binkert.org
1576143Snate@binkert.org  \image html "gem5_MS_Fig3.PNG" "MSHR and Write Buffer Blocks" width=6cm
1586143Snate@binkert.org
1596143Snate@binkert.org  Each memory request is assigned to corresponding MSHR object (READ or WRITE
1606143Snate@binkert.org  on diagram above) that represents particular block (cache line) of memory
1616143Snate@binkert.org  that has to be read or written in order to complete the command(s). As shown
1625522Snate@binkert.org  on gigure above, cached read/writes against the same cache line have a common
1636143Snate@binkert.org  MSHR object and will be completed with a single memory access.
1646143Snate@binkert.org
1656143Snate@binkert.org  The size of the block (and therefore the size of read/write access to lower
1666143Snate@binkert.org  memory) is:
1678233Snate@binkert.org    - The size of cache line for cached access & writeback;
1688233Snate@binkert.org    - As specified in CPU instruction for uncached access.
1698233Snate@binkert.org
1706143Snate@binkert.org  In general, Data Cache model distinguishes between just two memory types:
1716143Snate@binkert.org    - Normal Cached memory. It is always treated as write back, read and write
1726143Snate@binkert.org      allocate.
1736143Snate@binkert.org    - Normal uncached, Device and Strongly Ordered types are treated equally
1745522Snate@binkert.org      (as uncached memory)
1755522Snate@binkert.org
1765522Snate@binkert.org  \subsection gem5_MS_Ordering Memory Access Ordering
1775522Snate@binkert.org
1785604Snate@binkert.org  An unique order number is assigned to each CPU read/write request(as they appear on
1795604Snate@binkert.org  slave port). Order numbers of MSHR objects are copied from the first
1806143Snate@binkert.org  assigned read/write.
1816143Snate@binkert.org
1824762Snate@binkert.org  Memory read/writes from each of these two queues are executed in order (according
1834762Snate@binkert.org  to the assigned order number). When both queues are not empty the model will
1846143Snate@binkert.org  execute memory read from MSHR block unless WriteBuffer is full. It will,
1856727Ssteve.reinhardt@amd.com  however, always preserve the order of read/writes on the same
1866727Ssteve.reinhardt@amd.com  (or overlapping) memory cache line (block).
1876727Ssteve.reinhardt@amd.com
1884762Snate@binkert.org  In summary:
1896143Snate@binkert.org    - Order of accesses to cached memory is not preserved unless they target
1906143Snate@binkert.org      the same cache line. For example, the accesses #1, #5 & #10 will
1916143Snate@binkert.org      complete simultaneously in the same tick (still in order). The access
1926143Snate@binkert.org      #5 will complete before #3.
1936727Ssteve.reinhardt@amd.com    - Order of all uncached memory writes is preserved. Write#6 always
1946143Snate@binkert.org      completes before Write#13.
1957674Snate@binkert.org    - Order to all uncached memory reads is preserved. Read#2 always completes
1967674Snate@binkert.org      before Read#8.
1975604Snate@binkert.org    - The order of a read and a write uncached access is not necessarily
1986143Snate@binkert.org      preserved  - unless their access regions overlap. Therefore, Write#6
1996143Snate@binkert.org      always completes before Read#8 (they target the same memory block).
2006143Snate@binkert.org      However, Write#13 may complete before Read#8.
2014762Snate@binkert.org
2026143Snate@binkert.org
2034762Snate@binkert.org  \section gem5_MS_Bus COHERENT BUS OBJECT
2044762Snate@binkert.org
2054762Snate@binkert.org  \image html "gem5_MS_Fig4.PNG" "Coherent Bus Object" width=3cm
2066143Snate@binkert.org
2076143Snate@binkert.org  Coherent Bus object provides basic support for snoop protocol:
2084762Snate@binkert.org
2098233Snate@binkert.org  <b>All requests on the slave port</b> are forwarded to the appropriate master port. Requests
2108233Snate@binkert.org  for cached memory regions are also forwarded to other slave ports (as snoop
2118233Snate@binkert.org  requests).
2128233Snate@binkert.org
2136143Snate@binkert.org  <b>Master port replies</b> are forwarded to the appropriate slave port.
2146143Snate@binkert.org
2154762Snate@binkert.org  <b>Master port snoop requests</b> are forwarded to all slave ports.
2166143Snate@binkert.org
2174762Snate@binkert.org  <b>Slave port snoop replies</b> are forwarded to the port that was the source of the
2186143Snate@binkert.org  request. (Note that the source of snoop request can be either slave or
2194762Snate@binkert.org  master port.)
2206143Snate@binkert.org
2218233Snate@binkert.org  The bus declares itself blocked for a configurable period of time after
2228233Snate@binkert.org  any of the following events:
2238233Snate@binkert.org    - A packet is sent (or failed to be sent) to a slave port.
2246143Snate@binkert.org    - A reply message is sent to a master port.
2256143Snate@binkert.org    - Snoop response from one slave port is sent to another slave port.
2266143Snate@binkert.org
2276143Snate@binkert.org  The bus in blocked state rejects the following incoming messages:
2286143Snate@binkert.org    - Slave port requests.
2296143Snate@binkert.org    - Master port replies.
2306143Snate@binkert.org    - Master port snoop requests.
2316143Snate@binkert.org
2328233Snate@binkert.org  \section gem5_MS_SimpleMemory SIMPLE MEMORY OBJECT
2338233Snate@binkert.org
234955SN/A  It never blocks the access on slave port.
2358235Snate@binkert.org
2368235Snate@binkert.org  Memory read/write takes immediate effect. (Read or write is performed when
2376143Snate@binkert.org  the request is received).
2388235Snate@binkert.org
2398235Snate@binkert.org  Reply message is sent after a configurable period of time .
2408235Snate@binkert.org
2418235Snate@binkert.org  \section gem5_MS_MessageFlow MESSAGE FLOW
2428235Snate@binkert.org
2438235Snate@binkert.org  \subsection gem5_MS_Ordering Read Access
2448235Snate@binkert.org
2458235Snate@binkert.org  The following diagram shows read access that hits Data Cache line with Valid
2468235Snate@binkert.org  and Read flags:
2478235Snate@binkert.org
2488235Snate@binkert.org  \image html "gem5_MS_Fig5.PNG" "Read Hit (Read flag must be set in cache line)" width=3cm
2498235Snate@binkert.org
2508235Snate@binkert.org  Cache miss read access will generate the following sequence of messages:
2518235Snate@binkert.org
2528235Snate@binkert.org  \image html "gem5_MS_Fig6.PNG" "Read Miss with snoop reply" width=3cm
2538235Snate@binkert.org
2548235Snate@binkert.org  Note that bus object never gets response from both DCache2 and Memory object.
2555584Snate@binkert.org  It sends the very same ReadReq package (message) object to memory and data
2564382Sbinkertn@umich.edu  cache. When Data Cache wants to reply on snoop request it marks the message
2574202Sbinkertn@umich.edu  with MEM_INHIBIT flag that tells Memory object not to process the message.
2584382Sbinkertn@umich.edu
2594382Sbinkertn@umich.edu  \subsection gem5_MS_Ordering Write Access
2604382Sbinkertn@umich.edu
2615584Snate@binkert.org  The following diagram shows write access that hits DCache1 cache line with
2624382Sbinkertn@umich.edu  Valid & Write flags:
2634382Sbinkertn@umich.edu
2644382Sbinkertn@umich.edu  \image html "gem5_MS_Fig7.PNG" "Write Hit (with Write flag set in cache line)" width=3cm
2658232Snate@binkert.org
2665192Ssaidi@eecs.umich.edu  Next figure shows write access that hits DCache1 cache line with Valid but no
2678232Snate@binkert.org  Write flags – which qualifies as write miss. DCache1 issues UpgradeReq to
2688232Snate@binkert.org  obtain write permission. DCache2::snoopTiming will invalidate cache line that
2698232Snate@binkert.org  has been hit. Note that UpgradeResp message doesn’t carry data.
2705192Ssaidi@eecs.umich.edu
2718232Snate@binkert.org  \image html "gem5_MS_Fig8.PNG" "Write Miss – matching tag with no Write flag" width=3cm
2725192Ssaidi@eecs.umich.edu
2735799Snate@binkert.org  The next diagram shows write miss in DCache. ReadExReq invalidates cache line
2748232Snate@binkert.org  in DCache2. ReadExResp carries the content of memory cache line.
2755192Ssaidi@eecs.umich.edu
2765192Ssaidi@eecs.umich.edu  \image html "gem5_MS_Fig9.PNG" "Miss - no matching tag" width=3cm
2775192Ssaidi@eecs.umich.edu
2788232Snate@binkert.org*/
2795192Ssaidi@eecs.umich.edu