HMC.py revision 12340
110152Satgutier@umich.edu# Copyright (c) 2012-2013 ARM Limited 210152Satgutier@umich.edu# All rights reserved. 310152Satgutier@umich.edu# 410152Satgutier@umich.edu# The license below extends only to copyright in the software and shall 510152Satgutier@umich.edu# not be construed as granting a license to any other intellectual 610152Satgutier@umich.edu# property including but not limited to intellectual property relating 710152Satgutier@umich.edu# to a hardware implementation of the functionality of the software 810152Satgutier@umich.edu# licensed hereunder. You may use the software subject to the license 910152Satgutier@umich.edu# terms below provided that you ensure that this notice is replicated 1010152Satgutier@umich.edu# unmodified and in its entirety in all distributions of the software, 1110152Satgutier@umich.edu# modified or unmodified, in source code or in binary form. 1210152Satgutier@umich.edu# 1310152Satgutier@umich.edu# Copyright (c) 2015 The University of Bologna 1410152Satgutier@umich.edu# All rights reserved. 1510152Satgutier@umich.edu# 1610152Satgutier@umich.edu# Redistribution and use in source and binary forms, with or without 1710152Satgutier@umich.edu# modification, are permitted provided that the following conditions are 1810152Satgutier@umich.edu# met: redistributions of source code must retain the above copyright 1910152Satgutier@umich.edu# notice, this list of conditions and the following disclaimer; 2010152Satgutier@umich.edu# redistributions in binary form must reproduce the above copyright 2110152Satgutier@umich.edu# notice, this list of conditions and the following disclaimer in the 2210152Satgutier@umich.edu# documentation and/or other materials provided with the distribution; 2310152Satgutier@umich.edu# neither the name of the copyright holders nor the names of its 2410152Satgutier@umich.edu# contributors may be used to endorse or promote products derived from 2510152Satgutier@umich.edu# this software without specific prior written permission. 2610152Satgutier@umich.edu# 2710152Satgutier@umich.edu# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 2810152Satgutier@umich.edu# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 2910152Satgutier@umich.edu# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 3010152Satgutier@umich.edu# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 3110152Satgutier@umich.edu# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 3210152Satgutier@umich.edu# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 3310152Satgutier@umich.edu# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 3410152Satgutier@umich.edu# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 3510152Satgutier@umich.edu# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 3610152Satgutier@umich.edu# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 3710152Satgutier@umich.edu# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 3810152Satgutier@umich.edu# 3910152Satgutier@umich.edu# Authors: Erfan Azarkhish 4010152Satgutier@umich.edu# Abdul Mutaal Ahmad 4110152Satgutier@umich.edu 4210152Satgutier@umich.edu# A Simplified model of a complete HMC device. Based on: 4310152Satgutier@umich.edu# [1] http://www.hybridmemorycube.org/specification-download/ 4410152Satgutier@umich.edu# [2] High performance AXI-4.0 based interconnect for extensible smart memory 4510152Satgutier@umich.edu# cubes(E. Azarkhish et. al) 4610152Satgutier@umich.edu# [3] Low-Power Hybrid Memory Cubes With Link Power Management and Two-Level 4710152Satgutier@umich.edu# Prefetching (J. Ahn et. al) 4810152Satgutier@umich.edu# [4] Memory-centric system interconnect design with Hybrid Memory Cubes 4910152Satgutier@umich.edu# (G. Kim et. al) 5010152Satgutier@umich.edu# [5] Near Data Processing, Are we there yet? (M. Gokhale) 5110152Satgutier@umich.edu# http://www.cs.utah.edu/wondp/gokhale.pdf 5210152Satgutier@umich.edu# [6] openHMC - A Configurable Open-Source Hybrid Memory Cube Controller 5310152Satgutier@umich.edu# (J. Schmidt) 5410152Satgutier@umich.edu# [7] Hybrid Memory Cube performance characterization on data-centric 5510152Satgutier@umich.edu# workloads (M. Gokhale) 5610152Satgutier@umich.edu# 5710152Satgutier@umich.edu# This script builds a complete HMC device composed of vault controllers, 5810152Satgutier@umich.edu# serial links, the main internal crossbar, and an external hmc controller. 5910152Satgutier@umich.edu# 6010152Satgutier@umich.edu# - VAULT CONTROLLERS: 6110152Satgutier@umich.edu# Instances of the HMC_2500_1x32 class with their functionality specified in 6210152Satgutier@umich.edu# dram_ctrl.cc 6310152Satgutier@umich.edu# 6410152Satgutier@umich.edu# - THE MAIN XBAR: 6510152Satgutier@umich.edu# This component is simply an instance of the NoncoherentXBar class, and its 6610152Satgutier@umich.edu# parameters are tuned to [2]. 6710152Satgutier@umich.edu# 6810152Satgutier@umich.edu# - SERIAL LINKS CONTROLLER: 6910152Satgutier@umich.edu# SerialLink is a simple variation of the Bridge class, with the ability to 7010152Satgutier@umich.edu# account for the latency of packet serialization and controller latency. We 7110152Satgutier@umich.edu# assume that the serializer component at the transmitter side does not need 7210152Satgutier@umich.edu# to receive the whole packet to start the serialization. But the 7310152Satgutier@umich.edu# deserializer waits for the complete packet to check its integrity first. 7410152Satgutier@umich.edu# 7510152Satgutier@umich.edu# * Bandwidth of the serial links is not modeled in the SerialLink component 7610152Satgutier@umich.edu# itself. 7710152Satgutier@umich.edu# 7810152Satgutier@umich.edu# * Latency of serial link controller is composed of SerDes latency + link 7910152Satgutier@umich.edu# controller 8010152Satgutier@umich.edu# 8110152Satgutier@umich.edu# * It is inferred from the standard [1] and the literature [3] that serial 8210152Satgutier@umich.edu# links share the same address range and packets can travel over any of 8310152Satgutier@umich.edu# them so a load distribution mechanism is required among them. 8410152Satgutier@umich.edu# 8510152Satgutier@umich.edu# ----------------------------------------- 8610152Satgutier@umich.edu# | Host/HMC Controller | 8710152Satgutier@umich.edu# | ---------------------- | 8810152Satgutier@umich.edu# | | Link Aggregator | opt | 8910152Satgutier@umich.edu# | ---------------------- | 9010152Satgutier@umich.edu# | ---------------------- | 9110152Satgutier@umich.edu# | | Serial Link + Ser | * 4 | 9210152Satgutier@umich.edu# | ---------------------- | 9310152Satgutier@umich.edu# |--------------------------------------- 9410152Satgutier@umich.edu# ----------------------------------------- 9510152Satgutier@umich.edu# | Device 9610152Satgutier@umich.edu# | ---------------------- | 9710152Satgutier@umich.edu# | | Xbar | * 4 | 9810152Satgutier@umich.edu# | ---------------------- | 9910152Satgutier@umich.edu# | ---------------------- | 10010152Satgutier@umich.edu# | | Vault Controller | * 16 | 10110152Satgutier@umich.edu# | ---------------------- | 10210152Satgutier@umich.edu# | ---------------------- | 10310152Satgutier@umich.edu# | | Memory | | 10410152Satgutier@umich.edu# | ---------------------- | 10510152Satgutier@umich.edu# |---------------------------------------| 10610152Satgutier@umich.edu# 10710152Satgutier@umich.edu# In this version we have present 3 different HMC archiecture along with 10810152Satgutier@umich.edu# alongwith their corresponding test script. 10910152Satgutier@umich.edu# 11010152Satgutier@umich.edu# same: It has 4 crossbars in HMC memory. All the crossbars are connected 11110152Satgutier@umich.edu# to each other, providing complete memory range. This archicture also covers 11210152Satgutier@umich.edu# the added latency for sending a request to non-local vault(bridge in b/t 11310152Satgutier@umich.edu# crossbars). All the 4 serial links can access complete memory. So each 11410152Satgutier@umich.edu# link can be connected to separate processor. 11510152Satgutier@umich.edu# 11610152Satgutier@umich.edu# distributed: It has 4 crossbars inside the HMC. Crossbars are not 11710152Satgutier@umich.edu# connected.Through each crossbar only local vaults can be accessed. But to 11810152Satgutier@umich.edu# support this architecture we need a crossbar between serial links and 11910152Satgutier@umich.edu# processor. 12010152Satgutier@umich.edu# 12110152Satgutier@umich.edu# mixed: This is a hybrid architecture. It has 4 crossbars inside the HMC. 12210152Satgutier@umich.edu# 2 Crossbars are connected to only local vaults. From other 2 crossbar, a 12310152Satgutier@umich.edu# request can be forwarded to any other vault. 12410152Satgutier@umich.edu 12510152Satgutier@umich.eduimport argparse 12610152Satgutier@umich.edu 12710152Satgutier@umich.eduimport m5 12810152Satgutier@umich.edufrom m5.objects import * 12910152Satgutier@umich.edufrom m5.util import * 13010152Satgutier@umich.edu 13110152Satgutier@umich.edu 13210152Satgutier@umich.edudef add_options(parser): 13310152Satgutier@umich.edu # *****************************CROSSBAR PARAMETERS************************* 13410152Satgutier@umich.edu # Flit size of the main interconnect [1] 13510152Satgutier@umich.edu parser.add_argument("--xbar-width", default=32, action="store", type=int, 13610152Satgutier@umich.edu help="Data width of the main XBar (Bytes)") 13710152Satgutier@umich.edu 13810152Satgutier@umich.edu # Clock frequency of the main interconnect [1] 13910152Satgutier@umich.edu # This crossbar, is placed on the logic-based of the HMC and it has its 14010152Satgutier@umich.edu # own voltage and clock domains, different from the DRAM dies or from the 14110152Satgutier@umich.edu # host. 14210152Satgutier@umich.edu parser.add_argument("--xbar-frequency", default='1GHz', type=str, 14310152Satgutier@umich.edu help="Clock Frequency of the main XBar") 14410152Satgutier@umich.edu 14510152Satgutier@umich.edu # Arbitration latency of the HMC XBar [1] 14610152Satgutier@umich.edu parser.add_argument("--xbar-frontend-latency", default=1, action="store", 14710152Satgutier@umich.edu type=int, help="Arbitration latency of the XBar") 14810152Satgutier@umich.edu 14910152Satgutier@umich.edu # Latency to forward a packet via the interconnect [1](two levels of FIFOs 15010152Satgutier@umich.edu # at the input and output of the inteconnect) 15110152Satgutier@umich.edu parser.add_argument("--xbar-forward-latency", default=2, action="store", 15210152Satgutier@umich.edu type=int, help="Forward latency of the XBar") 15310152Satgutier@umich.edu 15410152Satgutier@umich.edu # Latency to forward a response via the interconnect [1](two levels of 15510152Satgutier@umich.edu # FIFOs at the input and output of the inteconnect) 15610152Satgutier@umich.edu parser.add_argument("--xbar-response-latency", default=2, action="store", 15710152Satgutier@umich.edu type=int, help="Response latency of the XBar") 15810152Satgutier@umich.edu 15910152Satgutier@umich.edu # number of cross which connects 16 Vaults to serial link[7] 16010152Satgutier@umich.edu parser.add_argument("--number-mem-crossbar", default=4, action="store", 16110152Satgutier@umich.edu type=int, help="Number of crossbar in HMC") 16210152Satgutier@umich.edu 16310152Satgutier@umich.edu # *****************************SERIAL LINK PARAMETERS********************** 16410152Satgutier@umich.edu # Number of serial links controllers [1] 16510152Satgutier@umich.edu parser.add_argument("--num-links-controllers", default=4, action="store", 16610152Satgutier@umich.edu type=int, help="Number of serial links") 16710152Satgutier@umich.edu 16810152Satgutier@umich.edu # Number of packets (not flits) to store at the request side of the serial 16910152Satgutier@umich.edu # link. This number should be adjusted to achive required bandwidth 17010152Satgutier@umich.edu parser.add_argument("--link-buffer-size-req", default=10, action="store", 17110152Satgutier@umich.edu type=int, help="Number of packets to buffer at the\ 17210152Satgutier@umich.edu request side of the serial link") 17310152Satgutier@umich.edu 17410152Satgutier@umich.edu # Number of packets (not flits) to store at the response side of the serial 17510152Satgutier@umich.edu # link. This number should be adjusted to achive required bandwidth 17610152Satgutier@umich.edu parser.add_argument("--link-buffer-size-rsp", default=10, action="store", 17710152Satgutier@umich.edu type=int, help="Number of packets to buffer at the\ 17810152Satgutier@umich.edu response side of the serial link") 17910152Satgutier@umich.edu 18010152Satgutier@umich.edu # Latency of the serial link composed by SER/DES latency (1.6ns [4]) plus 18110152Satgutier@umich.edu # the PCB trace latency (3ns Estimated based on [5]) 18210152Satgutier@umich.edu parser.add_argument("--link-latency", default='4.6ns', type=str, 18310152Satgutier@umich.edu help="Latency of the serial links") 18410152Satgutier@umich.edu 18510152Satgutier@umich.edu # Clock frequency of the each serial link(SerDes) [1] 18610152Satgutier@umich.edu parser.add_argument("--link-frequency", default='10GHz', type=str, 18710152Satgutier@umich.edu help="Clock Frequency of the serial links") 18810152Satgutier@umich.edu 18910152Satgutier@umich.edu # Clock frequency of serial link Controller[6] 19010152Satgutier@umich.edu # clk_hmc[Mhz]= num_lanes_per_link * lane_speed [Gbits/s] / 19110152Satgutier@umich.edu # data_path_width * 10^6 19210152Satgutier@umich.edu # clk_hmc[Mhz]= 16 * 10 Gbps / 256 * 10^6 = 625 Mhz 19310152Satgutier@umich.edu parser.add_argument("--link-controller-frequency", default='625MHz', 19410152Satgutier@umich.edu type=str, help="Clock Frequency of the link\ 19510152Satgutier@umich.edu controller") 19610152Satgutier@umich.edu 19710152Satgutier@umich.edu # Latency of the serial link controller to process the packets[1][6] 19810152Satgutier@umich.edu # (ClockDomain = 625 Mhz ) 19910152Satgutier@umich.edu # used here for calculations only 20010152Satgutier@umich.edu parser.add_argument("--link-ctrl-latency", default=4, action="store", 20110152Satgutier@umich.edu type=int, help="The number of cycles required for the\ 20210152Satgutier@umich.edu controller to process the packet") 20310152Satgutier@umich.edu 20410152Satgutier@umich.edu # total_ctrl_latency = link_ctrl_latency + link_latency 20510152Satgutier@umich.edu # total_ctrl_latency = 4(Cycles) * 1.6 ns + 4.6 ns 20610152Satgutier@umich.edu parser.add_argument("--total-ctrl-latency", default='11ns', type=str, 20710152Satgutier@umich.edu help="The latency experienced by every packet\ 20810152Satgutier@umich.edu regardless of size of packet") 20910152Satgutier@umich.edu 21010152Satgutier@umich.edu # Number of parallel lanes in each serial link [1] 21110152Satgutier@umich.edu parser.add_argument("--num-lanes-per-link", default=16, action="store", 21210152Satgutier@umich.edu type=int, help="Number of lanes per each link") 21310152Satgutier@umich.edu 21410152Satgutier@umich.edu # Number of serial links [1] 21510152Satgutier@umich.edu parser.add_argument("--num-serial-links", default=4, action="store", 21610152Satgutier@umich.edu type=int, help="Number of serial links") 21710152Satgutier@umich.edu 21810152Satgutier@umich.edu # speed of each lane of serial link - SerDes serial interface 10 Gb/s 21910152Satgutier@umich.edu parser.add_argument("--serial-link-speed", default=10, action="store", 22010152Satgutier@umich.edu type=int, help="Gbs/s speed of each lane of serial\ 22110152Satgutier@umich.edu link") 22210152Satgutier@umich.edu 22310152Satgutier@umich.edu # address range for each of the serial links 22410152Satgutier@umich.edu parser.add_argument("--serial-link-addr-range", default='1GB', type=str, 22510152Satgutier@umich.edu help="memory range for each of the serial links.\ 22610152Satgutier@umich.edu Default: 1GB") 22710152Satgutier@umich.edu 22810152Satgutier@umich.edu # *****************************PERFORMANCE MONITORING********************* 22910152Satgutier@umich.edu # The main monitor behind the HMC Controller 23010152Satgutier@umich.edu parser.add_argument("--enable-global-monitor", action="store_true", 23110152Satgutier@umich.edu help="The main monitor behind the HMC Controller") 23210152Satgutier@umich.edu 23310152Satgutier@umich.edu # The link performance monitors 23410152Satgutier@umich.edu parser.add_argument("--enable-link-monitor", action="store_true", 23510152Satgutier@umich.edu help="The link monitors") 23610152Satgutier@umich.edu 23710152Satgutier@umich.edu # link aggregator enable - put a cross between buffers & links 23810152Satgutier@umich.edu parser.add_argument("--enable-link-aggr", action="store_true", help="The\ 23910152Satgutier@umich.edu crossbar between port and Link Controller") 24010152Satgutier@umich.edu 24110152Satgutier@umich.edu parser.add_argument("--enable-buff-div", action="store_true", 24210152Satgutier@umich.edu help="Memory Range of Buffer is ivided between total\ 24310152Satgutier@umich.edu range") 24410152Satgutier@umich.edu 24510152Satgutier@umich.edu # *****************************HMC ARCHITECTURE ************************** 24610152Satgutier@umich.edu # Memory chunk for 16 vault - numbers of vault / number of crossbars 24710152Satgutier@umich.edu parser.add_argument("--mem-chunk", default=4, action="store", type=int, 24810152Satgutier@umich.edu help="Chunk of memory range for each cross bar in\ 24910152Satgutier@umich.edu arch 0") 25010152Satgutier@umich.edu 25110152Satgutier@umich.edu # size of req buffer within crossbar, used for modelling extra latency 25210152Satgutier@umich.edu # when the reuqest go to non-local vault 25310152Satgutier@umich.edu parser.add_argument("--xbar-buffer-size-req", default=10, action="store", 25410152Satgutier@umich.edu type=int, help="Number of packets to buffer at the\ 25510152Satgutier@umich.edu request side of the crossbar") 25610152Satgutier@umich.edu 25710152Satgutier@umich.edu # size of response buffer within crossbar, used for modelling extra latency 25810152Satgutier@umich.edu # when the response received from non-local vault 25910152Satgutier@umich.edu parser.add_argument("--xbar-buffer-size-resp", default=10, action="store", 26010152Satgutier@umich.edu type=int, help="Number of packets to buffer at the\ 26110152Satgutier@umich.edu response side of the crossbar") 26210152Satgutier@umich.edu # HMC device architecture. It affects the HMC host controller as well 26310152Satgutier@umich.edu parser.add_argument("--arch", type=str, choices=["same", "distributed", 26410152Satgutier@umich.edu "mixed"], default="distributed", help="same: HMC with\ 26510152Satgutier@umich.edu 4 links, all with same range.\ndistributed: HMC with\ 26610152Satgutier@umich.edu 4 links with distributed range.\nmixed: mixed with\ 26710152Satgutier@umich.edu same and distributed range.\nDefault: distributed") 26810152Satgutier@umich.edu # HMC device - number of vaults 26910152Satgutier@umich.edu parser.add_argument("--hmc-dev-num-vaults", default=16, action="store", 27010152Satgutier@umich.edu type=int, help="number of independent vaults within\ 27110152Satgutier@umich.edu the HMC device. Note: each vault has a memory\ 27210152Satgutier@umich.edu controller (valut controller)\nDefault: 16") 27310152Satgutier@umich.edu # HMC device - vault capacity or size 27410152Satgutier@umich.edu parser.add_argument("--hmc-dev-vault-size", default='256MB', type=str, 27510152Satgutier@umich.edu help="vault storage capacity in bytes. Default:\ 27610152Satgutier@umich.edu 256MB") 27710152Satgutier@umich.edu parser.add_argument("--mem-type", type=str, choices=["HMC_2500_1x32"], 27810152Satgutier@umich.edu default="HMC_2500_1x32", help="type of HMC memory to\ 27910152Satgutier@umich.edu use. Default: HMC_2500_1x32") 28010152Satgutier@umich.edu parser.add_argument("--mem-channels", default=1, action="store", type=int, 28110152Satgutier@umich.edu help="Number of memory channels") 28210152Satgutier@umich.edu parser.add_argument("--mem-ranks", default=1, action="store", type=int, 28310152Satgutier@umich.edu help="Number of ranks to iterate across") 28410152Satgutier@umich.edu parser.add_argument("--burst-length", default=256, action="store", 28510152Satgutier@umich.edu type=int, help="burst length in bytes. Note: the\ 28610152Satgutier@umich.edu cache line size will be set to this value.\nDefault:\ 28710152Satgutier@umich.edu 256") 28810152Satgutier@umich.edu 28910152Satgutier@umich.edu 29010152Satgutier@umich.edu# configure HMC host controller 29110152Satgutier@umich.edudef config_hmc_host_ctrl(opt, system): 29210152Satgutier@umich.edu 29310152Satgutier@umich.edu # create HMC host controller 29410152Satgutier@umich.edu system.hmc_host = SubSystem() 29510152Satgutier@umich.edu 29610152Satgutier@umich.edu # Create additional crossbar for arch1 29710152Satgutier@umich.edu if opt.arch == "distributed" or opt.arch == "mixed": 29810152Satgutier@umich.edu clk = '100GHz' 29910152Satgutier@umich.edu vd = VoltageDomain(voltage='1V') 30010152Satgutier@umich.edu # Create additional crossbar for arch1 30110152Satgutier@umich.edu system.membus = NoncoherentXBar(width=8) 30210152Satgutier@umich.edu system.membus.badaddr_responder = BadAddr() 30310152Satgutier@umich.edu system.membus.default = Self.badaddr_responder.pio 30410152Satgutier@umich.edu system.membus.width = 8 30510152Satgutier@umich.edu system.membus.frontend_latency = 3 30610152Satgutier@umich.edu system.membus.forward_latency = 4 30710152Satgutier@umich.edu system.membus.response_latency = 2 30810152Satgutier@umich.edu cd = SrcClockDomain(clock=clk, voltage_domain=vd) 30910152Satgutier@umich.edu system.membus.clk_domain = cd 31010152Satgutier@umich.edu 31110152Satgutier@umich.edu # create memory ranges for the serial links 31210152Satgutier@umich.edu slar = convert.toMemorySize(opt.serial_link_addr_range) 31310152Satgutier@umich.edu # Memmory ranges of serial link for arch-0. Same as the ranges of vault 31410152Satgutier@umich.edu # controllers (4 vaults to 1 serial link) 31510152Satgutier@umich.edu if opt.arch == "same": 31610152Satgutier@umich.edu ser_ranges = [AddrRange(0, (4*slar)-1) for i in 31710152Satgutier@umich.edu range(opt.num_serial_links)] 31810152Satgutier@umich.edu # Memmory ranges of serial link for arch-1. Distributed range accross 31910152Satgutier@umich.edu # links 32010152Satgutier@umich.edu if opt.arch == "distributed": 32110152Satgutier@umich.edu ser_ranges = [AddrRange(i*slar, ((i+1)*slar)-1) for i in 32210152Satgutier@umich.edu range(opt.num_serial_links)] 32310152Satgutier@umich.edu # Memmory ranges of serial link for arch-2 'Mixed' address distribution 32410152Satgutier@umich.edu # over links 32510152Satgutier@umich.edu if opt.arch == "mixed": 32610152Satgutier@umich.edu ser_range0 = AddrRange(0, (1*slar)-1) 32710152Satgutier@umich.edu ser_range1 = AddrRange(1*slar, 2*slar-1) 32810152Satgutier@umich.edu ser_range2 = AddrRange(0, (4*slar)-1) 32910152Satgutier@umich.edu ser_range3 = AddrRange(0, (4*slar)-1) 33010152Satgutier@umich.edu ser_ranges = [ser_range0, ser_range1, ser_range2, ser_range3] 33110152Satgutier@umich.edu 33210152Satgutier@umich.edu # Serial link Controller with 16 SerDes links at 10 Gbps with serial link 33310152Satgutier@umich.edu # ranges w.r.t to architecture 33410152Satgutier@umich.edu sl = [SerialLink(ranges=ser_ranges[i], 33510152Satgutier@umich.edu req_size=opt.link_buffer_size_req, 33610152Satgutier@umich.edu resp_size=opt.link_buffer_size_rsp, 33710152Satgutier@umich.edu num_lanes=opt.num_lanes_per_link, 33810152Satgutier@umich.edu link_speed=opt.serial_link_speed, 33910152Satgutier@umich.edu delay=opt.total_ctrl_latency) for i in 34010152Satgutier@umich.edu xrange(opt.num_serial_links)] 34110152Satgutier@umich.edu system.hmc_host.seriallink = sl 34210152Satgutier@umich.edu 34310152Satgutier@umich.edu # enable global monitor 34410152Satgutier@umich.edu if opt.enable_global_monitor: 34510152Satgutier@umich.edu system.hmc_host.lmonitor = [CommMonitor() for i in 34610152Satgutier@umich.edu xrange(opt.num_serial_links)] 34710152Satgutier@umich.edu 34810152Satgutier@umich.edu # set the clock frequency for serial link 34910152Satgutier@umich.edu for i in xrange(opt.num_serial_links): 35010152Satgutier@umich.edu clk = opt.link_controller_frequency 35110152Satgutier@umich.edu vd = VoltageDomain(voltage='1V') 35210152Satgutier@umich.edu scd = SrcClockDomain(clock=clk, voltage_domain=vd) 35310152Satgutier@umich.edu system.hmc_host.seriallink[i].clk_domain = scd 35410152Satgutier@umich.edu 35510152Satgutier@umich.edu # Connect membus/traffic gen to Serial Link Controller for differrent HMC 35610152Satgutier@umich.edu # architectures 35710152Satgutier@umich.edu hh = system.hmc_host 35810152Satgutier@umich.edu if opt.arch == "distributed": 35910152Satgutier@umich.edu mb = system.membus 36010152Satgutier@umich.edu for i in xrange(opt.num_links_controllers): 36110152Satgutier@umich.edu if opt.enable_global_monitor: 36210152Satgutier@umich.edu mb.master = hh.lmonitor[i].slave 36310152Satgutier@umich.edu hh.lmonitor[i].master = hh.seriallink[i].slave 36410152Satgutier@umich.edu else: 36510152Satgutier@umich.edu mb.master = hh.seriallink[i].slave 36610152Satgutier@umich.edu if opt.arch == "mixed": 36710152Satgutier@umich.edu mb = system.membus 36810152Satgutier@umich.edu if opt.enable_global_monitor: 36910152Satgutier@umich.edu mb.master = hh.lmonitor[0].slave 37010152Satgutier@umich.edu hh.lmonitor[0].master = hh.seriallink[0].slave 37110152Satgutier@umich.edu mb.master = hh.lmonitor[1].slave 37210152Satgutier@umich.edu hh.lmonitor[1].master = hh.seriallink[1].slave 37310152Satgutier@umich.edu else: 37410152Satgutier@umich.edu mb.master = hh.seriallink[0].slave 37510152Satgutier@umich.edu mb.master = hh.seriallink[1].slave 37610152Satgutier@umich.edu 37710152Satgutier@umich.edu if opt.arch == "same": 37810152Satgutier@umich.edu for i in xrange(opt.num_links_controllers): 37910152Satgutier@umich.edu if opt.enable_global_monitor: 38010152Satgutier@umich.edu hh.lmonitor[i].master = hh.seriallink[i].slave 38110152Satgutier@umich.edu 38210152Satgutier@umich.edu return system 38310152Satgutier@umich.edu 38410152Satgutier@umich.edu 38510152Satgutier@umich.edu# Create an HMC device 38610152Satgutier@umich.edudef config_hmc_dev(opt, system, hmc_host): 38710152Satgutier@umich.edu 38810152Satgutier@umich.edu # create HMC device 38910152Satgutier@umich.edu system.hmc_dev = SubSystem() 39010152Satgutier@umich.edu 39110152Satgutier@umich.edu # create memory ranges for the vault controllers 39210152Satgutier@umich.edu arv = convert.toMemorySize(opt.hmc_dev_vault_size) 39310152Satgutier@umich.edu addr_ranges_vaults = [AddrRange(i*arv, ((i+1)*arv-1)) for i in 39410152Satgutier@umich.edu range(opt.hmc_dev_num_vaults)] 39510152Satgutier@umich.edu system.mem_ranges = addr_ranges_vaults 39610152Satgutier@umich.edu 39710152Satgutier@umich.edu if opt.enable_link_monitor: 39810152Satgutier@umich.edu lm = [CommMonitor() for i in xrange(opt.num_links_controllers)] 39910152Satgutier@umich.edu system.hmc_dev.lmonitor = lm 40010152Satgutier@umich.edu 40110152Satgutier@umich.edu # 4 HMC Crossbars located in its logic-base (LoB) 40210152Satgutier@umich.edu xb = [NoncoherentXBar(width=opt.xbar_width, 40310152Satgutier@umich.edu frontend_latency=opt.xbar_frontend_latency, 40410152Satgutier@umich.edu forward_latency=opt.xbar_forward_latency, 40510152Satgutier@umich.edu response_latency=opt.xbar_response_latency) for i in 40610152Satgutier@umich.edu xrange(opt.number_mem_crossbar)] 40710152Satgutier@umich.edu system.hmc_dev.xbar = xb 40810152Satgutier@umich.edu 40910152Satgutier@umich.edu for i in xrange(opt.number_mem_crossbar): 41010152Satgutier@umich.edu clk = opt.xbar_frequency 41110152Satgutier@umich.edu vd = VoltageDomain(voltage='1V') 41210152Satgutier@umich.edu scd = SrcClockDomain(clock=clk, voltage_domain=vd) 41310152Satgutier@umich.edu system.hmc_dev.xbar[i].clk_domain = scd 41410152Satgutier@umich.edu 41510152Satgutier@umich.edu # Attach 4 serial link to 4 crossbar/s 41610152Satgutier@umich.edu for i in xrange(opt.num_serial_links): 41710152Satgutier@umich.edu if opt.enable_link_monitor: 41810152Satgutier@umich.edu system.hmc_host.seriallink[i].master = \ 41910152Satgutier@umich.edu system.hmc_dev.lmonitor[i].slave 42010152Satgutier@umich.edu system.hmc_dev.lmonitor[i].master = system.hmc_dev.xbar[i].slave 42110152Satgutier@umich.edu else: 42210152Satgutier@umich.edu system.hmc_host.seriallink[i].master = system.hmc_dev.xbar[i].slave 42310152Satgutier@umich.edu 42410152Satgutier@umich.edu # Connecting xbar with each other for request arriving at the wrong xbar, 42510152Satgutier@umich.edu # then it will be forward to correct xbar. Bridge is used to connect xbars 42610152Satgutier@umich.edu if opt.arch == "same": 42710152Satgutier@umich.edu numx = len(system.hmc_dev.xbar) 42810152Satgutier@umich.edu 42910152Satgutier@umich.edu # create a list of buffers 43010152Satgutier@umich.edu system.hmc_dev.buffers = [Bridge(req_size=opt.xbar_buffer_size_req, 43110152Satgutier@umich.edu resp_size=opt.xbar_buffer_size_resp) 43210152Satgutier@umich.edu for i in xrange(numx*(opt.mem_chunk-1))] 43310152Satgutier@umich.edu 43410152Satgutier@umich.edu # Buffer iterator 43510152Satgutier@umich.edu it = iter(range(len(system.hmc_dev.buffers))) 43610152Satgutier@umich.edu 43710152Satgutier@umich.edu # necesarry to add system_port to one of the xbar 43810152Satgutier@umich.edu system.system_port = system.hmc_dev.xbar[3].slave 43910152Satgutier@umich.edu 44010152Satgutier@umich.edu # iterate over all the crossbars and connect them as required 44110152Satgutier@umich.edu for i in range(numx): 44210152Satgutier@umich.edu for j in range(numx): 44310152Satgutier@umich.edu # connect xbar to all other xbars except itself 44410152Satgutier@umich.edu if i != j: 44510152Satgutier@umich.edu # get the next index of buffer 44610152Satgutier@umich.edu index = it.next() 44710152Satgutier@umich.edu 44810152Satgutier@umich.edu # Change the default values for ranges of bridge 44910152Satgutier@umich.edu system.hmc_dev.buffers[index].ranges = system.mem_ranges[ 45010152Satgutier@umich.edu j * int(opt.mem_chunk): 45110152Satgutier@umich.edu (j + 1) * int(opt.mem_chunk)] 45210152Satgutier@umich.edu 45310152Satgutier@umich.edu # Connect the bridge between corssbars 45410152Satgutier@umich.edu system.hmc_dev.xbar[i].master = system.hmc_dev.buffers[ 45510152Satgutier@umich.edu index].slave 45610152Satgutier@umich.edu system.hmc_dev.buffers[ 45710152Satgutier@umich.edu index].master = system.hmc_dev.xbar[j].slave 45810152Satgutier@umich.edu else: 45910152Satgutier@umich.edu # Don't connect the xbar to itself 46010152Satgutier@umich.edu pass 46110152Satgutier@umich.edu 46210152Satgutier@umich.edu # Two crossbars are connected to all other crossbars-Other 2 vault 46310152Satgutier@umich.edu # can only direct traffic to it local vaults 46410152Satgutier@umich.edu if opt.arch == "mixed": 46510152Satgutier@umich.edu system.hmc_dev.buffer30 = Bridge(ranges=system.mem_ranges[0:4]) 46610152Satgutier@umich.edu system.hmc_dev.xbar[3].master = system.hmc_dev.buffer30.slave 46710152Satgutier@umich.edu system.hmc_dev.buffer30.master = system.hmc_dev.xbar[0].slave 46810152Satgutier@umich.edu 46910152Satgutier@umich.edu system.hmc_dev.buffer31 = Bridge(ranges=system.mem_ranges[4:8]) 47010152Satgutier@umich.edu system.hmc_dev.xbar[3].master = system.hmc_dev.buffer31.slave 47110152Satgutier@umich.edu system.hmc_dev.buffer31.master = system.hmc_dev.xbar[1].slave 47210152Satgutier@umich.edu 47310152Satgutier@umich.edu system.hmc_dev.buffer32 = Bridge(ranges=system.mem_ranges[8:12]) 47410152Satgutier@umich.edu system.hmc_dev.xbar[3].master = system.hmc_dev.buffer32.slave 47510152Satgutier@umich.edu system.hmc_dev.buffer32.master = system.hmc_dev.xbar[2].slave 47610152Satgutier@umich.edu 47710152Satgutier@umich.edu system.hmc_dev.buffer20 = Bridge(ranges=system.mem_ranges[0:4]) 47810152Satgutier@umich.edu system.hmc_dev.xbar[2].master = system.hmc_dev.buffer20.slave 47910152Satgutier@umich.edu system.hmc_dev.buffer20.master = system.hmc_dev.xbar[0].slave 48010152Satgutier@umich.edu 48110152Satgutier@umich.edu system.hmc_dev.buffer21 = Bridge(ranges=system.mem_ranges[4:8]) 48210152Satgutier@umich.edu system.hmc_dev.xbar[2].master = system.hmc_dev.buffer21.slave 48310152Satgutier@umich.edu system.hmc_dev.buffer21.master = system.hmc_dev.xbar[1].slave 48410152Satgutier@umich.edu 48510152Satgutier@umich.edu system.hmc_dev.buffer23 = Bridge(ranges=system.mem_ranges[12:16]) 48610152Satgutier@umich.edu system.hmc_dev.xbar[2].master = system.hmc_dev.buffer23.slave 48710152Satgutier@umich.edu system.hmc_dev.buffer23.master = system.hmc_dev.xbar[3].slave 48810152Satgutier@umich.edu