dist_iface.hh revision 11290
110923SN/A/* 210923SN/A * Copyright (c) 2015 ARM Limited 310923SN/A * All rights reserved 410923SN/A * 510923SN/A * The license below extends only to copyright in the software and shall 610923SN/A * not be construed as granting a license to any other intellectual 710923SN/A * property including but not limited to intellectual property relating 810923SN/A * to a hardware implementation of the functionality of the software 910923SN/A * licensed hereunder. You may use the software subject to the license 1010923SN/A * terms below provided that you ensure that this notice is replicated 1110923SN/A * unmodified and in its entirety in all distributions of the software, 1210923SN/A * modified or unmodified, in source code or in binary form. 1310923SN/A * 1410923SN/A * Redistribution and use in source and binary forms, with or without 1510923SN/A * modification, are permitted provided that the following conditions are 1610923SN/A * met: redistributions of source code must retain the above copyright 1710923SN/A * notice, this list of conditions and the following disclaimer; 1810923SN/A * redistributions in binary form must reproduce the above copyright 1910923SN/A * notice, this list of conditions and the following disclaimer in the 2010923SN/A * documentation and/or other materials provided with the distribution; 2110923SN/A * neither the name of the copyright holders nor the names of its 2210923SN/A * contributors may be used to endorse or promote products derived from 2310923SN/A * this software without specific prior written permission. 2410923SN/A * 2510923SN/A * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 2610923SN/A * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 2710923SN/A * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 2810923SN/A * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 2910923SN/A * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 3010923SN/A * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 3110923SN/A * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 3210923SN/A * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 3310923SN/A * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 3410923SN/A * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 3510923SN/A * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 3610923SN/A * 3710923SN/A * Authors: Gabor Dozsa 3810923SN/A */ 3910923SN/A 4010923SN/A/* @file 4111290Sgabor.dozsa@arm.com * The interface class for dist gem5 simulations. 4210923SN/A * 4311290Sgabor.dozsa@arm.com * dist-gem5 is an extension to gem5 to enable parallel simulation of a 4410923SN/A * distributed system (e.g. simulation of a pool of machines 4511290Sgabor.dozsa@arm.com * connected by Ethernet links). A dist gem5 run consists of seperate gem5 4610923SN/A * processes running in parallel. Each gem5 process executes 4710923SN/A * the simulation of a component of the simulated distributed system. 4811290Sgabor.dozsa@arm.com * (An example component can be a dist-core board with an Ethernet NIC.) 4911290Sgabor.dozsa@arm.com * The DistIface class below provides services to transfer data and 5010923SN/A * control messages among the gem5 processes. The main such services are 5110923SN/A * as follows. 5210923SN/A * 5310923SN/A * 1. Send a data packet coming from a simulated Ethernet link. The packet 5410923SN/A * will be transferred to (all) the target(s) gem5 processes. The send 5510923SN/A * operation is always performed by the simulation thread, i.e. the gem5 5610923SN/A * thread that is processing the event queue associated with the simulated 5710923SN/A * Ethernet link. 5810923SN/A * 5910923SN/A * 2. Spawn a receiver thread to process messages coming in from the 6010923SN/A * from other gem5 processes. Each simulated Ethernet link has its own 6110923SN/A * associated receiver thread. The receiver thread saves the incoming packet 6210923SN/A * and schedule an appropriate receive event in the event queue. 6310923SN/A * 6410923SN/A * 3. Schedule a global barrier event periodically to keep the gem5 6510923SN/A * processes in sync. 6610923SN/A * Periodic barrier event to keep peer gem5 processes in sync. The basic idea 6710923SN/A * is that no gem5 process can go ahead further than the simulated link 6810923SN/A * transmission delay to ensure that a corresponding receive event can always 6910923SN/A * be scheduled for any message coming in from a peer gem5 process. 7010923SN/A * 7110923SN/A * 7210923SN/A * 7311290Sgabor.dozsa@arm.com * This interface is an abstract class. It can work with various low level 7410923SN/A * send/receive service implementations (e.g. TCP/IP, MPI,...). A TCP 7511290Sgabor.dozsa@arm.com * stream socket version is implemented in src/dev/net/tcp_iface.[hh,cc]. 7610923SN/A */ 7711290Sgabor.dozsa@arm.com#ifndef __DEV_DIST_IFACE_HH__ 7811290Sgabor.dozsa@arm.com#define __DEV_DIST_IFACE_HH__ 7910923SN/A 8010923SN/A#include <array> 8110923SN/A#include <mutex> 8210923SN/A#include <queue> 8310923SN/A#include <thread> 8410923SN/A#include <utility> 8510923SN/A 8611290Sgabor.dozsa@arm.com#include "dev/net/dist_packet.hh" 8711263SN/A#include "dev/net/etherpkt.hh" 8810923SN/A#include "sim/core.hh" 8910923SN/A#include "sim/drain.hh" 9010923SN/A#include "sim/global_event.hh" 9111290Sgabor.dozsa@arm.com#include "sim/serialize.hh" 9210923SN/A 9310923SN/Aclass EventManager; 9410923SN/A 9510923SN/A/** 9610923SN/A * The interface class to talk to peer gem5 processes. 9710923SN/A */ 9811290Sgabor.dozsa@arm.comclass DistIface : public Drainable, public Serializable 9910923SN/A{ 10010923SN/A public: 10111290Sgabor.dozsa@arm.com typedef DistHeaderPkt::Header Header; 10211290Sgabor.dozsa@arm.com 10311290Sgabor.dozsa@arm.com protected: 10411290Sgabor.dozsa@arm.com typedef DistHeaderPkt::MsgType MsgType; 10511290Sgabor.dozsa@arm.com typedef DistHeaderPkt::ReqType ReqType; 10610923SN/A 10710923SN/A private: 10811290Sgabor.dozsa@arm.com class SyncEvent; 10910923SN/A /** @class Sync 11010923SN/A * This class implements global sync operations among gem5 peer processes. 11110923SN/A * 11211290Sgabor.dozsa@arm.com * @note This class is used as a singleton object (shared by all DistIface 11310923SN/A * objects). 11410923SN/A */ 11511290Sgabor.dozsa@arm.com class Sync : public Serializable 11610923SN/A { 11711290Sgabor.dozsa@arm.com protected: 11810923SN/A /** 11911290Sgabor.dozsa@arm.com * The lock to protect access to the Sync object. 12010923SN/A */ 12110923SN/A std::mutex lock; 12210923SN/A /** 12310923SN/A * Condition variable for the simulation thread to wait on 12410923SN/A * until all receiver threads completes the current global 12510923SN/A * synchronisation. 12610923SN/A */ 12710923SN/A std::condition_variable cv; 12810923SN/A /** 12910923SN/A * Number of receiver threads that not yet completed the current global 13010923SN/A * synchronisation. 13110923SN/A */ 13210923SN/A unsigned waitNum; 13310923SN/A /** 13411290Sgabor.dozsa@arm.com * Flag is set if exit is permitted upon sync completion 13510923SN/A */ 13611290Sgabor.dozsa@arm.com bool doExit; 13710923SN/A /** 13811290Sgabor.dozsa@arm.com * Flag is set if taking a ckpt is permitted upon sync completion 13910923SN/A */ 14011290Sgabor.dozsa@arm.com bool doCkpt; 14111290Sgabor.dozsa@arm.com /** 14211290Sgabor.dozsa@arm.com * The repeat value for the next periodic sync 14311290Sgabor.dozsa@arm.com */ 14411290Sgabor.dozsa@arm.com Tick nextRepeat; 14511290Sgabor.dozsa@arm.com /** 14611290Sgabor.dozsa@arm.com * Tick for the very first periodic sync 14711290Sgabor.dozsa@arm.com */ 14811290Sgabor.dozsa@arm.com Tick firstAt; 14911290Sgabor.dozsa@arm.com /** 15011290Sgabor.dozsa@arm.com * Tick for the next periodic sync (if the event is not scheduled yet) 15111290Sgabor.dozsa@arm.com */ 15211290Sgabor.dozsa@arm.com Tick nextAt; 15310923SN/A 15411290Sgabor.dozsa@arm.com friend class SyncEvent; 15510923SN/A 15610923SN/A public: 15710923SN/A /** 15811290Sgabor.dozsa@arm.com * Initialize periodic sync params. 15910923SN/A * 16011290Sgabor.dozsa@arm.com * @param start Start tick for dist synchronisation 16111290Sgabor.dozsa@arm.com * @param repeat Frequency of dist synchronisation 16210923SN/A * 16310923SN/A */ 16411290Sgabor.dozsa@arm.com void init(Tick start, Tick repeat); 16510923SN/A /** 16611290Sgabor.dozsa@arm.com * Core method to perform a full dist sync. 16710923SN/A */ 16811290Sgabor.dozsa@arm.com virtual void run(bool same_tick) = 0; 16911290Sgabor.dozsa@arm.com /** 17011290Sgabor.dozsa@arm.com * Callback when the receiver thread gets a sync ack message. 17111290Sgabor.dozsa@arm.com */ 17211290Sgabor.dozsa@arm.com virtual void progress(Tick send_tick, 17311290Sgabor.dozsa@arm.com Tick next_repeat, 17411290Sgabor.dozsa@arm.com ReqType do_ckpt, 17511290Sgabor.dozsa@arm.com ReqType do_exit) = 0; 17610923SN/A 17711290Sgabor.dozsa@arm.com virtual void requestCkpt(ReqType req) = 0; 17811290Sgabor.dozsa@arm.com virtual void requestExit(ReqType req) = 0; 17911290Sgabor.dozsa@arm.com 18011290Sgabor.dozsa@arm.com void drainComplete(); 18111290Sgabor.dozsa@arm.com 18211290Sgabor.dozsa@arm.com virtual void serialize(CheckpointOut &cp) const override = 0; 18311290Sgabor.dozsa@arm.com virtual void unserialize(CheckpointIn &cp) override = 0; 18410923SN/A }; 18510923SN/A 18611290Sgabor.dozsa@arm.com class SyncNode: public Sync 18711290Sgabor.dozsa@arm.com { 18811290Sgabor.dozsa@arm.com private: 18911290Sgabor.dozsa@arm.com /** 19011290Sgabor.dozsa@arm.com * Exit requested 19111290Sgabor.dozsa@arm.com */ 19211290Sgabor.dozsa@arm.com ReqType needExit; 19311290Sgabor.dozsa@arm.com /** 19411290Sgabor.dozsa@arm.com * Ckpt requested 19511290Sgabor.dozsa@arm.com */ 19611290Sgabor.dozsa@arm.com ReqType needCkpt; 19711290Sgabor.dozsa@arm.com 19811290Sgabor.dozsa@arm.com public: 19911290Sgabor.dozsa@arm.com 20011290Sgabor.dozsa@arm.com SyncNode(); 20111290Sgabor.dozsa@arm.com ~SyncNode() {} 20211290Sgabor.dozsa@arm.com void run(bool same_tick) override; 20311290Sgabor.dozsa@arm.com void progress(Tick max_req_tick, 20411290Sgabor.dozsa@arm.com Tick next_repeat, 20511290Sgabor.dozsa@arm.com ReqType do_ckpt, 20611290Sgabor.dozsa@arm.com ReqType do_exit) override; 20711290Sgabor.dozsa@arm.com 20811290Sgabor.dozsa@arm.com void requestCkpt(ReqType req) override; 20911290Sgabor.dozsa@arm.com void requestExit(ReqType req) override; 21011290Sgabor.dozsa@arm.com 21111290Sgabor.dozsa@arm.com void serialize(CheckpointOut &cp) const override; 21211290Sgabor.dozsa@arm.com void unserialize(CheckpointIn &cp) override; 21311290Sgabor.dozsa@arm.com }; 21411290Sgabor.dozsa@arm.com 21511290Sgabor.dozsa@arm.com class SyncSwitch: public Sync 21611290Sgabor.dozsa@arm.com { 21711290Sgabor.dozsa@arm.com private: 21811290Sgabor.dozsa@arm.com /** 21911290Sgabor.dozsa@arm.com * Counter for recording exit requests 22011290Sgabor.dozsa@arm.com */ 22111290Sgabor.dozsa@arm.com unsigned numExitReq; 22211290Sgabor.dozsa@arm.com /** 22311290Sgabor.dozsa@arm.com * Counter for recording ckpt requests 22411290Sgabor.dozsa@arm.com */ 22511290Sgabor.dozsa@arm.com unsigned numCkptReq; 22611290Sgabor.dozsa@arm.com /** 22711290Sgabor.dozsa@arm.com * Number of connected simulated nodes 22811290Sgabor.dozsa@arm.com */ 22911290Sgabor.dozsa@arm.com unsigned numNodes; 23011290Sgabor.dozsa@arm.com 23111290Sgabor.dozsa@arm.com public: 23211290Sgabor.dozsa@arm.com SyncSwitch(int num_nodes); 23311290Sgabor.dozsa@arm.com ~SyncSwitch() {} 23411290Sgabor.dozsa@arm.com 23511290Sgabor.dozsa@arm.com void run(bool same_tick) override; 23611290Sgabor.dozsa@arm.com void progress(Tick max_req_tick, 23711290Sgabor.dozsa@arm.com Tick next_repeat, 23811290Sgabor.dozsa@arm.com ReqType do_ckpt, 23911290Sgabor.dozsa@arm.com ReqType do_exit) override; 24011290Sgabor.dozsa@arm.com 24111290Sgabor.dozsa@arm.com void requestCkpt(ReqType) override { 24211290Sgabor.dozsa@arm.com panic("Switch requested checkpoint"); 24311290Sgabor.dozsa@arm.com } 24411290Sgabor.dozsa@arm.com void requestExit(ReqType) override { 24511290Sgabor.dozsa@arm.com panic("Switch requested exit"); 24611290Sgabor.dozsa@arm.com } 24711290Sgabor.dozsa@arm.com 24811290Sgabor.dozsa@arm.com void serialize(CheckpointOut &cp) const override; 24911290Sgabor.dozsa@arm.com void unserialize(CheckpointIn &cp) override; 25011290Sgabor.dozsa@arm.com }; 25110923SN/A 25210923SN/A /** 25311290Sgabor.dozsa@arm.com * The global event to schedule periodic dist sync. It is used as a 25410923SN/A * singleton object. 25510923SN/A * 25610923SN/A * The periodic synchronisation works as follows. 25711290Sgabor.dozsa@arm.com * 1. A SyncEvent is scheduled as a global event when startup() is 25810923SN/A * called. 25911290Sgabor.dozsa@arm.com * 2. The process() method of the SyncEvent initiates a new barrier 26011290Sgabor.dozsa@arm.com * for each simulated Ethernet link. 26110923SN/A * 3. Simulation thread(s) then waits until all receiver threads 26211290Sgabor.dozsa@arm.com * complete the ongoing barrier. The global sync event is done. 26310923SN/A */ 26410923SN/A class SyncEvent : public GlobalSyncEvent 26510923SN/A { 26611290Sgabor.dozsa@arm.com private: 26711290Sgabor.dozsa@arm.com /** 26811290Sgabor.dozsa@arm.com * Flag to set when the system is draining 26911290Sgabor.dozsa@arm.com */ 27011290Sgabor.dozsa@arm.com bool _draining; 27110923SN/A public: 27210923SN/A /** 27311290Sgabor.dozsa@arm.com * Only the firstly instantiated DistIface object will 27411290Sgabor.dozsa@arm.com * call this constructor. 27510923SN/A */ 27611290Sgabor.dozsa@arm.com SyncEvent() : GlobalSyncEvent(Sim_Exit_Pri, 0), _draining(false) {} 27711290Sgabor.dozsa@arm.com 27811290Sgabor.dozsa@arm.com ~SyncEvent() {} 27910923SN/A /** 28011290Sgabor.dozsa@arm.com * Schedule the first periodic sync event. 28110923SN/A */ 28211290Sgabor.dozsa@arm.com void start(); 28310923SN/A /** 28411290Sgabor.dozsa@arm.com * This is a global event so process() will only be called by 28511290Sgabor.dozsa@arm.com * exactly one simulation thread. (See further comments in the .cc 28611290Sgabor.dozsa@arm.com * file.) 28710923SN/A */ 28811290Sgabor.dozsa@arm.com void process() override; 28911290Sgabor.dozsa@arm.com 29011290Sgabor.dozsa@arm.com bool draining() const { return _draining; } 29111290Sgabor.dozsa@arm.com void draining(bool fl) { _draining = fl; } 29211290Sgabor.dozsa@arm.com }; 29311290Sgabor.dozsa@arm.com /** 29411290Sgabor.dozsa@arm.com * Class to encapsulate information about data packets received. 29511290Sgabor.dozsa@arm.com 29611290Sgabor.dozsa@arm.com * @note The main purpose of the class to take care of scheduling receive 29711290Sgabor.dozsa@arm.com * done events for the simulated network link and store incoming packets 29811290Sgabor.dozsa@arm.com * until they can be received by the simulated network link. 29911290Sgabor.dozsa@arm.com */ 30011290Sgabor.dozsa@arm.com class RecvScheduler : public Serializable 30111290Sgabor.dozsa@arm.com { 30211290Sgabor.dozsa@arm.com private: 30311290Sgabor.dozsa@arm.com /** 30411290Sgabor.dozsa@arm.com * Received packet descriptor. This information is used by the receive 30511290Sgabor.dozsa@arm.com * thread to schedule receive events and by the simulation thread to 30611290Sgabor.dozsa@arm.com * process those events. 30711290Sgabor.dozsa@arm.com */ 30811290Sgabor.dozsa@arm.com struct Desc : public Serializable 30911290Sgabor.dozsa@arm.com { 31011290Sgabor.dozsa@arm.com EthPacketPtr packet; 31111290Sgabor.dozsa@arm.com Tick sendTick; 31211290Sgabor.dozsa@arm.com Tick sendDelay; 31311290Sgabor.dozsa@arm.com 31411290Sgabor.dozsa@arm.com Desc() : sendTick(0), sendDelay(0) {} 31511290Sgabor.dozsa@arm.com Desc(EthPacketPtr p, Tick s, Tick d) : 31611290Sgabor.dozsa@arm.com packet(p), sendTick(s), sendDelay(d) {} 31711290Sgabor.dozsa@arm.com Desc(const Desc &d) : 31811290Sgabor.dozsa@arm.com packet(d.packet), sendTick(d.sendTick), sendDelay(d.sendDelay) {} 31911290Sgabor.dozsa@arm.com 32011290Sgabor.dozsa@arm.com void serialize(CheckpointOut &cp) const override; 32111290Sgabor.dozsa@arm.com void unserialize(CheckpointIn &cp) override; 32211290Sgabor.dozsa@arm.com }; 32311290Sgabor.dozsa@arm.com /** 32411290Sgabor.dozsa@arm.com * The queue to store the receive descriptors. 32511290Sgabor.dozsa@arm.com */ 32611290Sgabor.dozsa@arm.com std::queue<Desc> descQueue; 32711290Sgabor.dozsa@arm.com /** 32811290Sgabor.dozsa@arm.com * The tick when the most recent receive event was processed. 32911290Sgabor.dozsa@arm.com * 33011290Sgabor.dozsa@arm.com * @note This information is necessary to simulate possible receiver 33111290Sgabor.dozsa@arm.com * link contention when calculating the receive tick for the next 33211290Sgabor.dozsa@arm.com * incoming data packet (see the calcReceiveTick() method) 33311290Sgabor.dozsa@arm.com */ 33411290Sgabor.dozsa@arm.com Tick prevRecvTick; 33511290Sgabor.dozsa@arm.com /** 33611290Sgabor.dozsa@arm.com * The receive done event for the simulated Ethernet link. 33711290Sgabor.dozsa@arm.com * 33811290Sgabor.dozsa@arm.com * @note This object is constructed by the simulated network link. We 33911290Sgabor.dozsa@arm.com * schedule this object for each incoming data packet. 34011290Sgabor.dozsa@arm.com */ 34111290Sgabor.dozsa@arm.com Event *recvDone; 34211290Sgabor.dozsa@arm.com /** 34311290Sgabor.dozsa@arm.com * The link delay in ticks for the simulated Ethernet link. 34411290Sgabor.dozsa@arm.com * 34511290Sgabor.dozsa@arm.com * @note This value is used for calculating the receive ticks for 34611290Sgabor.dozsa@arm.com * incoming data packets. 34711290Sgabor.dozsa@arm.com */ 34811290Sgabor.dozsa@arm.com Tick linkDelay; 34911290Sgabor.dozsa@arm.com /** 35011290Sgabor.dozsa@arm.com * The event manager associated with the simulated Ethernet link. 35111290Sgabor.dozsa@arm.com * 35211290Sgabor.dozsa@arm.com * @note It is used to access the event queue for scheduling receive 35311290Sgabor.dozsa@arm.com * done events for the link. 35411290Sgabor.dozsa@arm.com */ 35511290Sgabor.dozsa@arm.com EventManager *eventManager; 35611290Sgabor.dozsa@arm.com /** 35711290Sgabor.dozsa@arm.com * Calculate the tick to schedule the next receive done event. 35811290Sgabor.dozsa@arm.com * 35911290Sgabor.dozsa@arm.com * @param send_tick The tick the packet was sent. 36011290Sgabor.dozsa@arm.com * @param send_delay The simulated delay at the sender side. 36111290Sgabor.dozsa@arm.com * @param prev_recv_tick Tick when the last receive event was 36211290Sgabor.dozsa@arm.com * processed. 36311290Sgabor.dozsa@arm.com * 36411290Sgabor.dozsa@arm.com * @note This method tries to take into account possible receiver link 36511290Sgabor.dozsa@arm.com * contention and adjust receive tick for the incoming packets 36611290Sgabor.dozsa@arm.com * accordingly. 36711290Sgabor.dozsa@arm.com */ 36811290Sgabor.dozsa@arm.com Tick calcReceiveTick(Tick send_tick, 36911290Sgabor.dozsa@arm.com Tick send_delay, 37011290Sgabor.dozsa@arm.com Tick prev_recv_tick); 37111290Sgabor.dozsa@arm.com 37211290Sgabor.dozsa@arm.com /** 37311290Sgabor.dozsa@arm.com * Flag to set if receive ticks for pending packets need to be 37411290Sgabor.dozsa@arm.com * recalculated due to changed link latencies at a resume 37511290Sgabor.dozsa@arm.com */ 37611290Sgabor.dozsa@arm.com bool ckptRestore; 37710923SN/A 37810923SN/A public: 37910923SN/A /** 38011290Sgabor.dozsa@arm.com * Scheduler for the incoming data packets. 38111290Sgabor.dozsa@arm.com * 38211290Sgabor.dozsa@arm.com * @param em The event manager associated with the simulated Ethernet 38311290Sgabor.dozsa@arm.com * link. 38410923SN/A */ 38511290Sgabor.dozsa@arm.com RecvScheduler(EventManager *em) : 38611290Sgabor.dozsa@arm.com prevRecvTick(0), recvDone(nullptr), linkDelay(0), 38711290Sgabor.dozsa@arm.com eventManager(em), ckptRestore(false) {} 38810923SN/A 38910923SN/A /** 39011290Sgabor.dozsa@arm.com * Initialize network link parameters. 39110923SN/A * 39211290Sgabor.dozsa@arm.com * @note This method is called from the receiver thread (see 39311290Sgabor.dozsa@arm.com * recvThreadFunc()). 39411290Sgabor.dozsa@arm.com */ 39511290Sgabor.dozsa@arm.com void init(Event *recv_done, Tick link_delay); 39611290Sgabor.dozsa@arm.com /** 39711290Sgabor.dozsa@arm.com * Fetch the next packet that is to be received by the simulated network 39811290Sgabor.dozsa@arm.com * link. 39910923SN/A * 40011290Sgabor.dozsa@arm.com * @note This method is called from the process() method of the receive 40111290Sgabor.dozsa@arm.com * done event associated with the network link. 40210923SN/A */ 40311290Sgabor.dozsa@arm.com EthPacketPtr popPacket(); 40410923SN/A /** 40511290Sgabor.dozsa@arm.com * Push a newly arrived packet into the desc queue. 40611290Sgabor.dozsa@arm.com */ 40711290Sgabor.dozsa@arm.com void pushPacket(EthPacketPtr new_packet, 40811290Sgabor.dozsa@arm.com Tick send_tick, 40911290Sgabor.dozsa@arm.com Tick send_delay); 41011290Sgabor.dozsa@arm.com 41111290Sgabor.dozsa@arm.com void serialize(CheckpointOut &cp) const override; 41211290Sgabor.dozsa@arm.com void unserialize(CheckpointIn &cp) override; 41311290Sgabor.dozsa@arm.com /** 41411290Sgabor.dozsa@arm.com * Adjust receive ticks for pending packets when restoring from a 41511290Sgabor.dozsa@arm.com * checkpoint 41610923SN/A * 41711290Sgabor.dozsa@arm.com * @note Link speed and delay parameters may change at resume. 41810923SN/A */ 41911290Sgabor.dozsa@arm.com void resumeRecvTicks(); 42010923SN/A }; 42110923SN/A /** 42211290Sgabor.dozsa@arm.com * Tick to schedule the first dist sync event. 42311290Sgabor.dozsa@arm.com * This is just as optimization : we do not need any dist sync 42410923SN/A * event until the simulated NIC is brought up by the OS. 42510923SN/A */ 42610923SN/A Tick syncStart; 42710923SN/A /** 42811290Sgabor.dozsa@arm.com * Frequency of dist sync events in ticks. 42910923SN/A */ 43010923SN/A Tick syncRepeat; 43110923SN/A /** 43210923SN/A * Receiver thread pointer. 43311290Sgabor.dozsa@arm.com * Each DistIface object must have exactly one receiver thread. 43410923SN/A */ 43510923SN/A std::thread *recvThread; 43610923SN/A /** 43711290Sgabor.dozsa@arm.com * Meta information about data packets received. 43810923SN/A */ 43911290Sgabor.dozsa@arm.com RecvScheduler recvScheduler; 44010923SN/A 44111290Sgabor.dozsa@arm.com protected: 44210923SN/A /** 44310923SN/A * The rank of this process among the gem5 peers. 44410923SN/A */ 44510923SN/A unsigned rank; 44610923SN/A /** 44711290Sgabor.dozsa@arm.com * The number of gem5 processes comprising this dist simulation. 44811290Sgabor.dozsa@arm.com */ 44911290Sgabor.dozsa@arm.com unsigned size; 45011290Sgabor.dozsa@arm.com /** 45111290Sgabor.dozsa@arm.com * Number of DistIface objects (i.e. dist links in this gem5 process) 45211290Sgabor.dozsa@arm.com */ 45311290Sgabor.dozsa@arm.com static unsigned distIfaceNum; 45411290Sgabor.dozsa@arm.com /** 45511290Sgabor.dozsa@arm.com * Unique id for the dist link 45611290Sgabor.dozsa@arm.com */ 45711290Sgabor.dozsa@arm.com unsigned distIfaceId; 45811290Sgabor.dozsa@arm.com 45911290Sgabor.dozsa@arm.com bool isMaster; 46011290Sgabor.dozsa@arm.com 46111290Sgabor.dozsa@arm.com private: 46211290Sgabor.dozsa@arm.com /** 46311290Sgabor.dozsa@arm.com * Number of receiver threads (in this gem5 process) 46410923SN/A */ 46510923SN/A static unsigned recvThreadsNum; 46610923SN/A /** 46711290Sgabor.dozsa@arm.com * The singleton Sync object to perform dist synchronisation. 46811290Sgabor.dozsa@arm.com */ 46911290Sgabor.dozsa@arm.com static Sync *sync; 47011290Sgabor.dozsa@arm.com /** 47111290Sgabor.dozsa@arm.com * The singleton SyncEvent object to schedule periodic dist sync. 47211290Sgabor.dozsa@arm.com */ 47311290Sgabor.dozsa@arm.com static SyncEvent *syncEvent; 47411290Sgabor.dozsa@arm.com /** 47511290Sgabor.dozsa@arm.com * The very first DistIface object created becomes the master. We need 47610923SN/A * a master to co-ordinate the global synchronisation. 47710923SN/A */ 47811290Sgabor.dozsa@arm.com static DistIface *master; 47910923SN/A 48011290Sgabor.dozsa@arm.com private: 48110923SN/A /** 48211290Sgabor.dozsa@arm.com * Send out a data packet to the remote end. 48311290Sgabor.dozsa@arm.com * @param header Meta info about the packet (which needs to be transferred 48411290Sgabor.dozsa@arm.com * to the destination alongside the packet). 48511290Sgabor.dozsa@arm.com * @param packet Pointer to the packet to send. 48610923SN/A */ 48711290Sgabor.dozsa@arm.com virtual void sendPacket(const Header &header, const EthPacketPtr &packet) = 0; 48810923SN/A /** 48911290Sgabor.dozsa@arm.com * Send out a control command to the remote end. 49011290Sgabor.dozsa@arm.com * @param header Meta info describing the command (e.g. sync request) 49110923SN/A */ 49211290Sgabor.dozsa@arm.com virtual void sendCmd(const Header &header) = 0; 49310923SN/A /** 49411290Sgabor.dozsa@arm.com * Receive a header (i.e. meta info describing a data packet or a control command) 49511290Sgabor.dozsa@arm.com * from the remote end. 49611290Sgabor.dozsa@arm.com * @param header The meta info structure to store the incoming header. 49710923SN/A */ 49811290Sgabor.dozsa@arm.com virtual bool recvHeader(Header &header) = 0; 49911290Sgabor.dozsa@arm.com /** 50011290Sgabor.dozsa@arm.com * Receive a packet from the remote end. 50111290Sgabor.dozsa@arm.com * @param header Meta info about the incoming packet (obtanied by a previous 50211290Sgabor.dozsa@arm.com * call to the recvHedaer() method). 50311290Sgabor.dozsa@arm.com * @param Pointer to packet received. 50411290Sgabor.dozsa@arm.com */ 50511290Sgabor.dozsa@arm.com virtual void recvPacket(const Header &header, EthPacketPtr &packet) = 0; 50611290Sgabor.dozsa@arm.com /** 50711290Sgabor.dozsa@arm.com * Init hook for the underlaying transport 50811290Sgabor.dozsa@arm.com */ 50911290Sgabor.dozsa@arm.com virtual void initTransport() = 0; 51011290Sgabor.dozsa@arm.com /** 51111290Sgabor.dozsa@arm.com * spawn the receiver thread. 51211290Sgabor.dozsa@arm.com * @param recv_done The receive done event associated with the simulated 51311290Sgabor.dozsa@arm.com * Ethernet link. 51411290Sgabor.dozsa@arm.com * @param link_delay The link delay for the simulated Ethernet link. 51511290Sgabor.dozsa@arm.com */ 51611290Sgabor.dozsa@arm.com void spawnRecvThread(const Event *recv_done, Tick link_delay); 51710923SN/A /** 51810923SN/A * The function executed by a receiver thread. 51910923SN/A */ 52011290Sgabor.dozsa@arm.com void recvThreadFunc(Event *recv_done, Tick link_delay); 52110923SN/A 52210923SN/A public: 52310923SN/A 52410923SN/A /** 52510923SN/A * ctor 52611290Sgabor.dozsa@arm.com * @param dist_rank Rank of this gem5 process within the dist run 52711290Sgabor.dozsa@arm.com * @param sync_start Start tick for dist synchronisation 52811290Sgabor.dozsa@arm.com * @param sync_repeat Frequency for dist synchronisation 52910923SN/A * @param em The event manager associated with the simulated Ethernet link 53010923SN/A */ 53111290Sgabor.dozsa@arm.com DistIface(unsigned dist_rank, 53211290Sgabor.dozsa@arm.com unsigned dist_size, 53311290Sgabor.dozsa@arm.com Tick sync_start, 53411290Sgabor.dozsa@arm.com Tick sync_repeat, 53511290Sgabor.dozsa@arm.com EventManager *em, 53611290Sgabor.dozsa@arm.com bool is_switch, 53711290Sgabor.dozsa@arm.com int num_nodes); 53810923SN/A 53911290Sgabor.dozsa@arm.com virtual ~DistIface(); 54010923SN/A /** 54110923SN/A * Send out an Ethernet packet. 54210923SN/A * @param pkt The Ethernet packet to send. 54310923SN/A * @param send_delay The delay in ticks for the send completion event. 54410923SN/A */ 54510923SN/A void packetOut(EthPacketPtr pkt, Tick send_delay); 54610923SN/A /** 54711290Sgabor.dozsa@arm.com * Fetch the packet scheduled to be received next by the simulated 54811290Sgabor.dozsa@arm.com * network link. 54911290Sgabor.dozsa@arm.com * 55011290Sgabor.dozsa@arm.com * @note This method is called within the process() method of the link 55111290Sgabor.dozsa@arm.com * receive done event. It also schedules the next receive event if the 55211290Sgabor.dozsa@arm.com * receive queue is not empty. 55310923SN/A */ 55411290Sgabor.dozsa@arm.com EthPacketPtr packetIn() { return recvScheduler.popPacket(); } 55510923SN/A 55611168SN/A DrainState drain() override; 55711290Sgabor.dozsa@arm.com void drainResume() override; 55811290Sgabor.dozsa@arm.com void init(const Event *e, Tick link_delay); 55911290Sgabor.dozsa@arm.com void startup(); 56010923SN/A 56111290Sgabor.dozsa@arm.com void serialize(CheckpointOut &cp) const override; 56211290Sgabor.dozsa@arm.com void unserialize(CheckpointIn &cp) override; 56310923SN/A /** 56411290Sgabor.dozsa@arm.com * Initiate the exit from the simulation. 56511290Sgabor.dozsa@arm.com * @param delay Delay param from the m5 exit command. If Delay is zero 56611290Sgabor.dozsa@arm.com * then a collaborative exit is requested (i.e. all nodes have to call 56711290Sgabor.dozsa@arm.com * this method before the distributed simulation can exit). If Delay is 56811290Sgabor.dozsa@arm.com * not zero then exit is requested asap (and it will happen at the next 56911290Sgabor.dozsa@arm.com * sync tick). 57011290Sgabor.dozsa@arm.com * @return False if we are in distributed mode (i.e. exit can happen only 57111290Sgabor.dozsa@arm.com * at sync), True otherwise. 57210923SN/A */ 57311290Sgabor.dozsa@arm.com static bool readyToExit(Tick delay); 57411290Sgabor.dozsa@arm.com /** 57511290Sgabor.dozsa@arm.com * Initiate taking a checkpoint 57611290Sgabor.dozsa@arm.com * @param delay Delay param from the m5 checkpoint command. If Delay is 57711290Sgabor.dozsa@arm.com * zero then a collaborative checkpoint is requested (i.e. all nodes have 57811290Sgabor.dozsa@arm.com * to call this method before the checkpoint can be taken). If Delay is 57911290Sgabor.dozsa@arm.com * not zero then a checkpoint is requested asap (and it will happen at the 58011290Sgabor.dozsa@arm.com * next sync tick). 58111290Sgabor.dozsa@arm.com * @return False if we are in dist mode (i.e. exit can happen only at 58211290Sgabor.dozsa@arm.com * sync), True otherwise. 58311290Sgabor.dozsa@arm.com */ 58411290Sgabor.dozsa@arm.com static bool readyToCkpt(Tick delay, Tick period); 58511290Sgabor.dozsa@arm.com /** 58611290Sgabor.dozsa@arm.com * Getter for the dist rank param. 58711290Sgabor.dozsa@arm.com */ 58811290Sgabor.dozsa@arm.com static uint64_t rankParam(); 58911290Sgabor.dozsa@arm.com /** 59011290Sgabor.dozsa@arm.com * Getter for the dist size param. 59111290Sgabor.dozsa@arm.com */ 59211290Sgabor.dozsa@arm.com static uint64_t sizeParam(); 59311290Sgabor.dozsa@arm.com }; 59410923SN/A 59511290Sgabor.dozsa@arm.com#endif 596