dist_iface.hh revision 11168
11689SN/A/* 21689SN/A * Copyright (c) 2015 ARM Limited 31689SN/A * All rights reserved 41689SN/A * 51689SN/A * The license below extends only to copyright in the software and shall 61689SN/A * not be construed as granting a license to any other intellectual 71689SN/A * property including but not limited to intellectual property relating 81689SN/A * to a hardware implementation of the functionality of the software 91689SN/A * licensed hereunder. You may use the software subject to the license 101689SN/A * terms below provided that you ensure that this notice is replicated 111689SN/A * unmodified and in its entirety in all distributions of the software, 121689SN/A * modified or unmodified, in source code or in binary form. 131689SN/A * 141689SN/A * Redistribution and use in source and binary forms, with or without 151689SN/A * modification, are permitted provided that the following conditions are 161689SN/A * met: redistributions of source code must retain the above copyright 171689SN/A * notice, this list of conditions and the following disclaimer; 181689SN/A * redistributions in binary form must reproduce the above copyright 191689SN/A * notice, this list of conditions and the following disclaimer in the 201689SN/A * documentation and/or other materials provided with the distribution; 211689SN/A * neither the name of the copyright holders nor the names of its 221689SN/A * contributors may be used to endorse or promote products derived from 231689SN/A * this software without specific prior written permission. 241689SN/A * 251689SN/A * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 261689SN/A * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 272665Ssaidi@eecs.umich.edu * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 282665Ssaidi@eecs.umich.edu * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 292756Sksewell@umich.edu * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 301689SN/A * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 311689SN/A * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 322325SN/A * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 332325SN/A * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 341060SN/A * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 351060SN/A * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 361060SN/A * 372292SN/A * Authors: Gabor Dozsa 382292SN/A */ 391681SN/A 401060SN/A/* @file 412669Sktlim@umich.edu * The interface class for multi gem5 simulations. 421060SN/A * 431060SN/A * Multi gem5 is an extension to gem5 to enable parallel simulation of a 441858SN/A * distributed system (e.g. simulation of a pool of machines 452325SN/A * connected by Ethernet links). A multi gem5 run consists of seperate gem5 461717SN/A * processes running in parallel. Each gem5 process executes 472683Sktlim@umich.edu * the simulation of a component of the simulated distributed system. 481717SN/A * (An example component can be a multi-core board with an Ethernet NIC.) 491717SN/A * The MultiIface class below provides services to transfer data and 502292SN/A * control messages among the gem5 processes. The main such services are 512292SN/A * as follows. 522817Sksewell@umich.edu * 531060SN/A * 1. Send a data packet coming from a simulated Ethernet link. The packet 541060SN/A * will be transferred to (all) the target(s) gem5 processes. The send 552316SN/A * operation is always performed by the simulation thread, i.e. the gem5 562316SN/A * thread that is processing the event queue associated with the simulated 572680Sktlim@umich.edu * Ethernet link. 582817Sksewell@umich.edu * 592817Sksewell@umich.edu * 2. Spawn a receiver thread to process messages coming in from the 602669Sktlim@umich.edu * from other gem5 processes. Each simulated Ethernet link has its own 611060SN/A * associated receiver thread. The receiver thread saves the incoming packet 621060SN/A * and schedule an appropriate receive event in the event queue. 632733Sktlim@umich.edu * 641060SN/A * 3. Schedule a global barrier event periodically to keep the gem5 651060SN/A * processes in sync. 661060SN/A * Periodic barrier event to keep peer gem5 processes in sync. The basic idea 671464SN/A * is that no gem5 process can go ahead further than the simulated link 681061SN/A * transmission delay to ensure that a corresponding receive event can always 692733Sktlim@umich.edu * be scheduled for any message coming in from a peer gem5 process. 702292SN/A * 712292SN/A * 722632Sstever@eecs.umich.edu * 732817Sksewell@umich.edu * This interface is an abstract class (sendRaw() and recvRaw() 742817Sksewell@umich.edu * methods are pure virtual). It can work with various low level 752817Sksewell@umich.edu * send/receive service implementations (e.g. TCP/IP, MPI,...). A TCP 762817Sksewell@umich.edu * stream socket version is implemented in dev/src/tcp_iface.[hh,cc]. 772669Sktlim@umich.edu */ 781681SN/A#ifndef __DEV_MULTI_IFACE_HH__ 791685SN/A#define __DEV_MULTI_IFACE_HH__ 801681SN/A 811060SN/A#include <array> 821060SN/A#include <mutex> 832348SN/A#include <queue> 842348SN/A#include <thread> 852348SN/A#include <utility> 862348SN/A 872348SN/A#include "dev/etherpkt.hh" 881060SN/A#include "dev/multi_packet.hh" 892733Sktlim@umich.edu#include "sim/core.hh" 901060SN/A#include "sim/drain.hh" 911060SN/A#include "sim/global_event.hh" 922669Sktlim@umich.edu 932669Sktlim@umich.educlass EventManager; 942669Sktlim@umich.edu 952325SN/A/** 961060SN/A * The interface class to talk to peer gem5 processes. 971060SN/A */ 981061SN/Aclass MultiIface : public Drainable 991060SN/A{ 1002292SN/A public: 1012292SN/A /*! 1022292SN/A * The possible reasons a multi sync among gem5 peers is needed for. 1032292SN/A */ 1042817Sksewell@umich.edu enum 1051060SN/A class SyncTrigger { 1061060SN/A periodic, /*!< Regular periodic sync. This can be interrupted by a 1071060SN/A checkpoint sync request */ 1081060SN/A ckpt, /*!< sync before taking a checkpoint */ 1091060SN/A atomic /*!< sync that cannot be interrupted (e.g. sync at startup) */ 1102307SN/A }; 1112307SN/A 1121060SN/A private: 1131060SN/A typedef MultiHeaderPkt::MsgType MsgType; 1142292SN/A 1151060SN/A /** Sync State-Machine 1161060SN/A \dot 1171060SN/A digraph Sync { 1181060SN/A node [shape=box, fontsize=10]; 1191060SN/A idle -> busy 1201060SN/A [ label="new trigger\n by run()" fontsize=8 ]; 1212292SN/A busy -> busy 1221755SN/A [ label="new message by progress():\n(msg == SyncAck &&\nwaitNum > 1) || \n(msg==CkptSyncReq &&\ntrigger == ckpt)" fontsize=8 ]; 1231060SN/A busy -> idle 1241060SN/A [ label="new message by progress():\n(msg == SyncAck &&\nwaitNum == 1)" fontsize=8 ]; 1252292SN/A busy -> interrupted 1261755SN/A [ label="new message by progress():\n(msg == CkptSyncReq &&\ntrigger == periodic)" fontsize=8 ]; 1272292SN/A idle -> asyncCkpt 1282292SN/A [ label="new message by progress():\nmsg == CkptSyncReq" fontsize=8 ]; 1291060SN/A asyncCkpt -> asyncCkpt 1302292SN/A [ label="new message by progress():\nmsg == CkptSyncReq" fontsize=8 ]; 1311060SN/A asyncCkpt -> busy 1321060SN/A [ label="new trigger by run():\ntrigger == ckpt" fontsize=8 ]; 1331060SN/A asyncCkpt -> idle 1342292SN/A [ label="new trigger by run():\n(trigger == periodic &&\nwaitNum == 0) " fontsize=8 ]; 1351060SN/A asyncCkpt -> interrupted 1361060SN/A [ label="new trigger by run():\n(trigger == periodic &&\nwaitNum > 0) " fontsize=8 ]; 1372292SN/A interrupted -> interrupted 1381060SN/A [ label="new message by progress():\n(msg == CkptSyncReq &&\nwaitNum > 1)" fontsize=8 ]; 1391060SN/A interrupted -> idle 1401060SN/A [ label="new message by progress():\n(msg == CkptSyncReq &&\nwaitNum == 1)" fontsize=8 ]; 1412307SN/A } 1421060SN/A \enddot 1432307SN/A */ 1441060SN/A /** @class Sync 1451060SN/A * This class implements global sync operations among gem5 peer processes. 1462292SN/A * 1471060SN/A * @note This class is used as a singleton object (shared by all MultiIface 1481060SN/A * objects). 1491060SN/A */ 1501060SN/A class Sync 1511060SN/A { 1521060SN/A private: 1531060SN/A /*! 1542292SN/A * Internal state of the sync singleton object. 1552292SN/A */ 1562292SN/A enum class SyncState { 1571755SN/A busy, /*!< There is an on-going sync. */ 1581060SN/A interrupted, /*!< An on-going periodic sync was interrupted. */ 1592292SN/A asyncCkpt, /*!< A checkpoint (sim_exit) is already scheduled */ 1601684SN/A idle /*!< There is no active sync. */ 1611684SN/A }; 1622292SN/A /** 1632292SN/A * The lock to protect access to the MultiSync object. 1642292SN/A */ 1651684SN/A std::mutex lock; 1661684SN/A /** 1672292SN/A * Condition variable for the simulation thread to wait on 1681060SN/A * until all receiver threads completes the current global 1691060SN/A * synchronisation. 1702292SN/A */ 1712292SN/A std::condition_variable cv; 1721060SN/A /** 1732292SN/A * Number of receiver threads that not yet completed the current global 1742292SN/A * synchronisation. 1752292SN/A */ 1762292SN/A unsigned waitNum; 1772292SN/A /** 1782292SN/A * The trigger for the most recent sync. 1792292SN/A */ 1802292SN/A SyncTrigger trigger; 1812292SN/A /** 1822292SN/A * Map sync triggers to request messages. 1832292SN/A */ 1842292SN/A std::array<MsgType, 3> triggerToMsg = {{ 1852292SN/A MsgType::cmdPeriodicSyncReq, 1862292SN/A MsgType::cmdCkptSyncReq, 1872292SN/A MsgType::cmdAtomicSyncReq 1882292SN/A }}; 1892292SN/A 1902292SN/A /** 1912292SN/A * Current sync state. 1922292SN/A */ 1932292SN/A SyncState state; 1942292SN/A 1952292SN/A public: 1962292SN/A /** 1972292SN/A * Core method to perform a full multi sync. 1982292SN/A * 1992292SN/A * @param t Sync trigger. 2002292SN/A * @param sync_tick The tick the sync was expected to happen at. 2012292SN/A * @return true if the sync completed, false if it was interrupted. 2022292SN/A * 2032292SN/A * @note In case of an interrupted periodic sync, sync_tick can be less 2042292SN/A * than curTick() when we resume (i.e. re-run) it 2052292SN/A */ 2062292SN/A bool run(SyncTrigger t, Tick sync_tick); 2072292SN/A /** 2082292SN/A * Callback when the receiver thread gets a sync message. 2092292SN/A */ 2102292SN/A void progress(MsgType m); 2112292SN/A 2122292SN/A Sync() : waitNum(0), state(SyncState::idle) {} 2132292SN/A ~Sync() {} 2142292SN/A }; 2152292SN/A 2162292SN/A 2172325SN/A /** 2182292SN/A * The global event to schedule peridic multi sync. It is used as a 2192348SN/A * singleton object. 2202307SN/A * 2212292SN/A * The periodic synchronisation works as follows. 2222348SN/A * 1. A MultisyncEvent is scheduled as a global event when startup() is 2232316SN/A * called. 2242316SN/A * 2. The progress() method of the MultisyncEvent initiates a new barrier 2252348SN/A * for each simulated Ethernet links. 2261060SN/A * 3. Simulation thread(s) then waits until all receiver threads 2271060SN/A * completes the ongoing barrier. The global sync event is done. 2281060SN/A */ 2292316SN/A class SyncEvent : public GlobalSyncEvent 2302316SN/A { 2311060SN/A public: 2321858SN/A /** 2331060SN/A * Flag to indicate that the most recent periodic sync was interrupted 2341060SN/A * (by a checkpoint request). 2351060SN/A */ 2361060SN/A bool interrupted; 2371060SN/A /** 2381060SN/A * The tick when the most recent periodic synchronisation was scheduled 2391060SN/A * at. 2402292SN/A */ 2412292SN/A Tick scheduledAt; 2421060SN/A /** 2431060SN/A * Flag to indicate an on-going drain cycle. 2442292SN/A */ 2452292SN/A bool isDraining; 2461060SN/A 2472292SN/A public: 2482292SN/A /** 2492683Sktlim@umich.edu * Only the firstly instanstiated MultiIface object will 2501060SN/A * call this constructor. 2512292SN/A */ 2522292SN/A SyncEvent() : GlobalSyncEvent(Default_Pri, 0), interrupted(false), 2532683Sktlim@umich.edu scheduledAt(0), isDraining(false) {} 2541060SN/A 2551060SN/A ~SyncEvent() { assert (scheduled() == false); } 2561060SN/A /** 2572348SN/A * Schedule the first periodic sync event. 2581060SN/A * 2591060SN/A * @param start Start tick for multi synchronisation 2602455SN/A * @param repeat Frequency of multi synchronisation 2611060SN/A * 2622455SN/A */ 2631060SN/A void start(Tick start, Tick repeat); 2642455SN/A /** 2652455SN/A * Reschedule (if necessary) the periodic sync event. 2662455SN/A * 2671060SN/A * @param start Start tick for multi synchronisation 2681060SN/A * @param repeat Frequency of multi synchronisation 2691060SN/A * 2702669Sktlim@umich.edu * @note Useful if we have multiple MultiIface objects with 2711060SN/A * different 'start' and 'repeat' values for global sync. 2722455SN/A */ 2731060SN/A void adjust(Tick start, Tick repeat); 2742455SN/A /** 2752455SN/A * This is a global event so process() will be called by each 2762669Sktlim@umich.edu * simulation threads. (See further comments in the .cc file.) 2771060SN/A */ 2782292SN/A void process() override; 2791060SN/A /** 2802292SN/A * Schedule periodic sync when resuming from a checkpoint. 2811060SN/A */ 2822292SN/A void resume(); 2832292SN/A 2842292SN/A void serialize(const std::string &base, CheckpointOut &cp) const; 2852292SN/A void unserialize(const std::string &base, CheckpointIn &cp); 2862348SN/A }; 2872348SN/A 2882348SN/A /** 2892348SN/A * The receive thread needs to store the packet pointer and the computed 2902348SN/A * receive tick for each incoming data packet. This information is used 2912292SN/A * by the simulation thread when it processes the corresponding receive 2922292SN/A * event. (See more comments at the implemetation of the recvThreadFunc() 2932292SN/A * and RecvPacketIn() methods.) 2942292SN/A */ 2952292SN/A typedef std::pair<EthPacketPtr, Tick> RecvInfo; 2962292SN/A 2972292SN/A /** 2982292SN/A * Comparison predicate for RecvInfo, needed by the recvQueue. 2992348SN/A */ 3002292SN/A struct RecvInfoCompare { 3012292SN/A bool operator()(const RecvInfo &lhs, const RecvInfo &rhs) 3022348SN/A { 3032348SN/A return lhs.second > rhs.second; 3042292SN/A } 3052348SN/A }; 3062292SN/A 3072292SN/A /** 3082348SN/A * Customized priority queue used to store incoming data packets info by 3092348SN/A * the receiver thread. We need to expose the underlying container to 3101060SN/A * enable iterator access for serializing. 3112756Sksewell@umich.edu */ 3122756Sksewell@umich.edu class RecvQueue : public std::priority_queue<RecvInfo, 3132756Sksewell@umich.edu std::vector<RecvInfo>, 3142756Sksewell@umich.edu RecvInfoCompare> 3152756Sksewell@umich.edu { 3162756Sksewell@umich.edu public: 3171060SN/A std::vector<RecvInfo> &impl() { return c; } 3181060SN/A const std::vector<RecvInfo> &impl() const { return c; } 3191060SN/A }; 3202292SN/A 3211060SN/A /* 3221060SN/A * The priority queue to store RecvInfo items ordered by receive ticks. 3232292SN/A */ 3241060SN/A RecvQueue recvQueue; 3252292SN/A /** 3262292SN/A * The singleton Sync object to perform multi synchronisation. 3271060SN/A */ 3282325SN/A static Sync *sync; 3292325SN/A /** 3301060SN/A * The singleton SyncEvent object to schedule periodic multi sync. 3311061SN/A */ 3321060SN/A static SyncEvent *syncEvent; 3331060SN/A /** 3342292SN/A * Tick to schedule the first multi sync event. 3351060SN/A * This is just as optimization : we do not need any multi sync 3361062SN/A * event until the simulated NIC is brought up by the OS. 3372292SN/A */ 3382292SN/A Tick syncStart; 3392348SN/A /** 3402292SN/A * Frequency of multi sync events in ticks. 3412292SN/A */ 3422348SN/A Tick syncRepeat; 3432292SN/A /** 3441062SN/A * Receiver thread pointer. 3452348SN/A * Each MultiIface object must have exactly one receiver thread. 3461060SN/A */ 3471060SN/A std::thread *recvThread; 3481060SN/A /** 3491060SN/A * The event manager associated with the MultiIface object. 3502292SN/A */ 3511060SN/A EventManager *eventManager; 3522292SN/A 3532292SN/A /** 3542292SN/A * The receive done event for the simulated Ethernet link. 3552292SN/A * It is scheduled by the receiver thread for each incoming data 3562292SN/A * packet. 3572325SN/A */ 3582348SN/A Event *recvDone; 3592348SN/A 3602348SN/A /** 3612292SN/A * The packet that belongs to the currently scheduled recvDone event. 3622325SN/A */ 3632292SN/A EthPacketPtr scheduledRecvPacket; 3642325SN/A 3652325SN/A /** 3662292SN/A * The link delay in ticks for the simulated Ethernet link. 3672292SN/A */ 3682292SN/A Tick linkDelay; 3691060SN/A 3701060SN/A /** 3711060SN/A * The rank of this process among the gem5 peers. 3721060SN/A */ 3731060SN/A unsigned rank; 3741060SN/A /** 3751060SN/A * Total number of receiver threads (in this gem5 process). 3761060SN/A * During the simulation it should be constant and equal to the 3771060SN/A * number of MultiIface objects (i.e. simulated Ethernet 3781060SN/A * links). 3791060SN/A */ 3801060SN/A static unsigned recvThreadsNum; 3811060SN/A /** 3821060SN/A * The very first MultiIface object created becomes the master. We need 3831060SN/A * a master to co-ordinate the global synchronisation. 3841060SN/A */ 3851060SN/A static MultiIface *master; 3861060SN/A 3871060SN/A protected: 3881060SN/A /** 3891060SN/A * Low level generic send routine. 3901060SN/A * @param buf buffer that holds the data to send out 3911060SN/A * @param length number of bytes to send 3922292SN/A * @param dest_addr address of the target (simulated NIC). This may be 3932292SN/A * used by a subclass for optimization (e.g. optimize broadcast) 3942292SN/A */ 3952292SN/A virtual void sendRaw(void *buf, 3961060SN/A unsigned length, 3971060SN/A const MultiHeaderPkt::AddressType dest_addr) = 0; 3981060SN/A /** 3991060SN/A * Low level generic receive routine. 4002292SN/A * @param buf the buffer to store the incoming message 4012292SN/A * @param length buffer size (in bytes) 4022292SN/A */ 4032292SN/A virtual bool recvRaw(void *buf, unsigned length) = 0; 4042292SN/A /** 4052292SN/A * Low level request for synchronisation among gem5 processes. Only one 4061060SN/A * MultiIface object needs to call this (in each gem5 process) to trigger 4072292SN/A * a multi sync. 4082292SN/A * 4092292SN/A * @param sync_req Sync request command. 4102292SN/A * @param sync_tick The tick when sync is expected to happen in the sender. 4112292SN/A */ 4122292SN/A virtual void syncRaw(MsgType sync_req, Tick sync_tick) = 0; 4132292SN/A /** 4142292SN/A * The function executed by a receiver thread. 4152292SN/A */ 4162292SN/A void recvThreadFunc(); 4172292SN/A /** 4182292SN/A * Receive a multi header packet. Called by the receiver thread. 4191060SN/A * @param header the structure to store the incoming header packet. 4201060SN/A * @return false if any error occured during the receive, true otherwise 4211060SN/A * 4221061SN/A * A header packet can carry a control command (e.g. 'barrier leave') or 4231060SN/A * information about a data packet that is following the header packet 4241061SN/A * back to back. 4251060SN/A */ 4261061SN/A bool recvHeader(MultiHeaderPkt::Header &header); 4271060SN/A /** 4281061SN/A * Receive a data packet. Called by the receiver thread. 4291060SN/A * @param data_header The packet descriptor for the expected incoming data 4301061SN/A * packet. 4311060SN/A */ 4321060SN/A void recvData(const MultiHeaderPkt::Header &data_header); 4331060SN/A 4341060SN/A public: 4351060SN/A 4361060SN/A /** 4371060SN/A * ctor 4381060SN/A * @param multi_rank Rank of this gem5 process within the multi run 4391060SN/A * @param sync_start Start tick for multi synchronisation 4401060SN/A * @param sync_repeat Frequency for multi synchronisation 4411060SN/A * @param em The event manager associated with the simulated Ethernet link 4421060SN/A */ 4431060SN/A MultiIface(unsigned multi_rank, 4441060SN/A Tick sync_start, 4451060SN/A Tick sync_repeat, 4461060SN/A EventManager *em); 4472348SN/A 4482348SN/A virtual ~MultiIface(); 4492348SN/A /** 4502348SN/A * Send out an Ethernet packet. 4512348SN/A * @param pkt The Ethernet packet to send. 4522325SN/A * @param send_delay The delay in ticks for the send completion event. 4531060SN/A */ 4542348SN/A void packetOut(EthPacketPtr pkt, Tick send_delay); 4552348SN/A /** 4562325SN/A * Fetch the next packet from the receive queue. 4572292SN/A */ 4582348SN/A EthPacketPtr packetIn(); 4592325SN/A 4602325SN/A /** 4612292SN/A * spawn the receiver thread. 4622348SN/A * @param recv_done The receive done event associated with the simulated 4632325SN/A * Ethernet link. 4642325SN/A * @param link_delay The link delay for the simulated Ethernet link. 4652292SN/A */ 4662292SN/A void spawnRecvThread(Event *recv_done, 4672292SN/A Tick link_delay); 4682260SN/A /** 4692292SN/A * Initialize the random number generator with a different seed in each 4702292SN/A * peer gem5 process. 4712292SN/A */ 4722292SN/A void initRandom(); 4732680Sktlim@umich.edu 4742680Sktlim@umich.edu DrainState drain() override; 4751681SN/A 4762680Sktlim@umich.edu /** 4772190SN/A * Callback when draining is complete. 4782190SN/A */ 4792292SN/A void drainDone(); 4801060SN/A 4811060SN/A /** 4822348SN/A * Initialize the periodic synchronisation among peer gem5 processes. 4832348SN/A */ 4842348SN/A void startPeriodicSync(); 4852348SN/A 4862316SN/A void serialize(const std::string &base, CheckpointOut &cp) const; 4872316SN/A void unserialize(const std::string &base, CheckpointIn &cp); 4881858SN/A 4892292SN/A}; 4901060SN/A 4911060SN/A 4922292SN/A#endif 4931060SN/A