dist_iface.hh revision 10923
16313Sgblack@eecs.umich.edu/* 26313Sgblack@eecs.umich.edu * Copyright (c) 2015 ARM Limited 36313Sgblack@eecs.umich.edu * All rights reserved 46313Sgblack@eecs.umich.edu * 56313Sgblack@eecs.umich.edu * The license below extends only to copyright in the software and shall 66313Sgblack@eecs.umich.edu * not be construed as granting a license to any other intellectual 76313Sgblack@eecs.umich.edu * property including but not limited to intellectual property relating 86313Sgblack@eecs.umich.edu * to a hardware implementation of the functionality of the software 96313Sgblack@eecs.umich.edu * licensed hereunder. You may use the software subject to the license 106313Sgblack@eecs.umich.edu * terms below provided that you ensure that this notice is replicated 116313Sgblack@eecs.umich.edu * unmodified and in its entirety in all distributions of the software, 126313Sgblack@eecs.umich.edu * modified or unmodified, in source code or in binary form. 136313Sgblack@eecs.umich.edu * 146313Sgblack@eecs.umich.edu * Redistribution and use in source and binary forms, with or without 156313Sgblack@eecs.umich.edu * modification, are permitted provided that the following conditions are 166313Sgblack@eecs.umich.edu * met: redistributions of source code must retain the above copyright 176313Sgblack@eecs.umich.edu * notice, this list of conditions and the following disclaimer; 186313Sgblack@eecs.umich.edu * redistributions in binary form must reproduce the above copyright 196313Sgblack@eecs.umich.edu * notice, this list of conditions and the following disclaimer in the 206313Sgblack@eecs.umich.edu * documentation and/or other materials provided with the distribution; 216313Sgblack@eecs.umich.edu * neither the name of the copyright holders nor the names of its 226313Sgblack@eecs.umich.edu * contributors may be used to endorse or promote products derived from 236313Sgblack@eecs.umich.edu * this software without specific prior written permission. 246313Sgblack@eecs.umich.edu * 256313Sgblack@eecs.umich.edu * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 266313Sgblack@eecs.umich.edu * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 276313Sgblack@eecs.umich.edu * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 286313Sgblack@eecs.umich.edu * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 296313Sgblack@eecs.umich.edu * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 306313Sgblack@eecs.umich.edu * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 3111793Sbrandon.potter@amd.com * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 3211793Sbrandon.potter@amd.com * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 339376Sgblack@eecs.umich.edu * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 346336Sgblack@eecs.umich.edu * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 356336Sgblack@eecs.umich.edu * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 366313Sgblack@eecs.umich.edu * 379384SAndreas.Sandberg@arm.com * Authors: Gabor Dozsa 386336Sgblack@eecs.umich.edu */ 396313Sgblack@eecs.umich.edu 406313Sgblack@eecs.umich.edu/* @file 416313Sgblack@eecs.umich.edu * The interface class for multi gem5 simulations. 426313Sgblack@eecs.umich.edu * 436313Sgblack@eecs.umich.edu * Multi gem5 is an extension to gem5 to enable parallel simulation of a 446336Sgblack@eecs.umich.edu * distributed system (e.g. simulation of a pool of machines 459376Sgblack@eecs.umich.edu * connected by Ethernet links). A multi gem5 run consists of seperate gem5 469376Sgblack@eecs.umich.edu * processes running in parallel. Each gem5 process executes 476336Sgblack@eecs.umich.edu * the simulation of a component of the simulated distributed system. 486712Snate@binkert.org * (An example component can be a multi-core board with an Ethernet NIC.) 496336Sgblack@eecs.umich.edu * The MultiIface class below provides services to transfer data and 506336Sgblack@eecs.umich.edu * control messages among the gem5 processes. The main such services are 516336Sgblack@eecs.umich.edu * as follows. 526336Sgblack@eecs.umich.edu * 536336Sgblack@eecs.umich.edu * 1. Send a data packet coming from a simulated Ethernet link. The packet 546336Sgblack@eecs.umich.edu * will be transferred to (all) the target(s) gem5 processes. The send 556336Sgblack@eecs.umich.edu * operation is always performed by the simulation thread, i.e. the gem5 566336Sgblack@eecs.umich.edu * thread that is processing the event queue associated with the simulated 576336Sgblack@eecs.umich.edu * Ethernet link. 586336Sgblack@eecs.umich.edu * 596336Sgblack@eecs.umich.edu * 2. Spawn a receiver thread to process messages coming in from the 606336Sgblack@eecs.umich.edu * from other gem5 processes. Each simulated Ethernet link has its own 616336Sgblack@eecs.umich.edu * associated receiver thread. The receiver thread saves the incoming packet 626336Sgblack@eecs.umich.edu * and schedule an appropriate receive event in the event queue. 636336Sgblack@eecs.umich.edu * 646336Sgblack@eecs.umich.edu * 3. Schedule a global barrier event periodically to keep the gem5 656336Sgblack@eecs.umich.edu * processes in sync. 666336Sgblack@eecs.umich.edu * Periodic barrier event to keep peer gem5 processes in sync. The basic idea 676336Sgblack@eecs.umich.edu * is that no gem5 process can go ahead further than the simulated link 686336Sgblack@eecs.umich.edu * transmission delay to ensure that a corresponding receive event can always 696336Sgblack@eecs.umich.edu * be scheduled for any message coming in from a peer gem5 process. 706336Sgblack@eecs.umich.edu * 716336Sgblack@eecs.umich.edu * 726336Sgblack@eecs.umich.edu * 736336Sgblack@eecs.umich.edu * This interface is an abstract class (sendRaw() and recvRaw() 746336Sgblack@eecs.umich.edu * methods are pure virtual). It can work with various low level 756336Sgblack@eecs.umich.edu * send/receive service implementations (e.g. TCP/IP, MPI,...). A TCP 766336Sgblack@eecs.umich.edu * stream socket version is implemented in dev/src/tcp_iface.[hh,cc]. 776336Sgblack@eecs.umich.edu */ 786336Sgblack@eecs.umich.edu#ifndef __DEV_MULTI_IFACE_HH__ 796336Sgblack@eecs.umich.edu#define __DEV_MULTI_IFACE_HH__ 806336Sgblack@eecs.umich.edu 816336Sgblack@eecs.umich.edu#include <array> 826336Sgblack@eecs.umich.edu#include <mutex> 836336Sgblack@eecs.umich.edu#include <queue> 846336Sgblack@eecs.umich.edu#include <thread> 856336Sgblack@eecs.umich.edu#include <utility> 866336Sgblack@eecs.umich.edu 876336Sgblack@eecs.umich.edu#include "dev/etherpkt.hh" 886336Sgblack@eecs.umich.edu#include "dev/multi_packet.hh" 896336Sgblack@eecs.umich.edu#include "sim/core.hh" 906336Sgblack@eecs.umich.edu#include "sim/drain.hh" 916336Sgblack@eecs.umich.edu#include "sim/global_event.hh" 926336Sgblack@eecs.umich.edu 936336Sgblack@eecs.umich.educlass EventManager; 946336Sgblack@eecs.umich.edu 956336Sgblack@eecs.umich.edu/** 966336Sgblack@eecs.umich.edu * The interface class to talk to peer gem5 processes. 976336Sgblack@eecs.umich.edu */ 986336Sgblack@eecs.umich.educlass MultiIface : public Drainable 996336Sgblack@eecs.umich.edu{ 1006336Sgblack@eecs.umich.edu public: 1019376Sgblack@eecs.umich.edu /*! 1029376Sgblack@eecs.umich.edu * The possible reasons a multi sync among gem5 peers is needed for. 1036336Sgblack@eecs.umich.edu */ 1046336Sgblack@eecs.umich.edu enum 1056336Sgblack@eecs.umich.edu class SyncTrigger { 1066313Sgblack@eecs.umich.edu periodic, /*!< Regular periodic sync. This can be interrupted by a 1076313Sgblack@eecs.umich.edu checkpoint sync request */ 1086336Sgblack@eecs.umich.edu ckpt, /*!< sync before taking a checkpoint */ 1096336Sgblack@eecs.umich.edu atomic /*!< sync that cannot be interrupted (e.g. sync at startup) */ 1106336Sgblack@eecs.umich.edu }; 1116336Sgblack@eecs.umich.edu 1126336Sgblack@eecs.umich.edu private: 1136313Sgblack@eecs.umich.edu typedef MultiHeaderPkt::MsgType MsgType; 1146313Sgblack@eecs.umich.edu 1159384SAndreas.Sandberg@arm.com /** Sync State-Machine 1169384SAndreas.Sandberg@arm.com \dot 1179384SAndreas.Sandberg@arm.com digraph Sync { 1189384SAndreas.Sandberg@arm.com node [shape=box, fontsize=10]; 1199384SAndreas.Sandberg@arm.com idle -> busy 1209384SAndreas.Sandberg@arm.com [ label="new trigger\n by run()" fontsize=8 ]; 1219384SAndreas.Sandberg@arm.com busy -> busy 1229384SAndreas.Sandberg@arm.com [ label="new message by progress():\n(msg == SyncAck &&\nwaitNum > 1) || \n(msg==CkptSyncReq &&\ntrigger == ckpt)" fontsize=8 ]; 1239384SAndreas.Sandberg@arm.com busy -> idle 1249384SAndreas.Sandberg@arm.com [ label="new message by progress():\n(msg == SyncAck &&\nwaitNum == 1)" fontsize=8 ]; 1259384SAndreas.Sandberg@arm.com busy -> interrupted 1269384SAndreas.Sandberg@arm.com [ label="new message by progress():\n(msg == CkptSyncReq &&\ntrigger == periodic)" fontsize=8 ]; 1276313Sgblack@eecs.umich.edu idle -> asyncCkpt 12810698Sandreas.hansson@arm.com [ label="new message by progress():\nmsg == CkptSyncReq" fontsize=8 ]; 1296313Sgblack@eecs.umich.edu asyncCkpt -> asyncCkpt 1306336Sgblack@eecs.umich.edu [ label="new message by progress():\nmsg == CkptSyncReq" fontsize=8 ]; 1316336Sgblack@eecs.umich.edu asyncCkpt -> busy 1326336Sgblack@eecs.umich.edu [ label="new trigger by run():\ntrigger == ckpt" fontsize=8 ]; 13311324Ssteve.reinhardt@amd.com asyncCkpt -> idle 1346336Sgblack@eecs.umich.edu [ label="new trigger by run():\n(trigger == periodic &&\nwaitNum == 0) " fontsize=8 ]; 1356336Sgblack@eecs.umich.edu asyncCkpt -> interrupted 1366313Sgblack@eecs.umich.edu [ label="new trigger by run():\n(trigger == periodic &&\nwaitNum > 0) " fontsize=8 ]; 1376313Sgblack@eecs.umich.edu interrupted -> interrupted 1386313Sgblack@eecs.umich.edu [ label="new message by progress():\n(msg == CkptSyncReq &&\nwaitNum > 1)" fontsize=8 ]; 1396336Sgblack@eecs.umich.edu interrupted -> idle 1406313Sgblack@eecs.umich.edu [ label="new message by progress():\n(msg == CkptSyncReq &&\nwaitNum == 1)" fontsize=8 ]; 1416336Sgblack@eecs.umich.edu } 1426336Sgblack@eecs.umich.edu \enddot 1436336Sgblack@eecs.umich.edu */ 1449372Snilay@cs.wisc.edu /** @class Sync 1459372Snilay@cs.wisc.edu * This class implements global sync operations among gem5 peer processes. 1469372Snilay@cs.wisc.edu * 1479372Snilay@cs.wisc.edu * @note This class is used as a singleton object (shared by all MultiIface 14812368Sgabeblack@google.com * objects). 1499372Snilay@cs.wisc.edu */ 1509372Snilay@cs.wisc.edu class Sync 1516336Sgblack@eecs.umich.edu { 1526313Sgblack@eecs.umich.edu private: 1536313Sgblack@eecs.umich.edu /*! 1546313Sgblack@eecs.umich.edu * Internal state of the sync singleton object. 1556336Sgblack@eecs.umich.edu */ 1566313Sgblack@eecs.umich.edu enum class SyncState { 1576336Sgblack@eecs.umich.edu busy, /*!< There is an on-going sync. */ 1586336Sgblack@eecs.umich.edu interrupted, /*!< An on-going periodic sync was interrupted. */ 1596336Sgblack@eecs.umich.edu asyncCkpt, /*!< A checkpoint (sim_exit) is already scheduled */ 16011324Ssteve.reinhardt@amd.com idle /*!< There is no active sync. */ 16110899Snikos.nikoleris@gmail.com }; 16212368Sgabeblack@google.com /** 16312368Sgabeblack@google.com * The lock to protect access to the MultiSync object. 16410899Snikos.nikoleris@gmail.com */ 16510899Snikos.nikoleris@gmail.com std::mutex lock; 16612368Sgabeblack@google.com /** 16710899Snikos.nikoleris@gmail.com * Condition variable for the simulation thread to wait on 16810899Snikos.nikoleris@gmail.com * until all receiver threads completes the current global 16912368Sgabeblack@google.com * synchronisation. 17010899Snikos.nikoleris@gmail.com */ 17112368Sgabeblack@google.com std::condition_variable cv; 17210899Snikos.nikoleris@gmail.com /** 17310899Snikos.nikoleris@gmail.com * Number of receiver threads that not yet completed the current global 17412368Sgabeblack@google.com * synchronisation. 17510899Snikos.nikoleris@gmail.com */ 17610899Snikos.nikoleris@gmail.com unsigned waitNum; 17712368Sgabeblack@google.com /** 17810899Snikos.nikoleris@gmail.com * The trigger for the most recent sync. 17910899Snikos.nikoleris@gmail.com */ 18010899Snikos.nikoleris@gmail.com SyncTrigger trigger; 18110899Snikos.nikoleris@gmail.com /** 18212368Sgabeblack@google.com * Map sync triggers to request messages. 18310899Snikos.nikoleris@gmail.com */ 18410899Snikos.nikoleris@gmail.com std::array<MsgType, 3> triggerToMsg = {{ 18510899Snikos.nikoleris@gmail.com MsgType::cmdPeriodicSyncReq, 18610899Snikos.nikoleris@gmail.com MsgType::cmdCkptSyncReq, 18712368Sgabeblack@google.com MsgType::cmdAtomicSyncReq 18810899Snikos.nikoleris@gmail.com }}; 18910899Snikos.nikoleris@gmail.com 19010899Snikos.nikoleris@gmail.com /** 19110899Snikos.nikoleris@gmail.com * Current sync state. 19210899Snikos.nikoleris@gmail.com */ 19312368Sgabeblack@google.com SyncState state; 1946313Sgblack@eecs.umich.edu 1956313Sgblack@eecs.umich.edu public: 1966313Sgblack@eecs.umich.edu /** 1976336Sgblack@eecs.umich.edu * Core method to perform a full multi sync. 1986313Sgblack@eecs.umich.edu * 1996336Sgblack@eecs.umich.edu * @param t Sync trigger. 2006336Sgblack@eecs.umich.edu * @param sync_tick The tick the sync was expected to happen at. 2016336Sgblack@eecs.umich.edu * @return true if the sync completed, false if it was interrupted. 2026336Sgblack@eecs.umich.edu * 2036336Sgblack@eecs.umich.edu * @note In case of an interrupted periodic sync, sync_tick can be less 2046336Sgblack@eecs.umich.edu * than curTick() when we resume (i.e. re-run) it 2056336Sgblack@eecs.umich.edu */ 2066336Sgblack@eecs.umich.edu bool run(SyncTrigger t, Tick sync_tick); 2076336Sgblack@eecs.umich.edu /** 2086336Sgblack@eecs.umich.edu * Callback when the receiver thread gets a sync message. 2096336Sgblack@eecs.umich.edu */ 2106336Sgblack@eecs.umich.edu void progress(MsgType m); 2116336Sgblack@eecs.umich.edu 2126336Sgblack@eecs.umich.edu Sync() : waitNum(0), state(SyncState::idle) {} 2136336Sgblack@eecs.umich.edu ~Sync() {} 2146336Sgblack@eecs.umich.edu }; 2156336Sgblack@eecs.umich.edu 2166336Sgblack@eecs.umich.edu 2176336Sgblack@eecs.umich.edu /** 2186336Sgblack@eecs.umich.edu * The global event to schedule peridic multi sync. It is used as a 2199423SAndreas.Sandberg@arm.com * singleton object. 2209423SAndreas.Sandberg@arm.com * 2216336Sgblack@eecs.umich.edu * The periodic synchronisation works as follows. 2226336Sgblack@eecs.umich.edu * 1. A MultisyncEvent is scheduled as a global event when startup() is 2236336Sgblack@eecs.umich.edu * called. 2246336Sgblack@eecs.umich.edu * 2. The progress() method of the MultisyncEvent initiates a new barrier 2256336Sgblack@eecs.umich.edu * for each simulated Ethernet links. 2266336Sgblack@eecs.umich.edu * 3. Simulation thread(s) then waits until all receiver threads 2276336Sgblack@eecs.umich.edu * completes the ongoing barrier. The global sync event is done. 2286336Sgblack@eecs.umich.edu */ 2299376Sgblack@eecs.umich.edu class SyncEvent : public GlobalSyncEvent 2309376Sgblack@eecs.umich.edu { 2316336Sgblack@eecs.umich.edu public: 2326336Sgblack@eecs.umich.edu /** 2336336Sgblack@eecs.umich.edu * Flag to indicate that the most recent periodic sync was interrupted 2346336Sgblack@eecs.umich.edu * (by a checkpoint request). 2356336Sgblack@eecs.umich.edu */ 2369423SAndreas.Sandberg@arm.com bool interrupted; 2379423SAndreas.Sandberg@arm.com /** 2386336Sgblack@eecs.umich.edu * The tick when the most recent periodic synchronisation was scheduled 2396336Sgblack@eecs.umich.edu * at. 2406336Sgblack@eecs.umich.edu */ 2416336Sgblack@eecs.umich.edu Tick scheduledAt; 2426336Sgblack@eecs.umich.edu /** 2439423SAndreas.Sandberg@arm.com * Flag to indicate an on-going drain cycle. 2449423SAndreas.Sandberg@arm.com */ 2456336Sgblack@eecs.umich.edu bool isDraining; 2466336Sgblack@eecs.umich.edu 2476336Sgblack@eecs.umich.edu public: 2486336Sgblack@eecs.umich.edu /** 2496336Sgblack@eecs.umich.edu * Only the firstly instanstiated MultiIface object will 2506336Sgblack@eecs.umich.edu * call this constructor. 2516336Sgblack@eecs.umich.edu */ 2526336Sgblack@eecs.umich.edu SyncEvent() : GlobalSyncEvent(Default_Pri, 0), interrupted(false), 2536336Sgblack@eecs.umich.edu scheduledAt(0), isDraining(false) {} 2546336Sgblack@eecs.umich.edu 2556336Sgblack@eecs.umich.edu ~SyncEvent() { assert (scheduled() == false); } 2566336Sgblack@eecs.umich.edu /** 2576336Sgblack@eecs.umich.edu * Schedule the first periodic sync event. 2586336Sgblack@eecs.umich.edu * 2596336Sgblack@eecs.umich.edu * @param start Start tick for multi synchronisation 2606336Sgblack@eecs.umich.edu * @param repeat Frequency of multi synchronisation 2616336Sgblack@eecs.umich.edu * 2626336Sgblack@eecs.umich.edu */ 2636336Sgblack@eecs.umich.edu void start(Tick start, Tick repeat); 2646336Sgblack@eecs.umich.edu /** 2656336Sgblack@eecs.umich.edu * Reschedule (if necessary) the periodic sync event. 2666336Sgblack@eecs.umich.edu * 2676336Sgblack@eecs.umich.edu * @param start Start tick for multi synchronisation 2686336Sgblack@eecs.umich.edu * @param repeat Frequency of multi synchronisation 2696336Sgblack@eecs.umich.edu * 2706336Sgblack@eecs.umich.edu * @note Useful if we have multiple MultiIface objects with 2719376Sgblack@eecs.umich.edu * different 'start' and 'repeat' values for global sync. 2729376Sgblack@eecs.umich.edu */ 2736336Sgblack@eecs.umich.edu void adjust(Tick start, Tick repeat); 2746336Sgblack@eecs.umich.edu /** 2756336Sgblack@eecs.umich.edu * This is a global event so process() will be called by each 2766336Sgblack@eecs.umich.edu * simulation threads. (See further comments in the .cc file.) 2776336Sgblack@eecs.umich.edu */ 2786336Sgblack@eecs.umich.edu void process() M5_ATTR_OVERRIDE; 2796336Sgblack@eecs.umich.edu /** 2809376Sgblack@eecs.umich.edu * Schedule periodic sync when resuming from a checkpoint. 2819376Sgblack@eecs.umich.edu */ 2826336Sgblack@eecs.umich.edu void resume(); 2836336Sgblack@eecs.umich.edu 2846336Sgblack@eecs.umich.edu void serialize(const std::string &base, CheckpointOut &cp) const; 2856336Sgblack@eecs.umich.edu void unserialize(const std::string &base, CheckpointIn &cp); 2866336Sgblack@eecs.umich.edu }; 2876336Sgblack@eecs.umich.edu 2886336Sgblack@eecs.umich.edu /** 2896336Sgblack@eecs.umich.edu * The receive thread needs to store the packet pointer and the computed 2906336Sgblack@eecs.umich.edu * receive tick for each incoming data packet. This information is used 2916336Sgblack@eecs.umich.edu * by the simulation thread when it processes the corresponding receive 2926336Sgblack@eecs.umich.edu * event. (See more comments at the implemetation of the recvThreadFunc() 2936336Sgblack@eecs.umich.edu * and RecvPacketIn() methods.) 2946336Sgblack@eecs.umich.edu */ 2956336Sgblack@eecs.umich.edu typedef std::pair<EthPacketPtr, Tick> RecvInfo; 2966336Sgblack@eecs.umich.edu 2976336Sgblack@eecs.umich.edu /** 2986336Sgblack@eecs.umich.edu * Comparison predicate for RecvInfo, needed by the recvQueue. 2996336Sgblack@eecs.umich.edu */ 3006336Sgblack@eecs.umich.edu struct RecvInfoCompare { 3016336Sgblack@eecs.umich.edu bool operator()(const RecvInfo &lhs, const RecvInfo &rhs) 3026336Sgblack@eecs.umich.edu { 3036336Sgblack@eecs.umich.edu return lhs.second > rhs.second; 3046336Sgblack@eecs.umich.edu } 3056336Sgblack@eecs.umich.edu }; 3066336Sgblack@eecs.umich.edu 3076336Sgblack@eecs.umich.edu /** 3086336Sgblack@eecs.umich.edu * Customized priority queue used to store incoming data packets info by 3096336Sgblack@eecs.umich.edu * the receiver thread. We need to expose the underlying container to 3106336Sgblack@eecs.umich.edu * enable iterator access for serializing. 3116336Sgblack@eecs.umich.edu */ 3126336Sgblack@eecs.umich.edu class RecvQueue : public std::priority_queue<RecvInfo, 3136336Sgblack@eecs.umich.edu std::vector<RecvInfo>, 3146336Sgblack@eecs.umich.edu RecvInfoCompare> 3156336Sgblack@eecs.umich.edu { 3166336Sgblack@eecs.umich.edu public: 3176336Sgblack@eecs.umich.edu std::vector<RecvInfo> &impl() { return c; } 3186336Sgblack@eecs.umich.edu const std::vector<RecvInfo> &impl() const { return c; } 3196336Sgblack@eecs.umich.edu }; 3206336Sgblack@eecs.umich.edu 3216336Sgblack@eecs.umich.edu /* 3226336Sgblack@eecs.umich.edu * The priority queue to store RecvInfo items ordered by receive ticks. 3236336Sgblack@eecs.umich.edu */ 3246336Sgblack@eecs.umich.edu RecvQueue recvQueue; 3256336Sgblack@eecs.umich.edu /** 3266336Sgblack@eecs.umich.edu * The singleton Sync object to perform multi synchronisation. 3276336Sgblack@eecs.umich.edu */ 3286336Sgblack@eecs.umich.edu static Sync *sync; 3296336Sgblack@eecs.umich.edu /** 3306336Sgblack@eecs.umich.edu * The singleton SyncEvent object to schedule periodic multi sync. 3316336Sgblack@eecs.umich.edu */ 3326336Sgblack@eecs.umich.edu static SyncEvent *syncEvent; 3336336Sgblack@eecs.umich.edu /** 3346336Sgblack@eecs.umich.edu * Tick to schedule the first multi sync event. 3356336Sgblack@eecs.umich.edu * This is just as optimization : we do not need any multi sync 3366336Sgblack@eecs.umich.edu * event until the simulated NIC is brought up by the OS. 3376336Sgblack@eecs.umich.edu */ 3386336Sgblack@eecs.umich.edu Tick syncStart; 3396336Sgblack@eecs.umich.edu /** 3406336Sgblack@eecs.umich.edu * Frequency of multi sync events in ticks. 3416336Sgblack@eecs.umich.edu */ 3426336Sgblack@eecs.umich.edu Tick syncRepeat; 3436336Sgblack@eecs.umich.edu /** 3446336Sgblack@eecs.umich.edu * Receiver thread pointer. 3456336Sgblack@eecs.umich.edu * Each MultiIface object must have exactly one receiver thread. 3466336Sgblack@eecs.umich.edu */ 3476336Sgblack@eecs.umich.edu std::thread *recvThread; 3486336Sgblack@eecs.umich.edu /** 3496336Sgblack@eecs.umich.edu * The event manager associated with the MultiIface object. 3506336Sgblack@eecs.umich.edu */ 3516336Sgblack@eecs.umich.edu EventManager *eventManager; 3526336Sgblack@eecs.umich.edu 3536336Sgblack@eecs.umich.edu /** 3546336Sgblack@eecs.umich.edu * The receive done event for the simulated Ethernet link. 3556336Sgblack@eecs.umich.edu * It is scheduled by the receiver thread for each incoming data 3566336Sgblack@eecs.umich.edu * packet. 3576336Sgblack@eecs.umich.edu */ 3586336Sgblack@eecs.umich.edu Event *recvDone; 3596336Sgblack@eecs.umich.edu 3606336Sgblack@eecs.umich.edu /** 3616336Sgblack@eecs.umich.edu * The packet that belongs to the currently scheduled recvDone event. 3626336Sgblack@eecs.umich.edu */ 3636336Sgblack@eecs.umich.edu EthPacketPtr scheduledRecvPacket; 3646336Sgblack@eecs.umich.edu 3656336Sgblack@eecs.umich.edu /** 3666336Sgblack@eecs.umich.edu * The link delay in ticks for the simulated Ethernet link. 3676336Sgblack@eecs.umich.edu */ 3686336Sgblack@eecs.umich.edu Tick linkDelay; 3696336Sgblack@eecs.umich.edu 3706336Sgblack@eecs.umich.edu /** 3716336Sgblack@eecs.umich.edu * The rank of this process among the gem5 peers. 3726336Sgblack@eecs.umich.edu */ 3736336Sgblack@eecs.umich.edu unsigned rank; 3746336Sgblack@eecs.umich.edu /** 3756336Sgblack@eecs.umich.edu * Total number of receiver threads (in this gem5 process). 3766336Sgblack@eecs.umich.edu * During the simulation it should be constant and equal to the 3776336Sgblack@eecs.umich.edu * number of MultiIface objects (i.e. simulated Ethernet 3786336Sgblack@eecs.umich.edu * links). 3796336Sgblack@eecs.umich.edu */ 3806336Sgblack@eecs.umich.edu static unsigned recvThreadsNum; 3816336Sgblack@eecs.umich.edu /** 3826336Sgblack@eecs.umich.edu * The very first MultiIface object created becomes the master. We need 3836336Sgblack@eecs.umich.edu * a master to co-ordinate the global synchronisation. 3846336Sgblack@eecs.umich.edu */ 3856336Sgblack@eecs.umich.edu static MultiIface *master; 3866336Sgblack@eecs.umich.edu 3876336Sgblack@eecs.umich.edu protected: 3889376Sgblack@eecs.umich.edu /** 3899376Sgblack@eecs.umich.edu * Low level generic send routine. 3906336Sgblack@eecs.umich.edu * @param buf buffer that holds the data to send out 3916336Sgblack@eecs.umich.edu * @param length number of bytes to send 3926336Sgblack@eecs.umich.edu * @param dest_addr address of the target (simulated NIC). This may be 3936336Sgblack@eecs.umich.edu * used by a subclass for optimization (e.g. optimize broadcast) 3946336Sgblack@eecs.umich.edu */ 3956336Sgblack@eecs.umich.edu virtual void sendRaw(void *buf, 3966336Sgblack@eecs.umich.edu unsigned length, 3976336Sgblack@eecs.umich.edu const MultiHeaderPkt::AddressType dest_addr) = 0; 39810905Sandreas.sandberg@arm.com /** 3996336Sgblack@eecs.umich.edu * Low level generic receive routine. 4006336Sgblack@eecs.umich.edu * @param buf the buffer to store the incoming message 4016336Sgblack@eecs.umich.edu * @param length buffer size (in bytes) 4026336Sgblack@eecs.umich.edu */ 4036336Sgblack@eecs.umich.edu virtual bool recvRaw(void *buf, unsigned length) = 0; 40410905Sandreas.sandberg@arm.com /** 4056336Sgblack@eecs.umich.edu * Low level request for synchronisation among gem5 processes. Only one 4066336Sgblack@eecs.umich.edu * MultiIface object needs to call this (in each gem5 process) to trigger 4077533Ssteve.reinhardt@amd.com * a multi sync. 4087533Ssteve.reinhardt@amd.com * 4097533Ssteve.reinhardt@amd.com * @param sync_req Sync request command. 4107533Ssteve.reinhardt@amd.com * @param sync_tick The tick when sync is expected to happen in the sender. 4119376Sgblack@eecs.umich.edu */ 4129376Sgblack@eecs.umich.edu virtual void syncRaw(MsgType sync_req, Tick sync_tick) = 0; 4136313Sgblack@eecs.umich.edu /** 4146313Sgblack@eecs.umich.edu * The function executed by a receiver thread. 4159461Snilay@cs.wisc.edu */ 4169461Snilay@cs.wisc.edu void recvThreadFunc(); 4179461Snilay@cs.wisc.edu /** 4189461Snilay@cs.wisc.edu * Receive a multi header packet. Called by the receiver thread. 4199461Snilay@cs.wisc.edu * @param header the structure to store the incoming header packet. 4209461Snilay@cs.wisc.edu * @return false if any error occured during the receive, true otherwise 4216313Sgblack@eecs.umich.edu * 4229384SAndreas.Sandberg@arm.com * A header packet can carry a control command (e.g. 'barrier leave') or 4239384SAndreas.Sandberg@arm.com * information about a data packet that is following the header packet 4249384SAndreas.Sandberg@arm.com * back to back. 4259384SAndreas.Sandberg@arm.com */ 4269384SAndreas.Sandberg@arm.com bool recvHeader(MultiHeaderPkt::Header &header); 4279384SAndreas.Sandberg@arm.com /** 428 * Receive a data packet. Called by the receiver thread. 429 * @param data_header The packet descriptor for the expected incoming data 430 * packet. 431 */ 432 void recvData(const MultiHeaderPkt::Header &data_header); 433 434 public: 435 436 /** 437 * ctor 438 * @param multi_rank Rank of this gem5 process within the multi run 439 * @param sync_start Start tick for multi synchronisation 440 * @param sync_repeat Frequency for multi synchronisation 441 * @param em The event manager associated with the simulated Ethernet link 442 */ 443 MultiIface(unsigned multi_rank, 444 Tick sync_start, 445 Tick sync_repeat, 446 EventManager *em); 447 448 virtual ~MultiIface(); 449 /** 450 * Send out an Ethernet packet. 451 * @param pkt The Ethernet packet to send. 452 * @param send_delay The delay in ticks for the send completion event. 453 */ 454 void packetOut(EthPacketPtr pkt, Tick send_delay); 455 /** 456 * Fetch the next packet from the receive queue. 457 */ 458 EthPacketPtr packetIn(); 459 460 /** 461 * spawn the receiver thread. 462 * @param recv_done The receive done event associated with the simulated 463 * Ethernet link. 464 * @param link_delay The link delay for the simulated Ethernet link. 465 */ 466 void spawnRecvThread(Event *recv_done, 467 Tick link_delay); 468 /** 469 * Initialize the random number generator with a different seed in each 470 * peer gem5 process. 471 */ 472 void initRandom(); 473 474 DrainState drain() M5_ATTR_OVERRIDE; 475 476 /** 477 * Callback when draining is complete. 478 */ 479 void drainDone(); 480 481 /** 482 * Initialize the periodic synchronisation among peer gem5 processes. 483 */ 484 void startPeriodicSync(); 485 486 void serialize(const std::string &base, CheckpointOut &cp) const; 487 void unserialize(const std::string &base, CheckpointIn &cp); 488 489}; 490 491 492#endif 493