One stream of trade reports, many copies of every balance

Every balance in Naoto is a copy. The matching engine is the only thing that decides which trades happened, and it says so once, as a stream of trade reports sent over UDP multicast. The account service listens to that stream, and so does every order gateway, and each of them builds its own copy of the balances it cares about from the same reports in the same order.

Exchanges already publish market data this way, as sequenced multicast feeds (Nasdaq's ITCH over MoldUDP64 is the classic example). Naoto uses the same pattern for its own internal state, so a balance is something each service derives from the feed, not something stored in one place and kept in sync.

Why the engine doesn't send to anyone

The straightforward version is a TCP connection from the engine to each service that needs results. Then the engine does one send per consumer for every report, so its cost grows with the number of gateways. It also has to know who is listening, and a consumer that reads slowly fills its socket buffer until the engine's sends start failing or blocking, so the slowest service ends up setting the engine's pace.

With multicast the engine sends each packet once, to a group address, and the network copies it to whoever joined the group. The engine doesn't know how many gateways there are, and a gateway that joins or leaves the cluster costs it nothing. A slow reader only falls behind on its own copy.

This is also what lets the gateway stop asking the account service about every order (moving work off the hot path). Both are reading the same record of what happened, so once the gateway has a starting snapshot it can keep its copy current on its own, and the two only disagree about how far along they are, never about what happened.

Public market data works the same way on a second group: the engine emits book updates there, for anyone who wants to follow prices.

Packing reports into packets

A trade report is a fixed-size packed struct. Without the latency timestamps it's 48 bytes:

#pragma pack(push, 1)
struct OrderStateReport
{
    int64_t BoughtDelta;
    int64_t SoldDelta;
    int64_t SoldAttemptDelta;
    uint32_t SequenceId;
    uint32_t ClientId;
    uint32_t OrderId;
    uint32_t TradeId;
    uint16_t BoughtAssetId;
    uint16_t SoldAssetId;
    OrderState State;
};
#pragma pack(pop)

Because every report is the same size, how many fit in a packet is known at compile time. The emitter derives it from the MTU, minus the IP and UDP headers, and refuses to compile if a single one wouldn't fit:

static constexpr size_t ObjectsPerPacket =
    (MTU - sizeof(rte_ipv4_hdr) - sizeof(rte_udp_hdr)) / sizeof(T);

static_assert(ObjectsPerPacket > 0,
              "T does not fit within MTU - needs mbuf chaining, not "
              "supported here");

With a 1500-byte MTU that's 30 reports per packet. The receiver's side is the same arithmetic backwards: the payload length divided by sizeof(T) is the number of reports, and each one is a memcpy into a pooled object. There's no length prefix and no field to parse, so reading a packet can't go wrong halfway through a message.

Sending without the kernel

The emitter runs on its own core and owns a transmit queue on the network card through DPDK, so a send is a write into that queue instead of a syscall. That means doing the kernel's job for it: the emitter fills in the Ethernet, IP and UDP headers itself, and works out the group's MAC address from its IP (multicast MACs are 01:00:5E followed by the low 23 bits of the address), since there's no kernel networking stack to ask.

It's also where reports get their sequence numbers. The matching thread only pushes report pointers onto a queue; the emitter stamps each one as it packs it:

for (size_t i = 0; i < curStep; ++i)
{
    buffer[totalCopied + i]->SequenceId = SequenceId++;
}

So the numbering costs the matching thread nothing, and it follows exactly the order reports go out on the wire, because one thread does both. A whole batch of packets then leaves in one rte_eth_tx_burst call.

What a lost packet costs

Multicast is UDP underneath, so a packet can arrive late, twice, out of order, or not at all. Every receiver drops packets with bad checksums, puts reports into a ring by sequence number, and only releases them in order (the same number, doing two other jobs).

That ordering decides what a lost report does. A receiver never applies report 42 before it has 41, so a gap stops its copy at the gap instead of letting it go wrong. For a gateway that's the safe direction: the funds it reserved stay reserved until the fill that releases them shows up, so a client can't spend anything twice, but their balance stops moving until the gap is filled. For the account service it means its snapshots fall behind, and a snapshot that's behind is something the gateway already knows how to replay forward.

Filling the gap is the job of a retransmission service that's still being built. It will do nothing but listen to the stream and answer requests for reports a receiver missed, which keeps that work off the engine too.