Catching up a stale snapshot without blocking

The order gateway needs a client's confirmed balance, and the source of truth for it is the account service, a separate machine. The gateway only asks once, when the client logs in, so the account service never sits on the order path (moving work off the hot path). What comes back is a snapshot that's already a little stale by the time it lands. Instead of asking again, the gateway keeps a rolling buffer of every trade report it has processed itself, and fast-forwards the snapshot to the present by replaying the updates it already had.

A rolling buffer of processed updates

ClientStatesWriter keeps UpdatesBuffer, a fixed-size ring indexed by SequenceId & (Size - 1), and a LastSeq marking the newest one it's seen:

LastSeq = report->SequenceId;
UpdatesBuffer[report->SequenceId & (GatewayUpdateBufferSize - 1)] = *report;

This is populated continuously, independent of anything to do with the account service. By the time a snapshot shows up, the gateway may already have processed dozens of updates the snapshot doesn't reflect.

Replaying the gap

The snapshot itself carries a sequence id too, marking exactly which update it was accurate as of. That one fact is what makes the replay possible: the gap between "what the snapshot knew" and "what the gateway knows now" is just the range (response.SequenceId, LastSeq], and every update in that range is sitting in the ring buffer, ready to be replayed:

uint64_t seqId = response.SequenceId + 1;

while (seqId <= LastSeq)
{
    OrderStateReport &curReport = UpdatesBuffer[seqId++ & (GatewayUpdateBufferSize - 1)];

    if (curReport.ClientId == response.ClientId)
    {
        for (size_t i = 0; i < MaxPositions; ++i)
        {
            if (response.AssetId[i] == curReport.BoughtAssetId)
            {
                response.Confirmed[i] += curReport.BoughtDelta;
            }
            else if (response.AssetId[i] == curReport.SoldAssetId)
            {
                response.Confirmed[i] += curReport.SoldDelta;
                response.Attempt[i] += curReport.SoldDelta;
            }
        }
    }
}

The snapshot walks forward through every update it missed, applying each one's delta, until it lands exactly on LastSeq, which is by definition current. It takes no second request, and doesn't depend on the account service having caught up itself.

Why a sequence id and not a timestamp

The reason a sequence id, not a timestamp, is what the snapshot carries is that the replay is a discrete walk: response.SequenceId + 1, +2, and so on, each step landing on exactly one buffered update, no more and no less. There's no equivalent operation on timestamps, because "the update that happened right after this one" isn't a question a clock can answer, clocks don't have a notion of adjacency, only of ordering, and even the ordering is only as reliable as two machines' clocks agree. A sequence id is a count, and counts support "the next one" as a first-class operation.

The bounded window is deliberate

If the snapshot is older than GatewayUpdateBufferSize updates, the update it would need to start its replay from has already been overwritten by a newer one at the same slot. The code has to at least check for that:

if (UpdatesBuffer[response.SequenceId & (GatewayUpdateBufferSize - 1)].SequenceId != response.SequenceId
    && response.SequenceId != LastSeq) [[unlikely]]
{
    // stale beyond the replay window: this path needs an explicit resync,
    // not a silent replay from data that's no longer there
}

That's the real trade being made: a bounded buffer means bounded memory and a replay that's always a fixed amount of work, at the cost of an upper limit on how stale a snapshot is allowed to be before it needs a real resync instead of a replay. Unbounded history would remove that limit, at the cost of unbounded memory on a thread that has to keep pace with the report stream.

The same number, doing two other jobs

The replay is one of three things the sequence number ends up carrying, and the other two are why it was worth stamping on every report in the first place.

Trade reports reach the gateway over multicast, which can deliver packets out of order. So before any of the above happens, the receiver drops each report into a ring slot chosen by seq & (size - 1), the same indexing as UpdatesBuffer, and only releases reports downstream in strict sequence. The thread applying them to client state never sees one early, which is what lets LastSeq mean "everything up to here, with no holes". The bounded window above is the third job: the same number that orders the replay also tells the gateway when a snapshot is too old to replay at all.

A timestamp could do none of the three, for the same reason as above: all of them depend on knowing which report comes next. Recovering packets that were dropped outright, from a dedicated retransmission service, is the part still being built.