Catching up a stale snapshot without blocking

A snapshot fetched from another machine is out of date by the time it arrives, because updates kept happening while it was in flight. Market data feeds handle this with snapshot-plus-incremental recovery: buffer the incremental updates, then apply only the ones newer than the snapshot's sequence number.

In Naoto the order gateway needs a client's confirmed balance, and the source of truth for it is the account service, a separate machine. The gateway only asks once, when the client logs in, so the account service never sits on the order path (moving work off the hot path). It uses the same recovery, with one difference: it already processes every trade report itself, so instead of asking again, it keeps the recent ones in a rolling buffer and fast-forwards the snapshot by replaying the updates it already had.

A rolling buffer of processed updates

ClientStatesWriter keeps UpdatesBuffer, a fixed-size ring indexed by SequenceId & (Size - 1), and a LastSeq marking the newest one it's seen:

LastSeq = report->SequenceId;
UpdatesBuffer[report->SequenceId & (GatewayUpdateBufferSize - 1)] = *report;

This is populated continuously, independent of anything to do with the account service. By the time a snapshot shows up, the gateway may already have processed dozens of updates the snapshot doesn't reflect.

Replaying the gap

The snapshot itself carries a sequence id too, marking exactly which update it was accurate as of. That one fact is what makes the replay possible: the gap between "what the snapshot knew" and "what the gateway knows now" is just the range (response.SequenceId, LastSeq], and every update in that range is sitting in the ring buffer, ready to be replayed:

uint64_t seqId = response.SequenceId + 1;

while (seqId <= LastSeq)
{
    OrderStateReport &curReport = UpdatesBuffer[seqId++ & (GatewayUpdateBufferSize - 1)];

    if (curReport.ClientId == response.ClientId)
    {
        for (size_t i = 0; i < MaxPositions; ++i)
        {
            if (response.AssetId[i] == curReport.BoughtAssetId)
            {
                response.Confirmed[i] += curReport.BoughtDelta;
            }
            else if (response.AssetId[i] == curReport.SoldAssetId)
            {
                response.Confirmed[i] += curReport.SoldDelta;
                response.Attempt[i] += curReport.SoldDelta;
            }
        }
    }
}

The snapshot walks forward through every update it missed, applying each one's delta, until it lands exactly on LastSeq, which is by definition current. It takes no second request, and doesn't depend on the account service having caught up itself.

This only works because the snapshot is tagged with a sequence id rather than a time. The replay is a walk, response.SequenceId + 1, then +2, each step landing on exactly one buffered report. A timestamp can say which of two reports came first, if the two machines' clocks agree, but it can't say which report comes next, and next is the only question the walk asks.

The bounded window is deliberate

If the snapshot is older than GatewayUpdateBufferSize updates, the update it would need to start its replay from has already been overwritten by a newer one at the same slot. The code has to at least check for that:

if (UpdatesBuffer[response.SequenceId & (GatewayUpdateBufferSize - 1)].SequenceId != response.SequenceId
    && response.SequenceId != LastSeq) [[unlikely]]
{
    // stale beyond the replay window: this path needs an explicit resync,
    // not a silent replay from data that's no longer there
}

A bounded buffer means bounded memory and a replay that's never more than a fixed amount of work, and the price is a limit on how stale a snapshot can be before it needs a real resync instead. Unbounded history would remove the limit and put unbounded memory on a thread that has to keep pace with the report stream.

The same number, doing two other jobs

The replay is one of three jobs the sequence number ends up doing.

Trade reports reach the gateway over multicast, which can deliver packets out of order. So before any of the above happens, the receiver drops each report into a ring slot chosen by seq & (size - 1), the same indexing as UpdatesBuffer, and only releases reports downstream in strict sequence. The thread applying them to client state never sees one early, which is what lets LastSeq mean "everything up to here, with no holes". The bounded window above is the third job: the same number that orders the replay also tells the gateway when a snapshot is too old to replay at all.

All three depend on knowing which report comes next. The engine stamps the number on each report as it leaves (one stream, many copies covers that side).