Reliable risk checking on real-time client states

Risk checking against a client's real-time balance has a gap built into it: there is no real-time balance to check against. Every update has to be received, processed, and then made visible to the thread doing the risk check, and each of those steps takes time.

That gap never fully closes, because it comes from having separate threads at all. The risk check has to stay correct anyway, even while the state it's reading is behind what actually happened.

Why waiting for confirmation doesn't work

The tempting fix is to just wait: hold an order until the state catches up, then check it against numbers you know are current. That breaks down under real trading. In the time it takes one update to arrive, several more orders can land from the same client, and if none of them are checked against anything, several invalid trades can get accepted before the first update ever shows up to say otherwise.

The only design that survives that is a pessimistic one: an order only gets accepted if it can be proven valid right now, with whatever information is available right now.

Freezing funds immediately

Proving an order valid right now means reserving the funds for it right now, before the matching engine has said anything back.

That reservation is a plain check against two numbers:

int64_t confirmed = curState.GetConfirmedAt(assetIdx);
int64_t attempt =
    curState.GetAttemptAt(assetIdx) + localAttempt[assetIdx];

int64_t amount = order.Side == OrderSide::SELL
    ? order.Amount
    : order.Amount * order.Price;

if (amount > confirmed - attempt) [[unlikely]]
{
    return OrderConfirmationStatus::InsufficientFunds;
}

localAttempt[assetIdx] += amount;

confirmed is the client's actual balance, last known. attempt is everything already reserved against it, including this thread's own not-yet-visible reservations. An order is only accepted if what's left, confirmed - attempt, covers it, and the moment it's accepted, that amount gets added to the reservation immediately. The next order checked, a moment later, sees the updated reservation and can't spend the same funds twice.

One counter, two threads

That reservation has to live somewhere, and the obvious place, the client's shared state, is off limits. Client states are triple-buffered specifically so one writer thread can publish updates without a reader ever blocking, which only works because writers and readers never touch the same memory. Having the risk thread reach in and mutate that state directly would mean making it safe for two threads to write, and that safety costs both of them speed on every single access.

A thread-local counter instead

The reservation doesn't go into the shared state at all. It goes into localAttempt, a plain array that belongs entirely to the risk thread:

struct LocalAttempts
{
    std::array<int64_t, MaxPositions> Attempt;

    [[nodiscard]] int64_t &operator[](const uint16_t assetId) noexcept
    {
        return Attempt[assetId];
    }

    void Clear(void) noexcept
    {
        Attempt.fill(0);
    }
};

No atomics, no cross-thread write, because nothing outside this thread ever reads or writes it. The risk check folds it into every decision (GetAttemptAt(assetIdx) + localAttempt[assetIdx]), so as far as that thread is concerned, its own reservations are always instantly up to date, they never had to wait for a round trip through the shared state at all.

That only works if the shared state doesn't end up double-counting the same reservation once it does catch up. When the writer thread processes a trade report confirming an order was added to the book, it deliberately skips updating the shared reservation for that case:

States.SetClientAssets(clientFd, report->SoldDelta,
                       report->State == OrderState::ADD
                           ? 0 // Ignore add updates because
                               // they're already taken into
                               // account (local counter)
                           : report->SoldAttemptDelta,
                       report->SoldAssetId);

An ADD update means the order the local counter already reserved for just got confirmed, nothing new happened as far as risk is concerned, so the shared reservation delta for that update is forced to zero. Any other update, a fill or a cancel, is real news the local counter never knew about, and does get applied.

What this buys

The risk check never has to trust that updates arrive quickly, or in order, or at all before the next order shows up. It only has to trust that whatever it approved is reflected the instant it approves it, which is a promise a single thread can make to itself without asking anyone else. Client state staying temporarily behind is fine, because the one number that actually has to be current, this thread's own reservations, never depended on it catching up in the first place.

Trade reports reach the gateway over multicast, where packets can arrive out of order, and they're put back in sequence before anything applies them (the same number, doing two other jobs). Recovering packets that were dropped outright is the part still being built.