Checking risk against a balance that's always behind

Risk checking against a client's real-time balance has a gap built into it: there is no real-time balance to check against. Every update has to be received, processed, and then made visible to the thread doing the risk check, and each of those steps takes time.

That gap never fully closes, because it comes from having separate threads at all. The risk check has to stay correct anyway, even while the state it's reading is behind what actually happened.

Why waiting for confirmation doesn't work

The tempting fix is to just wait: hold an order until the state catches up, then check it against numbers you know are current. That breaks down under real trading. In the time it takes one update to arrive, several more orders can land from the same client, and if none of them are checked against anything, several invalid trades can get accepted before the first update ever shows up to say otherwise.

The only design that survives that is a pessimistic one: an order only gets accepted if it can be proven valid right now, with whatever information is available right now.

Freezing funds immediately

Proving an order valid right now means reserving the funds for it right now, before the matching engine has said anything back.

That reservation is a plain check against two numbers:

int64_t confirmed = curState.GetConfirmedAt(assetIdx);
int64_t attempt =
    curState.GetAttemptAt(assetIdx) + localAttempt[assetIdx];

int64_t amount = order.Side == OrderSide::SELL
    ? order.Amount
    : order.Amount * order.Price;

if (amount > confirmed - attempt) [[unlikely]]
{
    return OrderConfirmationStatus::InsufficientFunds;
}

localAttempt[assetIdx] += amount;

confirmed is the client's actual balance, last known. attempt is everything already reserved against it, including this thread's own not-yet-visible reservations. An order is only accepted if what's left, confirmed - attempt, covers it, and the moment it's accepted, that amount gets added to the reservation immediately. The next order checked, a moment later, sees the updated reservation and can't spend the same funds twice.

Where the reservation lives

The obvious place for the reservation is the client's shared state, and that's the one place it can't go. Client states are triple-buffered so that a single writer thread can publish them without the reader ever blocking, and that only works while the risk thread never writes to them. So the reservation goes into localAttempt, a plain array that belongs to the risk thread alone:

struct LocalAttempts
{
    std::array<int64_t, MaxPositions> Attempt;

    [[nodiscard]] int64_t &operator[](const uint16_t assetId) noexcept
    {
        return Attempt[assetId];
    }

    void Clear(void) noexcept
    {
        Attempt.fill(0);
    }
};

No atomics, no cross-thread write, because nothing outside this thread ever reads or writes it. The risk check folds it into every decision (GetAttemptAt(assetIdx) + localAttempt[assetIdx]), so as far as that thread is concerned its own reservations are always up to date, without a round trip through the shared state.

That only works if the shared state doesn't end up double-counting the same reservation once it does catch up. When the writer thread processes a trade report confirming an order was added to the book, it deliberately skips updating the shared reservation for that case:

States.SetClientAssets(clientFd, report->SoldDelta,
                       report->State == OrderState::ADD
                           ? 0 // Ignore add updates because
                               // they're already taken into
                               // account (local counter)
                           : report->SoldAttemptDelta,
                       report->SoldAssetId);

An ADD update means the order the local counter already reserved for just got confirmed, nothing new happened as far as risk is concerned, so the shared reservation delta for that update is forced to zero. Any other update, a fill or a cancel, is real news the local counter never knew about, and does get applied.

What it relies on

The check never has to trust that updates arrive quickly, or before the next order shows up. It only needs whatever it approved to count against the balance the moment it approves it, and a single thread can guarantee that to itself without asking anyone else. The shared state can lag as much as it likes, because the number that has to be current never depended on it.

That guarantee has one precondition outside the gateway: nobody else can be spending the same balance. A client is only ever authorized on one gateway, which is what makes this local view complete (one gateway per client).

How far behind it is

Lag never makes the check wrong, but it does cost the client something: funds that a fill or a cancel releases can't be spent until the risk thread sees them, and an order sent in that window gets rejected for funds the client actually has. So I measured the lag, from a trade report entering the writer's queue to the risk thread reading the new balance, through the gateway's real writer loop, with a report arriving every 20 µs on a fixed schedule, on the Ryzen 5 4600H laptop from the project page.

With the threads pinned as the gateway pins them today, the balance is 1.7 ms behind at P50 and 6.0 ms at P99. The writer shares CPU 10 with the account receiver and the ack sender, which never sleep either, and the scheduler hands each of the three a slice of about 2.8 ms, so a report that arrives while the writer is off the CPU waits up to two of the others' slices (where the threads sit).

Underneath that, with the writer alone on its CPU, the lag is 0.5 to 0.7 µs at P50 and under 1 µs at P99, and the rest is distance. The risk thread is on the other half of the CPU from the writer, each half with its own L3, and moving the writer and the trade report receiver to the risk thread's half brings the lag to 0.25 µs at P50 and 0.4 µs at P99.