When a descriptor changes mid-send

Every service in Naoto registers itself in etcd under a 10-second lease, and the gateway watches those keys. When a matching engine appears the gateway connects to it, and when its lease expires the connection is dropped and orders for that asset are rejected as a technical failure instead of hanging. That part is fairly standard. What took more thought is a thread that's in the middle of using a connection when the connection changes under it.

The descriptor a thread is still holding

The risk thread, which also sends each accepted order on to its engine, holds a matching engine's file descriptor while it sends. Meanwhile the etcd watcher, on another thread, can replace that connection and close the old descriptor, and the kernel is free to hand the same number to the next socket it opens, which could be a client's. A plain fd is just an integer, so nothing would notice an order going somewhere it shouldn't.

A lock around the send would fix it, and put the watcher and the risk thread on the same lock for every single order, to protect against something that almost never happens. So the fix had to cost the risk thread close to nothing in the common case.

A generation next to the descriptor

Each service connection is stored as a VersionedFd: the descriptor in the low 32 bits and a generation counter in the high 32, packed into one 64-bit atomic. The watcher is the only thread that switches it, and every switch bumps the generation:

void SwitchFd(int32_t fd) noexcept
{
    uint64_t old = Packed.load(std::memory_order_relaxed);
    uint32_t newGen = (old >> 32) + 1;
    Packed.store(pack(fd, newGen), std::memory_order_release);
}

Because the fd and the generation live in the same word, a reader can never see a new fd with an old generation or the other way around. One load gives it a consistent pair.

The risk thread loads the value once per order, sends on the fd inside it, then loads it again. If anything changed during the send it can't vouch for where those bytes went, so the order isn't confirmed to the client:

VersionedFd &curSlot = MatchingEngines[curOrder.AssetId];

uint64_t newVal = curSlot.load(std::memory_order_acquire);

if (newVal != curVal) [[unlikely]]
{
    return false;
}

That costs the hot path two atomic loads and shares no lock with the watcher.

Detection is half of it

Detecting the race means a misrouted order is never reported as accepted, but the bytes could still have gone out on a recycled descriptor. The other half is making the recycle impossible in the first place: the watcher closes the old descriptor only once the risk thread has observed the new generation, so the number can't be handed out again while a send might still be using it. That ordering is the next change on this path.

Reads have the same problem

The same value protects the other direction. When the gateway's connection to the account service changes, the thread receiving from it may be holding half a message from the old connection. It compares the generation it last read against the current one, and if they differ it throws the partial buffer away instead of stitching stale bytes onto the start of the new stream.

Client slots, one level up

Client connections have a version of this too. Client state is indexed directly by socket fd, which keeps the lookup to one array index, but it means a new client can land on the fd a previous one just released. The risk thread keeps its in-flight reservations locally, per fd, and a new session must never inherit the old one's.

So every snapshot the gateway installs for a slot bumps a session id, and the risk check compares it against the one it last saw before doing anything else:

if (curState.SessionId != localSessionId) [[unlikely]]
{
    localAttempt.Clear();
    localSessionId = curState.SessionId;
}

It's the same idea as the generation on a service connection, a counter next to a reused number so a thread can tell the number now means something else, just applied to clients instead of services.