When a descriptor changes mid-send

A file descriptor is just an integer, and the kernel reuses them: close one, and the next socket opened can get the same number. So a thread sending on a connection while another thread replaces it can end up writing to a completely different socket, and nothing will notice.

In Naoto connections change at runtime. Every service registers itself in etcd under a 10-second lease, and the gateway watches those keys. When a matching engine appears the gateway connects to it, and when its lease expires the connection is dropped and orders for that asset are rejected as a technical failure instead of hanging.

The descriptor a thread is still holding

The risk thread, which also sends each accepted order on to its engine, holds a matching engine's file descriptor while it sends. Meanwhile the etcd watcher, on another thread, can replace that connection and close the old descriptor, and the same number can come back as the next socket the gateway opens, which could be a client's. An order meant for an engine would go out to that client.

A lock around the send would fix it, and put the watcher and the risk thread on the same lock for every single order, to protect against something that almost never happens. So the fix had to cost the risk thread close to nothing in the common case.

A generation next to the descriptor

Each service connection is stored as a VersionedFd: the descriptor in the low 32 bits and a generation counter in the high 32, packed into one 64-bit atomic. The watcher is the only thread that switches it, and every switch bumps the generation:

void SwitchFd(int32_t fd) noexcept
{
    uint64_t old = Packed.load(std::memory_order_relaxed);
    uint32_t newGen = (old >> 32) + 1;
    Packed.store(pack(fd, newGen), std::memory_order_release);
}

Because the fd and the generation live in the same word, a reader can never see a new fd with an old generation or the other way around. One load gives it a consistent pair.

This is the ABA problem with a descriptor in place of a pointer, and the fix is the one tagged pointers use: a counter that changes even when the value comes back the same.

The risk thread loads the value once per order, sends on the fd inside it, then loads it again. If anything changed during the send it can't vouch for where those bytes went, so the order isn't confirmed to the client:

VersionedFd &curSlot = MatchingEngines[curOrder.AssetId];

uint64_t newVal = curSlot.load(std::memory_order_acquire);

if (newVal != curVal) [[unlikely]]
{
    return false;
}

That costs the hot path two atomic loads and shares no lock with the watcher.

Detection is half of it

Detecting the race means a misrouted order is never reported as accepted, but it doesn't stop the bytes. The watcher closes the old descriptor right after switching to the new one, so its number can be handed out again while a send is still using it, and if the send wins that race the order has already gone out on the recycled descriptor by the time the generation check notices.

Reads have the same problem

The same value protects the other direction. When the gateway's connection to the account service changes, the thread receiving from it may be holding half a message from the old connection. It compares the generation it last read against the current one, and if they differ it throws the partial buffer away instead of stitching stale bytes onto the start of the new stream.

Client slots, one level up

Client connections have a version of this too. Client state is indexed directly by socket fd, which keeps the lookup to one array index, but it means a new client can land on the fd a previous one just released. The risk thread keeps its in-flight reservations locally, per fd, and a new session must never inherit the old one's.

So every snapshot the gateway installs for a slot bumps a session id, and the risk check compares it against the one it last saw before doing anything else:

if (curState.SessionId != localSessionId) [[unlikely]]
{
    localAttempt.Clear();
    localSessionId = curState.SessionId;
}

It's the same idea as the generation on a service connection, a counter next to a reused number so a thread can tell the number now means something else, just applied to clients instead of services.