One writer for every piece of state

Most of the locks a system like Naoto would normally need exist because two things can write the same data. So rather than making concurrent writes cheap, the design makes sure they don't happen. That's the single-writer principle, usually applied to threads inside one process. Naoto applies it across services as well.

Naoto isn't one program. It's several services (order gateways, one matching engine per asset, and an account service) that can run on different machines, and together they're what I'll call the cluster. So two things writing the same data can mean different things depending on where you look:

• cluster: two services, like two gateways both spending one client's balance.
• process: two threads inside the same service, both touching the same data.
• thread: data that only one thread ever touches, so the question doesn't come up at all.

Below, each piece of state sits beside its only writer.

One engine per asset

An asset's order book belongs to exactly one matching engine, registered in etcd under that asset's id, and inside the engine to one thread. The book needs no locks, and ordering within an asset needs no protocol, since it's simply the order that thread processed things in. The engine is also the only source of trade reports, and every balance elsewhere in the cluster is a copy built from them (one stream, many copies).

The cost is that a single asset can't be spread over several machines. For an exchange that's the right trade, since an order book's operations depend on each other in strict order anyway, and parallelism comes from having many assets rather than from splitting one.

One gateway per client

A client is authorized on exactly one gateway, and the account service refuses the login anywhere else. That small rule removes a whole protocol. If two gateways could spend the same balance, they'd have to agree on reservations before accepting anything, which is a distributed lock on every order. With one gateway per client, that gateway already knows every reservation that exists for the client because it made all of them, so its local check is complete rather than an approximation.

This is what makes the risk check possible at all. It runs against a copy of the client's state that is always slightly behind, and it can still be pessimistic and correct, because nobody outside the gateway can have spent anything the gateway doesn't know about (checking risk against a balance that's always behind goes through that part).

One writer inside the gateway

Inside the gateway the same idea repeats. One thread writes client state while the risk thread reads it, which is what makes triple buffering work: with a single writer, publishing a new version is one atomic store, and there's no second writer to race with. The risk thread keeps its in-flight reservations in memory no other thread touches, so those need no synchronization either (a thread-local counter).

Threads hand work to each other under the same rule. Every queue between two threads in the services has exactly one producer and one consumer, so every one of them is a single-producer single-consumer queue: a fixed ring of slots where the producer only ever advances the tail and the consumer only ever advances the head, each on its own cache line. None of the services takes a lock anywhere.

Order ids without an id service

Even order ids follow the rule. Each gateway puts its own id in the top 16 bits and a local counter below, so the id space is split up front and each gateway owns its slice.

Ids are unique across the cluster without an id service, a round trip, or any coordination, and generating one is an increment.

What it costs

Single ownership moves the hard part to the moments ownership changes. A matching engine can disappear and be replaced, a client can reconnect and land on a different socket, and in each case something that was the only writer stops being one while another thread may still be using it. Handling those transitions is its own problem, covered in when a descriptor changes mid-send. But those are rare events, and paying for them there is much cheaper than paying a lock on every order.