Any instance, the same answer

Three backend instances behind nginx, each with its own caches. One rule and three mechanisms keep any of them from answering out of stale memory.

Project
STP: Vehicle Cost Platform, part 8 of 10
Published
October 5, 2026
Reading time
2 min read

Scaling a stateless service is easy: start more copies. Most services aren't stateless, though, because they keep caches, and a cache on one copy doesn't know what another copy just wrote. In STP, scaling the backend meant running several instances behind nginx, in round robin, each holding caches and graphs in memory. So the question was never whether we could start three, but whether any of them could answer from stale memory.

One rule for in-memory state

The rule we wrote down, for every developer and every coding agent working on the project: any state kept in memory must stay correct when another instance writes. A new cache, a new background task or any new shared state is checked against the stack with three instances before it ships:

STP_MULTI=1 pytest tests/stack

Three ways to stay right

Each piece of in-memory state uses one of three mechanisms.

  • A revision the caller sends. A draft's graph is identified by its revision, and the UI sends the revision it displays. An instance that holds revision 4 can't be asked about revision 5 and answer with 4: it either knows the requested revision or loads it.
  • An event that makes others forget. When a draft is saved, a bom-revised event on Kafka tells every other instance to drop what it holds for that BOM.
  • Reread from the database. Some data is too cheap to cache. The market price behind the margin is one row, so it's read from PostgreSQL on every request and grafted onto the cached result. Edit locks live in PostgreSQL for the same reason, so every instance sees the same holder.
a revisionthe caller sendsa Kafka eventthat makes others forgetPostgreSQLread again, not cacheda draft's graphUI sends the rev it showsresults computed for a BOMbom-revised: drop themsupplier prices, ratesoffer and rate eventsreconciled every minutemarket price, for the marginone row, every requestwho holds the edit locklives there, never cached
Each piece of in-memory state, and the mechanism that keeps it right when another instance writes.

The third one was a choice between options. We considered a market-price-updated event and turned it down: a topic, a consumer and their telemetry to save the read of one row isn't a good trade.

Never answered on an old revision

That's the property that matters most, so it was measured on its own. Over 10 autosaves with 3 instances, no request was ever answered on the old revision, and every instance refused it between 18 and 84 ms after the save (median 33 ms). That interval is only the time round robin takes to reach each instance, since the eviction itself had already happened by the first request.

No instance answers on an old revision: the one that saved knows the new one, the others forget theirs, and a request for the old one is refused. Measured over 10 autosaves with 3 instances, every instance refused the old revision within 18 to 84 ms of the save.

What three instances bought

On the demo car, with 12 clients in parallel on one machine, going from one instance to three took full computations from 46 to 48 per second up to 65 to 85, and reads from cache from 90 to 95 per second up to 165 to 172. Median latency roughly halved in both cases.

050100150200requests per secondfull computations1 instance46–483 instances65–85reads from cache1 instance90–953 instances165–172
Requests per second on the demo car, 1 instance against 3, on one shared machine. The darker part of each bar is the spread between two trials.

It's not ×3 because the client, nginx, Docker and three instances share the same machine, and each instance pays once for loading the graph. Those explanations are hypotheses, and the performance report says so.

Each instance also keeps its own caches, so a BOM everyone reads ends up computed once per instance. Routing by BOM id (consistent hashing in nginx) would keep the hit rate of a single instance. We didn't build it, because adding an instance already takes its share of the load, which is what the system needed.