Reconcile Lab
Two writers reach for one object. The store checks the version each write was read at and refuses the one that lost, and what the loop does next is the only thing this page changes.
Every write to a Kubernetes object carries the resourceVersion it was read at, and the API server refuses any write whose version is no longer current. A controller therefore has to decide what to do about a refusal, and there are only three answers. Retry at once, which is back on target fastest when it wins the next race and spends the most attempts when it does not. Retry with backoff, which concedes the object for a moment — and conceding is exactly what lets the other writer finish. Or stand down, which takes no write events at all and waits for the periodic sweep. Run the same contention through each and they charge visibly different costs.
What this demonstrates
The reconcile semantics themselves, end to end and in this process: the real Kubernetes API types from k8s.io/api, the real resourceVersion rule, the real conflict error that apimachinery defines, an object store that enforces the version on every write, and a loop that has to decide what to do when one comes back refused. Every conflict on this page arises from two writers racing; nothing here schedules one.
Cluster operation — scheduling, nodes, rollouts, everything an operator does at three in the morning — is a separate subject that takes separate evidence.
Where the work happens
The simulation runs on the server, in Go, and the figures reach this page over a server-sent event stream ten times a second. A session starts real goroutines on a real box and holds them, so the server admits 4 simulations at a time, one per address, and runs no more than 32 reconcile workers across all of them.
Every control below has a hard ceiling that is applied in the simulation, not in the browser: at most 200 objects, 250 competing writes a second, 8 concurrent reconciles, a sweep between 100 and 30000 ms, and a session no longer than 90 seconds. Ask for more and the server clamps it and tells you what it changed — the numbers on this page are always the numbers that ran.
Two figures are fixed so the rest can be read. One store call costs 2 ms of simulated time, paid by whoever makes it, so a refused write costs exactly what an accepted one does. And a pass gives up after 8 attempts, because a real controller hands the key back to its queue rather than holding a worker on one object forever. Backing off waits 5 ms and doubles, to a ceiling of 500 ms. Every object is driven toward 3 replicas; the competing writer only ever writes something else.
Idle. Pick a policy and press Run it.
Objects off target over the session. Flat at the top is a loop that never catches up.
- Write attempts
- 0
- Writes landed
- 0
- Reconcile passes
- 0
- Competing writes achieved
- —
- Times off target
- 0
- Restored
- 0
- p50 back on target
- —
- p99 back on target
- —
Times back on target are measured from the competing write that moved an object off target to the write that brought it back, and are read off a fixed bucket ladder — so each figure is the top of the bucket its percentile falls in and never reads lower than the truth.
What to try
- Leave the defaults and run it three times, changing only the policy. The same contention meets three answers, and the refused-write counts separate within seconds.
- Now set objects to 1 and run retry at once against retry with backoff. Both get slower and both waste far more, because every competing write now lands on the object you are reconciling — but retrying at once stays the quicker way back, and buys it with roughly three refused writes in four.
- Run stand down and watch the objects stay off target between sweeps. It takes no write events at all, so it can only be as fresh as its sweep interval — that is the difference between reacting to events and reading the world on a timer.
- Raise objects to 32 and leave everything else. The same competing writes spread over more objects, each one is contended less, and all three policies start to look alike — which is why a controller that never meets a conflict tells you nothing about the one you chose.
One result did not survive the move from the test bench to this page, and it is worth knowing why. The same simulation runs under a virtual clock in its own test suite, where every sleep is exact; there, at one object, retrying at once becomes the slower way back on target and backing off overtakes it, because two reconciles running to the microsecond can settle into a rhythm where they keep colliding. On this page, running against a real clock, that never happens: timer overshoot and ordinary scheduling jitter break the rhythm up on their own, and do for free what backing off pays for. The figures above are the real-clock ones.