Kafka Lab
A consumer that spends 20 ms on each message cannot exceed 50 messages a second, however high you set its rate limit. This page is a simulated model of that ceiling — drag the controls and watch a backlog build, then clear it without touching the producer.
This page has two halves, and they are different kinds of thing. The panel below is a model — no broker, no producer, no consumer, no server-side work of any kind, just arithmetic compiled to WebAssembly and run in your browser. The recorded run further down is not: those figures came off an actual Kafka broker in the real lab, and the raw output is committed and linked.
The ceiling
One consumer spending N milliseconds on a message tops out at 1000 ÷ N messages per second. That is arithmetic, not a tuning problem, and no rate limit changes it. The fixes are doing less work per message, or spreading the work across more consumers — which is what partitions are for. The consumer control below is there so you can prove that rather than take it on trust.
Every figure in this panel is simulated — nothing here is measured, and nothing here contacts a broker.
Loading the model…
What to try
- Raise the work per message to 40 ms. The ceiling halves to 25 while the producer keeps asking for 50, and the backlog climbs. The consumed figure sticks at the ceiling — that gap between what was asked for and what is achieved is the whole of backpressure.
- Now raise the rate limit. Drag the producer up and watch the backlog climb faster and the consumed rate not move at all. Turning the producer down would help, but that is not fixing the consumer.
- Put the producer back to 50, then drag Consumers to 3. The ceiling rises clear of the producer, the backlog turns over and it drains to zero. Note what fixed it: the producer is back where it started and the work per message is still where you left it in step 1 — adding consumers is what drained the queue.
- Keep adding consumers. Past 3 the ceiling stops moving. Kafka assigns whole partitions to group members, so a fourth consumer against a 3-partition topic gets nothing assigned and contributes nothing. At that point the fix is more partitions, not more processes.
What the model does and does not include
The model is three inputs and one line of integration: the ceiling is min(consumers, 3) × 1000 ÷ work, and the backlog integrates the difference between the producer rate and that ceiling, floored at empty. The cap is the topic's partition count: Kafka assigns whole partitions to group members, so beyond 3 consumers there is nothing left to assign and an extra one contributes nothing to the ceiling. That cap is why step 4 above stops paying off. There is deliberately no jitter, no random arrival process, no batching and no commit latency. Each of those would be defensible in a queueing model and each would make this page worse at its one job: halving the work should halve the ceiling, exactly, so you can check the arithmetic in your head. Variance invented for realism would be variance the page could not justify.
Measured: a recorded run against a real broker
Size for the tail, not for the fault rate
Sizing on “1% of commits lost ≈ 1% duplicates” is wrong by more than an order of magnitude. In the run below, 56 injected faults produced 1948 double applies — 35 duplicate applications per fault. A shorter sizing run of the same injector gave about 4.5×.
It is a range rather than a constant because the multiplier is broker batch composition. A consumer’s cursor is per partition, so rewinding it to one record replays that record and every later record of that partition already polled. As lag builds the consumer polls larger batches, so each rewind replays more. A crash does not lose one message; it loses everything applied since the last commit.
What was run
Kafka gives at-least-once delivery. The duplicate that costs you is the one arriving after its effect was already applied, because the process died between doing the work and recording that it had. The lab reproduces that with an in-process cursor rewind: the consumer applies a batch, then seeks its own consume cursor back instead of committing, so the effects ran and the offset never moved.
That models one process crashing between apply and commit. It does not model a consumer group rebalance moving a partition to another member, a restart with an empty store, or a second consumer instance with a store of its own. Those are different failures with different answers.
Both arms: 3 partitions, fault
rate 0.01, seed pf-s313, a 50000-key
idempotency store with a 10m window, and a full broker reset between them.
The only difference between the two columns is whether the store was consulted.
| Measured | at-least-once | idempotent |
|---|---|---|
| Records the loop finished | 6773 | 7515 |
| Effects run | 6773 | 5389 |
| Applied a second time | 1948 | 0 |
| Duplicates suppressed | 0 | 2126 |
| Keys faulted | 56 | 59 |
| Store evictions | 0 | 0 |
| Store expiries | 0 | 0 |
The two loss rows are what make the rest readable. The idempotency store is bounded: it holds 50000 keys for 10m, and a key it forgets reads as new when its redelivery arrives, so the effect runs again. Both counters are zero on both arms, which is what says these figures measure delivery rather than the size of the store. A redelivery arriving later than 10m after its first sighting is not caught, and neither is one whose key was pushed out by more than 50000 distinct keys since.
The store lives in one process’s memory. The guarantee does not survive a restart or a partition rebalance — a new process starts with an empty store and every message in flight is new to it. Kafka transactions and exactly-once semantics are out of scope here: they are not implemented, not measured and not approximated.
Every figure above is derived from capture
files committed in the linked repository and embedded in this site byte for
byte — metric scrapes and the fault sets, at commit
2ce565a.
Open the raw
capture directory, which also carries the full consumer logs and the image
digests of the run.
Run the real thing locally
The repository is a Docker Compose stack — a single-broker Kafka in KRaft mode, a Go producer and consumer, a control UI, Prometheus, Grafana and kafka-ui. One command brings it up, and the numbers there are measured rather than modeled.
git clone https://github.com/pigfox/kafka-lab
cd kafka-lab
./run.shTwo things there work differently from the model above, and both are worth seeing. The services take their settings from a compacted Kafka topic rather than from an HTTP call, so the demo configures itself over its own bus and the settings survive a restart with no database. And the achieved rates are read from Prometheus while the backlog is read from the group coordinator — measured quantities, from a broker that is really running.