The Keyed Worker Pool: Parallel Kafka Consumers That Keep Order
Kafka orders records inside a partition. The moment you hand a batch to an executor, that guarantee is gone. Four interactive simulations of the machinery that gets parallelism back without losing order: mailbox dispatch, the low-watermark commit, pause-based backpressure, and the rebalance gap no broker setting can close.
Kafka orders records inside a partition. The moment you hand a batch to an executor, that guarantee is gone — and the usual fix, adding partitions, buys parallelism in units nobody wants to manage. Ordering is only ever guaranteed within a partition consumed by a single thread.
Everything below keeps that invariant while running more than one thread: mailbox dispatch, the low-watermark commit, pause-based backpressure, and the rebalance gap no broker setting can close. Each one is running live. Drive the controls and watch where it holds, and where it quietly stops holding.
Dispatch: shared pool versus keyed mailboxes
Both modes poll the same partition, in offset order, with identical per-record latencies. Only the routing rule differs. A shared pool lets two records for the same key run concurrently; mailboxes never do.
IN KAFKA: the difference between concurrency=N capped by partition count and a dispatcher that decouples parallelism from partitions entirely
Partition 0 — log
Mailboxes
Worker threads
Completion order, per key
At most one thread drains a mailbox at a time. Order breaks stay at zero by construction.
The commit watermark
Records now complete out of offset order, so the consumer can no longer commit what it just finished. Tap cells to complete them in any order, then crash the consumer and see what survives.
IN KAFKA: the committed offset is a promise that everything below it is durably handled — the only safe value is the lowest offset still in flight
In flight — tap to complete
Leave one early offset incomplete and complete everything after it. The watermark pins — that is correct, and it is why a permanently failing key has to be quarantined on a timer rather than retried forever.
Backpressure: pause, do not queue
Dispatch is non-blocking, so poll() will pull faster than the pool drains. The only bounded answer is to keep polling for heartbeats while refusing to fetch. Push fetch above drain and watch each mode find its ceiling.
IN KAFKA: pause() and resume() on assigned partitions — a paused partition returns no records but still satisfies max.poll.interval.ms
In-flight depth over time
High mark 24, low mark 8, heap ceiling 72. The consumer keeps polling while paused, so the session never expires.
Rebalance, and the write guard behind it
On revoke you stop dispatching, wait for in-flight work, and commit. If the wait times out, work from the old owner lands after the new owner has started the same offsets. Shorten the drain window until records leak, then turn the guard off.
IN KAFKA: onPartitionsRevoked and the bounded drain — the one window where two consumers hold the same offsets at once
Consumer A — revoked
Consumer B — new owner
Write log — fund NAV row
A drain of 7 ticks outlasts every in-flight record. Drop it to 2 and three writes leak past the handoff.
The common thread: every mechanism here converts an ordering problem into a bookkeeping problem. Mailboxes track which key is busy, the watermark tracks the lowest thing unfinished, pause tracks how much is outstanding, and the version guard tracks what the database has already seen. None of them ask the broker for a stronger guarantee than a partition can give.