Consumer Groups and Partition Assignment

Parallel consumption from a partitioned log requires a rule that no two processes read the same partition at the same time. Without it, two consumers would apply the same records to their own views of the world and disagree. With it, the log becomes a set of independent ordered streams, each owned by one member of a group.

The rule has to hold while processes start, stop, crash, and hang. That is a distributed ownership problem, and the coordinator’s answer is a generation counter: every partition assignment belongs to a numbered generation, and a consumer whose generation is stale is not allowed to commit. Ownership is not a lock that gets released; it is a membership epoch that gets superseded.

The generation is the ownership token

A consumer group has a coordinator, one broker that owns the group’s metadata. Members send heartbeats to it. When membership changes — a new member joins, a member stops heartbeating, a member fails to poll within the configured interval — the coordinator increments the generation and starts a rebalance.

During the join phase, every member sends the topics it wants and its subscription metadata. The coordinator picks one member as the group leader and returns the full member list to it. The leader runs the assignment algorithm locally and returns a map from member to partitions; the coordinator distributes the result. From that point, a member may fetch only the partitions in its map, and a commit that carries an older generation is rejected.

Two details make this work under failure. First, assignment is computed by a client, not the broker, so the algorithm can change without upgrading the cluster. Second, the generation check means a consumer that was partitioned away cannot commit offsets for work it did while isolated. That check is what prevents split ownership from corrupting group progress, and it is why the duplicate window at rebalance is bounded by uncommitted work rather than by whatever a disconnected process believes.

Rebalance as a stop-the-world barrier

In the classic eager protocol, every member revokes all of its partitions when a rebalance begins, then waits for a new assignment. No records are processed by anyone during that interval. The cost is proportional to the number of partitions being moved and to the time it takes the slowest member to rejoin.

Eager rebalancing is easy to reason about and expensive at scale. A group with two hundred consumers and a single rolling restart triggers as many rebalances as there are restarts, and each one stops the whole group. The stops are short in a healthy cluster, but the tail is what matters: a member that is slow to rejoin holds the barrier for everyone, and the barrier is a global pause in consumption.

The cooperative protocol changes the shape of the pause. Members keep the partitions they already own and revoke only those that are moving. The rebalance becomes incremental: the leader computes a new assignment, each member compares it with its current set, and revokes only the difference before the next round. Ownership of unmoved partitions never lapses, so processing continues through most of the rebalance. The price is an extra round trip, because the protocol needs a phase in which revocations are acknowledged before the new assignment is applied.

A third option removes the pause for planned changes: static membership. A consumer that declares a stable identity is not removed from the group when it disconnects briefly, so a restart within the session timeout does not trigger a rebalance at all. The assignment is unchanged, the returning member resumes its partitions, and the only visible effect is a short gap in consumption. This trades responsiveness for stability: a crashed member that never returns keeps its partitions until the session timeout expires, so unplanned failure recovery is slower.

What the assignor is optimizing

Kafka’s default strategy list starts with the range assignor, which distributes partitions per topic in contiguous ranges: with three partitions and two consumers, the first gets partitions 0 and 1, the second gets 2. It is simple and deterministic, and it is unbalanced whenever the number of partitions is not a multiple of the number of consumers. Round-robin assignment spreads individual partitions across members in turn and evens the counts, at the cost of giving a consumer partitions from many topics. Sticky variants add a third objective: keep the previous assignment where possible, so that a rebalance moves the minimum number of partitions.

The objectives conflict. A perfectly balanced assignment after every membership change maximizes movement; a maximally stable assignment tolerates imbalance. Sticky assignment resolves this by preferring balance first and stability second, which is why a group that adds a member moves only the partitions needed to even the load.

A range assignment is short enough to state exactly:

def range_assign(partitions: list[int], members: list[str]) -> dict[str, list[int]]:
    """Contiguous ranges per member, as in the classic range assignor."""
    n = len(members)
    per_member, remainder = divmod(len(partitions), n)
    out, start = {}, 0
    for i, member in enumerate(members):
        size = per_member + (1 if i < remainder else 0)
        out[member] = partitions[start:start + size]
        start += size
    return out

With ten partitions and three members the result is four, three, three. With eleven and three it is four, four, three. The imbalance is bounded by one partition, which matters when partitions are few and members are many: two partitions and three consumers leaves one consumer idle no matter which assignor runs.

Parallelism is capped by partition count

The number of active consumers in a group cannot exceed the number of partitions. Additional members join, receive an empty assignment, and sit idle. This is not an implementation limit but a consequence of the ordering guarantee: if two consumers shared a partition, the total order within it would have no single reader to preserve it.

The practical consequence is that partition count is chosen for peak parallelism, not for storage. A topic created with four partitions can never be consumed by more than four members of a group, so a workload that later needs sixteen workers requires a new topic, a migration, or a repartitioning operation that changes key-to-partition mapping and therefore breaks per-key ordering across the change. Overprovisioning partitions costs metadata and file handles and lengthens rebalances, and underprovisioning caps throughput permanently. The usual compromise is to pick a partition count that covers expected peak concurrency with headroom, accepting the per-partition overhead.

Throughput per partition also has a ceiling set by sequential I/O and by the consumer’s processing rate, not by the broker’s disk in aggregate. Adding partitions adds parallelism until some shared resource saturates, and after that point the group can be perfectly balanced and still not faster.

Failure detection has its own timing

Heartbeats are sent on a shorter interval than the session timeout, so that a few lost heartbeats do not evict a healthy member. The relationship is what matters: the session timeout must be several heartbeat intervals, and long enough to cover the time a load balancer or orchestrator takes to restart a process, or the group will churn. A session timeout that is too short evicts members during a garbage collection pause or a brief network hiccup, and each eviction rebalances the group, which is the failure mode where a small hiccup becomes a group-wide stall.

A separate timeout covers processing. A member that heartbeats but does not call poll is still considered alive by the coordinator until the poll interval expires, at which point it is evicted to avoid a partition being held by a process that has stopped making progress. The two timeouts catch different failures: the session timeout catches a dead process, the poll interval catches a live but stuck one. Configuring only the first leaves the second at its default, which is usually far longer than a request timeout and therefore slow to notice a hung handler.

The interaction between rebalances and offset commits is where the duplicate arithmetic lives, described in acknowledgements, offsets, and visibility timeouts. The ordering guarantees that make the partition the unit of ownership in the first place are covered in ordering guarantees and head-of-line blocking.