Specify fencing guarantees for polling and consumer offsets

[diegomrsantos]

While investigating #4117 and implementing #4119, I documented how poll completions must be validated against the partition's current message history before they can update consumer progress or authorize a successful reply.

I would like to build on that work with a focused specification of Iggy's fencing guarantees, starting with polling and consumer offsets.

The initial document would describe:

• Which identities establish validity, including partition history, consumer membership, and partition assignment.
• What invalidates those identities, where validation occurs, and which effects it protects.
• What happens to pending reads, queued automatic commits, and manual offset operations across purge and reassignment.
• What clients can infer from rejection, timeout, successful replies, and durable offset commits.
• Which guarantees have regression coverage and which cases still need clarification.

Two cases I would like to clarify are manual commits derived from messages read before purge, and delayed requests across partition reassignment, including when a member later regains the same partition. These are questions about the intended contract. I have not established additional bugs in those paths.

#4119 would remain focused on the stale poll completion bug. #4128 separately tracks the durability of consumer offset cleanup during purge. Both provide useful examples of boundaries the specification should explain.

I propose documenting the current guarantees and intentional limitations first, then creating focused implementation issues for any agreed gaps. Existing consensus and session documentation could be referenced where relevant.

I would like to prepare the initial specification once we agree on scope. Is there an existing document this should extend, and would polling and consumer offsets be a useful starting point?

[justinmclean]

Thanks, this is worth doing. @hubcio @spetz this one needs you: it asks what the intended contract is, not what the code currently does.

[hubcio]

my take on the open questions, including the three from #4130:

• purge numbering: restarting at zero isn't promised in the docs as far as i can see. i lean towards keeping numbering and advancing the start offset, kafka style. a stale bookmark then just points below the start, like after retention, and most of the bookmark cleanup goes away. it's still a visible change, so it needs a release note and a check of what sdks and bdd tests assume.
• if we keep the reset, bookmarks need the committed purge generation on them, and an old-generation bookmark is ignored on load (format change for the offset files).
• restart / power loss: guarantee per partition. which side of the purge survives can follow the durability policy, but a node must never mix histories, e.g. an old bookmark on top of new messages.
• manual commit of an offset read before purge: today it lands in the new offset space and can skip new messages, because a store carries no read identity. harmless with kept numbering. with the reset it has to be rejected, which needs the generation on the wire.
• delayed requests across reassignment: the assignment has to be part of the fence, same as the purge generation is for history, so a request from a previous assignment gets rejected even when the same member gets the partition back.
  • polls: the deferred polling i'm working on (not merged yet) already does this. before a parked poll completes, the partition owner re-checks group id + generation, purge generation and partition history in one step, against the metadata it has applied. the generation bumps on every assignment change, including a regain, so a stale poll gets a retryable error, never a batch. plain immediate polls (code 100) only check membership at dispatch, so they need the same check at completion.
  • commits: not covered yet. a store gets at most a membership check when it arrives, so a commit computed under the old assignment still lands once the member owns the partition again. closing that needs the generation of the read on the wire, same as the purge case above.