Skip to content

dekaf: new fields added to the collection schema are excluded until new "Read" #3337

Description

@danielnelson

Reproduction Steps:

  1. Begin reading a dekaf topic with schema inference.
  2. Write a document: {"name": "alice"}
  3. Write a document with a new field not in the schema: {"name": "bob", "value": 42}

Expected:

Read documents as written:

{"name": "alice"}
{"name": "bob", "value": 42}

Actual:

New field is omitted from output:

{"name": "alice"}
{"name": "bob"}

Even after waiting for the spec_ttl expires and a new schema is picked up, the new field will be absent. In order to pick up the new field, you must disconnect or perform another operation that creates a new Read such as changing the read offset.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions