Skip to content

dekaf: incompatible field schema closes session until spec refresh #3338

Description

@danielnelson

Reproduction Steps:

  1. Begin reading a dekaf topic with schema inference.
  2. Write a document: {"name": "alice", "value": 42}
  3. Write a document with an incompatible field schema: {"name": "bob", "value": "howdy"}

Expected:

Read documents as written due to schema widening:

{"name": "alice", "value": 42}
{"name": "bob", "value": "howdy"}

Actual:

Disconnected:

%6|1786407316.978|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://127.0.0.1:10250/1]: sasl_plaintext://127.0.0.1:10250/1: Disconnected (after 21858ms in state UP)
%6|1786407317.015|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://127.0.0.1:10250/1]: sasl_plaintext://127.0.0.1:10250/1: Disconnected (after 12ms in state UP, 1 identical error(s) suppressed)
% ERROR: Local: Broker transport failure: sasl_plaintext://127.0.0.1:10250/1: Disconnected (after 21858ms in state UP)
% ERROR: Local: All broker connections are down: 1/1 brokers are down: terminating

Server logs:

2026-08-11T00:15:17.014501Z  WARN dekaf_session{session_client_id="rdkafka" task_name="dbn/test/dekaf-generic"}:serve{idle_timeout=60s stale_read_timeout=60s tls_handshake_timeout=10s addr=[::1]:50252}: dekaf: error=Extracting field Extractor(
    Extractor {
        ptr: Pointer(
            [
                Property(
                    "value",
                ),
            ],
        ),
        policy: SerPolicy {
            str_truncate_after: 18446744073709551615,
            array_truncate_after: 18446744073709551615,
            nested_obj_truncate_after: 18446744073709551615,
        },
        default: Null,
        magic: None,
    },
), schema: Long
Caused by:
    at , value "howdy" does not conform to AVRO schema "long"

Cannot read again until spec_ttl timeout refreshes the collection spec.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions