Skip to content

fix(fetcher): write dump units to the instance topic - #316

Merged
catinspace-au merged 1 commit into
mainfrom
fix/dump-unit-table
Nov 16, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/dump-unit-table

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

A shape: dump unit writes to <topic>-<unit> and stamps that name as _source. The engine creates one topic and one table per source, so a dump source's rows had no table. The loader warned Destination table does not exist, falling back to the default table and wrote them to dfe.main.

  • topic_per_dump_unit: false on a rest, db or file instance writes its dump units to <topic>, with the tails. _source is then the instance topic.
  • The default stays <topic>-<unit>, so a standalone deployment keeps its per-store topics.
  • The dfe-engine change sets the key on every keyed family it compiles. Sources created through the engine need no manual setting.
  • Schema, profile grammar, snapshot envelope and internals docs updated.
  • The secret resolver test has a separate resolver type. A shared call counter caused failures when it and a sibling test ran on different threads.

Streaming decoders (json_array, ndjson, lines) still reject timeout_secs. It would cut a long dump at a fixed time. The client's idle read timeout bounds each read (docs/reference/profile-grammar.md). A json_at, json, document or csv unit takes it.

Results on desktop-derek with Redpanda, ClickHouse, dfe-loader v1.18.73 and a local fixture server, for a feodo-style source with two dump units. The table was created from the engine's DDL:

  • without the key: 18 rows in dfe.main, 0 in dfe.feodo-c2, topics feodo-c2-replay_land and feodo-c2-blocklist_land
  • with the engine's stanza: 0 rows in dfe.main, 18 in dfe.feodo-c2, all _source = feodo-c2, feodo-c2_land at offset 18

A dump unit writes to `<topic>-<unit>` and sets `_source` to that name. The engine creates one topic and one table per source. The loader found no table for `feodo-c2-blocklist`. It warned and wrote the rows to the default table.

`topic_per_dump_unit: false` on a rest, db or file instance writes its dump units to `<topic>` with the tails. `_source` is the instance topic. The default stays `<topic>-<unit>`, so a standalone deployment keeps its per-store topics.

The secret resolver test shared a call counter with a sibling test and failed when the two ran on different threads. It has a separate resolver type.
@catinspace-au
catinspace-au merged commit c70edb1 into main Nov 16, 2026
24 checks passed
@catinspace-au
catinspace-au deleted the fix/dump-unit-table branch November 16, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant