Repository navigation
fix(fetcher): write dump units to the instance topic - #316
Merged
Merged
Conversation
A dump unit writes to `<topic>-<unit>` and sets `_source` to that name. The engine creates one topic and one table per source. The loader found no table for `feodo-c2-blocklist`. It warned and wrote the rows to the default table. `topic_per_dump_unit: false` on a rest, db or file instance writes its dump units to `<topic>` with the tails. `_source` is the instance topic. The default stays `<topic>-<unit>`, so a standalone deployment keeps its per-store topics. The secret resolver test shared a call counter with a sibling test and failed when the two ran on different threads. It has a separate resolver type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
shape: dumpunit writes to<topic>-<unit>and stamps that name as_source. The engine creates one topic and one table per source, so a dump source's rows had no table. The loader warnedDestination table does not exist, falling back to the default tableand wrote them todfe.main.topic_per_dump_unit: falseon a rest, db or file instance writes its dump units to<topic>, with the tails._sourceis then the instance topic.<topic>-<unit>, so a standalone deployment keeps its per-store topics.Streaming decoders (
json_array,ndjson,lines) still rejecttimeout_secs. It would cut a long dump at a fixed time. The client's idle read timeout bounds each read (docs/reference/profile-grammar.md). Ajson_at,json,documentorcsvunit takes it.Results on desktop-derek with Redpanda, ClickHouse, dfe-loader v1.18.73 and a local fixture server, for a feodo-style source with two dump units. The table was created from the engine's DDL:
dfe.main, 0 indfe.feodo-c2, topicsfeodo-c2-replay_landandfeodo-c2-blocklist_landdfe.main, 18 indfe.feodo-c2, all_source = feodo-c2,feodo-c2_landat offset 18