Skip to content

fix(appmgmt): send dump units to the source's topic - #1192

Open
catinspace-au wants to merge 1 commit into
mainfrom
fix/dump-unit-table
Open

catinspace-au wants to merge 1 commit into
mainfrom
fix/dump-unit-table

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

A fetcher source with a shape: dump unit lost its rows to dfe.main. dfe-fetcher wrote the unit to <source>-<unit>_land and stamped _source the same. The engine creates only <source>_land and the table <source>.

  • Compiled config for keyed fetcher families (rest, db, file) sets topic_per_dump_unit: false. Every unit lands on the source's topic and table.
  • A source's fetcher.config cannot set the key, enabled or topic.
  • The dfe-fetcher release with the key (fix(fetcher): write dump units to the instance topic dfe-fetcher#316) must land first. The fetcher rejects unknown instance keys.

Tests: the key is set on rest, db and file and refused in source config. 2216 tests pass under tests/unit/test_appmgmt, test_source and the app contract tests. dfe-fetcher#316 tests the data path with the compiled config.

dfe-fetcher lands a dump unit on `<topic>-<unit>`. The engine creates only the topic and table `<source>`. Dump unit rows went to the default table.

The compiled config for a keyed fetcher family (rest, db, file) sets `topic_per_dump_unit` to false. Every unit goes to the source's topic. A source's fetcher config cannot set the key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant