Skip to content

fix(api): store dotted keys nested so templates can read them back - #149

Open
arminfauland wants to merge 1 commit into
Disane87:mainfrom
arminfauland:fix/store-data-nested-runtime
Open

fix(api): store dotted keys nested so templates can read them back#149
arminfauland wants to merge 1 commit into
Disane87:mainfrom
arminfauland:fix/store-data-nested-runtime

Conversation

@arminfauland

Copy link
Copy Markdown

The bug

StoreDataAction writes the runtime value under the flat key:

this.storedData[key] = value;      // key = "amazon.downloadedInvoices"

But templates resolve {{storedData.amazon.downloadedInvoices}} as a path, and the DB loader convertJobDataToNestedObject() in ScrapeDataService builds a nested object from the very same dotted key.

So the flat entry is invisible to every template. Verified with the bundled Handlebars:

storedData written flat   → {{storedData.amazon.downloadedInvoices}} = ""
storedData written nested → {{storedData.amazon.downloadedInvoices}} = "VERSCHACHTELT"

The comment on that line reads "für Template-Zugriff in diesem Run" — which is exactly what does not work. A value stored during a run cannot be read back during that run; it only appears in the next run, via the detour through the database.

Why it is easy to miss

It almost works. A config that appends to a list —

{ "action": "transform", "name": "updatedList",
  "params": { "expression": "'{{storedData.x.list}}' != '' ? '{{storedData.x.list}},' & item : item" } },
{ "action": "storeData",
  "params": { "key": "x.list", "value": "{{previousData.updatedList}}", "persist": true } }

— grows by exactly one entry per run instead of one per loop iteration, because every iteration reads the value loaded at run start and the last write wins.

In my case a list meant to hold ~800 downloaded invoice ids held 12 — one per run. The scraper therefore re-downloaded everything on every run and relied on the downstream consumer to deduplicate by checksum. Which it did, silently, so nothing looked broken — until the source started rate-limiting the redundant traffic.

The fix

Mirror the loader: write the nested form as well. The flat key is kept, so configs that read the dotted key as a whole keep working.

Tests

Five added to the existing spec:

  • dotted key becomes a nested object
  • flat key is still present
  • deep nesting (a.b.c.d)
  • merges into an existing branch instead of replacing it
  • replaces a non-object value that would otherwise block the path

Full API suite: 1421 tests / 84 files, all passing. nx lint api clean.

`storeData` writes the runtime value under the flat key:

    this.storedData[key] = value;      // "amazon.downloadedInvoices"

But templates resolve `{{storedData.amazon.downloadedInvoices}}` as a *path*,
and the DB loader (`convertJobDataToNestedObject` in ScrapeDataService) builds
a nested object from the same dotted key. So the flat entry is invisible to
every template — the expression yields an empty string.

The comment on that line says "für Template-Zugriff in diesem Run", which is
exactly what did not work: a value stored during a run could not be read back
during that run. It only became visible in the *next* run, via the detour
through the database.

That is easy to miss because it almost works. A config that appends to a list
like

    updatedList = storedData.x.list != '' ? storedData.x.list & ',' & item : item
    storeData x.list = updatedList

grows by exactly one entry per run instead of one per loop iteration, since
every iteration reads the value loaded at run start. In my case a list meant to
hold ~800 downloaded invoice ids held 12 — one per run — so the scraper
re-downloaded everything on each run and relied on the downstream consumer to
deduplicate.

Fix: mirror the loader and write the nested form as well. The flat key is kept
so configs that read it as a whole keep working.

Adds five tests: nested write, flat key retained, deep nesting, merging into an
existing branch, and replacing a non-object value that would otherwise block
the path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LixHBPkhb8h5oDdMqSG4se
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant