fix(api): store dotted keys nested so templates can read them back - #149
Open
arminfauland wants to merge 1 commit into
Open
fix(api): store dotted keys nested so templates can read them back#149arminfauland wants to merge 1 commit into
arminfauland wants to merge 1 commit into
Conversation
`storeData` writes the runtime value under the flat key:
this.storedData[key] = value; // "amazon.downloadedInvoices"
But templates resolve `{{storedData.amazon.downloadedInvoices}}` as a *path*,
and the DB loader (`convertJobDataToNestedObject` in ScrapeDataService) builds
a nested object from the same dotted key. So the flat entry is invisible to
every template — the expression yields an empty string.
The comment on that line says "für Template-Zugriff in diesem Run", which is
exactly what did not work: a value stored during a run could not be read back
during that run. It only became visible in the *next* run, via the detour
through the database.
That is easy to miss because it almost works. A config that appends to a list
like
updatedList = storedData.x.list != '' ? storedData.x.list & ',' & item : item
storeData x.list = updatedList
grows by exactly one entry per run instead of one per loop iteration, since
every iteration reads the value loaded at run start. In my case a list meant to
hold ~800 downloaded invoice ids held 12 — one per run — so the scraper
re-downloaded everything on each run and relied on the downstream consumer to
deduplicate.
Fix: mirror the loader and write the nested form as well. The flat key is kept
so configs that read it as a whole keep working.
Adds five tests: nested write, flat key retained, deep nesting, merging into an
existing branch, and replacing a non-object value that would otherwise block
the path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LixHBPkhb8h5oDdMqSG4se
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
StoreDataActionwrites the runtime value under the flat key:But templates resolve
{{storedData.amazon.downloadedInvoices}}as a path, and the DB loaderconvertJobDataToNestedObject()inScrapeDataServicebuilds a nested object from the very same dotted key.So the flat entry is invisible to every template. Verified with the bundled Handlebars:
The comment on that line reads "für Template-Zugriff in diesem Run" — which is exactly what does not work. A value stored during a run cannot be read back during that run; it only appears in the next run, via the detour through the database.
Why it is easy to miss
It almost works. A config that appends to a list —
{ "action": "transform", "name": "updatedList", "params": { "expression": "'{{storedData.x.list}}' != '' ? '{{storedData.x.list}},' & item : item" } }, { "action": "storeData", "params": { "key": "x.list", "value": "{{previousData.updatedList}}", "persist": true } }— grows by exactly one entry per run instead of one per loop iteration, because every iteration reads the value loaded at run start and the last write wins.
In my case a list meant to hold ~800 downloaded invoice ids held 12 — one per run. The scraper therefore re-downloaded everything on every run and relied on the downstream consumer to deduplicate by checksum. Which it did, silently, so nothing looked broken — until the source started rate-limiting the redundant traffic.
The fix
Mirror the loader: write the nested form as well. The flat key is kept, so configs that read the dotted key as a whole keep working.
Tests
Five added to the existing spec:
a.b.c.d)Full API suite: 1421 tests / 84 files, all passing.
nx lint apiclean.