fix(scrape): repair download memory and dead break guard in amazon config - #150
Open
arminfauland wants to merge 1 commit into
Open
fix(scrape): repair download memory and dead break guard in amazon config#150arminfauland wants to merge 1 commit into
arminfauland wants to merge 1 commit into
Conversation
…nfig
Two independent bugs in the shipped Amazon configuration meant every run
re-downloaded every invoice.
1. `alreadyDownloaded` matched substrings
`amazon.downloadedInvoices` is a comma-joined string, but the check was
a plain `$contains`. An invoice key that is a substring of a stored key
counted as present, so it was never downloaded:
list = "…-306-1111111-2222222-10"
key = "…-306-1111111-2222222-1" → $contains → true (wrong)
Both sides are now wrapped in commas, which matches whole entries only.
Verified with jsonata: the old expression returns true, the new one false.
2. `shouldStoreOrderId` could never be true
The guard tested `firstDownloadedOrderId`, which is only created by the
*next* action. In JSONata a comparison against an undefined value yields
undefined, so `= null or = undefined` evaluates to false in both states:
unset → false set → false
The following `skipIf` uses `skipIfFalse: true`, so the step was always
skipped and `amazon.lastOrderId` was never written. That in turn left the
incremental `shouldBreak` — which compares against `lastOrderId` — as dead
code, so runs always walked the full order history.
`$not($exists(...))` expresses the original intent and evaluates to true
on the first iteration and false afterwards.
Both were found while investigating why a daily run kept re-fetching ~830
invoices: the stored list held 12 entries after 830 downloads.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LixHBPkhb8h5oDdMqSG4se
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent bugs in
config/sites/amazon.jsoncmake every run re-download every invoice. Found while investigating why a daily run kept fetching ~830 invoices it already had — the stored list held 12 entries after 830 downloads.1.
alreadyDownloadedmatched substringsamazon.downloadedInvoicesis a comma-joined string, but the check was a plain$contains. An invoice key that is a substring of a stored key counts as present and is therefore never downloaded:Affects any order with ten or more invoice links. Both sides are now wrapped in commas so only whole entries match.
Verified with
jsonataagainst the real expressions:…-2222222-1(only-10stored)…-2222222-102.
shouldStoreOrderIdcould never be trueThe guard tests
firstDownloadedOrderId— but that value is only created by the next action. In JSONata a comparison against an undefined operand yields undefined, so= null or = undefinedevaluates to false in both states:The following
skipIfusesskipIfFalse: true, so the step was always skipped andamazon.lastOrderIdwas never written. That leaves the incrementalshouldBreak— which compares the current order againstlastOrderId— as dead code, so runs always walk the entire order history instead of stopping at the last known order.$not($exists(...))expresses the original intent: true on the first iteration, false afterwards.Verification
Both expressions were extracted from the file and evaluated with the
jsonatapackage rather than reasoned about. Deployed on my own instance since 2026-08-04: the stored list now grows per invoice instead of per run, three consecutive runs downloaded 19 → 3 → 0 files, and the run time for one account dropped from 13.0 min to 1.7 min.Related: #149 — without that fix the list is invisible to templates within the same run, so these two fixes only take full effect together.