Skip to content

fix(scrape): repair download memory and dead break guard in amazon config - #150

Open
arminfauland wants to merge 1 commit into
Disane87:mainfrom
arminfauland:fix/amazon-config-download-memory
Open

fix(scrape): repair download memory and dead break guard in amazon config#150
arminfauland wants to merge 1 commit into
Disane87:mainfrom
arminfauland:fix/amazon-config-download-memory

Conversation

@arminfauland

Copy link
Copy Markdown

Two independent bugs in config/sites/amazon.jsonc make every run re-download every invoice. Found while investigating why a daily run kept fetching ~830 invoices it already had — the stored list held 12 entries after 830 downloads.

1. alreadyDownloaded matched substrings

amazon.downloadedInvoices is a comma-joined string, but the check was a plain $contains. An invoice key that is a substring of a stored key counts as present and is therefore never downloaded:

list = "…-306-1111111-2222222-10"
key  = "…-306-1111111-2222222-1"    → $contains → true   (wrong)

Affects any order with ten or more invoice links. Both sides are now wrapped in commas so only whole entries match.

Verified with jsonata against the real expressions:

key should be old new
…-2222222-1 (only -10 stored) false true false
…-2222222-10 true true true

2. shouldStoreOrderId could never be true

The guard tests firstDownloadedOrderId — but that value is only created by the next action. In JSONata a comparison against an undefined operand yields undefined, so = null or = undefined evaluates to false in both states:

unset → false          set → false

The following skipIf uses skipIfFalse: true, so the step was always skipped and amazon.lastOrderId was never written. That leaves the incremental shouldBreak — which compares the current order against lastOrderId — as dead code, so runs always walk the entire order history instead of stopping at the last known order.

$not($exists(...)) expresses the original intent: true on the first iteration, false afterwards.

Verification

Both expressions were extracted from the file and evaluated with the jsonata package rather than reasoned about. Deployed on my own instance since 2026-08-04: the stored list now grows per invoice instead of per run, three consecutive runs downloaded 19 → 3 → 0 files, and the run time for one account dropped from 13.0 min to 1.7 min.

Related: #149 — without that fix the list is invisible to templates within the same run, so these two fixes only take full effect together.

…nfig

Two independent bugs in the shipped Amazon configuration meant every run
re-downloaded every invoice.

1. `alreadyDownloaded` matched substrings

   `amazon.downloadedInvoices` is a comma-joined string, but the check was
   a plain `$contains`. An invoice key that is a substring of a stored key
   counted as present, so it was never downloaded:

       list = "…-306-1111111-2222222-10"
       key  = "…-306-1111111-2222222-1"   → $contains → true (wrong)

   Both sides are now wrapped in commas, which matches whole entries only.
   Verified with jsonata: the old expression returns true, the new one false.

2. `shouldStoreOrderId` could never be true

   The guard tested `firstDownloadedOrderId`, which is only created by the
   *next* action. In JSONata a comparison against an undefined value yields
   undefined, so `= null or = undefined` evaluates to false in both states:

       unset → false      set → false

   The following `skipIf` uses `skipIfFalse: true`, so the step was always
   skipped and `amazon.lastOrderId` was never written. That in turn left the
   incremental `shouldBreak` — which compares against `lastOrderId` — as dead
   code, so runs always walked the full order history.

   `$not($exists(...))` expresses the original intent and evaluates to true
   on the first iteration and false afterwards.

Both were found while investigating why a daily run kept re-fetching ~830
invoices: the stored list held 12 entries after 830 downloads.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LixHBPkhb8h5oDdMqSG4se
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant