Repository navigation
Conversation
…starved crawl The hourly armory crawl has made no calls since 2026-11-13: its queue seeds from the near-empty players table and three rosters, and every stored row is fresh for 20 days. - armory_spider_guilds queues each never-read guild that stored characters name (most characters first, per-run limit). Rosters still queue their members through armory_upsert_guild, which skips hidden characters. - armory_take: rows near the 30-day purge first, then the queue (never-read guilds before stored ones), then due refreshes. Characters that logged in or were seen by the addon in the last 3 days, or rank in the top 100 of their region and ruleset by item level, gear score or honorable kills, are due daily; others every 20 days. - Guild rosters get at most a quarter of the run's budget. - Default budget 6,000 -> 12,000 per region (24,000 of the client's 36,000 per hour, leaving room for auctions, sign-in and lookups). - armory_runs keeps a per-region count of runs with zero calls; the job fails on the sixth in a row. Without the migration the crawl still reads, then fails naming the file. Part of #1270, #1272. Signed-off-by: CDVolvik <cjimenez@r21digital.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1270 (proposal item 3, discovery crawl) and #1272.
What changed
The hourly armory crawl has logged
"calls":0 … "enqueued":0for both regions since 2026-11-13. Its queue seeds from the near-empty players table and three guild rosters, and every stored row stays fresh for 20 days, so it has nothing to read.Roster spider (
armory_spider_guilds, new). Each run first queues every never-read guild that a stored character names, guilds with the most stored characters first, up to the run's guild limit. Guilds already read or already queued are skipped, and the queue's primary key dedupes the rest. Freshly read characters already queue their guild inarmory_upsert_characters, and a roster read already queues its members througharmory_upsert_guild, which skips hidden characters. The spider also covers rows stored before that guild step existed, and guilds whose queue rows were dropped.Take order (
armory_take, replaced; same signature):0. Stored rows checked 25 or more days ago, so the 30-day purge never deletes a row the queue starved.
Tiered refresh. A character is due after 20 days. It is due after 1 day if it is "hot":
last_loginoraddon_seenfalls within the last 3 days, oritem_level,gear_scoreorhonorable_kills.All of these columns are already in the store. Linked characters stay daily, and the details revision and retry rules are unchanged. A daily refresh of an unchanged character costs one 304.
Budget. The default goes from 6,000 to 12,000 per region per run. One Battle.net client serves this crawl in both regions (24,000 an hour), the auction refresh (
--budget 100per region, 200 an hour) and Battle.net sign-in. That leaves about 11,800 of the documented 36,000 an hour for sign-ins and the on-demand lookup being built in another lane. Guild rosters get at most a quarter of a run's budget (budget / 8guilds at 2 calls each), so a long guild queue cannot starve character reads.Alert. The new
armory_runstable holds one row per region with the count of consecutive runs that made zero calls.armory_record_runupdates it after each completed, non-dry run. When the count reaches 6, the job exits non-zero withArmory <region> made no Blizzard calls for 6 runs in a row …. The table has RLS on, is closed to anon, authenticated and service_role, and gets select-only access forbackup_exporter(same pattern asarmory_links).Without the migration. If the new functions are missing (PostgREST 404), the crawl still reads as before, then fails with
Armory discovery functions are missing: apply supabase/migrations/20261117140000_armory_discovery.sql.Storage. This adds no per-character JSON. The only new table holds two rows. Hidden characters are never queued, taken or re-stored.
How you tested it
kb:
npx vitest run: 184 files, 2205 passed, 1 skipped.tsc --noEmitis clean. New crawl tests use the recorded roster and guild fixtures with fake API and store (no live calls). They cover:SQL: new
supabase/tests/armory_discovery.sql, run on PGlite bykb/test/armory-profile-sql.test.ts. The harness now also applies the migration twice, the second time after the backup role exists. The test covers:supabase/tests/armory.sqlnow expects 24 armory functions (it was 22) and also checksarmory_runs.I also applied all migrations on PGlite and ran each armory SQL test.
armory_discovery.sql,armory.sqlandarmory_progress.sqlpass.armory_profiles.sql,site_reads_armory.sqlandbackup_exporter.sqlfail on that full chain with or without this migration (ausers.namefixture andops_upload_attributions), so those failures are not from this PR.web: full
npx vitest run --maxWorkers=4 --testTimeout=60000: 521 files passed, 5 skipped; 5967 tests passed, 30 skipped.npm run typecheckandnpm run lintare clean apart from existing warnings in files this PR does not touch.I made no live Blizzard or Supabase calls.
For the maintainer
Apply
supabase/migrations/20261117140000_armory_discovery.sqlbefore or right after merging. Until it is applied, every hourly armory run reads as before and then fails, naming the file. It is safe to apply twice. Rollback steps are at the bottom of the file.No new secrets or workflow variables are needed. The workflow keeps the default budget. Pass
--budgetthere if you want a different number.Discovery still needs outside seeds. A guild roster only lists members of that guild, so the three seeded guilds form a closed set. The spider finds new guilds only through characters that come from somewhere else: addon named uploads (
/everloot players on), the on-demand lookup from the other lane, or manual submissions. Until those arrive, the daily tier keeps the crawl calling every hour (top 100 per ruleset plus active players). I could not query prod, so I don't know how many unread guilds the 347 stored characters name today.Existing pacing, unchanged here: both regions start at :17 and each paces itself to 50 calls a second. Together that is the documented 100 a second per client. Staggering the cron would add headroom.
Each regional job fails on its own sixth zero-call run.
The title reads
type(scope): plain sentenceCommits are signed off
No scraped, datamined or spoiler data
For an addon change: n/a
CHANGELOG.mdhas one line under[Unreleased]The area label is set
Docs are updated if behaviour changed (migration header and code comments)