Skip to content

feat(kb): spider guild rosters, tier armory refreshes and alert on a starved crawl - #1337

Draft
CDVolvik wants to merge 1 commit into
mainfrom
srdev/crawler-20261116
Draft

CDVolvik wants to merge 1 commit into
mainfrom
srdev/crawler-20261116

Conversation

@CDVolvik

Copy link
Copy Markdown
Collaborator

Part of #1270 (proposal item 3, discovery crawl) and #1272.

What changed

The hourly armory crawl has logged "calls":0 … "enqueued":0 for both regions since 2026-11-13. Its queue seeds from the near-empty players table and three guild rosters, and every stored row stays fresh for 20 days, so it has nothing to read.

Roster spider (armory_spider_guilds, new). Each run first queues every never-read guild that a stored character names, guilds with the most stored characters first, up to the run's guild limit. Guilds already read or already queued are skipped, and the queue's primary key dedupes the rest. Freshly read characters already queue their guild in armory_upsert_characters, and a roster read already queues its members through armory_upsert_guild, which skips hidden characters. The spider also covers rows stored before that guild step existed, and guilds whose queue rows were dropped.

Take order (armory_take, replaced; same signature):
0. Stored rows checked 25 or more days ago, so the 30-day purge never deletes a row the queue starved.

  1. The queue. For guilds, never-read ones come before stored ones, then oldest first.
  2. Due refreshes, oldest check first.

Tiered refresh. A character is due after 20 days. It is due after 1 day if it is "hot":

  • last_login or addon_seen falls within the last 3 days, or
  • it is in the top 100 of its region and ruleset by item_level, gear_score or honorable_kills.

All of these columns are already in the store. Linked characters stay daily, and the details revision and retry rules are unchanged. A daily refresh of an unchanged character costs one 304.

Budget. The default goes from 6,000 to 12,000 per region per run. One Battle.net client serves this crawl in both regions (24,000 an hour), the auction refresh (--budget 100 per region, 200 an hour) and Battle.net sign-in. That leaves about 11,800 of the documented 36,000 an hour for sign-ins and the on-demand lookup being built in another lane. Guild rosters get at most a quarter of a run's budget (budget / 8 guilds at 2 calls each), so a long guild queue cannot starve character reads.

Alert. The new armory_runs table holds one row per region with the count of consecutive runs that made zero calls. armory_record_run updates it after each completed, non-dry run. When the count reaches 6, the job exits non-zero with Armory <region> made no Blizzard calls for 6 runs in a row …. The table has RLS on, is closed to anon, authenticated and service_role, and gets select-only access for backup_exporter (same pattern as armory_links).

Without the migration. If the new functions are missing (PostgREST 404), the crawl still reads as before, then fails with Armory discovery functions are missing: apply supabase/migrations/20261117140000_armory_discovery.sql.

Storage. This adds no per-character JSON. The only new table holds two rows. Hidden characters are never queued, taken or re-stored.

How you tested it

  • kb: npx vitest run: 184 files, 2205 passed, 1 skipped. tsc --noEmit is clean. New crawl tests use the recorded roster and guild fixtures with fake API and store (no live calls). They cover:

    • spidered guilds plus roster members are enqueued and counted
    • guild rows are capped at a quarter of the budget and characters get the rest
    • five zero-call runs pass and the sixth fails
    • a run that read something records its call count
    • a missing migration still crawls, then fails naming the file
  • SQL: new supabase/tests/armory_discovery.sql, run on PGlite by kb/test/armory-profile-sql.test.ts. The harness now also applies the migration twice, the second time after the backup role exists. The test covers:

    • a roster queues only unseen members and never a hidden one
    • the spider respects its limit, dedupes, stays in its region and skips read guilds
    • never-read guilds are taken before stored ones
    • character order: near-purge rows, then the queue, then active and top-ranked rows; a cold row is not due
    • an addon sighting makes a row due daily
    • streaks count per region and reset after a run with calls
    • privileges are closed

    supabase/tests/armory.sql now expects 24 armory functions (it was 22) and also checks armory_runs.

  • I also applied all migrations on PGlite and ran each armory SQL test. armory_discovery.sql, armory.sql and armory_progress.sql pass. armory_profiles.sql, site_reads_armory.sql and backup_exporter.sql fail on that full chain with or without this migration (a users.name fixture and ops_upload_attributions), so those failures are not from this PR.

  • web: full npx vitest run --maxWorkers=4 --testTimeout=60000: 521 files passed, 5 skipped; 5967 tests passed, 30 skipped. npm run typecheck and npm run lint are clean apart from existing warnings in files this PR does not touch.

  • I made no live Blizzard or Supabase calls.

For the maintainer

  • Apply supabase/migrations/20261117140000_armory_discovery.sql before or right after merging. Until it is applied, every hourly armory run reads as before and then fails, naming the file. It is safe to apply twice. Rollback steps are at the bottom of the file.

  • No new secrets or workflow variables are needed. The workflow keeps the default budget. Pass --budget there if you want a different number.

  • Discovery still needs outside seeds. A guild roster only lists members of that guild, so the three seeded guilds form a closed set. The spider finds new guilds only through characters that come from somewhere else: addon named uploads (/everloot players on), the on-demand lookup from the other lane, or manual submissions. Until those arrive, the daily tier keeps the crawl calling every hour (top 100 per ruleset plus active players). I could not query prod, so I don't know how many unread guilds the 347 stored characters name today.

  • Existing pacing, unchanged here: both regions start at :17 and each paces itself to 50 calls a second. Together that is the documented 100 a second per client. Staggering the cron would add headroom.

  • Each regional job fails on its own sixth zero-call run.

  • The title reads type(scope): plain sentence

  • Commits are signed off

  • No scraped, datamined or spoiler data

  • For an addon change: n/a

  • CHANGELOG.md has one line under [Unreleased]

  • The area label is set

  • Docs are updated if behaviour changed (migration header and code comments)

…starved crawl

The hourly armory crawl has made no calls since 2026-11-13: its queue seeds
from the near-empty players table and three rosters, and every stored row
is fresh for 20 days.

- armory_spider_guilds queues each never-read guild that stored characters
  name (most characters first, per-run limit). Rosters still queue their
  members through armory_upsert_guild, which skips hidden characters.
- armory_take: rows near the 30-day purge first, then the queue (never-read
  guilds before stored ones), then due refreshes. Characters that logged in
  or were seen by the addon in the last 3 days, or rank in the top 100 of
  their region and ruleset by item level, gear score or honorable kills,
  are due daily; others every 20 days.
- Guild rosters get at most a quarter of the run's budget.
- Default budget 6,000 -> 12,000 per region (24,000 of the client's 36,000
  per hour, leaving room for auctions, sign-in and lookups).
- armory_runs keeps a per-region count of runs with zero calls; the job
  fails on the sixth in a row. Without the migration the crawl still reads,
  then fails naming the file.

Part of #1270, #1272.

Signed-off-by: CDVolvik <cjimenez@r21digital.com>
@CDVolvik CDVolvik added web everloot.app kb The knowledge base and MCP server infra CI, workflows, deploys and repository setup needs-decision Waiting for a maintainer decision labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infra CI, workflows, deploys and repository setup kb The knowledge base and MCP server needs-decision Waiting for a maintainer decision web everloot.app

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant