Skip to content

feat(web): look up a full character name in Blizzard's API when search finds nobody - #1338

Draft
CDVolvik wants to merge 1 commit into
srdev/search-20261116from
srdev/lookup-20261116
Draft

CDVolvik wants to merge 1 commit into
srdev/search-20261116from
srdev/lookup-20261116

Conversation

@CDVolvik

Copy link
Copy Markdown
Collaborator

Part of #1270 (proposal item 2, the on-demand lookup).

Stacked on #1335 (base srdev/search-20261116). After #1335 merges, this PR retargets to main.

What changed

When /search finds no stored character for a full "First Surname", the site now asks the Blizzard Profile API for that slug, US first, then EU, stores a hit and lists it.

Lookup (web/src/lib/armory-lookup.ts):

  • Uses the crawler's own code from kb/src/armory: ForeverApi for OAuth and the requests, and parseCharacter for the row. Nothing is duplicated.
  • One lookup makes 1 to 4 Blizzard calls: the profile in each region until one answers, then that region's equipment and PvP summary, which the stored row needs.
  • A hit is written through armory_upsert_characters, the crawler's write, with its hide guard. That same function already queues the character's guild for the crawler (discovered_guilds, source character). So item 4 of the brief needed no new queue call; a test checks the guild is queued.
  • The hit is the armory's one row in the Characters group. It links to the normal armory page and is marked exact.

Abuse controls (the lookup spends a paid third-party quota from a public endpoint):

  • Switch: EVERLOOT_ARMORY_LOOKUP=1, which also needs EVERLOOT_ARMORY=1. It is off by default and not set in wrangler.jsonc.
  • Names: only a first name and surname that pass the armory form's rules (characterSlugOf: 2–12 and 2–24 letters). A single word, three words or a one-letter surname never triggers a lookup.
  • Callers: only the /search page asks, with lookup=1, and only when the request is the browser's own fetch: Sec-Fetch-Site: same-origin, Sec-Fetch-Dest: empty, a Mozilla/ agent, and no bot, crawler, headless or script marker in the agent. The header type-ahead never sends lookup=1, because it would ask on every key. The repository had no user-agent bot handling to reuse. The route was already noindex and no-store, and that is unchanged.
  • Hidden characters: the new armory_lookup_regions(slug) returns only regions where the slug is neither hidden nor stored. A hidden slug is never fetched in its region. A hide that lands mid-lookup is refused by the upsert's guard.
  • Negative cache: the new armory_lookup_miss(slug) keeps a name that both regions answered 404 for, for six hours. While it is kept, the name is not asked again and costs no quota. Errors and timeouts are not cached as misses.
  • Rate limits: a new lookup scope on the existing consume_intake_quota limiter (the one the forms, MCP and market use): 20 lookups an hour per address and 300 an hour site-wide. That is at most 1,200 Blizzard calls an hour, against the client's documented 36,000. The existing PEOPLE_SEARCH_LIMIT (60 a minute) still applies first. The quota is consumed only after the hidden and miss checks, just before the first Blizzard call. If the quota cannot be read, the lookup is refused. This is the opposite of the per-minute limiter, which lets requests through when it cannot be read.
  • Time: the whole lookup has 1.8 s, and each Blizzard request is aborted at 1.8 s. The /search page already waits 3 s. A slow, failed or refused lookup returns the normal "nothing matches". When Blizzard refuses the token, the cached client is dropped, so the next lookup gets a new token.
  • Logs: a failure logs only the reason, never the searched name.

Migration supabase/migrations/20261117150000_armory_lookup.sql (not applied anywhere):

  • It adds the table armory_lookup_misses. RLS is on and no role has grants on it.
  • It adds the functions armory_lookup_regions and armory_lookup_miss. Both are security definer, granted to service_role only.
  • It redefines consume_intake_quota with the lookup scope added. The other scopes are unchanged.
  • The rollback is at the bottom of the file.
  • If the flag is on before the migration is applied, every lookup fails on the missing RPC or the invalid scope, and search behaves as it does today.

Also changed:

  • docs/ENVIRONMENT.md and web/.env.example list the new switch, and that the BNET client is now also used by the lookup.
  • One CHANGELOG line.

For the maintainer

  1. Apply 20261117150000_armory_lookup.sql (after fix(web): character search finds Forever characters by surname or full name #1335's migration).
  2. Set on the Worker (everloot-web):
    • BNET_CLIENT_ID (secret): the Battle.net client id. It is the same name the crawler (.github/workflows/armory.yml) and the site's Battle.net sign-in already read. If the Worker already has it for sign-in, nothing new is needed.
    • BNET_CLIENT_SECRET (secret): the same.
    • EVERLOOT_ARMORY_LOOKUP=1 (var): last, after the migration.
  3. Decide:
    • Should lookups use the sign-in client or a separate one? Blizzard's hourly limit is per client. On the crawler's client, lookups share its 36,000 an hour.
    • Are the quota numbers (20 per address and 300 site-wide per hour) and the six-hour miss cache right?

Not done / follow-ups

  • A character found by lookup is stored without the crawler's details (talents, stats, professions, achievements). Its checked_at is now, so the crawler won't read it again for 20 days unless someone presses refresh, which is allowed after an hour. Queueing these for an early details read belongs to the crawler lane (srdev/crawler-20261116).
  • The bot check is headers only, and headers can be forged. The quota is the real limit. Cloudflare bot management (cf.botManagement) is not used, because the code doesn't read it today.
  • supabase/tests/armory.sql pins 22 armory_* functions for the migrations the existing PGlite tests apply. It is not run against this migration, which adds two. The new supabase/tests/armory_lookup.sql checks their grants and security settings itself. When armory.sql next runs on all migrations, its count should become 24.
  • Not tried against the live Blizzard API from the Worker. All Blizzard traffic in the tests is mocked.

How you tested it

  • web/src/lib/armory-lookup.test.ts: real ForeverApi with a mocked fetch, on the memory store. Cases:

    • a US hit is 3 calls, stored, and its guild queued
    • a US miss then an EU hit is 4 calls, and only the EU character is stored
    • when both regions miss, the miss is cached, and the second search makes no call and uses no quota
    • the miss runs out after 6 h
    • a hidden US slug is never fetched in US; with both regions hidden there are no calls and nothing is stored
    • a hide during the lookup means nothing is stored
    • a used-up quota and an unreadable quota both make no calls
    • invalid names make no calls
    • a hanging Blizzard returns null within the budget, with no miss and nothing stored
    • a 503 is not cached as a miss
    • a refused token drops the client
    • the switch is off by default
    • the browser-only check rejects Googlebot, bingbot, HeadlessChrome, curl, cross-site requests and page loads
  • web/src/app/search/people/lookup.test.ts, on the route. It checks that:

    • lookup=1 from the browser runs one lookup counted by the quota
    • a used-up or unreadable quota refuses the lookup
    • no lookup runs with the switch off, without lookup=1 (header type-ahead), for Googlebot, or for a python script
  • people-search.test.ts: a full name nobody stored is looked up once and listed. There is no lookup when the armory has the name, when the armory failed, or when the armory switch is off.

  • intake-quota.test.ts: the lookup scope allows 20 an hour per address in development and sends p_scope: "lookup" to the RPC.

  • people-results.test.tsx: the /search page now asks with lookup=1. The header's search-bar.test.tsx is unchanged: it still asks without it.

  • SQL on PGlite (kb/test/armory-lookup-sql.test.ts, which runs supabase/tests/armory_lookup.sql) checks:

    • the grants
    • that the misses table can't be read outside its functions
    • the regions answered for stored and hidden slugs
    • the upsert refusing a hidden hit
    • the 6 h miss, extending it, and the expiry clean-up
    • bad slugs
    • the lookup quota at 20 per address and 300 global
    • that the market scope still works
  • cd web && npx vitest run --maxWorkers=4 --testTimeout=60000: 5997 passed, 30 skipped (528 files), 0 failed. This ran before the intake-quota test was added; that file passes alone (24 tests).

  • cd web && npm run typecheck: clean. npm run lint: 4 warnings, all in files this PR doesn't touch.

  • cd kb && npm test: 2203 passed, 1 skipped (186 files). npm run typecheck: clean.

  • cd web && npx next build: passes; /search/people stays a dynamic route.

  • The title reads type(scope): plain sentence

  • Commits are signed off

  • No scraped, datamined or spoiler data; all fixtures are invented names

  • For an addon change: n/a

  • CHANGELOG.md has one line under [Unreleased] (### Added, Site:)

  • The area label is set

  • Docs are updated (docs/ENVIRONMENT.md, web/.env.example, code comments, migration header)

…h finds nobody

When /search finds no stored character for a full "First Surname", ask the
Blizzard Profile API for that slug, US first, then EU (1 to 4 calls), with the
kb armory client and parser, and store a hit through armory_upsert_characters.
Behind EVERLOOT_ARMORY_LOOKUP=1, off by default.

Limits: only names that pass the armory form's rules, only the /search page's
own browser fetch (never the header type-ahead, crawlers or scripts), only
regions where the slug is neither hidden nor stored, a six-hour miss cache, a
'lookup' quota of 20 an hour per address and 300 site-wide, and a 1.8 s budget.

Part of #1270.

Signed-off-by: CDVolvik <cjimenez@r21digital.com>
@CDVolvik CDVolvik added web everloot.app privacy Personal data, consent and analytics labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

privacy Personal data, consent and analytics web everloot.app

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant