From e1b0f4c45a80ed636d45b859b3c84d04fda85517 Mon Sep 17 00:00:00 2001 From: poketonova Date: Wed, 10 Jun 2026 18:35:09 -0600 Subject: [PATCH] Document the static data pipeline (serverless functions are gone) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The app repo dropped its Python data endpoints (novaoc/rarebox#18): Riftbound + Japanese Pokémon prices and meta decks are now pre-built into public/*.json by a daily GitHub Actions workflow and served as static assets. - architecture/serverless.md rewritten as 'Static Data Pipeline': the assets, the refresh-data.yml workflow, fork configuration (repo guard, own User-Agent, enabling the cron), client consumption, running the scripts locally - api-integrations.md: new tcgcsv.com section; Riftbound prices are a TCGplayer product-id join (PriceCharting now fallback-only); meta decks scraped in CI, client reads /meta-decks/{game}.json - deployment.md: current vercel.json, api/og.py as the only function, self-hosting now needs only static hosting + the workflow steps Co-Authored-By: Claude Fable 5 --- docs/.vitepress/config.js | 2 +- docs/architecture/api-integrations.md | 51 +++++-- docs/architecture/serverless.md | 199 ++++++++++++++------------ docs/reference/deployment.md | 47 +++--- 4 files changed, 179 insertions(+), 120 deletions(-) diff --git a/docs/.vitepress/config.js b/docs/.vitepress/config.js index 29b6d98..6eea5ca 100644 --- a/docs/.vitepress/config.js +++ b/docs/.vitepress/config.js @@ -62,7 +62,7 @@ export default defineConfig({ { text: 'Project Structure', link: '/architecture/project-structure' }, { text: 'Data Flow & Persistence', link: '/architecture/data-flow' }, { text: 'API Integrations', link: '/architecture/api-integrations' }, - { text: 'Serverless Functions', link: '/architecture/serverless' }, + { text: 'Static Data Pipeline', link: '/architecture/serverless' }, { text: 'Price Snapshot System', link: '/architecture/snapshots' }, { text: 'Component Guide', link: '/architecture/components' }, ], diff --git a/docs/architecture/api-integrations.md b/docs/architecture/api-integrations.md index f308c14..2e3630a 100644 --- a/docs/architecture/api-integrations.md +++ b/docs/architecture/api-integrations.md @@ -114,18 +114,41 @@ Rarebox pulls data from several external APIs. Each has different characteristic - Open REST API, no API key required - 7 sets (Origins, Spiritforged, Unleashed, promo sets), 1000+ cards - Card images from Riot Games CDN (`cmsassets.rgpub.io`) -- Pagination: 50 cards per page -- **No prices** — card data only. Prices come from PriceCharting, merged variant-aware (see note below) -- `tcgplayer_id` field available for price lookups if needed +- Pagination: 100 cards per page +- **No prices** — card data only. Each card carries a `tcgplayer_id`, which is + what makes exact price joins possible (see note below) - Results cached in memory for 1 hour -::: note Price Source — variant-aware -riftcodex provides card metadata and images; prices come from PriceCharting. Because PriceCharting caps each search at 100 products, the browse provider runs **three queries per set** (base, `signature`, `alternate art`), de-duplicates, and filters by console name. Variant printings (`(Signature)`, `(Alternate Art)`) only ever take their own bracketed PriceCharting listing — **a variant never falls back to the plain card's price** (no price beats a wrong one). `(Overnumbered)` is the plain printing of an over-set-size champion. Search, trade analyzer, portfolio refresh, and the offline cache all hydrate from this same per-set price map. +::: note Price Source — exact join, variant-aware +riftcodex provides card metadata and images; prices are TCGplayer market +prices from the pre-built `/riftbound-prices.json` static asset (see +[Static Data Pipeline](/architecture/serverless)), joined on each card's +`tcgplayer_id`. Every printing is its own TCGplayer product, so matches are +exact — **a variant (`(Signature)`, `(Alternate Art)`) never falls back to +the plain card's price** (no price beats a wrong one), and promo sets +(PR/OPP/JDG) are fully covered. `(Overnumbered)` is the plain printing of an +over-set-size champion. If the asset yields no prices for a set (e.g. newer +than the last daily refresh), the provider falls back to the old +PriceCharting search (three queries per set around its 100-result cap, +filtered by console name, variant-aware). Search, trade analyzer, portfolio +refresh, and the offline cache all hydrate from this same per-set price map. ::: +## tcgcsv.com + +**Purpose:** Daily TCGplayer price dumps — Riftbound singles (category 89) and Japanese Pokémon cards (category 85 "Pokemon Japan"). + +**Called from:** CI only (`scripts/build_riftbound_prices.py`, `scripts/build_jp_prices.py` via the daily `refresh-data.yml` workflow) — **never the browser**. tcgcsv has no CORS and its terms allow backend scripts only. The app consumes the results as static assets (`/riftbound-prices.json`, `/jp-prices.json`). + +**Key details:** +- Requires an identifying `User-Agent` (`Rarebox/1.4 (+https://rarebox.io)`) — generic browser UAs get a 401. Forks must use their own (see [Static Data Pipeline](/architecture/serverless)) +- Updates once daily around 20:00 UTC; `last-updated.txt` is the change stamp, and the scripts re-sync only when it moves +- Japanese Pokémon join: tcgcsv group abbreviations match tcgdex set ids, product `Number` ("205/187") matches tcgdex `localId` +- ≤10k requests/day allowed; a full sync of both categories uses a few hundred + ## PriceCharting -**Purpose:** Sealed product prices (booster boxes, ETBs, tins), graded slab prices (grade-specific), and market prices for non-Pokémon TCG cards (Magic, Yu-Gi-Oh!, Lorcana, One Piece, Riftbound). +**Purpose:** Sealed product prices (booster boxes, ETBs, tins), graded slab prices (grade-specific), and market prices for non-Pokémon TCG cards (Magic, Yu-Gi-Oh!, Lorcana, One Piece; for Riftbound singles it's now only the fallback when the static price asset has nothing). **Called from:** Browser (client-side, direct JSON API calls) @@ -153,17 +176,23 @@ riftcodex provides card metadata and images; prices come from PriceCharting. Bec **Purpose:** Current tournament meta deck data — top decks, meta share percentages, championship points. -**Called from:** Vercel serverless function (`/api/search`) and client-side (`metaDecksApi.js`) +**Called from:** CI only (`scripts/build_meta_decks.py` via the daily `refresh-data.yml` workflow); the client (`metaDecksApi.js`) reads the result from the `/meta-decks/{game}.json` static asset **Key details:** - Data is **scraped** from Limitless TCG's website using httpx + BeautifulSoup (no official API) -- Core cards are resolved server-side with exact card match (set code + number) -- Client-side service fetches from serverless endpoint with localStorage fallback (24h cache) -- Static fallback decks available when live endpoint is unavailable +- Core cards are resolved in the build script with exact card match (set code + number) +- Client-side service fetches the static asset with localStorage fallback (24h cache) +- Static fallback decks available when the asset is missing or empty for a game - If Limitless changes their HTML structure, the scraper will break and need updating ::: warning Scraping Dependency -The Limitless TCG integration is the most fragile part of the system. It relies on HTML structure that can change without notice. If meta decks stop loading, the scraper likely needs updating. Check `/api/search.py` first. +The meta-deck scrapers are the most fragile part of the system — they rely on +HTML structure that can change without notice. If meta decks stop loading, +check `scripts/build_meta_decks.py` first. As of mid-2026 only the Pokémon +(Limitless) scraper yields decks: the MTG/One Piece/Yu-Gi-Oh selectors are +stale and the Lorcana/Riftbound sources 403; those games run on the client's +fallback decks. The daily workflow keeps the previous day's file whenever a +scraper fails, so a broken scraper degrades to stale decks, never none. ::: ## Pokellector diff --git a/docs/architecture/serverless.md b/docs/architecture/serverless.md index f5536b7..a11e17e 100644 --- a/docs/architecture/serverless.md +++ b/docs/architecture/serverless.md @@ -1,99 +1,116 @@ -# Serverless Functions - -Rarebox is primarily a client-side app, but a few operations need server-side execution. These run as **Vercel Functions** — Python serverless functions in the `api/` directory. - -## Why Server-Side? - -Three reasons a request needs to go through the server: - -1. **CORS restrictions** — some APIs don't allow browser-origin requests -2. **HTML scraping** — Limitless TCG doesn't have a public API; we scrape their site -3. **Response transformation** — some data needs processing before the client can use it (e.g., resolving card set codes to pokemontcg.io IDs) - -Everything else is fetched directly from the browser. - -## Endpoints - -### `GET /api/health` - -Simple health check. Returns 200 with a status message. Used for uptime monitoring. - -### `GET /api/search` - -Scrapes Limitless TCG for current meta deck data. This is the most complex endpoint: - -1. Fetches the meta standings page from Limitless -2. Parses HTML with BeautifulSoup to extract top decks, meta share, and CP -3. Resolves core card names to exact pokemontcg.io card IDs (set code + card number) -4. Returns structured JSON with deck names, card lists, and metadata - -**Performance:** Can take 5-15 seconds depending on Limitless response time and the number of card resolutions needed. Max duration is 30 seconds (configured in `vercel.json`). - -### `GET /api/price` - -Price proxy/lookup endpoint. Fetches and transforms price data that can't be accessed directly from the browser. - -### `GET /api/sealed` - -Fetches sealed product pricing from PriceCharting for items that need server-side access. - -## Client-Side Meta Decks - -In addition to the serverless endpoint, `metaDecksApi.js` provides: - -- **Live fetch** from `/api/search` with localStorage fallback (24h cache) -- **Static fallback decks** when the live endpoint is unavailable -- **Game-specific caching** — each TCG has its own cache key -- **Cache invalidation** — stale fallback data is removed when server data is available - -## Runtime Configuration - -From `vercel.json`: - -```json -{ - "functions": { - "api/**/*.py": { - "runtime": "@vercel/python@4.5.0", - "maxDuration": 30 - } - } -} +# Static Data Pipeline + +Rarebox is a **local-only app**: Vercel serves only code and static assets, and +your device makes every API call itself. There are no serverless data +endpoints — the `api/` directory contains exactly one function, `/api/og`, +which renders social-embed images for link-preview crawlers (Discord, +Telegram, X). The app never calls it. + +Some data sources can't be reached from a browser, though: + +1. **tcgcsv.com** (daily TCGplayer price dumps) — no CORS, and its terms of + use allow backend scripts only +2. **Meta-deck sites** (Limitless TCG and others) — HTML scraping, no public + API + +Instead of proxying these through serverless functions, a daily GitHub +Actions workflow pre-builds them into static JSON in `public/`, which deploys +with the app like any other asset. + +## The Assets + +| Asset | Built by | Contents | +|-------|----------|----------| +| `public/riftbound-prices.json` | `scripts/build_riftbound_prices.py` | TCGplayer market prices for all Riftbound cards (tcgcsv category 89), keyed by TCGplayer product id, `{ normal, foil }` per product | +| `public/jp-prices.json` | `scripts/build_jp_prices.py` | TCGplayer prices for 16k+ Japanese Pokémon cards (tcgcsv category 85 "Pokemon Japan"), keyed `{tcgdex set id}-{number}` (lowercase, no leading zeros) | +| `public/meta-decks/{game}.json` | `scripts/build_meta_decks.py` | Scraped tournament meta decks per game, cards resolved to ids/images/prices | + +## The Workflow + +`.github/workflows/refresh-data.yml` in the app repo: + +- Runs daily at **21:00 UTC** — an hour after tcgcsv's ~20:00 refresh — with a + **23:00 retry** for days tcgcsv runs late. Also triggerable manually via + `workflow_dispatch`. +- The price scripts compare tcgcsv's `last-updated.txt` stamp against the one + embedded in the committed JSON and **skip the sync when nothing changed**, + so the retry (and any manual run) is a cheap no-op. +- Both price scripts **refuse to overwrite** their output if the pull comes + back suspiciously small — a bad upstream day degrades to stale prices, + never broken prices. The meta-deck script likewise keeps yesterday's file + for any game whose scraper fails. +- Changed files are committed as `github-actions[bot]`; the push to `main` + triggers the normal Vercel static deploy. No change → no commit → no + deploy. +- Robustness rails: a `concurrency` group (runs can't overlap), 15-minute + job timeout, `git pull --rebase` before push (a concurrent push to `main` + doesn't lose the refresh), and a fork guard (below). + +## Running a Fork Correctly + +The workflow job is gated: + +```yaml +if: github.repository == 'novaoc/rarebox' ``` -- **Runtime:** Python via `@vercel/python@4.5.0` -- **Max duration:** 30 seconds per invocation -- **Dependencies:** `httpx` (async HTTP) and `beautifulsoup4` (HTML parsing), specified in `requirements.txt` - -## Developing Locally - -Vercel Functions can be tested locally using the Vercel CLI: +Forks inherit the workflow file, but their copy no-ops — otherwise every +enabled fork would hit tcgcsv and the deck sites with traffic identified as +Rarebox. **A fork still works without it**: the committed JSON ships with the +clone, just frozen at fork time. + +To make your fork self-refresh: + +1. **Edit the guard** in `.github/workflows/refresh-data.yml` to your + `owner/repo`. +2. **Change the `User-Agent`** in `scripts/build_riftbound_prices.py` and + `scripts/build_jp_prices.py` to identify *your* deployment. tcgcsv + requires an identifying UA — generic browser UAs get a 401 — and its + other rules (re-sync only on stamp change, ≤10k requests/day) are already + handled by the scripts. +3. **Enable the workflow** in your fork's Actions tab. GitHub disables + Actions in forks until you opt in, and inherited scheduled workflows stay + disabled even after that — both are deliberate, so nothing runs by + surprise. + +Note that GitHub also auto-disables scheduled workflows in public repos +after 60 days without repository activity; the daily data commits normally +keep the timer reset on an active deployment. + +## Client Consumption + +The app fetches the assets same-origin (works in plain `vite dev` too — +Vite serves `public/` at the root): + +- **`providers.js`** (Riftbound browse) joins `/riftbound-prices.json` on + each card's `tcgplayer_id` — exact per-printing matches, so a Signature + can never inherit its plain card's price. If the asset yields zero prices + for a set (e.g. a set newer than the last refresh), it falls back to the + old PriceCharting search. +- **`pokemonApi.js`** reads `/jp-prices.json` for Japanese card grids, + detail views, and portfolio refresh, falling back to tcgdex's Cardmarket + EUR data where TCGplayer has nothing. +- **`metaDecksApi.js`** fetches `/meta-decks/{game}.json` with a 24h + localStorage cache and built-in fallback decks when the asset is missing + or empty for a game. + +## Running the Scripts Locally ```bash -npm i -g vercel -vercel dev +pip install httpx beautifulsoup4 # meta decks only; price scripts are stdlib +python3 scripts/build_riftbound_prices.py +python3 scripts/build_jp_prices.py +python3 scripts/build_meta_decks.py [game ...] ``` -This starts a local dev server that emulates the serverless function runtime. API endpoints are available at `http://localhost:3000/api/*`. - -## Caching Recommendations +Each writes into `public/` and prints what it did. The price scripts exit +early with "up to date" unless tcgcsv has refreshed since the committed +stamp. -Currently, API responses aren't cached server-side. For production traffic, consider: +## History -- **Quick win:** Add `Cache-Control: s-maxage=3600` headers to responses. Vercel's CDN caches them automatically for the specified duration. -- **Meta decks:** Could use `s-maxage=86400` (24h) since tournament data changes daily at most. -- **Price lookups:** `s-maxage=3600` (1h) is a reasonable default. - -```python -# Example: adding cache headers to a response -from http.server import BaseHTTPRequestHandler - -class handler(BaseHTTPRequestHandler): - def do_GET(self): - # ... fetch and process data ... - self.send_response(200) - self.send_header('Content-Type', 'application/json') - self.send_header('Cache-Control', 's-maxage=3600') - self.end_headers() - self.wfile.write(json.dumps(result).encode()) -``` +Until mid-2026 Rarebox ran Python serverless functions on Vercel +(`/api/search`, `/api/price`, `/api/sealed`, `/api/health`, `/api/meta-decks`) +for scraping and price proxying. They were removed in favor of this pipeline +— the local-only rule means the only thing a server should do is hand the +device files. diff --git a/docs/reference/deployment.md b/docs/reference/deployment.md index 877a5f8..540a61a 100644 --- a/docs/reference/deployment.md +++ b/docs/reference/deployment.md @@ -6,13 +6,19 @@ Rarebox is deployed on **Vercel** with automatic deployments from the `main` bra ### How It Works -1. Push to `main` on GitHub +1. Push to `main` on GitHub — by you, or by the daily + [data-refresh workflow](/architecture/serverless) committing fresh price + and meta-deck JSON into `public/` 2. Vercel detects the push and starts a build 3. `npm run build` runs Vite, outputting static files to `dist/` -4. Python serverless functions in `api/` are bundled with `@vercel/python@4.5.0` -5. Static files are deployed to Vercel's CDN -6. Serverless functions are deployed as edge functions -7. Live at [rarebox.io](https://rarebox.io) within ~60 seconds +4. Static files are deployed to Vercel's CDN +5. The one remaining Python function, `api/og.py` (social-embed images for + crawlers — the app never calls it), is bundled with `@vercel/python@4.5.0` +6. Live at [rarebox.io](https://rarebox.io) within ~60 seconds + +The app itself is **local-only**: Vercel hands the device code and static +assets, and the device makes every data call itself. There are no serverless +data endpoints. ### `vercel.json` Configuration @@ -22,10 +28,6 @@ Rarebox is deployed on **Vercel** with automatic deployments from the `main` bra "outputDirectory": "dist", "framework": "vite", "rewrites": [ - { "source": "/api/health", "destination": "/api/health" }, - { "source": "/api/search", "destination": "/api/search" }, - { "source": "/api/price", "destination": "/api/price" }, - { "source": "/api/sealed", "destination": "/api/sealed" }, { "source": "/((?!api/).*)", "destination": "/index.html" } ], "functions": { @@ -38,8 +40,7 @@ Rarebox is deployed on **Vercel** with automatic deployments from the `main` bra ``` **Rewrites explained:** -- API routes (`/api/*`) pass through to serverless functions -- Everything else falls through to `/index.html` — this is what makes Vue Router's HTML5 history mode work (clean URLs without hash fragments) +- Everything except `/api/og` falls through to `/index.html` — this is what makes Vue Router's HTML5 history mode work (clean URLs without hash fragments). Static files in `public/` (including the pre-built price/meta-deck JSON) are served before rewrites apply. ### DNS @@ -55,9 +56,17 @@ Want to run your own instance? Here's what you need. 1. Fork `novaoc/rarebox` on GitHub 2. Create a new project on [vercel.com](https://vercel.com) 3. Import your fork -4. Vercel auto-detects the Vite framework and Python functions +4. Vercel auto-detects the Vite framework (and the one `api/og.py` function) 5. Deploy — no configuration needed +Your fork ships with working price and meta-deck data: the JSON assets in +`public/` are committed to the repo, just frozen at fork time. To keep them +refreshing daily, configure the data-refresh workflow — edit its +`if: github.repository == ...` guard to your fork, set your own `User-Agent` +in `scripts/build_*_prices.py` (tcgcsv requires an identifying UA), and +enable the workflow in your fork's Actions tab. Full steps in +[Static Data Pipeline → Running a Fork Correctly](/architecture/serverless#running-a-fork-correctly). + ### Option 2: Static Hosting + Separate API The frontend is a standard Vite SPA — it can be hosted anywhere that serves static files: @@ -67,10 +76,14 @@ npm run build # Upload contents of dist/ to any static host ``` -**But:** You'll need to handle the serverless functions separately. Options: -- Run them as a standalone Python server (see `price-server/` directory) -- Port them to your preferred serverless platform (AWS Lambda, Cloudflare Workers, etc.) -- Skip them entirely — the app works without them, you just lose meta deck data and some price proxying +Since the app is local-only, static hosting is all it needs — there are no +serverless data endpoints to port. Two things to know: + +- The price/meta-deck JSON in `public/` deploys as plain files; run the + `scripts/build_*.py` scripts (cron, CI, or by hand) and redeploy to keep + them fresh — see [Static Data Pipeline](/architecture/serverless) +- The only thing you lose without Vercel is `/api/og` social-embed images; + link previews fall back to the static OG tags ### Option 3: Docker @@ -91,7 +104,7 @@ COPY --from=build /app/dist /usr/share/nginx/html COPY nginx.conf /etc/nginx/conf.d/default.conf ``` -This only serves the frontend. You'd need a separate container or service for the Python API functions. +This serves the full app — the data assets are part of the build output. Only `/api/og` social embeds would be missing (see Option 2). ## Analytics