Skip to content

Document the static data pipeline (serverless data endpoints are gone) #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
poketonova wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Document the static data pipeline (serverless data endpoints are gone) #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/.vitepress/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ export default defineConfig({
{ text: 'Project Structure', link: '/architecture/project-structure' },
{ text: 'Data Flow & Persistence', link: '/architecture/data-flow' },
{ text: 'API Integrations', link: '/architecture/api-integrations' },
{ text: 'Serverless Functions', link: '/architecture/serverless' },
{ text: 'Static Data Pipeline', link: '/architecture/serverless' },
{ text: 'Price Snapshot System', link: '/architecture/snapshots' },
{ text: 'Component Guide', link: '/architecture/components' },
],
Expand Down
51 changes: 40 additions & 11 deletions docs/architecture/api-integrations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,18 +114,41 @@ Rarebox pulls data from several external APIs. Each has different characteristic
- Open REST API, no API key required
- 7 sets (Origins, Spiritforged, Unleashed, promo sets), 1000+ cards
- Card images from Riot Games CDN (`cmsassets.rgpub.io`)
- Pagination: 50 cards per page
- **No prices** — card data only. Prices come from PriceCharting, merged variant-aware (see note below)
- `tcgplayer_id` field available for price lookups if needed
- Pagination: 100 cards per page
- **No prices** — card data only. Each card carries a `tcgplayer_id`, which is
what makes exact price joins possible (see note below)
- Results cached in memory for 1 hour

::: note Price Source — variant-aware
riftcodex provides card metadata and images; prices come from PriceCharting. Because PriceCharting caps each search at 100 products, the browse provider runs **three queries per set** (base, `signature`, `alternate art`), de-duplicates, and filters by console name. Variant printings (`(Signature)`, `(Alternate Art)`) only ever take their own bracketed PriceCharting listing — **a variant never falls back to the plain card's price** (no price beats a wrong one). `(Overnumbered)` is the plain printing of an over-set-size champion. Search, trade analyzer, portfolio refresh, and the offline cache all hydrate from this same per-set price map.
::: note Price Source — exact join, variant-aware
riftcodex provides card metadata and images; prices are TCGplayer market
prices from the pre-built `/riftbound-prices.json` static asset (see
[Static Data Pipeline](/architecture/serverless)), joined on each card's
`tcgplayer_id`. Every printing is its own TCGplayer product, so matches are
exact — **a variant (`(Signature)`, `(Alternate Art)`) never falls back to
the plain card's price** (no price beats a wrong one), and promo sets
(PR/OPP/JDG) are fully covered. `(Overnumbered)` is the plain printing of an
over-set-size champion. If the asset yields no prices for a set (e.g. newer
than the last daily refresh), the provider falls back to the old
PriceCharting search (three queries per set around its 100-result cap,
filtered by console name, variant-aware). Search, trade analyzer, portfolio
refresh, and the offline cache all hydrate from this same per-set price map.
:::

## tcgcsv.com

**Purpose:** Daily TCGplayer price dumps — Riftbound singles (category 89) and Japanese Pokémon cards (category 85 "Pokemon Japan").

**Called from:** CI only (`scripts/build_riftbound_prices.py`, `scripts/build_jp_prices.py` via the daily `refresh-data.yml` workflow) — **never the browser**. tcgcsv has no CORS and its terms allow backend scripts only. The app consumes the results as static assets (`/riftbound-prices.json`, `/jp-prices.json`).

**Key details:**
- Requires an identifying `User-Agent` (`Rarebox/1.4 (+https://rarebox.io)`) — generic browser UAs get a 401. Forks must use their own (see [Static Data Pipeline](/architecture/serverless))
- Updates once daily around 20:00 UTC; `last-updated.txt` is the change stamp, and the scripts re-sync only when it moves
- Japanese Pokémon join: tcgcsv group abbreviations match tcgdex set ids, product `Number` ("205/187") matches tcgdex `localId`
- ≤10k requests/day allowed; a full sync of both categories uses a few hundred

## PriceCharting

**Purpose:** Sealed product prices (booster boxes, ETBs, tins), graded slab prices (grade-specific), and market prices for non-Pokémon TCG cards (Magic, Yu-Gi-Oh!, Lorcana, One Piece, Riftbound).
**Purpose:** Sealed product prices (booster boxes, ETBs, tins), graded slab prices (grade-specific), and market prices for non-Pokémon TCG cards (Magic, Yu-Gi-Oh!, Lorcana, One Piece; for Riftbound singles it's now only the fallback when the static price asset has nothing).

**Called from:** Browser (client-side, direct JSON API calls)

Expand Down Expand Up @@ -153,17 +176,23 @@ riftcodex provides card metadata and images; prices come from PriceCharting. Bec

**Purpose:** Current tournament meta deck data — top decks, meta share percentages, championship points.

**Called from:** Vercel serverless function (`/api/search`) and client-side (`metaDecksApi.js`)
**Called from:** CI only (`scripts/build_meta_decks.py` via the daily `refresh-data.yml` workflow); the client (`metaDecksApi.js`) reads the result from the `/meta-decks/{game}.json` static asset

**Key details:**
- Data is **scraped** from Limitless TCG's website using httpx + BeautifulSoup (no official API)
- Core cards are resolved server-side with exact card match (set code + number)
- Client-side service fetches from serverless endpoint with localStorage fallback (24h cache)
- Static fallback decks available when live endpoint is unavailable
- Core cards are resolved in the build script with exact card match (set code + number)
- Client-side service fetches the static asset with localStorage fallback (24h cache)
- Static fallback decks available when the asset is missing or empty for a game
- If Limitless changes their HTML structure, the scraper will break and need updating

::: warning Scraping Dependency
The Limitless TCG integration is the most fragile part of the system. It relies on HTML structure that can change without notice. If meta decks stop loading, the scraper likely needs updating. Check `/api/search.py` first.
The meta-deck scrapers are the most fragile part of the system — they rely on
HTML structure that can change without notice. If meta decks stop loading,
check `scripts/build_meta_decks.py` first. As of mid-2026 only the Pokémon
(Limitless) scraper yields decks: the MTG/One Piece/Yu-Gi-Oh selectors are
stale and the Lorcana/Riftbound sources 403; those games run on the client's
fallback decks. The daily workflow keeps the previous day's file whenever a
scraper fails, so a broken scraper degrades to stale decks, never none.
:::

## Pokellector
Expand Down
199 changes: 108 additions & 91 deletions docs/architecture/serverless.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,99 +1,116 @@
# Serverless Functions

Rarebox is primarily a client-side app, but a few operations need server-side execution. These run as **Vercel Functions** — Python serverless functions in the `api/` directory.

## Why Server-Side?

Three reasons a request needs to go through the server:

1. **CORS restrictions** — some APIs don't allow browser-origin requests
2. **HTML scraping** — Limitless TCG doesn't have a public API; we scrape their site
3. **Response transformation** — some data needs processing before the client can use it (e.g., resolving card set codes to pokemontcg.io IDs)

Everything else is fetched directly from the browser.

## Endpoints

### `GET /api/health`

Simple health check. Returns 200 with a status message. Used for uptime monitoring.

### `GET /api/search`

Scrapes Limitless TCG for current meta deck data. This is the most complex endpoint:

1. Fetches the meta standings page from Limitless
2. Parses HTML with BeautifulSoup to extract top decks, meta share, and CP
3. Resolves core card names to exact pokemontcg.io card IDs (set code + card number)
4. Returns structured JSON with deck names, card lists, and metadata

**Performance:** Can take 5-15 seconds depending on Limitless response time and the number of card resolutions needed. Max duration is 30 seconds (configured in `vercel.json`).

### `GET /api/price`

Price proxy/lookup endpoint. Fetches and transforms price data that can't be accessed directly from the browser.

### `GET /api/sealed`

Fetches sealed product pricing from PriceCharting for items that need server-side access.

## Client-Side Meta Decks

In addition to the serverless endpoint, `metaDecksApi.js` provides:

- **Live fetch** from `/api/search` with localStorage fallback (24h cache)
- **Static fallback decks** when the live endpoint is unavailable
- **Game-specific caching** — each TCG has its own cache key
- **Cache invalidation** — stale fallback data is removed when server data is available

## Runtime Configuration

From `vercel.json`:

```json
{
"functions": {
"api/**/*.py": {
"runtime": "@vercel/python@4.5.0",
"maxDuration": 30
}
}
}
# Static Data Pipeline

Rarebox is a **local-only app**: Vercel serves only code and static assets, and
your device makes every API call itself. There are no serverless data
endpoints — the `api/` directory contains exactly one function, `/api/og`,
which renders social-embed images for link-preview crawlers (Discord,
Telegram, X). The app never calls it.

Some data sources can't be reached from a browser, though:

1. **tcgcsv.com** (daily TCGplayer price dumps) — no CORS, and its terms of
use allow backend scripts only
2. **Meta-deck sites** (Limitless TCG and others) — HTML scraping, no public
API

Instead of proxying these through serverless functions, a daily GitHub
Actions workflow pre-builds them into static JSON in `public/`, which deploys
with the app like any other asset.

## The Assets

| Asset | Built by | Contents |
|-------|----------|----------|
| `public/riftbound-prices.json` | `scripts/build_riftbound_prices.py` | TCGplayer market prices for all Riftbound cards (tcgcsv category 89), keyed by TCGplayer product id, `{ normal, foil }` per product |
| `public/jp-prices.json` | `scripts/build_jp_prices.py` | TCGplayer prices for 16k+ Japanese Pokémon cards (tcgcsv category 85 "Pokemon Japan"), keyed `{tcgdex set id}-{number}` (lowercase, no leading zeros) |
| `public/meta-decks/{game}.json` | `scripts/build_meta_decks.py` | Scraped tournament meta decks per game, cards resolved to ids/images/prices |

## The Workflow

`.github/workflows/refresh-data.yml` in the app repo:

- Runs daily at **21:00 UTC** — an hour after tcgcsv's ~20:00 refresh — with a
**23:00 retry** for days tcgcsv runs late. Also triggerable manually via
`workflow_dispatch`.
- The price scripts compare tcgcsv's `last-updated.txt` stamp against the one
embedded in the committed JSON and **skip the sync when nothing changed**,
so the retry (and any manual run) is a cheap no-op.
- Both price scripts **refuse to overwrite** their output if the pull comes
back suspiciously small — a bad upstream day degrades to stale prices,
never broken prices. The meta-deck script likewise keeps yesterday's file
for any game whose scraper fails.
- Changed files are committed as `github-actions[bot]`; the push to `main`
triggers the normal Vercel static deploy. No change → no commit → no
deploy.
- Robustness rails: a `concurrency` group (runs can't overlap), 15-minute
job timeout, `git pull --rebase` before push (a concurrent push to `main`
doesn't lose the refresh), and a fork guard (below).

## Running a Fork Correctly

The workflow job is gated:

```yaml
if: github.repository == 'novaoc/rarebox'
```

- **Runtime:** Python via `@vercel/python@4.5.0`
- **Max duration:** 30 seconds per invocation
- **Dependencies:** `httpx` (async HTTP) and `beautifulsoup4` (HTML parsing), specified in `requirements.txt`

## Developing Locally

Vercel Functions can be tested locally using the Vercel CLI:
Forks inherit the workflow file, but their copy no-ops — otherwise every
enabled fork would hit tcgcsv and the deck sites with traffic identified as
Rarebox. **A fork still works without it**: the committed JSON ships with the
clone, just frozen at fork time.

To make your fork self-refresh:

1. **Edit the guard** in `.github/workflows/refresh-data.yml` to your
`owner/repo`.
2. **Change the `User-Agent`** in `scripts/build_riftbound_prices.py` and
`scripts/build_jp_prices.py` to identify *your* deployment. tcgcsv
requires an identifying UA — generic browser UAs get a 401 — and its
other rules (re-sync only on stamp change, ≤10k requests/day) are already
handled by the scripts.
3. **Enable the workflow** in your fork's Actions tab. GitHub disables
Actions in forks until you opt in, and inherited scheduled workflows stay
disabled even after that — both are deliberate, so nothing runs by
surprise.

Note that GitHub also auto-disables scheduled workflows in public repos
after 60 days without repository activity; the daily data commits normally
keep the timer reset on an active deployment.

## Client Consumption

The app fetches the assets same-origin (works in plain `vite dev` too —
Vite serves `public/` at the root):

- **`providers.js`** (Riftbound browse) joins `/riftbound-prices.json` on
each card's `tcgplayer_id` — exact per-printing matches, so a Signature
can never inherit its plain card's price. If the asset yields zero prices
for a set (e.g. a set newer than the last refresh), it falls back to the
old PriceCharting search.
- **`pokemonApi.js`** reads `/jp-prices.json` for Japanese card grids,
detail views, and portfolio refresh, falling back to tcgdex's Cardmarket
EUR data where TCGplayer has nothing.
- **`metaDecksApi.js`** fetches `/meta-decks/{game}.json` with a 24h
localStorage cache and built-in fallback decks when the asset is missing
or empty for a game.

## Running the Scripts Locally

```bash
npm i -g vercel
vercel dev
pip install httpx beautifulsoup4 # meta decks only; price scripts are stdlib
python3 scripts/build_riftbound_prices.py
python3 scripts/build_jp_prices.py
python3 scripts/build_meta_decks.py [game ...]
```

This starts a local dev server that emulates the serverless function runtime. API endpoints are available at `http://localhost:3000/api/*`.

## Caching Recommendations
Each writes into `public/` and prints what it did. The price scripts exit
early with "up to date" unless tcgcsv has refreshed since the committed
stamp.

Currently, API responses aren't cached server-side. For production traffic, consider:
## History

- **Quick win:** Add `Cache-Control: s-maxage=3600` headers to responses. Vercel's CDN caches them automatically for the specified duration.
- **Meta decks:** Could use `s-maxage=86400` (24h) since tournament data changes daily at most.
- **Price lookups:** `s-maxage=3600` (1h) is a reasonable default.

```python
# Example: adding cache headers to a response
from http.server import BaseHTTPRequestHandler

class handler(BaseHTTPRequestHandler):
def do_GET(self):
# ... fetch and process data ...
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.send_header('Cache-Control', 's-maxage=3600')
self.end_headers()
self.wfile.write(json.dumps(result).encode())
```
Until mid-2026 Rarebox ran Python serverless functions on Vercel
(`/api/search`, `/api/price`, `/api/sealed`, `/api/health`, `/api/meta-decks`)
for scraping and price proxying. They were removed in favor of this pipeline
— the local-only rule means the only thing a server should do is hand the
device files.
47 changes: 30 additions & 17 deletions docs/reference/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,19 @@ Rarebox is deployed on **Vercel** with automatic deployments from the `main` bra

### How It Works

1. Push to `main` on GitHub
1. Push to `main` on GitHub — by you, or by the daily
[data-refresh workflow](/architecture/serverless) committing fresh price
and meta-deck JSON into `public/`
2. Vercel detects the push and starts a build
3. `npm run build` runs Vite, outputting static files to `dist/`
4. Python serverless functions in `api/` are bundled with `@vercel/python@4.5.0`
5. Static files are deployed to Vercel's CDN
6. Serverless functions are deployed as edge functions
7. Live at [rarebox.io](https://rarebox.io) within ~60 seconds
4. Static files are deployed to Vercel's CDN
5. The one remaining Python function, `api/og.py` (social-embed images for
crawlers — the app never calls it), is bundled with `@vercel/python@4.5.0`
6. Live at [rarebox.io](https://rarebox.io) within ~60 seconds

The app itself is **local-only**: Vercel hands the device code and static
assets, and the device makes every data call itself. There are no serverless
data endpoints.

### `vercel.json` Configuration

Expand All @@ -22,10 +28,6 @@ Rarebox is deployed on **Vercel** with automatic deployments from the `main` bra
"outputDirectory": "dist",
"framework": "vite",
"rewrites": [
{ "source": "/api/health", "destination": "/api/health" },
{ "source": "/api/search", "destination": "/api/search" },
{ "source": "/api/price", "destination": "/api/price" },
{ "source": "/api/sealed", "destination": "/api/sealed" },
{ "source": "/((?!api/).*)", "destination": "/index.html" }
],
"functions": {
Expand All @@ -38,8 +40,7 @@ Rarebox is deployed on **Vercel** with automatic deployments from the `main` bra
```

**Rewrites explained:**
- API routes (`/api/*`) pass through to serverless functions
- Everything else falls through to `/index.html` — this is what makes Vue Router's HTML5 history mode work (clean URLs without hash fragments)
- Everything except `/api/og` falls through to `/index.html` — this is what makes Vue Router's HTML5 history mode work (clean URLs without hash fragments). Static files in `public/` (including the pre-built price/meta-deck JSON) are served before rewrites apply.

### DNS

Expand All @@ -55,9 +56,17 @@ Want to run your own instance? Here's what you need.
1. Fork `novaoc/rarebox` on GitHub
2. Create a new project on [vercel.com](https://vercel.com)
3. Import your fork
4. Vercel auto-detects the Vite framework and Python functions
4. Vercel auto-detects the Vite framework (and the one `api/og.py` function)
5. Deploy — no configuration needed

Your fork ships with working price and meta-deck data: the JSON assets in
`public/` are committed to the repo, just frozen at fork time. To keep them
refreshing daily, configure the data-refresh workflow — edit its
`if: github.repository == ...` guard to your fork, set your own `User-Agent`
in `scripts/build_*_prices.py` (tcgcsv requires an identifying UA), and
enable the workflow in your fork's Actions tab. Full steps in
[Static Data Pipeline → Running a Fork Correctly](/architecture/serverless#running-a-fork-correctly).

### Option 2: Static Hosting + Separate API

The frontend is a standard Vite SPA — it can be hosted anywhere that serves static files:
Expand All @@ -67,10 +76,14 @@ npm run build
# Upload contents of dist/ to any static host
```

**But:** You'll need to handle the serverless functions separately. Options:
- Run them as a standalone Python server (see `price-server/` directory)
- Port them to your preferred serverless platform (AWS Lambda, Cloudflare Workers, etc.)
- Skip them entirely — the app works without them, you just lose meta deck data and some price proxying
Since the app is local-only, static hosting is all it needs — there are no
serverless data endpoints to port. Two things to know:

- The price/meta-deck JSON in `public/` deploys as plain files; run the
`scripts/build_*.py` scripts (cron, CI, or by hand) and redeploy to keep
them fresh — see [Static Data Pipeline](/architecture/serverless)
- The only thing you lose without Vercel is `/api/og` social-embed images;
link previews fall back to the static OG tags

### Option 3: Docker

Expand All @@ -91,7 +104,7 @@ COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
```

This only serves the frontend. You'd need a separate container or service for the Python API functions.
This serves the full app — the data assets are part of the build output. Only `/api/og` social embeds would be missing (see Option 2).

## Analytics

Expand Down