pretape — the on-chain equity tape

English · 한국어

Collects and serves on-chain prices of tokenized equities on Solana, in two sections:

• Pre-IPO (6 assets / 11 mints: SpaceX, OpenAI, Anthropic, Anduril, Polymarket, Kalshi): the same company trades at materially different prices across issuers (e.g. OpenAI PreStocks vs Tessera ≈ 34% spread, 2026-08-13), no public reference price exists, and nobody else shows that in one place.
• Tokenized stocks vs exchange (75 assets / 156 mints, every multi-issuer public equity in the tokens.xyz registry: AAPL, TSLA, NVDA…): each token is measured against the real exchange price of its underlying stock (Yahoo, 74 anchor tickers), producing a premium/discount series. Tokens trade 24/7 while exchanges don't — the off-hours premium is on-chain price discovery that no exchange feed contains.

One Bun process does everything: polls DexScreener per mint (pre-IPO every 60s, public equities every 5th cycle), polls Yahoo per anchor ticker every 5 min, writes raw pairs + liquidity-weighted aggregate ticks + underlying ticks to SQLite, serves the JSON API and the live board over Hono.

The time series is the product. DexScreener has no historical API — every minute the collector is down is data that can never be recovered. The premium series doubly so: it needs both legs captured at the same moment. Deploy first, polish later.

Provenance

Built out of a fork of tokens.xyz (MIT, Solana Foundation): data/preipo-seed.json (pre-IPO) and data/equity-seed.json (multi-issuer public equities) are slices of its hand-curated asset registry — the mint↔issuer↔redeemability identity graph that makes the cross-issuer comparison possible. Underlying tickers and split scales in the equity seed are curated here and live-verified by scripts/verify-seed.ts. See LICENSE for third-party notices. The original build history lives in the fork's apps/preipo commits.

Quick start

bun install
bun run dev                        # collector + API on :8080, db at ./dev.db
curl localhost:8080/healthz
sqlite3 dev.db "SELECT variant_id, price_usd, status FROM variant_ticks ORDER BY id DESC LIMIT 11"
bun test src                       # unit + fixture + integration tests

Env vars (see .env.example): PREIPO_DB_PATH, PREIPO_POLL_INTERVAL_MS, PREIPO_PUBLIC_EVERY_N_CYCLES, PREIPO_STOCK_POLL_INTERVAL_MS, PREIPO_PORT.

Data model (SQLite, WAL)

table	what	retention
polls	one row per collection cycle (per-mint fetch stats, errors)	forever
raw_pairs	every pair DexScreener returned, all numeric fields — enough to re-run any future aggregation	90 days (nightly prune)
raw_responses	gzip'd {mint: raw body} map every 15th cycle — schema-drift insurance	forever
variant_ticks	aggregated price per variant per cycle, labeled with algo_version	forever
underlying_ticks	anchor exchange price per ticker per stock cycle (Yahoo v8 chart)	forever

A whole cycle is written in one transaction. Changing any aggregation threshold ⇒ bump ALGO_VERSION in src/aggregate/aggregate.ts; old ticks stay labeled, and raw_pairs can be replayed into new-version ticks.

Aggregation (v1)

Per mint: keep Solana pairs where the queried mint is the base token (kills meme-coin pairs where it's the quote) → require finite price and liquidity ≥ $1k → drop pairs outside ±30% of the liquidity-weighted median (kills broken pools like the $0.56 SPACEX pool) → price = liquidity-weighted mean of survivors. Zero base pairs → no_pairs (true for Ondo SPCXon today); everything filtered → all_filtered.

Premium vs the underlying (public equities only)

premium = token_price / (anchor_price × anchor_scale) − 1, computed at read time from the two stored series — never persisted, so both legs stay replayable. anchor_scale covers split drift (Netflix split 10:1 in Nov 2025; the tokens still represent one pre-split share, so NFLX anchors at ×10). Outside US market hours the anchor is the last close and the premium reads as "what the chain thinks the stock is worth while the exchange is shut." sk-hynix has no USD anchor (KRX listing) and is tracked spread-only.

Anchor source: Yahoo v8 chart API, no key, one request per ticker per stock cycle with a browser UA. Ticker map + split scales are curated in data/extract-equity-seed.ts and checked by bun scripts/verify-seed.ts (anchors must resolve in USD; |premium| > 20% flags a wrong mapping).

DexScreener facts (verified 2026-08-13, do not re-learn the hard way)

• GET /latest/dex/tokens/{mint} — free, no key, ~300 req/min limit. We use ~11 req/min.
• Every response caps at 30 pairs total. Batching several mints into one call silently drops pairs — that's why the poller fetches per mint.
• priceUsd is a string. Fields go missing. Parse tolerantly.
• Responses include meme pairs where our mint is the quote side, and broken pools 99% away from the real price. Both are filtered at aggregation but kept in raw_pairs.

Deploy (Fly.io, single machine + volume + Litestream→R2)

One-time setup:

fly launch --no-deploy --copy-config   # uses fly.toml; pick an app name/org
fly volumes create preipo_data --size 10 --region iad
# Cloudflare R2: create bucket (e.g. preipo-backup) + S3 API token, then:
fly secrets set \
  LITESTREAM_BUCKET=preipo-backup \
  LITESTREAM_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com \
  LITESTREAM_ACCESS_KEY_ID=... \
  LITESTREAM_SECRET_ACCESS_KEY=...
fly deploy
fly scale count 1                      # SQLite = exactly one writer, ever

Verify after deploy:

curl https://<app>.fly.dev/healthz     # degraded:false within ~2 min
fly logs                               # litestream "replicating to" line

degraded covers both legs of the premium series: collectorDegraded (no successful DexScreener poll in 5 min) OR anchorsDegraded (no underlying tick written in 3 stock cycles — e.g. Yahoo started blocking). Alert on it; the endpoint itself always answers 200 while the process lives.

Without the Litestream secrets the entrypoint still runs the collector (backup replication is skipped with a loud log line) — set them ASAP.

auto_stop_machines = "off" is load-bearing: the collector must never sleep. Deploys use strategy = "immediate" (seconds of downtime beats two SQLite writers). kill_timeout = 30s gives the in-flight cycle time to flush.

Restore drill (do this once for real)

litestream restore -config litestream.yml -o /tmp/restored.db /data/preipo.db
sqlite3 /tmp/restored.db "SELECT count(*), max(ts_ms) FROM variant_ticks"

If the volume dies, the entrypoint auto-restores from R2 on next boot (litestream restore -if-db-not-exists).

Chores

• Seed refresh (manual, occasionally): the upstream tokens.xyz registry gains new mints. data/extract-preipo-seed.ts and data/extract-equity-seed.ts only run inside a tokens.xyz repo checkout — copy them to <tokens-repo>/apps/preipo/data/, run them there with bun, and bring the refreshed JSON back here. Then run bun scripts/verify-seed.ts (live-checks every anchor ticker and flags split drift) before committing.
• Prune runs automatically (raw_pairs > 90 days, once per day).

Known limits

• The 30-pair cap can hide pools beyond the top 30 even per mint (SPCX already returns exactly 30). Liquidity-weighting makes this mostly harmless; the newer token-pairs/v1 endpoint is the escape hatch if it stops being so.
• Ondo (SPCXon) and Tessera venues without DEX pairs show no_pairs — a Jupiter price fallback is the planned M3 fix (new source column).
• Cross-issuer spreads mix instruments with different per-token share units (SpaceX PreStocks vs xStock ≈ 3.7×) and different redeemability tiers. The board labels tiers and never normalizes units; unit research is M3.