test: check the real API, and detect the spec drifting under a published version

[robertabbott]

Why

3.0.0 is published. Every suite in this repo proves a request body is legal against a vendored snapshot of the OpenAPI document. None prove the API honors the option, or that the snapshot still matches reality — precisely the gap that let the 2.x types rot for two months while all 18 tests stayed green. With a published version depending on it, that is a live risk rather than a latent one.

tests/live/ closes it from the other side: real requests, validated with the same ajv validators the contract suite uses. A response that stops matching the vendored spec fails here even though nothing in this repo changed — which makes this the upstream drift detector the parity tests cannot be. Those read a pinned snapshot and only move when someone runs pnpm spec:refresh.

Verified load-bearing, not assumed

Adding a bogus required property to SearchResponse in the vendored spec, via a YAML round-trip so the document stays valid:

State	Result
baseline	1 passed
drifted spec	FAIL
restored	1 passed

A first attempt at that proof broke the YAML and silently ran no tests — worth knowing if anyone repeats it, because "no tests" reads as green at a glance.

What the 16 tests cover

All of this shipped in 3.0.0 and had never touched a real API. Run against a live API: 16 passed in 14s.

• A query-only body is accepted and returns the same shape and result count as one restating the old fast/US/en-US defaults. This is the check that matters most — 3.0.0 stopped sending those three on the evidence that the spec documents them, and "the document says X is the default" is a different claim from "the API applies X when the key is absent."
• All ten new search options are accepted, rather than 400ing the way deferDataRetrieval silently did.
• includeDomains / excludeDomains actually filter. An empty result set counts as a pass; a leak is the failure.
• Both contentFormat values are honored and populate the matching payload field.
• quoteOnly returns a price with no content, and an over-ceiling maxRows clamps rather than failing — the behavior the README warns about and nothing could previously verify.
• A live response still validates against the vendored spec — the drift detector itself.

Guardrails, because this suite costs money

• Excluded from pnpm test by vitest.config.ts; included only by vitest.live.config.ts. Verified: the default run is still 8 files / 84 tests with no live file.
• Skips without a key instead of failing. Verified: 16 skipped, 0 failed, no red.
• Never on pull_request. Weekly cron plus manual dispatch only — a fork cannot read secrets, so a PR trigger would fail for every outside contributor.
• Serial, one retry. The tests compare responses across requests and Tako throttles per key.
• No host or token literal committed. The base URL comes from the environment and defaults to the same host the SDK does. Re-grepped to confirm.

Also fixes a defect I shipped last time

lint:package, added in #11, assumed dist/ already existed. CI hid it because pnpm build runs first, but on a fresh checkout it reported five "file does not exist" errors that look like packaging faults and are not — I hit it setting up this worktree. It now builds first, matching test:package.

Verification

• Default suite unchanged: 8 files, 84 tests
• Live suite: 16 passed against a real API; 16 skipped with no key
• Drift detector proven to fail on a drifted spec, then pass again
• typecheck, conformance compile, lint:package, test:package all clean — the last two now from a dist-less tree
• All three workflow YAML files parse
• The scheduled run has not fired yet, and needs TAKO_API_KEY in repo secrets plus LIVE_TESTS_ENABLED=true in repo variables before it will

What this does not do

No routing eval. Tool descriptions changed in #9 and again in #11 with zero measurement; this suite tests the API, not which tool a model picks. Still the largest untested surface, and it needs a fixture set and a model budget rather than a slot in this PR.

nodeIds/strict are covered for acceptance only — real pinning needs ids from /v1/graph/search, which this SDK does not wrap, and the spec 400s on a malformed id.

🤖 Generated with Claude Code