fix(openapi): reduce large import memory usage

[aryasaatvik]

Summary

Large OpenAPI imports can exhaust a Cloudflare Worker while loading unrelated integration bindings and constructing the full operation catalog. Scope storage reads in SQL and compile operation bindings in chunks before persisting them.

Stage	Behavior
Operation lookup	Query current and legacy integration key prefixes; preserve literal prefix filtering and integration checks, excluding overlapping current keys from legacy results
Generic import	Parse once, persist shared definitions, then compile and persist operation chunks
Refresh response	Report counts and name changes from persistence results without reloading the new bindings

Generic specs still require a full parsed document. The importer uses its existing bounded persistence chunks; the database adapter retains its atomic upsert behavior.

Verification

• bun run format:check and bun run typecheck pass.
• oxlint --deny-warnings passes for every changed source/test file. Full bun run lint retains the same 80 tracked diagnostics as dev, plus local ignored scratchpad findings; no new diagnostics remain.
• TURBO_TEST_CONCURRENCY=3 bun run test --continue=always: 42/43 tasks pass, including 329 OpenAPI tests, 877 SDK tests, and 73 FumaDB tests. Nine self-host suites fail at Better Auth device-authorization initialization with Zod 4.5.4; the same failure reproduces on unmodified dev with the same locked dependencies.
• A 16,548-operation regression verifies both add and update write multiple chunks of at most 500 rows, with complete row totals.
• Cloudflare workerd scenario OpenAPI · large imports report complete counts and preserve unrelated catalogs passes: import 1,201 operations alongside 1,001 unrelated operations, shrink to 1,002, reject invalid input and pathless updates without clearing the catalog, retry, read schemas, and verify the browser catalog.
• Full public DataForSEO YAML: 4,943,574 bytes and 570 operations import and refresh successfully under local workerd and a deployed Cloudflare Worker. The deployed refresh returns HTTP 200 with outcome: ok and all 570 tools; schema lookup also succeeds.

Screenshots

Checklist

• Added a patch changeset.
• Added storage and workerd regression coverage.
• No secrets, credentials, or private data in the diff.

[greptile-apps]

Greptile Summary

This PR reduces memory usage during large OpenAPI imports by narrowing plugin-storage reads to integration-specific prefixes and compiling operation bindings in bounded persistence chunks.

• Preserves current and legacy operation-key compatibility while excluding overlapping rows.
• Computes refresh counts and tool-name changes from persistence results without reloading the new catalog.
• Adds storage, large-catalog, and end-to-end regression coverage.
• The changes since the previous review strengthen batch-size coverage without changing production behavior.

Confidence Score: 5/5

The PR appears safe to merge, with both previous findings resolved and no new actionable issues introduced since the previous review.

Current operation lookup excludes overlap between legacy and hashed prefixes, and the D1 adapter change that threatened atomicity is absent. The latest test-only change preserves the exercised OpenAPI configuration while adding bounded-batch assertions.

Important Files Changed

Filename	Overview
packages/core/sdk/src/executor.ts	Pushes plugin-storage key-prefix filtering into the database query while retaining post-query validation.
packages/plugins/openapi/src/sdk/plugin.ts	Parses generic specifications once, persists operations in chunks, and derives response metadata from persistence results.
packages/plugins/openapi/src/sdk/store.ts	Scopes operation reads to current and legacy key prefixes and prevents overlapping current rows from being duplicated.
packages/plugins/openapi/src/sdk/plugin.test.ts	Verifies that large add and update operations persist through multiple batches capped at 500 entries.
e2e/scenarios/openapi-large-catalog.test.ts	Covers large imports, shrinking catalogs, failed updates, retries, schema reads, isolation, and browser visibility.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Resolve OpenAPI input] --> B[Parse document]
    B --> C[Persist shared spec and definitions]
    C --> D[Compile operation chunk]
    D --> E[Persist bounded batch]
    E --> F{More operations?}
    F -->|Yes| D
    F -->|No| G[Return persisted count and names]
    G --> H[Refresh connection catalogs]

Loading

_{Reviews (3): Last reviewed commit: "test(openapi): enforce bounded import an..." | Re-trigger Greptile}

[aryasaatvik]

The final base-to-head diff contains no changes to either FumaDB adapter file. git diff origin/dev...0fae0d2e4 -- packages/core/fumadb is empty. The adapter's single native batch is the existing atomic behavior on dev, not a new unbounded import path introduced by this PR.

The new regression in 0fae0d2 observes the actual plugin-storage putMany boundary for a 16,548-operation spec. Both addSpec and updateSpec write multiple chunks, every chunk contains at most 500 rows, and the row totals equal the complete catalog. The caller therefore bounds each D1 upsert while the adapter preserves atomicity within that call. The Cloudflare workerd scenario and full deployed 570-operation refresh also pass with the unchanged adapter.

The two earlier findings are fixed and have explanatory replies on their threads: current-prefix exclusion prevents duplicates for the op slug, and reverting the adapter split preserves upsert atomicity. Please assess the final dev-to-head diff with the caller's chunk bound in view.