diff --git a/CLAUDE.md b/CLAUDE.md index b9b15bd..43bcbd3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,9 +6,9 @@ Frontend on :3500, backend on :3501, Mastra Studio on :4111, Convex dashboard on ## Setup -1. Copy `.env.example` to `.env` and fill in your keys: +1. Local mode creates `.env` automatically and collects TinyFish + LLM provider credentials in the setup UI. Production still uses env keys: - `TINYFISH_API_KEY` — from https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2 - - `OPENROUTER_API_KEY` — from https://openrouter.ai/settings/keys + - `OPENROUTER_API_KEY` — production default LLM provider key - `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` — from Clerk API Keys - `CLERK_SECRET_KEY` — from Clerk API Keys - `CLERK_JWT_ISSUER_DOMAIN` — your Frontend API URL (e.g. `https://your-app.clerk.accounts.dev`) @@ -26,9 +26,9 @@ Frontend uses Convex React hooks (`useQuery`, `useMutation`) with `ConvexProvide Backend is Fastify + Mastra. Fastify serves the HTTP API (Clerk JWT auth on protected routes via `backend/src/clerk-auth.ts`). Mastra (`backend/src/mastra/`) is the workflow orchestration layer — it wraps pipelines into inspectable workflows with a Studio UI. Both run as separate Docker services sharing the same source code. -The schema inference pipeline: frontend calls `POST /infer-schema` → Fastify verifies the Clerk JWT → calls `inferSchema()` in `backend/src/pipeline/schema-inference.ts` → Claude Sonnet 4.6 via OpenRouter → returns a Zod-validated `DatasetSchema` → frontend maps it to editable columns in the wizard. +The schema inference pipeline: frontend calls `POST /infer-schema` → Fastify verifies the Clerk JWT → calls `inferSchema()` in `backend/src/pipeline/schema-inference.ts` → selected LLM provider/model → returns a Zod-validated `DatasetSchema` → frontend maps it to editable columns in the wizard. -The populate pipeline: frontend calls `POST /populate` with `{ datasetId, datasetName, description, columns }` → Fastify verifies the Clerk JWT → triggers `populateWorkflow` which: (1) clears existing rows, (2) builds a prompt from the schema, (3) runs the populate agent (Claude Sonnet 4.6) which searches the web via TinyFish APIs, then inserts rows into Convex one by one. Rows appear in realtime on the frontend via Convex reactive queries. +The populate pipeline: frontend calls `POST /populate` with `{ datasetId, datasetName, description, columns }` → Fastify verifies the Clerk JWT → triggers `populateWorkflow` which: (1) clears existing rows, (2) builds a prompt from the schema, (3) runs the populate agent using the selected LLM provider/model. The agent searches the web via TinyFish APIs, then inserts rows into Convex one by one. Rows appear in realtime on the frontend via Convex reactive queries. Convex functions use `ctx.auth.getUserIdentity()` to get the authenticated user. The `ownerId` field on datasets stores `identity.subject` (Clerk user ID). Do not pass `ownerId` from the client. @@ -36,7 +36,7 @@ Convex functions use `ctx.auth.getUserIdentity()` to get the authenticated user. Root `.env` is the only local env file. Docker Compose, package scripts, and Convex CLI helper targets all read it. Key variables: - `TINYFISH_API_KEY` — used by the populate agent for web search and fetch (get one at https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2) -- `OPENROUTER_API_KEY` — used by backend and Mastra for AI model calls +- `OPENROUTER_API_KEY` — production default LLM provider key; local mode can use OpenRouter, OpenAI, Anthropic, or custom OpenAI-compatible via setup UI - `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY`, `CLERK_SECRET_KEY` — shared by frontend and backend - `CONVEX_SELF_HOSTED_ADMIN_KEY` — used by backend for system-level Convex writes diff --git a/README.md b/README.md index 3be15fd..7537cf1 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ On first launch, BigSet sends you to setup. You'll connect two services: | Service | What it's for | Get your key | |---------|--------------|-------------| | **TinyFish** | Web search + page fetching | [tinyfish.ai/api-keys](https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2) | -| **OpenRouter** | LLM calls (schema inference + agents) | [openrouter.ai/settings/keys](https://openrouter.ai/settings/keys) | +| **LLM provider** | Schema inference + agents | OpenRouter, OpenAI, Anthropic, or custom OpenAI-compatible | Local API keys are stored in your OS keychain. @@ -214,23 +214,22 @@ Once everything is ready, you'll see: | **Mastra Studio** (workflow inspector) | [localhost:4111](http://localhost:4111) | Open [localhost:3500](http://localhost:3500). The setup screen will ask for -TinyFish and OpenRouter credentials and save them to your OS keychain for this -workspace. +TinyFish credentials plus an LLM provider (OpenRouter, OpenAI, Anthropic, or a +custom OpenAI-compatible endpoint) and save local keys to your OS keychain for +this workspace. -### Step 3: Connect TinyFish and OpenRouter +### Step 3: Connect TinyFish and an LLM provider -TinyFish powers web search and page fetching. OpenRouter routes LLM calls to -the models BigSet uses for schema inference and agents. +TinyFish powers web search and page fetching. Your LLM provider powers schema +inference and dataset-building agents. 1. Create a TinyFish key at [agent.tinyfish.ai/api-keys](https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2) -2. Create an OpenRouter key at [openrouter.ai/settings/keys](https://openrouter.ai/settings/keys) -3. Paste both into BigSet's setup screen - -OpenRouter is pay-as-you-go; $5-10 is plenty to start. +2. Choose an LLM provider: OpenRouter, OpenAI, Anthropic, or custom OpenAI-compatible +3. Paste the provider key and model name into BigSet's setup screen > **Note:** root `.env` is the only local env file. If you edit Convex functions in `frontend/convex/`, run `make convex-push` to deploy the changes. -> **Free tier:** cloud signed-in accounts get **2,500 row operations per calendar month** (resets on the 1st, UTC). Local mode bypasses the cloud quota and uses your TinyFish/OpenRouter accounts directly. +> **Free tier:** cloud signed-in accounts get **2,500 row operations per calendar month** (resets on the 1st, UTC). Local mode bypasses the cloud quota and uses your TinyFish + LLM provider accounts directly. ### Step 4 (optional): Load curated datasets @@ -257,7 +256,7 @@ This is idempotent; safe to run multiple times. 7. **Configures Convex auth** — sets `BIGSET_LOCAL_MODE=1` for the local app. 8. **Deploys Convex schema** — pushes the table schema and functions from `frontend/convex/` to the running instance. 9. **Starts remaining services** — brings up the frontend, backend, and Mastra. These read the now-populated `.env` including the admin key. -10. **Streams logs** — tails all container logs so you can see what's happening. `Ctrl+C` to stop watching (containers keep running). +10. **Streams app logs** — tails frontend, backend, and Mastra logs. `Ctrl+C` to stop watching (containers keep running). Convex logs are still available with `docker compose -f docker-compose.dev.yml logs -f convex`. ### Commands @@ -311,7 +310,7 @@ If you want a completely fresh start: `make clean` then `make dev`. | Auth | Local auth (dev); [Clerk](https://clerk.com) (cloud) | | Database | [Convex](https://convex.dev) (self-hosted) | | Data Collection | [TinyFish](https://www.tinyfish.ai?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2) APIs (Search, Fetch, Browser) | -| AI orchestration | [Mastra](https://mastra.ai) workflows + [Vercel AI SDK](https://sdk.vercel.ai) + [OpenRouter](https://openrouter.ai) → Claude Sonnet (schema inference + populate agent) | +| AI orchestration | [Mastra](https://mastra.ai) workflows + [Vercel AI SDK](https://sdk.vercel.ai) + local LLM provider (OpenRouter, OpenAI, Anthropic, or custom OpenAI-compatible) | | Table view | [TanStack Table](https://tanstack.com/table) + [react-window](https://github.com/bvaughn/react-window) virtualization | | Exports | CSV (built-in) + XLSX ([SheetJS](https://sheetjs.com), dynamic-imported) | | Analytics | [PostHog](https://posthog.com) — events, session replay, error tracking (optional) | diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 53218a0..741b444 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -15,7 +15,7 @@ To add a new protected route, register it inside the scoped plugin in `src/index ## Schema Inference Pipeline -`src/pipeline/schema-inference.ts` — takes a natural language prompt and returns a structured `DatasetSchema` (Zod-validated, defined in `src/pipeline/types.ts`). Uses Claude Sonnet 4.6 via OpenRouter (`@openrouter/ai-sdk-provider` + Vercel AI SDK). Auto-retries once on validation failure by feeding the error back to the model. +`src/pipeline/schema-inference.ts` — takes a natural language prompt and returns a structured `DatasetSchema` (Zod-validated, defined in `src/pipeline/types.ts`). Uses the configured LLM provider/model via `src/config/llm.ts` + Vercel AI SDK. Auto-retries once on validation failure by feeding the error back to the model. The pipeline is a pure function (`inferSchema(prompt) → DatasetSchema`). It is called by both Fastify (for the HTTP API) and Mastra (for workflow orchestration). @@ -26,7 +26,7 @@ The pipeline is a pure function (`inferSchema(prompt) → DatasetSchema`). It is - `src/mastra/index.ts` — registers workflows with the `Mastra` instance (the populate agent is built per-run, not registered as a singleton) - `src/mastra/workflows/infer-schema.ts` — `inferSchemaWorkflow`, a single-step workflow wrapping `inferSchema()` - `src/mastra/workflows/populate.ts` — `populateWorkflow`, 3-step workflow: clear rows → build prompt → run populate agent -- `src/mastra/agents/populate.ts` — `buildPopulateAgent(authorizedDatasetId, authContext, columns)`, builds the orchestrator agent (Claude Sonnet 4.6) with 3 tools: `search_web`, `fetch_page`, `investigate_row`. No write access — all inserts go through investigate subagents. +- `src/mastra/agents/populate.ts` — `buildPopulateAgent(authorizedDatasetId, authContext, columns)`, builds the orchestrator agent with the configured LLM provider/model and 3 tools: `search_web`, `fetch_page`, `investigate_row`. No write access — all inserts go through investigate subagents. - `src/mastra/agents/investigate.ts` — `buildInvestigateAgent(authorizedDatasetId, authContext, columns)`, builds a per-entity subagent with `insert_row`, `list_rows`, `search_web`, `fetch_page`. Researches one entity, inserts at most one row, returns structured feedback (`INSERTED/SUMMARY/CLUES/REASON`). - `src/mastra/tools/investigate-tool.ts` — `buildInvestigateTool(authorizedDatasetId, authContext, columns)` creates the `investigate_row` tool. The orchestrator calls it to hand off a lead; it spawns a fresh investigate agent, runs it (maxSteps: 25), parses the structured output, and returns it to the orchestrator. Errors are caught and returned as structured failures so the orchestrator can self-correct. - `src/mastra/tools/dataset-tools.ts` — `buildPopulateTools(authorizedDatasetId, authContext)` factory returning 5 Convex-backed tools: `insert_row`, `list_rows`, `get_row`, `update_row`, `delete_row`. The dataset id is captured by closure so the LLM cannot redirect writes to other datasets; `authContext` (Clerk userId + workflow run id) is captured for caller-attribution in security logs and the `CAPABILITY_VIOLATION` PostHog event. See the security note at the top of the file. @@ -48,7 +48,7 @@ Required env vars (see `.env.example`): - `CONVEX_URL` — Convex instance URL - `CONVEX_SELF_HOSTED_ADMIN_KEY` — for system-level Convex writes (internal mutations) - `CLERK_SECRET_KEY`, `CLERK_PUBLISHABLE_KEY` — for JWT verification -- `OPENROUTER_API_KEY` — for AI model calls +- `OPENROUTER_API_KEY` — production default LLM provider key; local mode can use OpenRouter, OpenAI, Anthropic, or custom OpenAI-compatible via setup UI - `TINYFISH_API_KEY` — for web search and fetch (populate agent). Get one at https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2 In Docker, these are interpolated from the root `.env` file via `docker-compose.dev.yml`. diff --git a/backend/package-lock.json b/backend/package-lock.json index e231b48..5feeb3d 100644 --- a/backend/package-lock.json +++ b/backend/package-lock.json @@ -8,6 +8,20 @@ "name": "bigset-backend", "version": "0.1.0", "dependencies": { + "@ai-sdk/alibaba": "^1.0.26", + "@ai-sdk/anthropic": "^3.0.81", + "@ai-sdk/deepinfra": "^2.0.52", + "@ai-sdk/deepseek": "^2.0.35", + "@ai-sdk/fireworks": "^2.0.53", + "@ai-sdk/google": "^3.0.80", + "@ai-sdk/groq": "^3.0.39", + "@ai-sdk/huggingface": "^1.0.50", + "@ai-sdk/mistral": "^3.0.37", + "@ai-sdk/openai": "^3.0.68", + "@ai-sdk/openai-compatible": "^2.0.48", + "@ai-sdk/provider": "^3.0.10", + "@ai-sdk/togetherai": "^2.0.53", + "@ai-sdk/xai": "^3.0.93", "@clerk/backend": "^3.4.11", "@fastify/cors": "^11.0.0", "@mastra/core": "^1.36.0", @@ -18,6 +32,8 @@ "dotenv": "^16.4.0", "fastify": "^5.0.0", "fastify-plugin": "^5.1.0", + "just-bash": "^3.0.1", + "playwright-core": "^1.60.0", "posthog-node": "^5.35.1", "resend": "^6.12.3", "zod": "^4.4.3" @@ -57,6 +73,89 @@ } } }, + "node_modules/@ai-sdk/alibaba": { + "version": "1.0.26", + "resolved": "https://registry.npmjs.org/@ai-sdk/alibaba/-/alibaba-1.0.26.tgz", + "integrity": "sha512-8j4QPWKDTraUlXh3+6AotaLA4CewOdLMBCtk8SATWpjwSSCfrUS8ExGEuwNSXx1eLIDiGVjwIb+hbtZL1e6fLw==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/openai-compatible": "2.0.48", + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/anthropic": { + "version": "3.0.81", + "resolved": "https://registry.npmjs.org/@ai-sdk/anthropic/-/anthropic-3.0.81.tgz", + "integrity": "sha512-B1JDd9Ugq9R5AgIaW3674lhGCMMYJcPUxnrZh8fzbGojgg4QvHFRv6eZahGQAUsmGHbcf74G9bdSBDLWQGY2GA==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/deepinfra": { + "version": "2.0.52", + "resolved": "https://registry.npmjs.org/@ai-sdk/deepinfra/-/deepinfra-2.0.52.tgz", + "integrity": "sha512-/S4WchqBHlZr0wZmpWfDafM6omNz5i1tIo7WeQ6Hd5y/Jz1le1uWW0AEb1J0ehw+uYMEHc/bLA9RMBf1zehIOg==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/openai-compatible": "2.0.48", + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/deepseek": { + "version": "2.0.35", + "resolved": "https://registry.npmjs.org/@ai-sdk/deepseek/-/deepseek-2.0.35.tgz", + "integrity": "sha512-9DhYurbAvcurOEGN6u2myYDybrrzGfcrkG8hwmFjwTrePW6KCMggm0YxP7e8RkLYcQKqCEMgFlyEB4BM6EmiKg==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/fireworks": { + "version": "2.0.53", + "resolved": "https://registry.npmjs.org/@ai-sdk/fireworks/-/fireworks-2.0.53.tgz", + "integrity": "sha512-HjeiGsdxSzrCkOf2l2V+K+opzlqxBtduBq6BCiohAdgQk2KdZmI/67SMkBM6Kdze/BjUXiZlv0d7zNICPhxVDA==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/openai-compatible": "2.0.48", + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/@ai-sdk/gateway": { "version": "3.0.116", "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-3.0.116.tgz", @@ -74,6 +173,103 @@ "zod": "^3.25.76 || ^4.1.8" } }, + "node_modules/@ai-sdk/google": { + "version": "3.0.80", + "resolved": "https://registry.npmjs.org/@ai-sdk/google/-/google-3.0.80.tgz", + "integrity": "sha512-5ORbm/yFUPO0MEvZsxBMN0cdKw2+lwU/wVn5KN3KF8Dmk1LughuDuUohMh/7iU/XFTiyB0OvmTW/tdV/J7O9zg==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/groq": { + "version": "3.0.39", + "resolved": "https://registry.npmjs.org/@ai-sdk/groq/-/groq-3.0.39.tgz", + "integrity": "sha512-BZAr6DjCbzWQ0Qn1/TSsHo/bmCt4JaAMb4A7HCSUZBQCAcOjne/03D0sVjHnQhUC3TpwcmYiv7tHAviK7BluRw==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/huggingface": { + "version": "1.0.50", + "resolved": "https://registry.npmjs.org/@ai-sdk/huggingface/-/huggingface-1.0.50.tgz", + "integrity": "sha512-2qn7UAP4q2YrQstKtyRSJro8ujicwgEgPChKMiTbTnAH7SDagI2TfciU+hC9adOP1dM9HDqFP+lVnl4+VBv6Aw==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/openai-compatible": "2.0.48", + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4" + } + }, + "node_modules/@ai-sdk/mistral": { + "version": "3.0.37", + "resolved": "https://registry.npmjs.org/@ai-sdk/mistral/-/mistral-3.0.37.tgz", + "integrity": "sha512-KkdaMjs4C2y+vrZWJE990E3ZxBFiOTHQ94ZlquuIttpphcqJTMxNoIpnKT/4UzMVWXL0BUEE2vs+1UEVXkN8Kg==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/openai": { + "version": "3.0.68", + "resolved": "https://registry.npmjs.org/@ai-sdk/openai/-/openai-3.0.68.tgz", + "integrity": "sha512-FCs/DPr4M95UyZ/ABHJmTmCEYRCka/4J0Bna0nsd78QCdGIS0X/zhn+fVzB7mZJo7464uOWYUjROx9PGNGOb0w==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/openai-compatible": { + "version": "2.0.48", + "resolved": "https://registry.npmjs.org/@ai-sdk/openai-compatible/-/openai-compatible-2.0.48.tgz", + "integrity": "sha512-z9MC6M4Oh/yUY/F/eszOtO8wc2nMz99XmZQKd2gWTtyIfe716xTfrKe3aYZKg20NZDtyjqPPKPSR+wqz7q1T7Q==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/@ai-sdk/provider": { "version": "3.0.10", "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-3.0.10.tgz", @@ -177,6 +373,40 @@ "node": ">=18" } }, + "node_modules/@ai-sdk/togetherai": { + "version": "2.0.53", + "resolved": "https://registry.npmjs.org/@ai-sdk/togetherai/-/togetherai-2.0.53.tgz", + "integrity": "sha512-M/qsqM1HMlFpWHxHTEorENCLFmBefwxhGTB+XVsjUh77gfyt1io8eg96c/CyWZTxCPa5GxLfl9mhPxp8wjWATg==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/openai-compatible": "2.0.48", + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/xai": { + "version": "3.0.93", + "resolved": "https://registry.npmjs.org/@ai-sdk/xai/-/xai-3.0.93.tgz", + "integrity": "sha512-HxazLIcSTgI0UQoq6ua0rcSR8+eXuNy0Qh4jkCY9EAWedYn6CQ0XD/j34U4/JYtO738xJdY+tE95okdqWqXkHA==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/openai-compatible": "2.0.48", + "@ai-sdk/provider": "3.0.10", + "@ai-sdk/provider-utils": "4.0.27" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/@babel/code-frame": { "version": "7.29.0", "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz", @@ -650,6 +880,16 @@ "node": ">=6.9.0" } }, + "node_modules/@borewit/text-codec": { + "version": "0.2.2", + "resolved": "https://registry.npmjs.org/@borewit/text-codec/-/text-codec-0.2.2.tgz", + "integrity": "sha512-DDaRehssg1aNrH4+2hnj1B7vnUGEjU6OIlyRdkMd0aUdIUvKXrJfXsy8LVtXAy7DRvYVluWbMspsRhz2lcW0mQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, "node_modules/@clack/core": { "version": "1.3.1", "resolved": "https://registry.npmjs.org/@clack/core/-/core-1.3.1.tgz", @@ -1380,6 +1620,48 @@ "node": ">=12" } }, + "node_modules/@jitl/quickjs-ffi-types": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/@jitl/quickjs-ffi-types/-/quickjs-ffi-types-0.32.0.tgz", + "integrity": "sha512-v9T+GQpmk43VDJ7d72sf0Nexhk+ArvtUihW27dy7lqAl0zBObFKtSBBIm5RBjwIhE8VwsPPm9PNuvPvNqLWUEg==", + "license": "MIT" + }, + "node_modules/@jitl/quickjs-wasmfile-debug-asyncify": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/@jitl/quickjs-wasmfile-debug-asyncify/-/quickjs-wasmfile-debug-asyncify-0.32.0.tgz", + "integrity": "sha512-EX8zbXwGqCgAE764M+qvkHtyXDi/FUoMBea0JnES7vCM3P7a2+EOZOjGv85wtZ2sJhI1oJ+nekmqpOODFDY+hw==", + "license": "MIT", + "dependencies": { + "@jitl/quickjs-ffi-types": "0.32.0" + } + }, + "node_modules/@jitl/quickjs-wasmfile-debug-sync": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/@jitl/quickjs-wasmfile-debug-sync/-/quickjs-wasmfile-debug-sync-0.32.0.tgz", + "integrity": "sha512-LeYWrPGC1uNCTBWvibo3ZLJj0CSVNYUXvJpXMCmuQ5Sap2cCACc3uvGvYV4homHHBAzfw5akoTqMMS4YFRtw+Q==", + "license": "MIT", + "dependencies": { + "@jitl/quickjs-ffi-types": "0.32.0" + } + }, + "node_modules/@jitl/quickjs-wasmfile-release-asyncify": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/@jitl/quickjs-wasmfile-release-asyncify/-/quickjs-wasmfile-release-asyncify-0.32.0.tgz", + "integrity": "sha512-3oSwPfja12ICz4aIblB58cuY8JlEq5Txt8Cut4VLo+LH47QN+mzCnSgnbB03hWzg1LBcc+VyyI9UOag7a1NF+Q==", + "license": "MIT", + "dependencies": { + "@jitl/quickjs-ffi-types": "0.32.0" + } + }, + "node_modules/@jitl/quickjs-wasmfile-release-sync": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/@jitl/quickjs-wasmfile-release-sync/-/quickjs-wasmfile-release-sync-0.32.0.tgz", + "integrity": "sha512-BKNDI/TPBfGlLNGYpLrhcDGXmIk4xHm4MRAisOBnOzpXVn9HZWsfmMAc9WMBrAHjvvds6HOikKeaOBKdPdpVrg==", + "license": "MIT", + "dependencies": { + "@jitl/quickjs-ffi-types": "0.32.0" + } + }, "node_modules/@jridgewell/gen-mapping": { "version": "0.3.13", "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", @@ -1763,6 +2045,12 @@ "zod": "^3.25.0 || ^4.0.0" } }, + "node_modules/@mixmark-io/domino": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@mixmark-io/domino/-/domino-2.2.0.tgz", + "integrity": "sha512-Y28PR25bHXUg88kCV7nivXrP2Nj2RueZ3/l/jdx6J9f8J4nsEGcgX0Qe6lt7Pa+J79+kPiJU3LguR6O/6zrLOw==", + "license": "BSD-2-Clause" + }, "node_modules/@modelcontextprotocol/sdk": { "version": "1.29.0", "resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz", @@ -1803,6 +2091,21 @@ } } }, + "node_modules/@mongodb-js/zstd": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/@mongodb-js/zstd/-/zstd-7.0.0.tgz", + "integrity": "sha512-mQ2s0pYYiav+tzCDR05Zptem8Ey2v8s11lri5RKGhTtL4COVCvVCk5vtyRYNT+9L8qSfyOqqefF9UtnW8mC5jA==", + "hasInstallScript": true, + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "node-addon-api": "^8.5.0", + "prebuild-install": "^7.1.3" + }, + "engines": { + "node": ">= 20.19.0" + } + }, "node_modules/@napi-rs/keyring": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/@napi-rs/keyring/-/keyring-1.3.0.tgz", @@ -2037,6 +2340,18 @@ "node": ">= 10" } }, + "node_modules/@nodable/entities": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@nodable/entities/-/entities-2.2.0.tgz", + "integrity": "sha512-9uGyhaQavEUMC8AIddIjau4NsnsXhou+j5sBAGojCM1oxmQpVKTWR/9JxABD6UAv12vpIms55fPZKFQEhG6uBg==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/nodable" + } + ], + "license": "MIT" + }, "node_modules/@nodelib/fs.scandir": { "version": "2.1.5", "resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz", @@ -2781,6 +3096,29 @@ "url": "https://github.com/sponsors/tannerlinsley" } }, + "node_modules/@tokenizer/inflate": { + "version": "0.4.1", + "resolved": "https://registry.npmjs.org/@tokenizer/inflate/-/inflate-0.4.1.tgz", + "integrity": "sha512-2mAv+8pkG6GIZiF1kNg1jAjh27IDxEPKwdGul3snfztFerfPGI1LjDezZp3i7BElXompqEtPmoPx6c2wgtWsOA==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "token-types": "^6.1.1" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, + "node_modules/@tokenizer/token": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/@tokenizer/token/-/token-0.3.0.tgz", + "integrity": "sha512-OvjF+z51L3ov0OyAU0duzsYuvO01PH7x4t6DJx+guahgTnBHkhJdG7soQeTSFLWN3efnHyibZ4Z8l2EuWwJN3A==", + "license": "MIT" + }, "node_modules/@types/babel__traverse": { "version": "7.28.0", "resolved": "https://registry.npmjs.org/@types/babel__traverse/-/babel__traverse-7.28.0.tgz", @@ -3051,6 +3389,18 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, + "node_modules/anynum": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/anynum/-/anynum-1.0.0.tgz", + "integrity": "sha512-xjR9/zBVnUOP6ztMIIgShjsxui80nQUQH+5xJnvrYLs+90bF25/KJqaAi8mk+B4RDtX1Nspi6fmp4YTEts8SfA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, "node_modules/arch": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/arch/-/arch-2.2.0.tgz", @@ -3308,7 +3658,7 @@ "version": "1.5.1", "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", - "dev": true, + "devOptional": true, "funding": [ { "type": "github", @@ -3338,6 +3688,58 @@ "node": ">=6.0.0" } }, + "node_modules/bl": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", + "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==", + "license": "MIT", + "optional": true, + "dependencies": { + "buffer": "^5.5.0", + "inherits": "^2.0.4", + "readable-stream": "^3.4.0" + } + }, + "node_modules/bl/node_modules/buffer": { + "version": "5.7.1", + "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", + "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT", + "optional": true, + "dependencies": { + "base64-js": "^1.3.1", + "ieee754": "^1.1.13" + } + }, + "node_modules/bl/node_modules/readable-stream": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", + "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", + "license": "MIT", + "optional": true, + "dependencies": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + }, + "engines": { + "node": ">= 6" + } + }, "node_modules/body-parser": { "version": "2.2.2", "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", @@ -3661,6 +4063,13 @@ } } }, + "node_modules/chownr": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", + "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", + "license": "ISC", + "optional": true + }, "node_modules/cli-boxes": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/cli-boxes/-/cli-boxes-3.0.0.tgz", @@ -4599,11 +5008,27 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/deep-extend": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", - "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", - "dev": true, + "node_modules/decompress-response": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", + "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", + "license": "MIT", + "optional": true, + "dependencies": { + "mimic-response": "^3.1.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/deep-extend": { + "version": "0.6.0", + "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", + "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", + "devOptional": true, "license": "MIT", "engines": { "node": ">=4.0.0" @@ -4637,6 +5062,16 @@ "node": ">=6" } }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "license": "Apache-2.0", + "optional": true, + "engines": { + "node": ">=8" + } + }, "node_modules/devlop": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/devlop/-/devlop-1.1.0.tgz", @@ -4650,6 +5085,15 @@ "url": "https://github.com/sponsors/wooorm" } }, + "node_modules/diff": { + "version": "8.0.4", + "resolved": "https://registry.npmjs.org/diff/-/diff-8.0.4.tgz", + "integrity": "sha512-DPi0FmjiSU5EvQV0++GFDOJ9ASQUVFh5kD+OzOnYdi7n3Wpm9hWWGfB/O2blfHcMVTL5WkQXSnRiK9makhrcnw==", + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.3.1" + } + }, "node_modules/dotenv": { "version": "16.6.1", "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", @@ -4726,7 +5170,7 @@ "version": "1.4.5", "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==", - "dev": true, + "devOptional": true, "license": "MIT", "dependencies": { "once": "^1.4.0" @@ -4945,6 +5389,16 @@ "url": "https://github.com/sindresorhus/execa?sponsor=1" } }, + "node_modules/expand-template": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", + "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", + "license": "(MIT OR WTFPL)", + "optional": true, + "engines": { + "node": ">=6" + } + }, "node_modules/express": { "version": "5.2.1", "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", @@ -5172,6 +5626,45 @@ "fast-string-width": "^3.0.2" } }, + "node_modules/fast-xml-builder": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/fast-xml-builder/-/fast-xml-builder-1.2.0.tgz", + "integrity": "sha512-00aAWieqff+ZJhsXA4g1g7M8k+7AYoMUUHF+/zFb5U6Uv/P0Vl4QZo84/IcufzYalLuEj9928bXN9PbbFzMF0Q==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "path-expression-matcher": "^1.5.0", + "xml-naming": "^0.1.0" + } + }, + "node_modules/fast-xml-parser": { + "version": "5.9.0", + "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.9.0.tgz", + "integrity": "sha512-duBuXbyIhEeNO4GjFuVqr0nF047oNwr18aum+zJyqo0MUG/n7Afgs3Qv3D6VN3ONedUKxiuFlPiMGIa0Z11chA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "@nodable/entities": "^2.2.0", + "fast-xml-builder": "^1.2.0", + "is-unsafe": "^1.0.1", + "path-expression-matcher": "^1.5.0", + "strnum": "^2.4.0", + "xml-naming": "^0.1.0" + }, + "bin": { + "fxparser": "src/cli/cli.js" + } + }, "node_modules/fastify": { "version": "5.8.5", "resolved": "https://registry.npmjs.org/fastify/-/fastify-5.8.5.tgz", @@ -5263,6 +5756,24 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/file-type": { + "version": "21.3.4", + "resolved": "https://registry.npmjs.org/file-type/-/file-type-21.3.4.tgz", + "integrity": "sha512-Ievi/yy8DS3ygGvT47PjSfdFoX+2isQueoYP1cntFW1JLYAuS4GD7NUPGg4zv2iZfV52uDyk5w5Z0TdpRS6Q1g==", + "license": "MIT", + "dependencies": { + "@tokenizer/inflate": "^0.4.1", + "strtok3": "^10.3.4", + "token-types": "^6.1.1", + "uint8array-extras": "^1.4.0" + }, + "engines": { + "node": ">=20" + }, + "funding": { + "url": "https://github.com/sindresorhus/file-type?sponsor=1" + } + }, "node_modules/fill-range": { "version": "7.1.1", "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", @@ -5358,6 +5869,13 @@ "node": ">= 0.8" } }, + "node_modules/fs-constants": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz", + "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", + "license": "MIT", + "optional": true + }, "node_modules/fs-extra": { "version": "11.3.5", "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.3.5.tgz", @@ -5486,6 +6004,13 @@ "url": "https://github.com/privatenumber/get-tsconfig?sponsor=1" } }, + "node_modules/github-from-package": { + "version": "0.0.0", + "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", + "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", + "license": "MIT", + "optional": true + }, "node_modules/glob": { "version": "10.5.0", "resolved": "https://registry.npmjs.org/glob/-/glob-10.5.0.tgz", @@ -5660,7 +6185,6 @@ "version": "1.2.1", "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", - "dev": true, "funding": [ { "type": "github", @@ -5696,7 +6220,7 @@ "version": "1.3.8", "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz", "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", - "dev": true, + "devOptional": true, "license": "ISC" }, "node_modules/ip-address": { @@ -5885,6 +6409,18 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/is-unsafe": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/is-unsafe/-/is-unsafe-1.0.1.tgz", + "integrity": "sha512-CLK2+VdgERgD96EYm5lUQssZYlRg2tkZnbsxZoacmSiRxiFJ4Nk4SzjCl+Ur+v3kXIY9dTIdb3IH22y1mZ56LA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, "node_modules/is-wsl": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/is-wsl/-/is-wsl-2.2.0.tgz", @@ -6060,6 +6596,88 @@ "graceful-fs": "^4.1.6" } }, + "node_modules/just-bash": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/just-bash/-/just-bash-3.0.1.tgz", + "integrity": "sha512-YVyzCN08fKarUnwqy7rKOAcX+2MLYLnYInuowmUXn3mqhrtd4ieZNBuzdQG+qYV9DqnIWuv9Whiph0WRIWsBtw==", + "license": "Apache-2.0", + "dependencies": { + "diff": "^8.0.2", + "fast-xml-parser": "^5.7.3", + "file-type": "^21.2.0", + "ini": "^6.0.0", + "minimatch": "^10.1.1", + "modern-tar": "^0.7.3", + "papaparse": "^5.5.3", + "quickjs-emscripten": "^0.32.0", + "re2js": "^1.2.1", + "seek-bzip": "^2.0.0", + "smol-toml": "^1.6.0", + "sprintf-js": "^1.1.3", + "sql.js": "^1.13.0", + "turndown": "^7.2.2", + "yaml": "^2.8.2" + }, + "bin": { + "just-bash": "dist/bin/just-bash.js", + "just-bash-shell": "dist/bin/shell/shell.js" + }, + "optionalDependencies": { + "@mongodb-js/zstd": "^7.0.0", + "node-liblzma": "^2.0.3" + } + }, + "node_modules/just-bash/node_modules/balanced-match": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", + "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", + "license": "MIT", + "engines": { + "node": "18 || 20 || >=22" + } + }, + "node_modules/just-bash/node_modules/brace-expansion": { + "version": "5.0.6", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz", + "integrity": "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==", + "license": "MIT", + "dependencies": { + "balanced-match": "^4.0.2" + }, + "engines": { + "node": "18 || 20 || >=22" + } + }, + "node_modules/just-bash/node_modules/ini": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/ini/-/ini-6.0.0.tgz", + "integrity": "sha512-IBTdIkzZNOpqm7q3dRqJvMaldXjDHWkEDfrwGEQTs5eaQMWV+djAhR+wahyNNMAa+qpbDUhBMVt4ZKNwpPm7xQ==", + "license": "ISC", + "engines": { + "node": "^20.17.0 || >=22.9.0" + } + }, + "node_modules/just-bash/node_modules/minimatch": { + "version": "10.2.5", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", + "integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==", + "license": "BlueOak-1.0.0", + "dependencies": { + "brace-expansion": "^5.0.5" + }, + "engines": { + "node": "18 || 20 || >=22" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" + } + }, + "node_modules/just-bash/node_modules/sprintf-js": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.3.tgz", + "integrity": "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==", + "license": "BSD-3-Clause" + }, "node_modules/kind-of": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz", @@ -7174,6 +7792,19 @@ "node": ">=6" } }, + "node_modules/mimic-response": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", + "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", + "license": "MIT", + "optional": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/minimatch": { "version": "9.0.9", "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.9.tgz", @@ -7194,7 +7825,7 @@ "version": "1.2.8", "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", - "dev": true, + "devOptional": true, "license": "MIT", "funding": { "url": "https://github.com/sponsors/ljharb" @@ -7210,6 +7841,13 @@ "node": ">=16 || 14 >=14.17" } }, + "node_modules/mkdirp-classic": { + "version": "0.5.3", + "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", + "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", + "license": "MIT", + "optional": true + }, "node_modules/mlly": { "version": "1.8.2", "resolved": "https://registry.npmjs.org/mlly/-/mlly-1.8.2.tgz", @@ -7223,6 +7861,15 @@ "ufo": "^1.6.3" } }, + "node_modules/modern-tar": { + "version": "0.7.6", + "resolved": "https://registry.npmjs.org/modern-tar/-/modern-tar-0.7.6.tgz", + "integrity": "sha512-sweCIVXzx1aIGTCdzcMlSZt1h8k5Tmk08VNAuRk3IU28XamGiOH5ypi11g6De2CH7PhYqSSnGy2A/EFhbWnVKg==", + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", @@ -7247,6 +7894,13 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, + "node_modules/napi-build-utils": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", + "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", + "license": "MIT", + "optional": true + }, "node_modules/negotiator": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz", @@ -7256,6 +7910,63 @@ "node": ">= 0.6" } }, + "node_modules/node-abi": { + "version": "3.92.0", + "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.92.0.tgz", + "integrity": "sha512-KdHvFWZjEKDf0cakgFjebl371GPsISX2oZHcuyKqM7DtogIsHrqKeLTo8wBHxaXRAQlY2PsPlZmfo+9ZCxEREQ==", + "license": "MIT", + "optional": true, + "dependencies": { + "semver": "^7.3.5" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/node-addon-api": { + "version": "8.8.0", + "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-8.8.0.tgz", + "integrity": "sha512-c5Ko1fZJIJmzhFIkhRN76WTq+fC6tWnGy9CXA0fA+XygsWZmEwG8vmbkNqxMyoaa0Tin4djul49NzdVcJJcjeA==", + "license": "MIT", + "optional": true, + "engines": { + "node": "^18 || ^20 || >= 21" + } + }, + "node_modules/node-gyp-build": { + "version": "4.8.4", + "resolved": "https://registry.npmjs.org/node-gyp-build/-/node-gyp-build-4.8.4.tgz", + "integrity": "sha512-LA4ZjwlnUblHVgq0oBF3Jl/6h/Nvs5fzBLwdEF4nuxnFdsfajde4WfxtJr3CaiH+F6ewcIB/q4jQ4UzPyid+CQ==", + "license": "MIT", + "optional": true, + "bin": { + "node-gyp-build": "bin.js", + "node-gyp-build-optional": "optional.js", + "node-gyp-build-test": "build-test.js" + } + }, + "node_modules/node-liblzma": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/node-liblzma/-/node-liblzma-2.2.0.tgz", + "integrity": "sha512-s0KzNOWwOJJgPG6wxg6cKohnAl9Wk/oW1KrQaVzJBjQwVcUGPQCzpR46Ximygjqj/3KhOrtJXnYMp/xYAXp75g==", + "hasInstallScript": true, + "license": "LGPL-3.0", + "optional": true, + "dependencies": { + "node-addon-api": "^8.5.0", + "node-gyp-build": "^4.8.4" + }, + "bin": { + "nxz": "lib/cli/nxz.js" + }, + "engines": { + "node": ">=16.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/oorabona" + } + }, "node_modules/node-releases": { "version": "2.0.44", "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.44.tgz", @@ -7436,6 +8147,12 @@ "dev": true, "license": "BlueOak-1.0.0" }, + "node_modules/papaparse": { + "version": "5.5.3", + "resolved": "https://registry.npmjs.org/papaparse/-/papaparse-5.5.3.tgz", + "integrity": "sha512-5QvjGxYVjxO59MGU2lHVYpRWBBtKHnlIAcSe1uNFCkkptUh63NFRj0FJQm7nR67puEruUci/ZkjmEFrjCAyP4A==", + "license": "MIT" + }, "node_modules/parse-ms": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/parse-ms/-/parse-ms-4.0.0.tgz", @@ -7457,6 +8174,21 @@ "node": ">= 0.8" } }, + "node_modules/path-expression-matcher": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/path-expression-matcher/-/path-expression-matcher-1.5.0.tgz", + "integrity": "sha512-cbrerZV+6rvdQrrD+iGMcZFEiiSrbv9Tfdkvnusy6y0x0GKBXREFg/Y65GhIfm0tnLntThhzCnfKwp1WRjeCyQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/path-is-inside": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/path-is-inside/-/path-is-inside-1.0.2.tgz", @@ -7623,6 +8355,18 @@ "pathe": "^2.0.1" } }, + "node_modules/playwright-core": { + "version": "1.60.0", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.60.0.tgz", + "integrity": "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==", + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/postal-mime": { "version": "2.7.4", "resolved": "https://registry.npmjs.org/postal-mime/-/postal-mime-2.7.4.tgz", @@ -7649,6 +8393,34 @@ } } }, + "node_modules/prebuild-install": { + "version": "7.1.3", + "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", + "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", + "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.", + "license": "MIT", + "optional": true, + "dependencies": { + "detect-libc": "^2.0.0", + "expand-template": "^2.0.3", + "github-from-package": "0.0.0", + "minimist": "^1.2.3", + "mkdirp-classic": "^0.5.3", + "napi-build-utils": "^2.0.0", + "node-abi": "^3.3.0", + "pump": "^3.0.0", + "rc": "^1.2.7", + "simple-get": "^4.0.0", + "tar-fs": "^2.0.0", + "tunnel-agent": "^0.6.0" + }, + "bin": { + "prebuild-install": "bin.js" + }, + "engines": { + "node": ">=10" + } + }, "node_modules/prettier": { "version": "3.8.3", "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.8.3.tgz", @@ -7738,7 +8510,7 @@ "version": "3.0.4", "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz", "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==", - "dev": true, + "devOptional": true, "license": "MIT", "dependencies": { "end-of-stream": "^1.1.0", @@ -7804,6 +8576,31 @@ "integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==", "license": "MIT" }, + "node_modules/quickjs-emscripten": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/quickjs-emscripten/-/quickjs-emscripten-0.32.0.tgz", + "integrity": "sha512-So0Sqw869y/S2oE3Nuc0uT3Dhqgvsj8FSrwBdsuTosVsG8ME5/OcudU1GxsrIFdFABgy17GHnTVO9TYV/bLQcA==", + "license": "MIT", + "dependencies": { + "@jitl/quickjs-wasmfile-debug-asyncify": "0.32.0", + "@jitl/quickjs-wasmfile-debug-sync": "0.32.0", + "@jitl/quickjs-wasmfile-release-asyncify": "0.32.0", + "@jitl/quickjs-wasmfile-release-sync": "0.32.0", + "quickjs-emscripten-core": "0.32.0" + }, + "engines": { + "node": ">=16.0.0" + } + }, + "node_modules/quickjs-emscripten-core": { + "version": "0.32.0", + "resolved": "https://registry.npmjs.org/quickjs-emscripten-core/-/quickjs-emscripten-core-0.32.0.tgz", + "integrity": "sha512-QFnPfjFey8EqknSrSxe1hZrf1/8z7/6s1QzGOmKo6++02r7QRRX7ZoyNaZh7JuVjWsVW87KnQrbZqnHkOAzUyg==", + "license": "MIT", + "dependencies": { + "@jitl/quickjs-ffi-types": "0.32.0" + } + }, "node_modules/range-parser": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", @@ -7832,7 +8629,7 @@ "version": "1.2.8", "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", - "dev": true, + "devOptional": true, "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", "dependencies": { "deep-extend": "^0.6.0", @@ -7848,12 +8645,18 @@ "version": "2.0.1", "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", - "dev": true, + "devOptional": true, "license": "MIT", "engines": { "node": ">=0.10.0" } }, + "node_modules/re2js": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/re2js/-/re2js-1.3.3.tgz", + "integrity": "sha512-s/I5zEAo79SUK0Qw4dpZKpiMwbQ6Gz0KU2NRr7eaO4x/p2g7Vvmn3hdeXDg8VsaUjfj/ora+e9oi27LX/C9+mw==", + "license": "MIT" + }, "node_modules/readable-stream": { "version": "4.7.0", "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz", @@ -8195,7 +8998,7 @@ "version": "5.2.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", - "dev": true, + "devOptional": true, "funding": [ { "type": "github", @@ -8278,6 +9081,28 @@ ], "license": "BSD-3-Clause" }, + "node_modules/seek-bzip": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/seek-bzip/-/seek-bzip-2.0.0.tgz", + "integrity": "sha512-SMguiTnYrhpLdk3PwfzHeotrcwi8bNV4iemL9tx9poR/yeaMYwB9VzR1w7b57DuWpuqR8n6oZboi0hj3AxZxQg==", + "license": "MIT", + "dependencies": { + "commander": "^6.0.0" + }, + "bin": { + "seek-bunzip": "bin/seek-bunzip", + "seek-table": "bin/seek-bzip-table" + } + }, + "node_modules/seek-bzip/node_modules/commander": { + "version": "6.2.1", + "resolved": "https://registry.npmjs.org/commander/-/commander-6.2.1.tgz", + "integrity": "sha512-U7VdrJFnJgo4xjrHpTzu0yrHPGImdsmD95ZlgYSEajAn2JKzDhDTPG9kBTefmObL2w/ngeZnilk+OV9CG3d7UA==", + "license": "MIT", + "engines": { + "node": ">= 6" + } + }, "node_modules/semver": { "version": "7.8.0", "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.0.tgz", @@ -8602,6 +9427,53 @@ "url": "https://github.com/sponsors/isaacs" } }, + "node_modules/simple-concat": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", + "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT", + "optional": true + }, + "node_modules/simple-get": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz", + "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT", + "optional": true, + "dependencies": { + "decompress-response": "^6.0.0", + "once": "^1.3.1", + "simple-concat": "^1.0.0" + } + }, "node_modules/sisteransi": { "version": "1.0.5", "resolved": "https://registry.npmjs.org/sisteransi/-/sisteransi-1.0.5.tgz", @@ -8609,6 +9481,18 @@ "dev": true, "license": "MIT" }, + "node_modules/smol-toml": { + "version": "1.6.1", + "resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.6.1.tgz", + "integrity": "sha512-dWUG8F5sIIARXih1DTaQAX4SsiTXhInKf1buxdY9DIg4ZYPZK5nGM1VRIYmEbDbsHt7USo99xSLFu5Q1IqTmsg==", + "license": "BSD-3-Clause", + "engines": { + "node": ">= 18" + }, + "funding": { + "url": "https://github.com/sponsors/cyyynthia" + } + }, "node_modules/sonic-boom": { "version": "4.2.1", "resolved": "https://registry.npmjs.org/sonic-boom/-/sonic-boom-4.2.1.tgz", @@ -8633,6 +9517,12 @@ "integrity": "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==", "license": "BSD-3-Clause" }, + "node_modules/sql.js": { + "version": "1.14.1", + "resolved": "https://registry.npmjs.org/sql.js/-/sql.js-1.14.1.tgz", + "integrity": "sha512-gcj8zBWU5cFsi9WUP+4bFNXAyF1iRpA3LLyS/DP5xlrNzGmPIizUeBggKa8DbDwdqaKwUcTEnChtd2grWo/x/A==", + "license": "MIT" + }, "node_modules/standardwebhooks": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.0.0.tgz", @@ -8674,7 +9564,7 @@ "version": "1.3.0", "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", - "dev": true, + "devOptional": true, "license": "MIT", "dependencies": { "safe-buffer": "~5.2.0" @@ -8818,6 +9708,37 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/strnum": { + "version": "2.4.0", + "resolved": "https://registry.npmjs.org/strnum/-/strnum-2.4.0.tgz", + "integrity": "sha512-sHrVyWWdq28RbhjuJdZsA1SnGRJV6NiXbk6AXBxDOsgAcA+lmpUZCYjOdLBxkXMwis6RRe7dlZt4VlIWFVzkmg==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "anynum": "^1.0.0" + } + }, + "node_modules/strtok3": { + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/strtok3/-/strtok3-10.3.5.tgz", + "integrity": "sha512-ki4hZQfh5rX0QDLLkOCj+h+CVNkqmp/CMf8v8kZpkNVK6jGQooMytqzLZYUVYIZcFZ6yDB70EfD8POcFXiF5oA==", + "license": "MIT", + "dependencies": { + "@tokenizer/token": "^0.3.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, "node_modules/supports-color": { "version": "7.2.0", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz", @@ -8853,6 +9774,51 @@ "standardwebhooks": "1.0.0" } }, + "node_modules/tar-fs": { + "version": "2.1.4", + "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz", + "integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==", + "license": "MIT", + "optional": true, + "dependencies": { + "chownr": "^1.1.1", + "mkdirp-classic": "^0.5.2", + "pump": "^3.0.0", + "tar-stream": "^2.1.4" + } + }, + "node_modules/tar-fs/node_modules/readable-stream": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", + "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", + "license": "MIT", + "optional": true, + "dependencies": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/tar-fs/node_modules/tar-stream": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz", + "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==", + "license": "MIT", + "optional": true, + "dependencies": { + "bl": "^4.0.3", + "end-of-stream": "^1.4.1", + "fs-constants": "^1.0.0", + "inherits": "^2.0.3", + "readable-stream": "^3.1.1" + }, + "engines": { + "node": ">=6" + } + }, "node_modules/tar-stream": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.2.0.tgz", @@ -8952,6 +9918,24 @@ "node": ">=0.6" } }, + "node_modules/token-types": { + "version": "6.1.2", + "resolved": "https://registry.npmjs.org/token-types/-/token-types-6.1.2.tgz", + "integrity": "sha512-dRXchy+C0IgK8WPC6xvCHFRIWYUbqqdEIKPaKo/AcTUNzwLTK6AH7RjdLWsEZcAN/TBdtfUw3PYEgPr5VPr6ww==", + "license": "MIT", + "dependencies": { + "@borewit/text-codec": "^0.2.1", + "@tokenizer/token": "^0.3.0", + "ieee754": "^1.2.1" + }, + "engines": { + "node": ">=14.16" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, "node_modules/tokenx": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/tokenx/-/tokenx-1.3.0.tgz", @@ -9477,6 +10461,32 @@ "@esbuild/win32-x64": "0.28.0" } }, + "node_modules/tunnel-agent": { + "version": "0.6.0", + "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", + "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "safe-buffer": "^5.0.1" + }, + "engines": { + "node": "*" + } + }, + "node_modules/turndown": { + "version": "7.2.4", + "resolved": "https://registry.npmjs.org/turndown/-/turndown-7.2.4.tgz", + "integrity": "sha512-I8yFsfRzmzK0WV1pNNOA4A7y4RDfFxPRxb3t+e3ui14qSGOxGtiSP6GjeX+Y6CHb7HYaFj7ECUD7VE5kQMZWGQ==", + "license": "MIT", + "dependencies": { + "@mixmark-io/domino": "^2.2.0" + }, + "engines": { + "node": ">=18", + "npm": ">=9" + } + }, "node_modules/type-fest": { "version": "2.19.0", "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-2.19.0.tgz", @@ -9552,6 +10562,18 @@ "dev": true, "license": "MIT" }, + "node_modules/uint8array-extras": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/uint8array-extras/-/uint8array-extras-1.5.0.tgz", + "integrity": "sha512-rvKSBiC5zqCCiDZ9kAOszZcDvdAHwwIKJG33Ykj43OKcWsnmcBRL09YTU4nOeHZ8Y2a7l1MgTd08SBe9A8Qj6A==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/undici-types": { "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", @@ -9727,7 +10749,7 @@ "version": "1.0.2", "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", - "dev": true, + "devOptional": true, "license": "MIT" }, "node_modules/uuid": { @@ -9936,6 +10958,21 @@ } } }, + "node_modules/xml-naming": { + "version": "0.1.0", + "resolved": "https://registry.npmjs.org/xml-naming/-/xml-naming-0.1.0.tgz", + "integrity": "sha512-k8KO9hrMyNk6tUWqUfkTEZbezRRpONVOzUTnc97VnCvyj6Tf9lyUR9EDAIeiVLv56jsMcoXEwjW8Kv5yPY52lw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "engines": { + "node": ">=16.0.0" + } + }, "node_modules/xxhash-wasm": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/xxhash-wasm/-/xxhash-wasm-1.1.0.tgz", @@ -9953,7 +10990,6 @@ "version": "2.9.0", "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz", "integrity": "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA==", - "dev": true, "license": "ISC", "bin": { "yaml": "bin.mjs" diff --git a/backend/package.json b/backend/package.json index 11df78b..cf620a8 100644 --- a/backend/package.json +++ b/backend/package.json @@ -10,6 +10,20 @@ "mastra:dev": "node ../scripts/with-root-env.mjs mastra dev" }, "dependencies": { + "@ai-sdk/alibaba": "^1.0.26", + "@ai-sdk/anthropic": "^3.0.81", + "@ai-sdk/deepinfra": "^2.0.52", + "@ai-sdk/deepseek": "^2.0.35", + "@ai-sdk/fireworks": "^2.0.53", + "@ai-sdk/google": "^3.0.80", + "@ai-sdk/groq": "^3.0.39", + "@ai-sdk/huggingface": "^1.0.50", + "@ai-sdk/mistral": "^3.0.37", + "@ai-sdk/openai": "^3.0.68", + "@ai-sdk/openai-compatible": "^2.0.48", + "@ai-sdk/provider": "^3.0.10", + "@ai-sdk/togetherai": "^2.0.53", + "@ai-sdk/xai": "^3.0.93", "@clerk/backend": "^3.4.11", "@fastify/cors": "^11.0.0", "@mastra/core": "^1.36.0", @@ -20,6 +34,8 @@ "dotenv": "^16.4.0", "fastify": "^5.0.0", "fastify-plugin": "^5.1.0", + "just-bash": "^3.0.1", + "playwright-core": "^1.60.0", "posthog-node": "^5.35.1", "resend": "^6.12.3", "zod": "^4.4.3" diff --git a/backend/prompts/schema-inference.txt b/backend/prompts/schema-inference.txt index 9752429..1f3b7b0 100644 --- a/backend/prompts/schema-inference.txt +++ b/backend/prompts/schema-inference.txt @@ -3,7 +3,7 @@ You are a data engineering assistant that converts natural-language prompts into Your job is to: 1. Identify the universe of entities the user wants to collect. Each entity becomes one row in the dataset. -2. Pick a clear primary key — the column whose values uniquely identify each row. This is usually a name, ID, or canonical URL. Exactly one column must have `is_primary_key: true`, and its `name` must equal `primary_key`. The primary key column must have `nullable: false` and `is_enumerable: true`. +2. Pick a clear primary key — the column whose values uniquely identify each row. This is usually a name, ID, or canonical URL. Exactly one column must have `is_primary_key: true`, and `primary_key` must be a one-item array containing that column name. The primary key column must have `nullable: false` and `is_enumerable: true`. 3. Choose useful columns. Each column captures one fact about the entity. Use snake_case names. Mark `is_enumerable: true` only on columns whose values can be used to list all rows (typically just the primary key, and occasionally one or two others when a source page lists them alongside the primary key). 4. Set `retrieval_strategy`: - `search_fetch` — the data lives on a static page or sitemap that can be fetched as HTML. @@ -11,6 +11,12 @@ Your job is to: - `hybrid` — unclear; the pipeline will try search_fetch first and fall back to browser. 5. Set `source_hint` to a specific URL whenever possible (e.g. `https://www.ycombinator.com/companies?industry=Fintech`). Avoid vague descriptions. 6. Write a `retrieval_hint` for each column describing where/how the value can be found later. Downstream agents will use this to fill the column for each row. +7. Set `codification_profile` conservatively: + - `disabled` for broad web research, arbitrary unrelated domains, or snippet-only work with no stable page family. + - `candidate` when rows likely share stable page families and a reusable Playwright extractor might work after seeing a representative row. Use `candidate`, not `disabled`, when the only concern is that the stable source has anti-bot or automation-blocking reputation. BigSet uses TinyFish Browser for interactive browser access, and TinyFish can often get through surfaces that plain fetches or commodity browser automation cannot; the extractor builder can inspect a real page and decline if it is actually blocked. + - `required` when one authoritative browser-heavy source/directory is clearly the intended repeated extraction path. + - `unknown` only when evidence is genuinely insufficient; prefer `disabled` for broad web datasets. + - For marketplace/catalog identifiers with deterministic product pages, include URL-template families instead of disabling codification. Use the site/schema's actual identifier and route shape; do not hardcode a source-specific template unless it is implied by the user prompt or discovered source hint. Rules: diff --git a/backend/src/abort-registry.ts b/backend/src/abort-registry.ts index 3911ea3..636ef1c 100644 --- a/backend/src/abort-registry.ts +++ b/backend/src/abort-registry.ts @@ -35,6 +35,35 @@ export function getSignal(datasetId: string): AbortSignal | undefined { return controllers.get(datasetId)?.signal; } +export function datasetAbortError(): DOMException { + return new DOMException("Run was stopped", "AbortError"); +} + +export function isDatasetRunAborted(datasetId: string): boolean { + return getSignal(datasetId)?.aborted === true; +} + +export function throwIfDatasetRunAborted(datasetId: string): void { + if (isDatasetRunAborted(datasetId)) { + throw datasetAbortError(); + } +} + +export function isAbortLikeError(err: unknown): boolean { + if (err instanceof DOMException && err.name === "AbortError") return true; + return err instanceof Error && err.name === "AbortError"; +} + +/** Number of dataset runs currently active in this process. */ +export function activeDatasetRunCount(): number { + return controllers.size; +} + +/** True when any dataset run is currently active in this process. */ +export function hasActiveDatasetRuns(): boolean { + return controllers.size > 0; +} + /** * Fire the abort signal for a dataset's active run. * Does NOT remove the entry — deregisterDataset() handles that. diff --git a/backend/src/config/agent-output-tokens.ts b/backend/src/config/agent-output-tokens.ts new file mode 100644 index 0000000..ffdd8ec --- /dev/null +++ b/backend/src/config/agent-output-tokens.ts @@ -0,0 +1,6 @@ +export const AGENT_MAX_OUTPUT_TOKENS = { + POPULATE_ORCHESTRATOR: 20_000, + INVESTIGATE_SUBAGENT: 16_000, + REFRESH_AGENT: 12_000, + EXTRACTOR_BUILDER: 20_000, +} as const; diff --git a/backend/src/config/llm.ts b/backend/src/config/llm.ts new file mode 100644 index 0000000..78ee252 --- /dev/null +++ b/backend/src/config/llm.ts @@ -0,0 +1,680 @@ +import type { LanguageModelV3 } from "@ai-sdk/provider"; +import { createAlibaba } from "@ai-sdk/alibaba"; +import { createAnthropic } from "@ai-sdk/anthropic"; +import { createDeepInfra } from "@ai-sdk/deepinfra"; +import { createDeepSeek } from "@ai-sdk/deepseek"; +import { createFireworks } from "@ai-sdk/fireworks"; +import { createGoogleGenerativeAI } from "@ai-sdk/google"; +import { createGroq } from "@ai-sdk/groq"; +import { createHuggingFace } from "@ai-sdk/huggingface"; +import { createMistral } from "@ai-sdk/mistral"; +import { createOpenAI } from "@ai-sdk/openai"; +import { createOpenAICompatible } from "@ai-sdk/openai-compatible"; +import { createTogetherAI } from "@ai-sdk/togetherai"; +import { createXai } from "@ai-sdk/xai"; +import { createOpenRouter } from "@openrouter/ai-sdk-provider"; + +import { env } from "../env.js"; +import { FETCH_TIMEOUT_MS } from "../fetch-timeout.js"; + +export const LLM_PROVIDER_TYPES = [ + "openrouter", + "openai", + "anthropic", + "google", + "xai", + "deepseek", + "qwen", + "mistral", + "groq", + "togetherai", + "deepinfra", + "fireworks", + "huggingface", + "ollama", + "lmstudio", + "custom", +] as const; + +export type LlmProviderType = (typeof LLM_PROVIDER_TYPES)[number]; + +export type ModelRoleKey = + | "schemaInference" + | "populateOrchestrator" + | "investigateSubagent" + | "extractorBuilder"; + +export interface LlmProviderConfig { + provider: LlmProviderType; + apiKey: string; + defaultModel: string; + baseUrl?: string; + source: "local" | "env"; +} + +export interface LlmProviderInput { + provider: LlmProviderType; + apiKey: string; + defaultModel?: string; + baseUrl?: string; +} + +export const LLM_PROVIDER_LABELS: Record = { + openrouter: "OpenRouter", + openai: "OpenAI", + anthropic: "Anthropic", + google: "Google Gemini", + xai: "xAI", + deepseek: "DeepSeek", + qwen: "Qwen", + mistral: "Mistral AI", + groq: "Groq", + togetherai: "Together.ai", + deepinfra: "DeepInfra", + fireworks: "Fireworks AI", + huggingface: "Hugging Face", + ollama: "Ollama", + lmstudio: "LM Studio", + custom: "Custom OpenAI-compatible", +}; + +export const LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE: Record< + LlmProviderType, + Record +> = { + openrouter: { + schemaInference: env.SCHEMA_INFERENCE_MODEL, + populateOrchestrator: env.POPULATE_ORCHESTRATOR_MODEL, + investigateSubagent: env.INVESTIGATE_SUBAGENT_MODEL, + extractorBuilder: env.EXTRACTOR_BUILDER_MODEL, + }, + openai: { + schemaInference: "gpt-5.4-mini", + populateOrchestrator: "gpt-5.4-mini", + investigateSubagent: "gpt-5.4-mini", + extractorBuilder: "gpt-5.4-mini", + }, + anthropic: { + schemaInference: "claude-sonnet-4-6", + populateOrchestrator: "claude-haiku-4-5-20251001", + investigateSubagent: "claude-haiku-4-5-20251001", + extractorBuilder: "claude-haiku-4-5-20251001", + }, + google: { + schemaInference: "gemini-3.5-flash", + populateOrchestrator: "gemini-3.5-flash", + investigateSubagent: "gemini-3.5-flash", + extractorBuilder: "gemini-3.5-flash", + }, + xai: { + schemaInference: "grok-4.3", + populateOrchestrator: "grok-4.3", + investigateSubagent: "grok-4.3", + extractorBuilder: "grok-4.3", + }, + deepseek: { + schemaInference: "deepseek-chat", + populateOrchestrator: "deepseek-chat", + investigateSubagent: "deepseek-chat", + extractorBuilder: "deepseek-chat", + }, + qwen: { + schemaInference: "qwen-plus", + populateOrchestrator: "qwen-plus", + investigateSubagent: "qwen-plus", + extractorBuilder: "qwen-plus", + }, + mistral: { + schemaInference: "mistral-large-latest", + populateOrchestrator: "mistral-large-latest", + investigateSubagent: "mistral-large-latest", + extractorBuilder: "mistral-large-latest", + }, + groq: { + schemaInference: "openai/gpt-oss-120b", + populateOrchestrator: "openai/gpt-oss-120b", + investigateSubagent: "openai/gpt-oss-120b", + extractorBuilder: "openai/gpt-oss-120b", + }, + togetherai: { + schemaInference: "Qwen/Qwen3.5-397B-A17B", + populateOrchestrator: "Qwen/Qwen3.5-397B-A17B", + investigateSubagent: "Qwen/Qwen3.5-397B-A17B", + extractorBuilder: "Qwen/Qwen3.5-397B-A17B", + }, + deepinfra: { + schemaInference: "meta-llama/Llama-3.3-70B-Instruct-Turbo", + populateOrchestrator: "meta-llama/Llama-3.3-70B-Instruct-Turbo", + investigateSubagent: "meta-llama/Llama-3.3-70B-Instruct-Turbo", + extractorBuilder: "meta-llama/Llama-3.3-70B-Instruct-Turbo", + }, + fireworks: { + schemaInference: "accounts/fireworks/models/kimi-k2p5", + populateOrchestrator: "accounts/fireworks/models/kimi-k2p5", + investigateSubagent: "accounts/fireworks/models/kimi-k2p5", + extractorBuilder: "accounts/fireworks/models/kimi-k2p5", + }, + huggingface: { + schemaInference: "deepseek-ai/DeepSeek-V3-0324", + populateOrchestrator: "deepseek-ai/DeepSeek-V3-0324", + investigateSubagent: "deepseek-ai/DeepSeek-V3-0324", + extractorBuilder: "deepseek-ai/DeepSeek-V3-0324", + }, + ollama: { + schemaInference: "", + populateOrchestrator: "", + investigateSubagent: "", + extractorBuilder: "", + }, + lmstudio: { + schemaInference: "", + populateOrchestrator: "", + investigateSubagent: "", + extractorBuilder: "", + }, + custom: { + schemaInference: "", + populateOrchestrator: "", + investigateSubagent: "", + extractorBuilder: "", + }, +}; + +export const LLM_PROVIDER_DEFAULT_MODELS: Record = { + openrouter: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.openrouter.schemaInference, + openai: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.openai.schemaInference, + anthropic: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.anthropic.schemaInference, + google: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.google.schemaInference, + xai: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.xai.schemaInference, + deepseek: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.deepseek.schemaInference, + qwen: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.qwen.schemaInference, + mistral: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.mistral.schemaInference, + groq: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.groq.schemaInference, + togetherai: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.togetherai.schemaInference, + deepinfra: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.deepinfra.schemaInference, + fireworks: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.fireworks.schemaInference, + huggingface: LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE.huggingface.schemaInference, + ollama: "", + lmstudio: "", + custom: "", +}; + +export function isLlmProviderType(value: unknown): value is LlmProviderType { + return ( + typeof value === "string" && + (LLM_PROVIDER_TYPES as readonly string[]).includes(value) + ); +} + +export function llmProviderLabel(provider: LlmProviderType): string { + return LLM_PROVIDER_LABELS[provider]; +} + +export function defaultModelForLlmProvider(provider: LlmProviderType): string { + return LLM_PROVIDER_DEFAULT_MODELS[provider]; +} + +export function defaultModelForLlmProviderRole( + provider: LlmProviderType, + role: ModelRoleKey, +): string { + return LLM_PROVIDER_DEFAULT_MODELS_BY_ROLE[provider][role]; +} + +export function defaultBaseUrlForLlmProvider( + provider: LlmProviderType, +): string | undefined { + if (provider === "openrouter") { + return process.env.OPENROUTER_BASE_URL || "https://openrouter.ai/api/v1"; + } + if (provider === "google") { + return ( + process.env.GOOGLE_GENERATIVE_AI_BASE_URL || + "https://generativelanguage.googleapis.com/v1beta" + ); + } + if (provider === "xai") { + return process.env.XAI_BASE_URL || "https://api.x.ai/v1"; + } + if (provider === "deepseek") { + return process.env.DEEPSEEK_BASE_URL || "https://api.deepseek.com"; + } + if (provider === "qwen") { + return ( + process.env.QWEN_BASE_URL || + "https://dashscope-intl.aliyuncs.com/compatible-mode/v1" + ); + } + if (provider === "mistral") { + return process.env.MISTRAL_BASE_URL || "https://api.mistral.ai/v1"; + } + if (provider === "groq") { + return process.env.GROQ_BASE_URL || "https://api.groq.com/openai/v1"; + } + if (provider === "togetherai") { + return process.env.TOGETHER_BASE_URL || "https://api.together.xyz/v1"; + } + if (provider === "deepinfra") { + return process.env.DEEPINFRA_BASE_URL || "https://api.deepinfra.com/v1"; + } + if (provider === "fireworks") { + return ( + process.env.FIREWORKS_BASE_URL || + "https://api.fireworks.ai/inference/v1" + ); + } + if (provider === "huggingface") { + return process.env.HUGGINGFACE_BASE_URL || "https://router.huggingface.co/v1"; + } + if (provider === "ollama") { + return process.env.OLLAMA_BASE_URL || "http://localhost:11434/v1"; + } + if (provider === "lmstudio") { + return process.env.LM_STUDIO_BASE_URL || "http://localhost:1234/v1"; + } + return undefined; +} + +function isOpenAiCompatibleProvider(provider: LlmProviderType): boolean { + return provider === "custom" || provider === "ollama" || provider === "lmstudio"; +} + +function providerAllowsMissingApiKey(provider: LlmProviderType): boolean { + return isOpenAiCompatibleProvider(provider); +} + +function isLoopbackHost(hostname: string): boolean { + return ["localhost", "127.0.0.1", "0.0.0.0", "::1", "[::1]"].includes( + hostname, + ); +} + +function normalizeLocalLoopbackForBackend(parsed: URL): void { + if (env.IS_LOCAL_MODE && env.IS_DOCKER && isLoopbackHost(parsed.hostname)) { + // In Docker local dev, from the container, + // localhost points at the container, not the host machine where LM Studio + // and other local OpenAI-compatible servers usually listen. + parsed.hostname = "host.docker.internal"; + } +} + +export function normalizeBaseUrl(baseUrl?: string): string | undefined { + const trimmed = baseUrl?.trim(); + if (!trimmed) return undefined; + const parsed = new URL(trimmed); + if (parsed.protocol !== "https:" && parsed.protocol !== "http:") { + throw new Error("Base URL must start with http:// or https://"); + } + normalizeLocalLoopbackForBackend(parsed); + return parsed.toString().replace(/\/+$/, ""); +} + +export function normalizeCustomBaseUrl(baseUrl?: string): string | undefined { + const normalized = normalizeBaseUrl(baseUrl); + if (!normalized) return undefined; + + const parsed = new URL(normalized); + if (parsed.pathname === "" || parsed.pathname === "/") { + parsed.pathname = "/v1"; + } + return parsed.toString().replace(/\/+$/, ""); +} + +export function normalizeLlmProviderInput( + input: LlmProviderInput, + source: "local" | "env", +): LlmProviderConfig { + const provider = input.provider; + const apiKey = input.apiKey.trim(); + if (!apiKey && !providerAllowsMissingApiKey(provider)) { + throw new Error(`${llmProviderLabel(provider)} API key is required`); + } + + const baseUrl = + isOpenAiCompatibleProvider(provider) + ? normalizeCustomBaseUrl( + input.baseUrl ?? defaultBaseUrlForLlmProvider(provider), + ) + : normalizeBaseUrl(input.baseUrl) ?? defaultBaseUrlForLlmProvider(provider); + + if (isOpenAiCompatibleProvider(provider) && !baseUrl) { + throw new Error(`${llmProviderLabel(provider)} requires a base URL`); + } + + const defaultModel = + input.defaultModel?.trim() || defaultModelForLlmProvider(provider); + + return { + provider, + apiKey, + defaultModel, + baseUrl, + source, + }; +} + +export function createLanguageModel( + config: LlmProviderConfig, + modelId?: string, +): LanguageModelV3 { + const resolvedModelId = (modelId?.trim() || config.defaultModel).trim(); + if (!resolvedModelId) { + throw new Error("Model name is required"); + } + + switch (config.provider) { + case "openrouter": { + const provider = createOpenRouter({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "openai": { + const provider = createOpenAI({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "anthropic": { + const provider = createAnthropic({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "google": { + const provider = createGoogleGenerativeAI({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "xai": { + const provider = createXai({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "deepseek": { + const provider = createDeepSeek({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "qwen": { + const provider = createAlibaba({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "mistral": { + const provider = createMistral({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "groq": { + const provider = createGroq({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "togetherai": { + const provider = createTogetherAI({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "deepinfra": { + const provider = createDeepInfra({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "fireworks": { + const provider = createFireworks({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "huggingface": { + const provider = createHuggingFace({ + apiKey: config.apiKey, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + case "ollama": + case "lmstudio": + case "custom": { + if (!config.baseUrl) { + throw new Error(`${llmProviderLabel(config.provider)} requires a base URL`); + } + const provider = createOpenAICompatible({ + name: config.provider, + apiKey: config.apiKey || undefined, + baseURL: config.baseUrl, + }); + return provider(resolvedModelId); + } + } +} + +export function modelsUrlForLlmProvider( + provider: LlmProviderType, + baseUrl?: string, +): string { + const resolvedBaseUrl = ( + baseUrl || + defaultBaseUrlForLlmProvider(provider) || + "https://api.openai.com/v1" + ).replace(/\/+$/, ""); + + if (provider === "deepinfra") { + return resolvedBaseUrl.endsWith("/openai") + ? `${resolvedBaseUrl}/models` + : `${resolvedBaseUrl}/openai/models`; + } + + return `${resolvedBaseUrl}/models`; +} + +type ProviderVerificationRequest = { + url: string; + headers: Record; + method?: "GET" | "POST"; + body?: string; + fallbackStatuses?: number[]; +}; + +function openAiStyleModelsVerificationRequest( + config: LlmProviderConfig, +): ProviderVerificationRequest { + return { + url: modelsUrlForLlmProvider(config.provider, config.baseUrl), + headers: { Authorization: `Bearer ${config.apiKey}` }, + }; +} + +function qwenChatVerificationRequest( + config: LlmProviderConfig, +): ProviderVerificationRequest { + const baseUrl = ( + config.baseUrl || + defaultBaseUrlForLlmProvider("qwen") || + "https://dashscope-intl.aliyuncs.com/compatible-mode/v1" + ).replace(/\/+$/, ""); + + return { + url: `${baseUrl}/chat/completions`, + method: "POST", + headers: { + Authorization: `Bearer ${config.apiKey}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + model: config.defaultModel || defaultModelForLlmProvider("qwen"), + messages: [{ role: "user", content: "ping" }], + max_tokens: 1, + }), + }; +} + +function providerVerificationRequests( + config: LlmProviderConfig, +): ProviderVerificationRequest[] { + switch (config.provider) { + case "openrouter": { + const baseUrl = (config.baseUrl || "https://openrouter.ai/api/v1").replace( + /\/+$/, + "", + ); + return [{ + url: `${baseUrl}/key`, + headers: { Authorization: `Bearer ${config.apiKey}` }, + }]; + } + case "openai": { + const baseUrl = (config.baseUrl || "https://api.openai.com/v1").replace( + /\/+$/, + "", + ); + return [{ + url: `${baseUrl}/models`, + headers: { Authorization: `Bearer ${config.apiKey}` }, + }]; + } + case "anthropic": { + const baseUrl = (config.baseUrl || "https://api.anthropic.com/v1").replace( + /\/+$/, + "", + ); + return [{ + url: `${baseUrl}/models?limit=1`, + headers: { + "x-api-key": config.apiKey, + "anthropic-version": "2023-06-01", + }, + }]; + } + case "google": { + const baseUrl = ( + config.baseUrl || "https://generativelanguage.googleapis.com/v1beta" + ).replace(/\/+$/, ""); + return [{ + url: `${baseUrl}/models`, + headers: { "x-goog-api-key": config.apiKey }, + }]; + } + case "qwen": { + return [ + { + ...openAiStyleModelsVerificationRequest(config), + fallbackStatuses: [404, 405], + }, + qwenChatVerificationRequest(config), + ]; + } + case "xai": + case "deepseek": + case "mistral": + case "groq": + case "togetherai": + case "deepinfra": + case "fireworks": + case "huggingface": { + return [openAiStyleModelsVerificationRequest(config)]; + } + case "ollama": + case "lmstudio": + case "custom": { + if (!config.baseUrl) { + throw new Error(`${llmProviderLabel(config.provider)} requires a base URL`); + } + const baseUrl = config.baseUrl.replace(/\/+$/, ""); + return [{ + url: `${baseUrl}/models`, + headers: config.apiKey + ? { Authorization: `Bearer ${config.apiKey}` } + : {}, + }]; + } + } +} + +export async function verifyLlmProviderConfig( + config: LlmProviderConfig, +): Promise { + const requests = providerVerificationRequests(config); + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS); + let lastResponseStatus: number | undefined; + let lastUrl: string | undefined; + + try { + for (const request of requests) { + lastUrl = request.url; + const response = await fetch(request.url, { + method: request.method ?? "GET", + headers: request.headers, + body: request.body, + signal: controller.signal, + }); + + if (response.ok) return; + + lastResponseStatus = response.status; + if (request.fallbackStatuses?.includes(response.status)) { + continue; + } + if (response.status === 401 || response.status === 403) { + throw new Error( + `${llmProviderLabel(config.provider)} rejected that API key.`, + ); + } + throw new Error( + `${llmProviderLabel(config.provider)} verification failed with HTTP ${response.status}.`, + ); + } + + throw new Error( + `${llmProviderLabel(config.provider)} verification failed with HTTP ${lastResponseStatus ?? "unknown"}.`, + ); + } catch (err) { + if (err instanceof Error && err.name === "AbortError") { + throw new Error( + `${llmProviderLabel(config.provider)} verification timed out after ${FETCH_TIMEOUT_MS / 1000} seconds.`, + ); + } + if (err instanceof Error && err.message === "fetch failed") { + const displayUrl = (lastUrl ?? requests[0]?.url ?? "").replace( + "host.docker.internal", + "localhost", + ); + const localHint = + config.provider === "ollama" + ? " Start Ollama and confirm the OpenAI-compatible endpoint is enabled." + : config.provider === "lmstudio" + ? " Start the LM Studio local server and confirm the port." + : config.provider === "custom" + ? " Check that the endpoint is running and reachable." + : ""; + throw new Error( + `${llmProviderLabel(config.provider)} verification failed: could not reach ${displayUrl}.${localHint}`, + ); + } + throw err; + } finally { + clearTimeout(timeout); + } +} diff --git a/backend/src/config/models.ts b/backend/src/config/models.ts index 1d28cbf..727b80a 100644 --- a/backend/src/config/models.ts +++ b/backend/src/config/models.ts @@ -1,12 +1,19 @@ /** * Backend configuration for AI models. * - * Defines the typed interfaces and constants for OpenRouter model management. + * Defines the typed interfaces and constants for model management. */ import { api, internal, convex } from "../convex.js"; import { env } from "../env.js"; -import { requireOpenRouterApiKey } from "../local-credentials.js"; +import { getLlmProviderConfig, requireOpenRouterApiKey } from "../local-credentials.js"; +import { FETCH_TIMEOUT_MS } from "../fetch-timeout.js"; +import { + defaultBaseUrlForLlmProvider, + defaultModelForLlmProviderRole, + modelsUrlForLlmProvider, + type ModelRoleKey, +} from "./llm.js"; export interface OpenRouterModel { modelName: string; @@ -17,17 +24,292 @@ export interface OpenRouterModel { } /** - * Default model slugs for each agent role. - * Read from environment variables so operators can change defaults - * without touching code. Falls back to typed literals when env vars - * are unset (useful for local dev without a .env file). + * Default model identifiers for each agent role. + * Read from environment variables so operators can change production defaults + * without touching code. Local mode falls back to the selected LLM provider's + * default model first. */ export const DEFAULT_MODEL_IDS = { SCHEMA_INFERENCE: env.SCHEMA_INFERENCE_MODEL, POPULATE_ORCHESTRATOR: env.POPULATE_ORCHESTRATOR_MODEL, INVESTIGATE_SUBAGENT: env.INVESTIGATE_SUBAGENT_MODEL, + EXTRACTOR_BUILDER: env.EXTRACTOR_BUILDER_MODEL, } as const; +const ROW_EXTRACTOR_CONCURRENCY_MIN = 1; +export const ROW_EXTRACTOR_CONCURRENCY_MAX = 100; +const ROW_EXTRACTOR_BROWSER_ATTEMPTS_MIN = 1; +export const ROW_EXTRACTOR_BROWSER_ATTEMPTS_MAX = 10; + +export function normalizeRowExtractorConcurrency(value: unknown): number { + return normalizeIntegerSetting( + value, + 5, + ROW_EXTRACTOR_CONCURRENCY_MIN, + ROW_EXTRACTOR_CONCURRENCY_MAX, + ); +} + +export function normalizeRowExtractorBrowserAttempts(value: unknown): number { + return normalizeIntegerSetting( + value, + 2, + ROW_EXTRACTOR_BROWSER_ATTEMPTS_MIN, + ROW_EXTRACTOR_BROWSER_ATTEMPTS_MAX, + ); +} + +function normalizeIntegerSetting( + value: unknown, + fallback: number, + min: number, + max: number, +): number { + const parsed = typeof value === "number" ? value : Number(value); + if (!Number.isFinite(parsed)) return fallback; + return Math.min(max, Math.max(min, Math.trunc(parsed))); +} + +const DEFAULT_ROW_EXTRACTOR_CONCURRENCY = normalizeRowExtractorConcurrency( + env.ROW_EXTRACTOR_CONCURRENCY, +); +const DEFAULT_ROW_EXTRACTOR_BROWSER_ATTEMPTS = + normalizeRowExtractorBrowserAttempts(env.ROW_EXTRACTOR_BROWSER_ATTEMPTS); + +const OPENAI_MODEL_EXCLUDE_PATTERNS = [ + "audio", + "babbage", + "dall-e", + "davinci", + "embedding", + "image", + "instruct", + "moderation", + "realtime", + "sora", + "transcribe", + "tts", + "whisper", +]; + +const GOOGLE_MODEL_EXCLUDE_PATTERNS = [ + "audio", + "embedding", + "imagen", + "image", + "live", + "lyria", + "nano-banana", + "robotics", + "tts", + "veo", +]; + +const TEXT_MODEL_EXCLUDE_PATTERNS = [ + "audio", + "babbage", + "dall-e", + "embedding", + "image", + "moderation", + "rerank", + "safeguard", + "sdxl", + "speech", + "stable-diffusion", + "transcribe", + "tts", + "video", + "voice", + "wan", + "whisper", +]; + +const QWEN_MODELS: OpenRouterModel[] = [ + { + modelName: "qwen-plus", + canonicalSlug: "qwen-plus", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen3.5-plus", + canonicalSlug: "qwen3.5-plus", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen3-max", + canonicalSlug: "qwen3-max", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen-max", + canonicalSlug: "qwen-max", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen-flash", + canonicalSlug: "qwen-flash", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen3-235b-a22b-instruct-2507", + canonicalSlug: "qwen3-235b-a22b-instruct-2507", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen3-235b-a22b-thinking-2507", + canonicalSlug: "qwen3-235b-a22b-thinking-2507", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, + { + modelName: "qwen3-coder-plus", + canonicalSlug: "qwen3-coder-plus", + contextLength: 0, + completionCost: 0, + promptCost: 0, + }, +]; + +function isOpenAITextModelId(id: string): boolean { + const lower = id.toLowerCase(); + if (OPENAI_MODEL_EXCLUDE_PATTERNS.some((pattern) => lower.includes(pattern))) { + return false; + } + return ( + lower.startsWith("gpt-") || + lower.startsWith("o1") || + lower.startsWith("o3") || + lower.startsWith("o4") || + lower.startsWith("chatgpt-") + ); +} + +function isGenericTextModelId(id: string): boolean { + const lower = id.toLowerCase(); + return !TEXT_MODEL_EXCLUDE_PATTERNS.some((pattern) => + lower.includes(pattern), + ); +} + +function isGoogleTextModelId(id: string): boolean { + const lower = id.toLowerCase(); + if (GOOGLE_MODEL_EXCLUDE_PATTERNS.some((pattern) => lower.includes(pattern))) { + return false; + } + return ( + lower.startsWith("gemini-") || + lower.startsWith("gemma-") || + lower.startsWith("deep-research-") + ); +} + +function isMistralTextModelId(id: string): boolean { + const lower = id.toLowerCase(); + return ( + isGenericTextModelId(id) && + (lower.startsWith("mistral-") || + lower.startsWith("magistral-") || + lower.startsWith("ministral-") || + lower.startsWith("codestral-") || + lower.startsWith("devstral-") || + lower.startsWith("pixtral-")) + ); +} + +function isProviderTextModelId( + id: string, + provider: Awaited>, +): boolean { + if (!provider) return true; + switch (provider.provider) { + case "openrouter": + return id.includes("/"); + case "openai": + return isOpenAITextModelId(id) && !id.includes("/"); + case "anthropic": + return id.startsWith("claude-") && !id.includes("/"); + case "google": + return isGoogleTextModelId(id) && !id.includes("/"); + case "xai": + return id.startsWith("grok-") && !id.includes("imagine"); + case "deepseek": + return id.startsWith("deepseek-"); + case "qwen": + return id.startsWith("qwen") || id.startsWith("qwq-"); + case "mistral": + return isMistralTextModelId(id); + case "groq": + case "togetherai": + case "deepinfra": + case "fireworks": + case "huggingface": + return isGenericTextModelId(id); + case "ollama": + case "lmstudio": + case "custom": + return true; + } +} + +function googleModelIdFromName(name: string): string { + return name.replace(/^models\//, ""); +} + +function sortModels(models: OpenRouterModel[]): OpenRouterModel[] { + return models.sort((a, b) => a.modelName.localeCompare(b.modelName)); +} + +function isModelCompatibleWithProvider( + modelId: string | undefined, + provider: Awaited>, +): modelId is string { + if (!modelId) return false; + return isProviderTextModelId(modelId, provider); +} + +function modelForProvider( + savedModel: string | undefined, + role: ModelRoleKey, + envDefault: string, + provider: Awaited>, +): string { + if (isModelCompatibleWithProvider(savedModel, provider)) return savedModel; + if (provider?.provider) return defaultModelForLlmProviderRole(provider.provider, role); + return envDefault; +} + +async function fetchJsonWithTimeout( + url: string, + headers: Record, +): Promise { + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS); + + try { + const response = await fetch(url, { headers, signal: controller.signal }); + if (!response.ok) { + throw new Error(`Model list request failed with HTTP ${response.status}.`); + } + return (await response.json()) as T; + } finally { + clearTimeout(timeout); + } +} + /** * Model roles for the settings UI. */ @@ -35,6 +317,7 @@ export const MODEL_ROLES = [ { key: "schemaInference", label: "Schema Inference" }, { key: "populateOrchestrator", label: "Populate Orchestrator" }, { key: "investigateSubagent", label: "Investigate Subagent" }, + { key: "extractorBuilder", label: "Extractor Builder" }, ] as const; /** @@ -58,6 +341,123 @@ export async function getCachedModels(): Promise { return fetched; } +export async function fetchModelsForCurrentLlmProvider(): Promise { + const config = await getLlmProviderConfig(); + if (!config) { + throw new Error("LLM provider is not configured."); + } + + if (config.provider === "openrouter") { + return await getCachedModels(); + } + + if (config.provider === "anthropic") { + const baseUrl = (config.baseUrl || "https://api.anthropic.com/v1").replace(/\/+$/, ""); + const json = await fetchJsonWithTimeout<{ + data?: Array<{ + id: string; + display_name?: string; + max_input_tokens?: number; + }>; + }>(`${baseUrl}/models?limit=100`, { + "x-api-key": config.apiKey, + "anthropic-version": "2023-06-01", + }); + + return sortModels( + (json.data ?? []).map((model) => ({ + modelName: model.display_name ?? model.id, + canonicalSlug: model.id, + contextLength: model.max_input_tokens ?? 0, + completionCost: 0, + promptCost: 0, + })), + ); + } + + if (config.provider === "google") { + const baseUrl = ( + config.baseUrl || + defaultBaseUrlForLlmProvider("google") || + "https://generativelanguage.googleapis.com/v1beta" + ).replace(/\/+$/, ""); + const json = await fetchJsonWithTimeout<{ + models?: Array<{ + name: string; + baseModelId?: string; + displayName?: string; + inputTokenLimit?: number; + outputTokenLimit?: number; + supportedActions?: string[]; + supportedGenerationMethods?: string[]; + }>; + }>(`${baseUrl}/models`, { + "x-goog-api-key": config.apiKey, + }); + + return sortModels( + (json.models ?? []) + .map((model) => { + const modelId = model.baseModelId || googleModelIdFromName(model.name); + return { + model, + modelId, + actions: + model.supportedActions ?? model.supportedGenerationMethods ?? [], + }; + }) + .filter(({ modelId, actions }) => { + return ( + isGoogleTextModelId(modelId) && + (actions.length === 0 || actions.includes("generateContent")) + ); + }) + .map(({ model, modelId }) => ({ + modelName: model.displayName ?? modelId, + canonicalSlug: modelId, + contextLength: model.inputTokenLimit ?? 0, + completionCost: 0, + promptCost: 0, + })), + ); + } + + if (config.provider === "qwen") { + return sortModels([...QWEN_MODELS]); + } + + const baseUrl = ( + config.baseUrl || + defaultBaseUrlForLlmProvider(config.provider) || + "https://api.openai.com/v1" + ).replace(/\/+$/, ""); + const headers: Record = + ["custom", "ollama", "lmstudio"].includes(config.provider) && !config.apiKey + ? {} + : { Authorization: `Bearer ${config.apiKey}` }; + const json = await fetchJsonWithTimeout<{ + data?: Array<{ + id: string; + display_name?: string; + name?: string; + context_length?: number; + contextLength?: number; + }>; + }>(modelsUrlForLlmProvider(config.provider, baseUrl), headers); + + const models = (json.data ?? []) + .filter((model) => isProviderTextModelId(model.id, config)) + .map((model) => ({ + modelName: model.display_name ?? model.name ?? model.id, + canonicalSlug: model.id, + contextLength: model.context_length ?? model.contextLength ?? 0, + completionCost: 0, + promptCost: 0, + })); + + return sortModels(models); +} + /** * Validate that a model slug exists in the cached model list. * Throws with a clear message if the slug is not found. @@ -65,7 +465,11 @@ export async function getCachedModels(): Promise { */ export async function validateModelSlug( slug: string, - role: "schemaInference" | "populateOrchestrator" | "investigateSubagent" + role: + | "schemaInference" + | "populateOrchestrator" + | "investigateSubagent" + | "extractorBuilder" ): Promise { const models = await getCachedModels(); const found = models.some((m) => m.canonicalSlug === slug); @@ -96,19 +500,33 @@ export async function upsertModelConfig( schemaInference?: string; populateOrchestrator?: string; investigateSubagent?: string; + extractorBuilder?: string; + rowExtractorConcurrency?: number; + rowExtractorBrowserAttempts?: number; } ): Promise { + const llmConfig = await getLlmProviderConfig(); await convex.mutation(internal.modelConfig.upsertInternal, { userId, + provider: llmConfig?.provider ?? "openrouter", schemaInference: config.schemaInference ?? undefined, populateOrchestrator: config.populateOrchestrator ?? undefined, investigateSubagent: config.investigateSubagent ?? undefined, + extractorBuilder: config.extractorBuilder ?? undefined, + rowExtractorConcurrency: + config.rowExtractorConcurrency !== undefined + ? normalizeRowExtractorConcurrency(config.rowExtractorConcurrency) + : undefined, + rowExtractorBrowserAttempts: + config.rowExtractorBrowserAttempts !== undefined + ? normalizeRowExtractorBrowserAttempts(config.rowExtractorBrowserAttempts) + : undefined, }); } /** * Fetch the model configuration for a specific user from Convex. - * If the user has no saved config, returns the system defaults from env. + * If the user has no saved config, returns the selected provider default or env defaults. * Callers always get a complete config — never null. */ export async function getModelConfig( @@ -117,12 +535,47 @@ export async function getModelConfig( schemaInference: string; populateOrchestrator: string; investigateSubagent: string; + extractorBuilder: string; + rowExtractorConcurrency: number; + rowExtractorBrowserAttempts: number; }> { - const config = await convex.query(internal.modelConfig.getInternal, { userId }); + const llmConfig = await getLlmProviderConfig(); + const config = await convex.query(internal.modelConfig.getInternal, { + userId, + provider: llmConfig?.provider ?? "openrouter", + }); return { - schemaInference: config?.schemaInference ?? DEFAULT_MODEL_IDS.SCHEMA_INFERENCE, - populateOrchestrator: config?.populateOrchestrator ?? DEFAULT_MODEL_IDS.POPULATE_ORCHESTRATOR, - investigateSubagent: config?.investigateSubagent ?? DEFAULT_MODEL_IDS.INVESTIGATE_SUBAGENT, + schemaInference: modelForProvider( + config?.schemaInference, + "schemaInference", + DEFAULT_MODEL_IDS.SCHEMA_INFERENCE, + llmConfig, + ), + populateOrchestrator: modelForProvider( + config?.populateOrchestrator, + "populateOrchestrator", + DEFAULT_MODEL_IDS.POPULATE_ORCHESTRATOR, + llmConfig, + ), + investigateSubagent: modelForProvider( + config?.investigateSubagent, + "investigateSubagent", + DEFAULT_MODEL_IDS.INVESTIGATE_SUBAGENT, + llmConfig, + ), + extractorBuilder: modelForProvider( + config?.extractorBuilder, + "extractorBuilder", + DEFAULT_MODEL_IDS.EXTRACTOR_BUILDER, + llmConfig, + ), + rowExtractorConcurrency: normalizeRowExtractorConcurrency( + config?.rowExtractorConcurrency ?? DEFAULT_ROW_EXTRACTOR_CONCURRENCY, + ), + rowExtractorBrowserAttempts: normalizeRowExtractorBrowserAttempts( + config?.rowExtractorBrowserAttempts ?? + DEFAULT_ROW_EXTRACTOR_BROWSER_ATTEMPTS, + ), }; } diff --git a/backend/src/env.ts b/backend/src/env.ts index adcec52..c9b2366 100644 --- a/backend/src/env.ts +++ b/backend/src/env.ts @@ -1,4 +1,5 @@ import { config as loadDotenv } from "dotenv"; +import { existsSync } from "node:fs"; import { fileURLToPath } from "node:url"; loadDotenv({ path: fileURLToPath(new URL("../../.env", import.meta.url)) }); @@ -18,10 +19,17 @@ function numberFromEnv(name: string, fallback: number): number { return Number.isFinite(parsed) ? parsed : fallback; } +function positiveIntegerFromEnv(name: string, fallback: number): number { + const parsed = numberFromEnv(name, fallback); + return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback; +} + export const env = { PROD: process.env.PROD, IS_PROD: process.env.PROD === "1", IS_LOCAL_MODE: process.env.PROD !== "1", + IS_DOCKER: + process.env.BIGSET_RUNNING_IN_DOCKER === "1" || existsSync("/.dockerenv"), CLIENT_ORIGIN: process.env.CLIENT_ORIGIN || "http://localhost:3500", CONVEX_URL: required("CONVEX_URL"), PORT: numberFromEnv("PORT", 3501), @@ -45,13 +53,25 @@ export const env = { LOCAL_KEYCHAIN_TIMEOUT_MS: numberFromEnv("LOCAL_KEYCHAIN_TIMEOUT_MS", 5_000), // Default models — used when a user has not saved a preference. - // Each must be a valid OpenRouter model slug. + // In production these are still interpreted as OpenRouter model slugs; in + // local mode the selected LLM provider's default model is used first. SCHEMA_INFERENCE_MODEL: process.env.SCHEMA_INFERENCE_MODEL ?? "anthropic/claude-sonnet-4.6", POPULATE_ORCHESTRATOR_MODEL: process.env.POPULATE_ORCHESTRATOR_MODEL ?? "qwen/qwen3.7-max", + POPULATE_ORCHESTRATOR_MAX_STEPS: positiveIntegerFromEnv( + "POPULATE_ORCHESTRATOR_MAX_STEPS", + 80, + ), INVESTIGATE_SUBAGENT_MODEL: process.env.INVESTIGATE_SUBAGENT_MODEL ?? "qwen/qwen3.7-max", + EXTRACTOR_BUILDER_MODEL: + process.env.EXTRACTOR_BUILDER_MODEL ?? "anthropic/claude-sonnet-4.6", + ROW_EXTRACTOR_CONCURRENCY: numberFromEnv("ROW_EXTRACTOR_CONCURRENCY", 5), + ROW_EXTRACTOR_BROWSER_ATTEMPTS: numberFromEnv( + "ROW_EXTRACTOR_BROWSER_ATTEMPTS", + 2, + ), // Resend (transactional email). Optional — when RESEND_API_KEY is unset // the email module no-ops with a log line, so local dev works without diff --git a/backend/src/index.ts b/backend/src/index.ts index 4d63f42..df461c0 100644 --- a/backend/src/index.ts +++ b/backend/src/index.ts @@ -5,8 +5,15 @@ import { z } from "zod"; import { env } from "./env.js"; import clerkAuthPlugin, { requireAuth, getUserEmail } from "./clerk-auth.js"; -import { inferSchema } from "./pipeline/schema-inference.js"; +import { + finalizeSchemaContracts, + inferSchema, +} from "./pipeline/schema-inference.js"; import { datasetContextSchema, type DatasetContext } from "./pipeline/populate.js"; +import { + normalizeCodificationProfile, + schemaCodificationProfileToRuntime, +} from "./pipeline/codification.js"; import { populateWorkflow } from "./mastra/workflows/populate.js"; import { updateWorkflow } from "./mastra/workflows/update.js"; import { convex, internal } from "./convex.js"; @@ -14,7 +21,14 @@ import { sendTransactionalEmail } from "./email/send.js"; import { datasetReadyTemplate } from "./email/templates/dataset-ready.js"; import { capture, shutdown as shutdownAnalytics } from "./analytics/posthog.js"; import { EVENTS } from "./analytics/events.js"; -import { registerDataset, deregisterDataset, abortDataset } from "./abort-registry.js"; +import { + activeDatasetRunCount, + abortDataset, + deregisterDataset, + hasActiveDatasetRuns, + isDatasetRunAborted, + registerDataset, +} from "./abort-registry.js"; import { LOCAL_USER_ID, clearLegacyPlaintextLocalCredentials, @@ -22,9 +36,17 @@ import { getLocalSetupStatus, requireLocalSetupComplete, saveLocalCredential, + saveLocalLlmProviderConfig, + setActiveLocalLlmProvider, verifyOpenRouterApiKey, verifyTinyFishApiKey, } from "./local-credentials.js"; +import { + isLlmProviderType, + normalizeLlmProviderInput, + verifyLlmProviderConfig, + type LlmProviderInput, +} from "./config/llm.js"; /** Domain part of an email, for analytics (we never log full addresses). */ function emailDomain(email: string): string { @@ -48,8 +70,18 @@ type DatasetUpdateBeginOutcome = | "already_building" | "already_updating"; type UpdateWorkflowRun = Awaited>; +type RuntimeModelConfig = { + schemaInference: string; + populateOrchestrator: string; + investigateSubagent: string; + extractorBuilder: string; + rowExtractorConcurrency: number; + rowExtractorBrowserAttempts: number; +}; const refreshCadenceSchema = z.enum(["manual", "30m", "6h", "12h", "daily", "weekly"]); +const datasetColumnTypeSchema = z.enum(["text", "number", "boolean", "url", "date"]); +const retrievalStrategySchema = z.enum(["search_fetch", "browser", "hybrid"]); const cliCreateDatasetSchema = z.object({ prompt: z.string().trim().min(1), maxRowCount: z.number().int().min(1).max(2500).default(100), @@ -58,6 +90,23 @@ const cliCreateDatasetSchema = z.object({ const cliDatasetIdParamsSchema = z.object({ datasetId: z.string().min(1), }); +const finalizeSchemaRequestSchema = z.object({ + prompt: z.string().trim().min(1), + datasetName: z.string().trim().min(1).optional(), + columns: z + .array( + z.object({ + name: z.string().trim().min(1), + type: datasetColumnTypeSchema, + description: z.string().optional(), + isPrimaryKey: z.boolean().optional(), + }), + ) + .min(1), + retrievalStrategy: retrievalStrategySchema.optional(), + sourceHint: z.string().optional(), + modelSlug: z.string().optional(), +}); function cliDatasetNameFromSchemaName(name: string): string { return name @@ -76,6 +125,12 @@ function mapSchemaTypeToDatasetType( return "text"; } +function mapDatasetTypeToSchemaType( + type: "text" | "number" | "boolean" | "url" | "date", +): "string" | "number" | "boolean" | "url" | "date" { + return type === "text" ? "string" : type; +} + function requireLocalCli(reply: FastifyReply): string | null { if (!env.IS_LOCAL_MODE) { void reply.code(404).send({ error: "Not found" }); @@ -117,6 +172,30 @@ async function beginDatasetPopulate( return claim.outcome; } +async function withCodificationProfile( + input: DatasetContext, + logger: FastifyBaseLogger, +): Promise { + if (input.codificationProfile) return input; + + const codificationProfile = normalizeCodificationProfile(undefined, input); + try { + await convex.mutation(internal.datasets.setCodificationProfileInternal, { + id: input.datasetId, + codificationProfile, + }); + } catch (err) { + logger.warn( + { err, datasetId: input.datasetId }, + "Failed to persist legacy codification profile; continuing with in-memory profile", + ); + } + return { + ...input, + codificationProfile, + }; +} + async function sendDatasetReadyNotification({ logger, clerk, @@ -200,7 +279,7 @@ async function ensureLocalSetupReady(reply: FastifyReply): Promise { return true; } catch { await reply.code(428).send({ - error: "Local setup is incomplete. Connect TinyFish and OpenRouter first.", + error: "Local setup is incomplete. Connect TinyFish and an LLM provider first.", }); return false; } @@ -245,6 +324,36 @@ async function finaliseRunAsLive({ } } +async function clearPendingUpdateStatuses({ + logger, + datasetId, + reason, +}: { + logger: FastifyBaseLogger; + datasetId: string; + reason: string; +}): Promise { + try { + const cleared = await convex.mutation( + internal.datasetRows.clearAllPendingUpdateStatus, + { datasetId }, + ); + if (cleared > 0) { + logger.info( + { datasetId, cleared, reason }, + "Cleared pending update row statuses", + ); + } + return cleared; + } catch (err) { + logger.error( + { err, datasetId, reason }, + "Failed to clear pending update row statuses", + ); + return null; + } +} + async function beginDatasetUpdate( datasetId: string, ownerId: string, @@ -269,11 +378,7 @@ async function runUpdateWorkflowInBackground({ authorizedUserId: string; logger: FastifyBaseLogger; clerk: ClerkClient; - modelConfig: { - schemaInference: string; - populateOrchestrator: string; - investigateSubagent: string; - }; + modelConfig: RuntimeModelConfig; }): Promise { const datasetId = input.datasetId; // registerDataset is called by the route handler before void-ing this @@ -331,11 +436,16 @@ async function runUpdateWorkflowInBackground({ workflowType: "update", }); } catch (err) { - // Note: a user-triggered stop is NOT handled here. The update workflow's - // refreshRowsStep detects the abort internally, clears pending row - // statuses, and returns normally — so run.start() returns { status: - // "success" } and the success path above handles the live transition. - // This catch only fires on genuine failures. + if (isDatasetRunAborted(datasetId)) { + await finaliseStoppedUpdateRun({ + logger, + clerk, + datasetId, + authorizedUserId, + }); + return; + } + const lastStatusError = statusErrorMessage(err); logger.error({ err, datasetId }, "Update background workflow failed"); @@ -350,6 +460,11 @@ async function runUpdateWorkflowInBackground({ ); return; } + await clearPendingUpdateStatuses({ + logger, + datasetId, + reason: "failed update workflow", + }); await setDatasetPopulateStatus(datasetId, "failed", lastStatusError); } catch (statusErr) { logger.error( @@ -373,11 +488,7 @@ async function runScheduledUpdateWorkflowInBackground({ run: UpdateWorkflowRun; authorizedUserId: string; logger: FastifyBaseLogger; - modelConfig: { - schemaInference: string; - populateOrchestrator: string; - investigateSubagent: string; - }; + modelConfig: RuntimeModelConfig; }): Promise { const datasetId = input.datasetId; registerDataset(datasetId); @@ -409,15 +520,55 @@ async function runScheduledUpdateWorkflowInBackground({ await convex.mutation(internal.datasets.completeScheduledRefreshInternal, { id: datasetId, + runId: run.runId, now: Date.now(), }); } catch (err) { + if (isDatasetRunAborted(datasetId)) { + logger.info({ datasetId }, "Scheduled update workflow stopped; transitioning to live"); + const cleared = await clearPendingUpdateStatuses({ + logger, + datasetId, + reason: "stopped scheduled update workflow", + }); + + try { + if (cleared === null) { + await convex.mutation(internal.datasets.failScheduledRefreshInternal, { + id: datasetId, + runId: run.runId, + now: Date.now(), + lastStatusError: "Workflow stopped but pending row cleanup failed", + }); + } else { + await convex.mutation(internal.datasets.completeScheduledRefreshInternal, { + id: datasetId, + runId: run.runId, + now: Date.now(), + }); + } + } catch (statusErr) { + logger.error( + { err: statusErr, datasetId }, + "Failed to record stopped scheduled refresh", + ); + } + return; + } + const lastStatusError = statusErrorMessage(err); logger.error({ err, datasetId }, "Scheduled update workflow failed"); + await clearPendingUpdateStatuses({ + logger, + datasetId, + reason: "failed scheduled update workflow", + }); + try { await convex.mutation(internal.datasets.failScheduledRefreshInternal, { id: datasetId, + runId: run.runId, now: Date.now(), lastStatusError, }); @@ -432,6 +583,69 @@ async function runScheduledUpdateWorkflowInBackground({ } } +async function finaliseStoppedUpdateRun({ + logger, + clerk, + datasetId, + authorizedUserId, +}: { + logger: FastifyBaseLogger; + clerk: ClerkClient; + datasetId: string; + authorizedUserId: string; +}): Promise { + logger.info({ datasetId }, "Update workflow stopped by user; transitioning to live"); + + const cleared = await clearPendingUpdateStatuses({ + logger, + datasetId, + reason: "stopped update workflow", + }); + + if (cleared === null) { + try { + await setDatasetPopulateStatus( + datasetId, + "failed", + "Workflow stopped but pending row cleanup failed", + ); + } catch (fallbackErr) { + logger.error( + { err: fallbackErr, datasetId }, + "Could not update dataset status after stopped update cleanup failure", + ); + } + return; + } + + try { + await finaliseRunAsLive({ + logger, + clerk, + datasetId, + authorizedUserId, + workflowType: "update", + }); + } catch (stopErr) { + logger.error( + { err: stopErr, datasetId }, + "Failed to finalise stopped update run; marking as failed", + ); + try { + await setDatasetPopulateStatus( + datasetId, + "failed", + "Workflow stopped but could not be finalised", + ); + } catch (fallbackErr) { + logger.error( + { err: fallbackErr, datasetId }, + "Could not update dataset status after stopped update finalisation failure", + ); + } + } +} + async function runPopulateWorkflowInBackground({ input, run, @@ -447,11 +661,7 @@ async function runPopulateWorkflowInBackground({ authorizedUserId: string; logger: FastifyBaseLogger; clerk: ClerkClient; - modelConfig: { - schemaInference: string; - populateOrchestrator: string; - investigateSubagent: string; - }; + modelConfig: RuntimeModelConfig; }): Promise { const datasetId = input.datasetId; @@ -467,6 +677,16 @@ async function runPopulateWorkflowInBackground({ }, }); + if (controller.signal.aborted) { + await finaliseStoppedPopulateRun({ + logger, + clerk, + datasetId, + authorizedUserId, + }); + return; + } + logger.info( { workflowStatus: result.status, @@ -509,21 +729,12 @@ async function runPopulateWorkflowInBackground({ }); } catch (err) { if (controller.signal.aborted) { - // User pressed Stop — treat whatever was collected as the final dataset. - logger.info({ datasetId }, "Populate workflow stopped by user; transitioning to live"); - try { - await finaliseRunAsLive({ logger, clerk, datasetId, authorizedUserId }); - } catch (stopErr) { - logger.error({ err: stopErr, datasetId }, "Failed to finalise stopped populate run; marking as failed"); - // Ensure the dataset always leaves "building" — without this fallback, - // a failed finalisation leaves the dataset with no active registry entry - // and no way for /stop to act on it again. - try { - await setDatasetPopulateStatus(datasetId, "failed", "Workflow stopped but could not be finalised"); - } catch (fallbackErr) { - logger.error({ err: fallbackErr, datasetId }, "Could not update dataset status after stop finalisation failure"); - } - } + await finaliseStoppedPopulateRun({ + logger, + clerk, + datasetId, + authorizedUserId, + }); return; } @@ -557,6 +768,43 @@ async function runPopulateWorkflowInBackground({ } } +async function finaliseStoppedPopulateRun({ + logger, + clerk, + datasetId, + authorizedUserId, +}: { + logger: FastifyBaseLogger; + clerk: ClerkClient; + datasetId: string; + authorizedUserId: string; +}): Promise { + logger.info({ datasetId }, "Populate workflow stopped by user; transitioning to live"); + try { + await finaliseRunAsLive({ logger, clerk, datasetId, authorizedUserId }); + } catch (stopErr) { + logger.error( + { err: stopErr, datasetId }, + "Failed to finalise stopped populate run; marking as failed", + ); + // Ensure the dataset always leaves "building" — without this fallback, + // a failed finalisation leaves the dataset with no active registry entry + // and no way for /stop to act on it again. + try { + await setDatasetPopulateStatus( + datasetId, + "failed", + "Workflow stopped but could not be finalised", + ); + } catch (fallbackErr) { + logger.error( + { err: fallbackErr, datasetId }, + "Could not update dataset status after stop finalisation failure", + ); + } + } +} + async function backfillDatasetRefreshSettings( logger: FastifyBaseLogger, ): Promise { @@ -587,6 +835,14 @@ function startLocalRefreshScheduler( ticking = true; try { + if (hasActiveDatasetRuns()) { + logger.debug( + { activeRuns: activeDatasetRunCount() }, + "Skipping scheduled refresh tick while dataset run is active", + ); + return; + } + if (env.IS_LOCAL_MODE) { const setup = await getLocalSetupStatus(); if (!setup.complete) return; @@ -631,15 +887,22 @@ function startLocalRefreshScheduler( const dataset = claim.dataset; const { getModelConfig } = await import("./config/models.js"); const modelConfig = await getModelConfig(dataset.ownerId); - - void runScheduledUpdateWorkflowInBackground({ - input: { + const input = await withCodificationProfile( + { datasetId: dataset.datasetId, datasetName: dataset.datasetName, description: dataset.description, maxRowCount: dataset.maxRowCount ?? 100, columns: dataset.columns, + retrievalStrategy: dataset.retrievalStrategy, + sourceHint: dataset.sourceHint, + codificationProfile: dataset.codificationProfile, }, + logger, + ); + + await runScheduledUpdateWorkflowInBackground({ + input, run, authorizedUserId: dataset.ownerId, logger, @@ -752,6 +1015,54 @@ fastify.post("/local-setup/tinyfish", async (req, reply) => { } }); +fastify.post("/local-setup/llm-provider", async (req, reply) => { + if (!env.IS_LOCAL_MODE) { + return reply.code(404).send({ error: "Not found" }); + } + + const body = (req.body ?? {}) as Partial; + const provider = body.provider; + if (!isLlmProviderType(provider)) { + return reply.code(400).send({ error: "Choose a supported LLM provider" }); + } + + try { + const apiKey = body.apiKey?.trim() ?? ""; + const isKeylessProvider = + provider === "custom" || provider === "ollama" || provider === "lmstudio"; + const isNewKeylessProvider = + isKeylessProvider && (provider !== "custom" || !!body.baseUrl?.trim()); + + if (!apiKey && !isNewKeylessProvider) { + const status = await getLocalSetupStatus(); + const savedProvider = status.services.llmProviders?.[provider]; + if (!savedProvider?.configured) { + return reply.code(400).send({ error: `${savedProvider?.providerLabel ?? provider} API key is required` }); + } + await setActiveLocalLlmProvider(provider); + return await getLocalSetupStatus(); + } + + const config = normalizeLlmProviderInput( + { + provider, + apiKey, + baseUrl: body.baseUrl, + defaultModel: body.defaultModel, + }, + "local", + ); + await verifyLlmProviderConfig(config); + await saveLocalLlmProviderConfig(config, "api_key"); + return await getLocalSetupStatus(); + } catch (err) { + const message = err instanceof Error ? err.message : "LLM provider verification failed"; + req.log.warn({ err }, "LLM provider local setup verification failed"); + return reply.code(400).send({ error: message }); + } +}); + +// Backward-compatible endpoint for older setup UI builds. fastify.post("/local-setup/openrouter-key", async (req, reply) => { if (!env.IS_LOCAL_MODE) { return reply.code(404).send({ error: "Not found" }); @@ -823,6 +1134,18 @@ fastify.get("/openrouter/models", async (req, reply) => { } }); +fastify.get("/llm-provider/models", async (req, reply) => { + const { fetchModelsForCurrentLlmProvider } = await import("./config/models.js"); + try { + const models = await fetchModelsForCurrentLlmProvider(); + return { models }; + } catch (err) { + const message = err instanceof Error ? err.message : "Failed to load models"; + req.log.error(err, "Failed to load current LLM provider models"); + return reply.code(500).send({ error: message }); + } +}); + // ──────────────────────────────────────────────────────────────────────── // Local trusted CLI routes // ──────────────────────────────────────────────────────────────────────── @@ -856,12 +1179,20 @@ fastify.post("/cli/datasets", async (req, reply) => { } try { - const schema = await inferSchema(parsed.data.prompt); + const { getModelConfig } = await import("./config/models.js"); + const modelConfig = await getModelConfig(ownerId); + const schema = await inferSchema( + parsed.data.prompt, + modelConfig.schemaInference, + ); const columns = schema.columns.map((column) => ({ name: column.name, type: mapSchemaTypeToDatasetType(column.type), description: column.retrieval_hint, isPrimaryKey: column.is_primary_key || undefined, + nullable: column.nullable, + validationRegex: column.validation_regex || undefined, + normalizationHint: column.normalization_hint || undefined, })); const datasetId = await convex.mutation( @@ -875,6 +1206,7 @@ fastify.post("/cli/datasets", async (req, reply) => { columns, retrievalStrategy: schema.retrieval_strategy, sourceHint: schema.source_hint, + codificationProfile: schemaCodificationProfileToRuntime(schema.codification_profile), }, ); @@ -971,15 +1303,22 @@ fastify.post("/cli/datasets/:datasetId/populate", async (req, reply) => { const modelConfig = await getModelConfig(ownerId); const run = await populateWorkflow.createRun(); const controller = registerDataset(dataset._id); - - void runPopulateWorkflowInBackground({ - input: { + const input = await withCodificationProfile( + { datasetId: dataset._id, datasetName: dataset.name, description: dataset.description, maxRowCount: dataset.maxRowCount ?? 100, columns: dataset.columns, + retrievalStrategy: dataset.retrievalStrategy, + sourceHint: dataset.sourceHint, + codificationProfile: dataset.codificationProfile, }, + req.log, + ); + + void runPopulateWorkflowInBackground({ + input, run, controller, authorizedUserId: ownerId, @@ -1074,39 +1413,78 @@ await fastify.register(async (instance) => { }); instance.post("/settings/models", async (req, reply) => { - const { upsertModelConfig, validateModelSlug, getCachedModels } = await import("./config/models.js"); + const { upsertModelConfig, fetchModelsForCurrentLlmProvider } = await import("./config/models.js"); const body = req.body as { schemaInference?: string | null; populateOrchestrator?: string | null; investigateSubagent?: string | null; + extractorBuilder?: string | null; + rowExtractorConcurrency?: number | null; + rowExtractorBrowserAttempts?: number | null; + }; + const config = { + schemaInference: typeof body.schemaInference === "string" ? body.schemaInference.trim() || undefined : undefined, + populateOrchestrator: typeof body.populateOrchestrator === "string" ? body.populateOrchestrator.trim() || undefined : undefined, + investigateSubagent: typeof body.investigateSubagent === "string" ? body.investigateSubagent.trim() || undefined : undefined, + extractorBuilder: typeof body.extractorBuilder === "string" ? body.extractorBuilder.trim() || undefined : undefined, + rowExtractorConcurrency: + typeof body.rowExtractorConcurrency === "number" + ? body.rowExtractorConcurrency + : undefined, + rowExtractorBrowserAttempts: + typeof body.rowExtractorBrowserAttempts === "number" + ? body.rowExtractorBrowserAttempts + : undefined, }; - const toValidate: Array<{ role: "schemaInference" | "populateOrchestrator" | "investigateSubagent"; slug: string }> = []; - if (body.schemaInference) toValidate.push({ role: "schemaInference", slug: body.schemaInference }); - if (body.populateOrchestrator) toValidate.push({ role: "populateOrchestrator", slug: body.populateOrchestrator }); - if (body.investigateSubagent) toValidate.push({ role: "investigateSubagent", slug: body.investigateSubagent }); + const toValidate: Array<{ + role: + | "schemaInference" + | "populateOrchestrator" + | "investigateSubagent" + | "extractorBuilder"; + slug: string; + }> = []; + if (config.schemaInference) toValidate.push({ role: "schemaInference", slug: config.schemaInference }); + if (config.populateOrchestrator) toValidate.push({ role: "populateOrchestrator", slug: config.populateOrchestrator }); + if (config.investigateSubagent) toValidate.push({ role: "investigateSubagent", slug: config.investigateSubagent }); + if (config.extractorBuilder) toValidate.push({ role: "extractorBuilder", slug: config.extractorBuilder }); if (toValidate.length > 0) { try { - const models = await getCachedModels(); - for (const { role, slug } of toValidate) { - const found = models.some((m) => m.canonicalSlug === slug); - if (!found) { - return reply.code(400).send({ - error: `Invalid model slug "${slug}" for ${role}. Refresh the model list first or choose a different model.`, - }); - } + const models = await fetchModelsForCurrentLlmProvider(); + const invalidModelSlugs = toValidate.filter( + ({ slug }) => !models.some((m) => m.canonicalSlug === slug), + ); + + if (invalidModelSlugs.length > 0) { + req.log.warn( + { invalidModelSlugs }, + "Rejected model slugs that were not returned by the current LLM provider", + ); + const [firstInvalidModelSlug] = invalidModelSlugs; + return reply.code(400).send({ + error: + invalidModelSlugs.length === 1 && firstInvalidModelSlug + ? `Invalid model slug "${firstInvalidModelSlug.slug}" for ${firstInvalidModelSlug.role}` + : "One or more model slugs are not available for the current LLM provider", + details: { invalidModelSlugs }, + }); } } catch (err) { - req.log.error(err, "Failed to validate model slugs — allowing save"); + req.log.error(err, "Failed to validate model slugs"); + return reply.code(502).send({ error: "Failed to validate model preferences" }); } } try { await upsertModelConfig(req.auth!.userId, { - schemaInference: body.schemaInference ?? undefined, - populateOrchestrator: body.populateOrchestrator ?? undefined, - investigateSubagent: body.investigateSubagent ?? undefined, + schemaInference: config.schemaInference, + populateOrchestrator: config.populateOrchestrator, + investigateSubagent: config.investigateSubagent, + extractorBuilder: config.extractorBuilder, + rowExtractorConcurrency: config.rowExtractorConcurrency, + rowExtractorBrowserAttempts: config.rowExtractorBrowserAttempts, }); return { success: true }; } catch (err) { @@ -1142,6 +1520,64 @@ await fastify.register(async (instance) => { } }); + instance.post("/finalize-schema", async (req, reply) => { + const parsed = finalizeSchemaRequestSchema.safeParse(req.body); + if (!parsed.success) { + return reply.code(400).send({ + error: "Invalid request", + details: parsed.error.flatten().fieldErrors, + }); + } + if (!(await ensureLocalSetupReady(reply))) return; + + try { + const auth = req.auth; + let modelSlug = parsed.data.modelSlug; + + if (!modelSlug && auth) { + const { getModelConfig } = await import("./config/models.js"); + const config = await getModelConfig(auth.userId); + if (config?.schemaInference) { + modelSlug = config.schemaInference; + } + } + + const schema = await finalizeSchemaContracts( + { + prompt: parsed.data.prompt, + datasetName: parsed.data.datasetName, + columns: parsed.data.columns.map((column) => ({ + name: column.name, + type: mapDatasetTypeToSchemaType(column.type), + description: column.description, + isPrimaryKey: column.isPrimaryKey, + })), + retrievalStrategy: parsed.data.retrievalStrategy, + sourceHint: parsed.data.sourceHint, + }, + modelSlug, + ); + + if (schema.columns.length !== parsed.data.columns.length) { + req.log.warn( + { + expectedColumnCount: parsed.data.columns.length, + actualColumnCount: schema.columns.length, + }, + "Schema finalization returned a different column count", + ); + return reply.code(502).send({ + error: "Schema finalization failed. Please try again.", + }); + } + + return schema; + } catch (err) { + req.log.error(err, "Schema finalization failed"); + return reply.code(502).send({ error: "Schema finalization failed. Please try again." }); + } + }); + instance.post("/populate", async (req, reply) => { const parsed = datasetContextSchema.safeParse(req.body); if (!parsed.success) { @@ -1200,12 +1636,19 @@ await fastify.register(async (instance) => { // arriving before registerDataset runs inside the background function // would incorrectly force-transition an active run to "failed". const controller = registerDataset(parsed.data.datasetId); - - void runPopulateWorkflowInBackground({ - input: { + const input = await withCodificationProfile( + { ...parsed.data, maxRowCount: dataset.maxRowCount ?? parsed.data.maxRowCount, + retrievalStrategy: dataset.retrievalStrategy ?? parsed.data.retrievalStrategy, + sourceHint: dataset.sourceHint ?? parsed.data.sourceHint, + codificationProfile: dataset.codificationProfile ?? parsed.data.codificationProfile, }, + req.log, + ); + + void runPopulateWorkflowInBackground({ + input, run, controller, authorizedUserId: auth.userId, @@ -1262,6 +1705,13 @@ await fastify.register(async (instance) => { throw new Error(`Unexpected update claim outcome: ${updateOutcome}`); } + const dataset = await convex.query(internal.datasets.getInternal, { + id: parsed.data.datasetId, + }); + if (!dataset) { + return reply.code(404).send({ error: "Dataset not found" }); + } + let run: UpdateWorkflowRun; try { run = await updateWorkflow.createRun(); @@ -1279,9 +1729,19 @@ await fastify.register(async (instance) => { // arriving before registerDataset runs inside the background function // would incorrectly force-transition an active run to "failed". registerDataset(parsed.data.datasetId); + const input = await withCodificationProfile( + { + ...parsed.data, + maxRowCount: dataset.maxRowCount ?? parsed.data.maxRowCount, + retrievalStrategy: dataset.retrievalStrategy ?? parsed.data.retrievalStrategy, + sourceHint: dataset.sourceHint ?? parsed.data.sourceHint, + codificationProfile: dataset.codificationProfile ?? parsed.data.codificationProfile, + }, + req.log, + ); void runUpdateWorkflowInBackground({ - input: parsed.data, + input, run, authorizedUserId: auth.userId, logger: req.log, diff --git a/backend/src/local-credential-types.ts b/backend/src/local-credential-types.ts index bcb4235..f899fc1 100644 --- a/backend/src/local-credential-types.ts +++ b/backend/src/local-credential-types.ts @@ -1,4 +1,22 @@ -export const LOCAL_CREDENTIAL_SERVICES = ["tinyfish", "openrouter"] as const; +export const LOCAL_CREDENTIAL_SERVICES = [ + "tinyfish", + "openrouter", + "openai", + "anthropic", + "google", + "xai", + "deepseek", + "qwen", + "mistral", + "groq", + "togetherai", + "deepinfra", + "fireworks", + "huggingface", + "ollama", + "lmstudio", + "custom", +] as const; export type LocalCredentialService = (typeof LOCAL_CREDENTIAL_SERVICES)[number]; export type ConnectionMethod = "api_key" | "oauth"; diff --git a/backend/src/local-credentials.ts b/backend/src/local-credentials.ts index a6a5b4b..e650cf0 100644 --- a/backend/src/local-credentials.ts +++ b/backend/src/local-credentials.ts @@ -5,6 +5,19 @@ import { getKeychainCredential, setKeychainCredential, } from "./local-keychain-client.js"; +import { + LLM_PROVIDER_TYPES, + defaultBaseUrlForLlmProvider, + defaultModelForLlmProvider, + defaultModelForLlmProviderRole, + isLlmProviderType, + llmProviderLabel, + normalizeLlmProviderInput, + type LlmProviderConfig, + type LlmProviderInput, + type ModelRoleKey, + type LlmProviderType, +} from "./config/llm.js"; import type { ConnectionMethod, LocalCredentialService, @@ -12,18 +25,35 @@ import type { export const LOCAL_USER_ID = "local_user_default"; +const MODEL_ROLE_KEYS: ModelRoleKey[] = [ + "schemaInference", + "populateOrchestrator", + "investigateSubagent", + "extractorBuilder", +]; + export interface ServiceSetupStatus { configured: boolean; source: "local" | "env" | null; connectionMethod: ConnectionMethod | null; verifiedAt: number | null; + provider?: LlmProviderType; + providerLabel?: string; + baseUrl?: string; + defaultModel?: string; } export interface LocalSetupStatus { mode: "local" | "production"; required: boolean; complete: boolean; - services: Record; + services: { + tinyfish: ServiceSetupStatus; + llm: ServiceSetupStatus; + llmProviders?: Record; + /** Deprecated compatibility alias for older UI code. */ + openrouter?: ServiceSetupStatus; + }; } function isPlaceholder(value: string, service: LocalCredentialService): boolean { @@ -35,30 +65,150 @@ function isPlaceholder(value: string, service: LocalCredentialService): boolean function envCredential(service: LocalCredentialService): string | undefined { const value = - service === "tinyfish" ? process.env.TINYFISH_API_KEY : env.OPENROUTER_API_KEY; + service === "tinyfish" + ? process.env.TINYFISH_API_KEY + : service === "openrouter" + ? env.OPENROUTER_API_KEY + : undefined; if (!value || isPlaceholder(value, service)) return undefined; return value; } +function llmProviderService(provider: LlmProviderType): LocalCredentialService { + return provider; +} + +function nonEmptyModelId(value: unknown): string | null { + return typeof value === "string" && value.trim() ? value.trim() : null; +} + +function localLlmProviderAllowsKeylessRead( + provider: LlmProviderType, +): boolean { + return ( + provider === "custom" || provider === "ollama" || provider === "lmstudio" + ); +} + +function localModelConfigRole( + config: unknown, + role: ModelRoleKey, +): string | null { + if (!config || typeof config !== "object") return null; + return nonEmptyModelId( + (config as Partial>)[role], + ); +} + +function localProviderRoleModelConfigured( + provider: LlmProviderType, + role: ModelRoleKey, + config: unknown, +): boolean { + return Boolean( + localModelConfigRole(config, role) || + nonEmptyModelId(defaultModelForLlmProviderRole(provider, role)), + ); +} + +async function localLlmModelRolesConfigured( + provider: LlmProviderType, +): Promise { + if ( + MODEL_ROLE_KEYS.every((role) => + nonEmptyModelId(defaultModelForLlmProviderRole(provider, role)), + ) + ) { + return true; + } + + try { + const config = await convex.query(internal.modelConfig.getInternal, { + userId: LOCAL_USER_ID, + provider, + }); + return MODEL_ROLE_KEYS.every((role) => + localProviderRoleModelConfigured(provider, role, config), + ); + } catch { + return false; + } +} + async function localCredential(service: LocalCredentialService): Promise<{ apiKey: string; connectionMethod: ConnectionMethod; verifiedAt: number | null; keychainAccount: string; + llmProvider?: LlmProviderType; + llmBaseUrl?: string; + llmDefaultModel?: string; } | null> { if (!env.IS_LOCAL_MODE) return null; - const keychain = await getKeychainCredential(service); - if (!keychain?.apiKey) return null; const row = await convex.query(internal.localCredentials.getInternal, { service, }); + const rowData = row as + | { + keychainAccount?: string; + connectionMethod?: ConnectionMethod; + verifiedAt?: number; + llmProvider?: unknown; + llmBaseUrl?: unknown; + llmDefaultModel?: unknown; + } + | null; + + const rowProvider = isLlmProviderType(rowData?.llmProvider) + ? rowData.llmProvider + : undefined; + const rowBaseUrl = + typeof rowData?.llmBaseUrl === "string" ? rowData.llmBaseUrl : undefined; + const rowDefaultModel = + typeof rowData?.llmDefaultModel === "string" + ? rowData.llmDefaultModel + : undefined; + const rowKeychainAccount = + typeof rowData?.keychainAccount === "string" + ? rowData.keychainAccount + : undefined; + const isKeylessLocalLlmProviderRow = Boolean( + rowProvider && + service === rowProvider && + localLlmProviderAllowsKeylessRead(rowProvider) && + rowBaseUrl, + ); + + const keychain = + !isKeylessLocalLlmProviderRow || rowKeychainAccount + ? await getKeychainCredential(service) + : null; + + if (keychain?.apiKey) { + return { + apiKey: keychain.apiKey, + connectionMethod: rowData?.connectionMethod ?? "api_key", + verifiedAt: rowData?.verifiedAt ?? null, + keychainAccount: keychain.keychainAccount, + llmProvider: rowProvider, + llmBaseUrl: rowBaseUrl, + llmDefaultModel: rowDefaultModel, + }; + } + + if (!isKeylessLocalLlmProviderRow) { + return null; + } return { - apiKey: keychain.apiKey, - connectionMethod: row?.connectionMethod ?? "api_key", - verifiedAt: row?.verifiedAt ?? null, - keychainAccount: keychain.keychainAccount, + apiKey: "", + connectionMethod: rowData?.connectionMethod ?? "api_key", + verifiedAt: rowData?.verifiedAt ?? null, + keychainAccount: rowKeychainAccount ?? "", + llmProvider: rowProvider, + llmBaseUrl: rowBaseUrl, + llmDefaultModel: rowDefaultModel, }; } @@ -72,6 +222,57 @@ async function localCredentialForStatus( } } +async function activeLlmProviderForStatus(): Promise { + if (!env.IS_LOCAL_MODE) return "openrouter"; + + try { + const active = await convex.query(internal.localCredentials.getInternal, { + service: "llm", + }); + const activeProvider = (active as { llmProvider?: unknown } | null) + ?.llmProvider; + if (isLlmProviderType(activeProvider)) return activeProvider; + } catch { + // Convex functions may be one push behind during local development. Fall + // back instead of making every backend status/settings route return 500. + } + + const legacy = await localCredentialForStatus("openrouter"); + if (legacy?.llmProvider) return legacy.llmProvider; + return "openrouter"; +} + +async function localCredentialForLlmProvider( + provider: LlmProviderType, +): Promise>> { + const direct = await localCredentialForStatus(llmProviderService(provider)); + if (direct && (!direct.llmProvider || direct.llmProvider === provider)) { + return direct; + } + + if (provider !== "openrouter") { + const legacy = await localCredentialForStatus("openrouter"); + if (legacy?.llmProvider === provider) return legacy; + } + + return null; +} + +export async function setActiveLocalLlmProvider( + provider: LlmProviderType, +): Promise { + if (!env.IS_LOCAL_MODE) { + throw new Error("Local credential storage is disabled when PROD=1."); + } + + await convex.mutation(internal.localCredentials.upsertInternal, { + service: "llm", + connectionMethod: "api_key", + verifiedAt: Date.now(), + llmProvider: provider, + }); +} + export async function resolveCredential( service: LocalCredentialService, ): Promise<{ apiKey: string; source: "local" | "env" } | null> { @@ -86,14 +287,55 @@ export async function resolveCredential( return null; } +export async function getLlmProviderConfig(): Promise { + if (env.IS_LOCAL_MODE) { + const provider = await activeLlmProviderForStatus(); + const local = await localCredentialForLlmProvider(provider); + if (!local) return null; + + return normalizeLlmProviderInput( + { + provider, + apiKey: local.apiKey, + baseUrl: + local.llmBaseUrl ?? defaultBaseUrlForLlmProvider(provider), + defaultModel: + local.llmDefaultModel || defaultModelForLlmProvider(provider), + }, + "local", + ); + } + + const apiKey = envCredential("openrouter"); + if (!apiKey) return null; + return normalizeLlmProviderInput( + { + provider: "openrouter", + apiKey, + baseUrl: process.env.OPENROUTER_BASE_URL, + defaultModel: env.SCHEMA_INFERENCE_MODEL, + }, + "env", + ); +} + +export async function requireLlmProviderConfig(): Promise { + const config = await getLlmProviderConfig(); + if (!config) { + throw new Error("LLM provider is not configured. Complete local setup first."); + } + return config; +} + export async function getOpenRouterApiKey(): Promise { - return (await resolveCredential("openrouter"))?.apiKey; + const config = await getLlmProviderConfig(); + return config?.provider === "openrouter" ? config.apiKey : undefined; } export async function requireOpenRouterApiKey(): Promise { const apiKey = await getOpenRouterApiKey(); if (!apiKey) { - throw new Error("OpenRouter is not configured. Complete local setup first."); + throw new Error("OpenRouter is not configured as the current LLM provider."); } return apiKey; } @@ -140,7 +382,24 @@ export async function requireLocalSetupComplete(): Promise { export async function getLocalSetupStatus(): Promise { if (!env.IS_LOCAL_MODE) { const tinyfish = envCredential("tinyfish"); - const openrouter = envCredential("openrouter"); + const llmConfig = await getLlmProviderConfig(); + const llm: ServiceSetupStatus = llmConfig + ? { + configured: true, + source: llmConfig.source, + connectionMethod: "api_key", + verifiedAt: null, + provider: llmConfig.provider, + providerLabel: llmProviderLabel(llmConfig.provider), + baseUrl: llmConfig.baseUrl, + defaultModel: llmConfig.defaultModel, + } + : { + configured: false, + source: null, + connectionMethod: null, + verifiedAt: null, + }; return { mode: "production", required: false, @@ -152,18 +411,13 @@ export async function getLocalSetupStatus(): Promise { connectionMethod: tinyfish ? "api_key" : null, verifiedAt: null, }, - openrouter: { - configured: !!openrouter, - source: openrouter ? "env" : null, - connectionMethod: openrouter ? "api_key" : null, - verifiedAt: null, - }, + llm, + openrouter: llm, }, }; } const tinyfishLocal = await localCredentialForStatus("tinyfish"); - const openrouterLocal = await localCredentialForStatus("openrouter"); const tinyfish: ServiceSetupStatus = tinyfishLocal ? { @@ -179,25 +433,50 @@ export async function getLocalSetupStatus(): Promise { verifiedAt: null, }; - const openrouter: ServiceSetupStatus = openrouterLocal - ? { - configured: true, - source: "local", - connectionMethod: openrouterLocal.connectionMethod, - verifiedAt: openrouterLocal.verifiedAt, - } - : { - configured: false, - source: null, - connectionMethod: null, - verifiedAt: null, - }; + const providerStatuses = {} as Record; + for (const provider of LLM_PROVIDER_TYPES) { + const credential = await localCredentialForLlmProvider(provider); + providerStatuses[provider] = credential + ? { + configured: true, + source: "local", + connectionMethod: credential.connectionMethod, + verifiedAt: credential.verifiedAt, + provider, + providerLabel: llmProviderLabel(provider), + baseUrl: + credential.llmBaseUrl ?? defaultBaseUrlForLlmProvider(provider), + defaultModel: + credential.llmDefaultModel || defaultModelForLlmProvider(provider), + } + : { + configured: false, + source: null, + connectionMethod: null, + verifiedAt: null, + provider, + providerLabel: llmProviderLabel(provider), + baseUrl: defaultBaseUrlForLlmProvider(provider), + defaultModel: defaultModelForLlmProvider(provider), + }; + } + + const llmProvider = await activeLlmProviderForStatus(); + const llm = providerStatuses[llmProvider]; + const llmModelRolesConfigured = llm.configured + ? await localLlmModelRolesConfigured(llmProvider) + : false; return { mode: "local", required: true, - complete: tinyfish.configured && openrouter.configured, - services: { tinyfish, openrouter }, + complete: tinyfish.configured && llm.configured && llmModelRolesConfigured, + services: { + tinyfish, + llm, + llmProviders: providerStatuses, + openrouter: providerStatuses.openrouter, + }, }; } @@ -215,7 +494,48 @@ export async function saveLocalCredential( keychainAccount, connectionMethod, verifiedAt: Date.now(), + ...(service === "openrouter" + ? { + llmProvider: "openrouter" as const, + llmBaseUrl: defaultBaseUrlForLlmProvider("openrouter"), + llmDefaultModel: defaultModelForLlmProvider("openrouter"), + } + : {}), + }); + + if (service === "openrouter") { + await setActiveLocalLlmProvider("openrouter"); + } +} + +export async function saveLocalLlmProviderConfig( + input: LlmProviderInput, + connectionMethod: ConnectionMethod = "api_key", +): Promise { + if (!env.IS_LOCAL_MODE) { + throw new Error("Local credential storage is disabled when PROD=1."); + } + + const config = normalizeLlmProviderInput(input, "local"); + const keychainAccount = config.apiKey + ? ( + await setKeychainCredential( + llmProviderService(config.provider), + config.apiKey, + ) + ).keychainAccount + : undefined; + await convex.mutation(internal.localCredentials.upsertInternal, { + service: llmProviderService(config.provider), + keychainAccount: keychainAccount ?? null, + connectionMethod, + verifiedAt: Date.now(), + llmProvider: config.provider, + llmBaseUrl: config.baseUrl, + llmDefaultModel: config.defaultModel, }); + await setActiveLocalLlmProvider(config.provider); + return config; } export async function clearLegacyPlaintextLocalCredentials(): Promise { diff --git a/backend/src/mastra/agents/investigate.ts b/backend/src/mastra/agents/investigate.ts index 63a7b8c..593aa6d 100644 --- a/backend/src/mastra/agents/investigate.ts +++ b/backend/src/mastra/agents/investigate.ts @@ -1,41 +1,67 @@ import { Agent } from "@mastra/core/agent"; -import { createOpenRouter } from "@openrouter/ai-sdk-provider"; +import { createLanguageModel, type LlmProviderConfig } from "../../config/llm.js"; import { buildPopulateTools } from "../tools/dataset-tools.js"; -import { searchWebTool, fetchPageTool } from "../tools/web-tools.js"; +import { buildWebTools } from "../tools/web-tools.js"; import type { AuthContext } from "../workflows/populate.js"; import type { PopulateColumn } from "../../pipeline/populate.js"; -function buildInvestigateInstructions(columns: PopulateColumn[]): string { +interface InvestigateAgentOptions { + insertDefaults?: { + data?: Record; + lockColumns?: string[]; + sources?: string[]; + cellSources?: Record; + rowSummary?: string; + howFoundPrefix?: string; + }; + membershipSourceHint?: string; + abortSignal?: AbortSignal; +} + +function buildInvestigateInstructions( + columns: PopulateColumn[], + membershipSourceHint?: string, +): string { const columnNames = columns.map((c) => c.name); + const dataExample = columnNames + .map((n) => `{"column": "${n}", "value": "value"}`) + .join(", "); const columnsDesc = columns .map( (c) => - `- "${c.name}" (${c.type})${c.isPrimaryKey ? " [PRIMARY KEY]" : ""}${c.description ? `: ${c.description}` : ""}`, + `- "${c.name}" (${c.type})${c.isPrimaryKey ? " [PRIMARY KEY]" : ""}${c.nullable === false ? " [REQUIRED]" : c.nullable === true ? " [OPTIONAL]" : ""}${c.validationRegex ? ` validation_regex=${JSON.stringify(c.validationRegex)}` : ""}${c.normalizationHint ? ` normalization_hint=${JSON.stringify(c.normalizationHint)}` : ""}${c.description ? `: ${c.description}` : ""}`, ) .join("\n"); - return `You research one entity and insert one row. Be fast — you have very few steps. + return `You research one entity and insert one row. Work efficiently, but do not sacrifice data quality. Columns: ${columnsDesc} +${membershipSourceHint ? `\nAuthoritative membership source: ${membershipSourceHint}` : ""} RULES: - Do NOT fetch the same URL twice. If a fetch worked, use the data you got. -- You have at most 6 tool calls total. Budget them: 1 fetch + 1 search + 1 fetch + 1 insert = done. -- ALWAYS insert a row, even if some fields are incomplete. Use "" for unknown fields. Partial real data is better than no row. +- Try to stay under 6 tool calls when that is enough, but do not stop early if more searching/fetching is needed to verify requested columns. +- Your goal is to fill every column from real, source-backed evidence. Some columns may be impossible to verify; use "" only after a reasonable targeted attempt. - Never fabricate values. Use "" for anything you cannot verify. +- Treat any browser-extracted candidate values in the prompt as hints, not truth. Re-verify primary key values and source-backed facts before inserting. +- If a primary key cannot be verified from a real source, do not insert the row. For URL primary keys, fetch or otherwise verify the exact current URL; if it 404s, redirects to a different entity, or cannot be justified by source-backed evidence, report INSERTED: false. +- If an authoritative membership source is listed, primary keys must be justified by that source family. Other sources may enrich fields, but they cannot prove that the entity belongs in the dataset. +- Optional/nullable columns should still be researched. Optional only means the row can be inserted blank if you cannot verify the value. +- Normalize values to the schema contract before inserting. Follow validation_regex and normalization_hint when present. +- For every primary key, include cell_sources that justify that exact primary-key value. For URL primary keys, cell_sources must include the exact verified URL. - insert_row rejects duplicates based on primary key columns. If you get a "Duplicate" error, do NOT retry — report INSERTED: false and move on. TOOL CALL FORMAT — every tool call argument must be a JSON object wrapped in curly braces: search_web: {"query": "your search terms"} fetch_page: {"url": "https://example.com"} - insert_row: {"data": {${columnNames.map((n) => `"${n}": "value"`).join(", ")}}, "sources": ["https://url-you-fetched.com"], "row_summary": "one line about this entity", "how_found": "step by step guide on how to extract the data so an agent in the future can do it too"} + insert_row: {"data": [${dataExample}], "sources": ["https://url-you-fetched.com"], "cell_sources": [{"column": "column_name", "sources": ["https://url-that-justifies-this-cell.com"]}], "row_summary": "one line about this entity", "how_found": "step by step guide on how to extract the data so an agent in the future can do it too"} WORKFLOW: 1. Fetch 1-2 of the provided URLs to get real data (if URLs were given). -2. If you need more, run ONE search and fetch the best result. -3. Call insert_row with whatever real data you have. Use "" for missing fields. - Include "sources" (URLs you fetched), "row_summary" (one line about this entity), and "how_found" (a step by step guide on how you found this data. eg, 1. fetch the contents of this url "", 2. Look for the pricing field, and title name field, 3. etc...) +2. For unresolved columns, run targeted searches/fetches until the value is verified, clearly unavailable, or further tool calls are unlikely to help. +3. Call insert_row with the best verified row you can produce. Use "" for any field that remains unknown after the targeted attempt. + Include "sources" (URLs you fetched for the row), "cell_sources" (only URLs that justify the exact cell value for that column), "row_summary" (one line about this entity), and "how_found" (a step by step guide on how you found this data. eg, 1. fetch the contents of this url "", 2. Look for the pricing field, and title name field, 3. etc...) 4. Write your final response: INSERTED: true/false SUMMARY: one line @@ -55,28 +81,35 @@ export function buildInvestigateAgent( authorizedDatasetId: string, authContext: AuthContext, columns: PopulateColumn[], - openRouterApiKey: string, + llmConfig: LlmProviderConfig, + options: InvestigateAgentOptions = {}, ): Agent { const modelSlug = authContext.modelConfig!.investigateSubagent; - const openrouter = createOpenRouter({ - apiKey: openRouterApiKey, - baseURL: process.env.OPENROUTER_BASE_URL, + const webTools = buildWebTools({ + datasetId: authorizedDatasetId, + abortSignal: options.abortSignal, }); const { insert_row } = buildPopulateTools( authorizedDatasetId, authContext, + { + ...options, + columns, + enforcePrimaryKeySources: true, + membershipSourceHint: options.membershipSourceHint, + abortSignal: options.abortSignal, + }, ); return new Agent({ id: "investigate-agent", name: "Dataset Investigate Agent", - instructions: buildInvestigateInstructions(columns), - model: openrouter(modelSlug), + instructions: buildInvestigateInstructions(columns, options.membershipSourceHint), + model: createLanguageModel(llmConfig, modelSlug), tools: { insert_row, - search_web: searchWebTool, - fetch_page: fetchPageTool, + ...webTools, }, }); } diff --git a/backend/src/mastra/agents/populate.ts b/backend/src/mastra/agents/populate.ts index eb9b26f..a64d497 100644 --- a/backend/src/mastra/agents/populate.ts +++ b/backend/src/mastra/agents/populate.ts @@ -1,11 +1,19 @@ import { Agent } from "@mastra/core/agent"; -import { createOpenRouter } from "@openrouter/ai-sdk-provider"; -import { buildSubagentTool } from "../tools/investigate-tool.js"; -import { searchWebTool, fetchPageTool } from "../tools/web-tools.js"; +import { createLanguageModel, type LlmProviderConfig } from "../../config/llm.js"; +import { buildSubagentTools } from "../tools/investigate-tool.js"; +import { buildWebTools } from "../tools/web-tools.js"; import type { AuthContext } from "../workflows/populate.js"; -import type { PopulateColumn } from "../../pipeline/populate.js"; +import type { CodificationProfile, PopulateColumn } from "../../pipeline/populate.js"; import type { RunMetrics } from "../run-metrics.js"; +export interface PopulateAgentDatasetContext { + datasetName: string; + description: string; + retrievalStrategy?: "search_fetch" | "browser" | "hybrid"; + sourceHint?: string; + codificationProfile?: CodificationProfile; +} + function buildInstructions(maxRowCount: number): string { return `You are an expert dataset builder. You conduct research using your web tools. You do broad research to see which rows to add, and then you spin up sub-agents that can do the deep research and fill in each row for you. @@ -19,7 +27,9 @@ If the dataset is to look at YC Companies, collect links for the YC Startup regi 3. See what the subagent reports back with, if all good and it gives you some information, use that to give better instuctions to subsequent sub agents. -Keep going until you have ${maxRowCount} rows, then finish immediately. If run_subagent reports ROW_LIMIT_REACHED, stop calling tools and finish the run. +When you find a batch of candidates, prefer queue_subagents so enumeration can continue while row workers build/reuse extractors and investigate rows in parallel. Use run_subagent when you need immediate feedback for one lead. Call drain_subagents before you finish if queued rows are still running. + +Keep going until you have ${maxRowCount} rows, then finish immediately. If run_subagent or queued work reports ROW_LIMIT_REACHED, stop calling tools and finish the run. This process should become faster overtime as you just find new rows to go and build, and you keep invoking sub agents in parallel to fill them in. @@ -40,32 +50,31 @@ export function buildPopulateAgent( authorizedDatasetId: string, authContext: AuthContext, columns: PopulateColumn[], - openRouterApiKey: string, + llmConfig: LlmProviderConfig, maxRowCount: number, + datasetContext: PopulateAgentDatasetContext, metrics?: RunMetrics, ): Agent { const modelSlug = authContext.modelConfig!.populateOrchestrator; - const openrouter = createOpenRouter({ - apiKey: openRouterApiKey, - baseURL: process.env.OPENROUTER_BASE_URL, - }); + const webTools = buildWebTools({ datasetId: authorizedDatasetId }); + const subagentTools = buildSubagentTools( + authorizedDatasetId, + authContext, + columns, + llmConfig, + maxRowCount, + datasetContext, + metrics, + ); return new Agent({ id: "populate-agent", name: "Dataset Populate Orchestrator", instructions: buildInstructions(maxRowCount), - model: openrouter(modelSlug), + model: createLanguageModel(llmConfig, modelSlug), tools: { - search_web: searchWebTool, - fetch_page: fetchPageTool, - run_subagent: buildSubagentTool( - authorizedDatasetId, - authContext, - columns, - openRouterApiKey, - maxRowCount, - metrics, - ), + ...webTools, + ...subagentTools, }, }); } diff --git a/backend/src/mastra/agents/refresh.ts b/backend/src/mastra/agents/refresh.ts index 144065a..532c136 100644 --- a/backend/src/mastra/agents/refresh.ts +++ b/backend/src/mastra/agents/refresh.ts @@ -1,12 +1,15 @@ import { Agent } from "@mastra/core/agent"; -import { createOpenRouter } from "@openrouter/ai-sdk-provider"; +import { createLanguageModel, type LlmProviderConfig } from "../../config/llm.js"; import { buildPopulateTools } from "../tools/dataset-tools.js"; -import { searchWebTool, fetchPageTool } from "../tools/web-tools.js"; +import { buildWebTools } from "../tools/web-tools.js"; import type { AuthContext } from "../workflows/populate.js"; import type { PopulateColumn } from "../../pipeline/populate.js"; function buildRefreshInstructions(columns: PopulateColumn[]): string { const columnNames = columns.map((c) => c.name); + const dataExample = columnNames + .map((n) => `{"column": "${n}", "value": "value"}`) + .join(", "); const columnsDesc = columns .map( (c) => @@ -25,7 +28,7 @@ RULES: - If no "Previously found via" steps are provided, fall back to fetching the source URLs directly. - If a source returns a 404, timeout, or is blocked, note it and move to the next. - Compare the fetched data with the existing row data carefully. -- If data has MEANINGFULLY changed (not just formatting differences), call update_row with the FULL updated data object (all columns, not just changed ones), plus updated sources, row_summary, and how_found. +- If data has MEANINGFULLY changed (not just formatting differences), call update_row with the FULL updated row data (all columns, not just changed ones), plus updated sources, row_summary, and how_found. - If NO sources work (all 404/blocked), try ONE web search using the primary key values to find a current source. - If the data is unchanged, do NOT call update_row. Just report your findings. - Never fabricate values. If you can't verify a field, keep the existing value. @@ -33,7 +36,7 @@ RULES: TOOL CALL FORMAT — every tool call argument must be a JSON object wrapped in curly braces: fetch_page: {"url": "https://example.com"} search_web: {"query": "your search terms"} - update_row: {"rowId": "", "data": {${columnNames.map((n) => `"${n}": "value"`).join(", ")}}, "sources": ["https://..."], "row_summary": "one line about this entity", "how_found": "how you verified this data"} + update_row: {"rowId": "", "data": [${dataExample}], "sources": ["https://..."], "row_summary": "one line about this entity", "how_found": "how you verified this data"} WORKFLOW: 1. Fetch the provided source URLs (1-2 calls). @@ -51,13 +54,10 @@ export function buildRefreshAgent( authorizedDatasetId: string, authContext: AuthContext, columns: PopulateColumn[], - openRouterApiKey: string, + llmConfig: LlmProviderConfig, ): Agent { const modelSlug = authContext.modelConfig!.investigateSubagent; - const openrouter = createOpenRouter({ - apiKey: openRouterApiKey, - baseURL: process.env.OPENROUTER_BASE_URL, - }); + const webTools = buildWebTools({ datasetId: authorizedDatasetId }); const { update_row } = buildPopulateTools( authorizedDatasetId, authContext, @@ -66,11 +66,10 @@ export function buildRefreshAgent( id: "refresh-agent", name: "Dataset Refresh Agent", instructions: buildRefreshInstructions(columns), - model: openrouter(modelSlug), + model: createLanguageModel(llmConfig, modelSlug), tools: { update_row, - search_web: searchWebTool, - fetch_page: fetchPageTool, + ...webTools, }, }); } diff --git a/backend/src/mastra/tools/dataset-tools.ts b/backend/src/mastra/tools/dataset-tools.ts index 1fc016e..397b151 100644 --- a/backend/src/mastra/tools/dataset-tools.ts +++ b/backend/src/mastra/tools/dataset-tools.ts @@ -3,7 +3,12 @@ import { z } from "zod"; import { convex, internal } from "../../convex.js"; import { capture } from "../../analytics/posthog.js"; import { EVENTS } from "../../analytics/events.js"; +import { + isAbortLikeError, + throwIfDatasetRunAborted, +} from "../../abort-registry.js"; import type { AuthContext } from "../workflows/populate.js"; +import type { PopulateColumn } from "../../pipeline/populate.js"; /** * Capability-scoped dataset tools for the populate agent. @@ -58,6 +63,58 @@ const writeResultSchema = z.object({ const ROW_NOT_FOUND_MSG = "Row not found. It may have been deleted, or the id belongs to a different dataset. Use list_rows to see valid row ids."; +const rowDataCellSchema = z.object({ + column: z.string().min(1), + value: z.string(), +}); + +type RowDataCell = z.infer; + +const cellSourcesSchema = z.object({ + column: z.string().min(1), + sources: z.array(z.string()).min(1), +}); + +type CellSourcesInput = z.infer; + +interface InsertDefaults { + data?: Record; + lockColumns?: string[]; + sources?: string[]; + cellSources?: Record; + rowSummary?: string; + howFoundPrefix?: string; +} + +interface PopulateToolOptions { + insertDefaults?: InsertDefaults; + columns?: PopulateColumn[]; + enforcePrimaryKeySources?: boolean; + membershipSourceHint?: string; + abortSignal?: AbortSignal; +} + +function abortSignalMessage(signal: AbortSignal): string { + const reason = signal.reason; + if (reason instanceof Error) return reason.message; + return reason ? String(reason) : "Run was stopped"; +} + +function throwIfAbortSignalAborted(signal: AbortSignal | undefined): void { + if (!signal?.aborted) return; + const reason = signal.reason; + if (reason instanceof Error) throw reason; + throw new DOMException(abortSignalMessage(signal), "AbortError"); +} + +function rowDataCellsToRecord(data: RowDataCell[]): Record { + const row: Record = {}; + for (const cell of data) { + row[cell.column] = cell.value; + } + return row; +} + function cleanDataKeys(data: Record): Record { const cleaned: Record = {}; for (const [key, value] of Object.entries(data)) { @@ -66,6 +123,153 @@ function cleanDataKeys(data: Record): Record { return cleaned; } +function hasMeaningfulValue(value: unknown): boolean { + if (value === undefined || value === null) return false; + if (typeof value === "string") return value.trim().length > 0; + return true; +} + +function uniqueStrings(values: Array): string[] { + return [...new Set(values.filter((value): value is string => Boolean(value)))]; +} + +function cellSourcesToRecord( + entries: CellSourcesInput[] | undefined, +): Record | undefined { + if (!entries || entries.length === 0) return undefined; + + const output: Record = {}; + for (const entry of entries) { + const column = entry.column.replace(/^["`]+|["`]+$/g, ""); + const sources = uniqueStrings(entry.sources); + if (column && sources.length > 0) output[column] = sources; + } + return Object.keys(output).length > 0 ? output : undefined; +} + +function mergeCellSources( + defaults: Record | undefined, + proposed: Record | undefined, +): Record | undefined { + const merged: Record = {}; + for (const [column, sources] of Object.entries(defaults ?? {})) { + const cleanSources = uniqueStrings(sources); + if (cleanSources.length > 0) merged[column] = cleanSources; + } + for (const [column, sources] of Object.entries(proposed ?? {})) { + const cleanSources = uniqueStrings([...(merged[column] ?? []), ...sources]); + if (cleanSources.length > 0) merged[column] = cleanSources; + } + return Object.keys(merged).length > 0 ? merged : undefined; +} + +function mergeInsertData( + proposedData: Record, + defaults: InsertDefaults | undefined, +): Record { + if (!defaults?.data) return proposedData; + + const merged = cleanDataKeys({ + ...defaults.data, + ...proposedData, + }); + + for (const column of defaults.lockColumns ?? []) { + const value = defaults.data[column]; + if (hasMeaningfulValue(value)) merged[column] = value; + } + + return merged; +} + +export function validatePrimaryKeySources( + data: Record, + rowSources: string[], + cellSources: Record | undefined, + columns: PopulateColumn[] | undefined, + enforcePrimaryKeySources: boolean | undefined, + membershipSourceHint?: string, +): string | undefined { + if (!enforcePrimaryKeySources || !columns || columns.length === 0) return undefined; + + const membershipHosts = membershipHostsFromHint(membershipSourceHint); + const primaryColumns = columns.filter((column) => column.isPrimaryKey); + for (const column of primaryColumns) { + const value = data[column.name]; + if (!hasMeaningfulValue(value)) { + return `Primary key "${column.name}" is missing. Verify the primary key before inserting.`; + } + + const primaryKeySources = cellSources?.[column.name] ?? []; + if (primaryKeySources.length === 0) { + return `Primary key "${column.name}" must include cell_sources that justify the exact primary-key value.`; + } + + if ( + membershipHosts.length > 0 && + !primaryKeySources.some((source) => membershipHosts.includes(normalizeHost(source))) + ) { + return `Primary key "${column.name}" must be justified by the authoritative source family (${membershipHosts.join(", ")}). Third-party enrichment sources can fill other columns but cannot admit this row.`; + } + + if (!isUrlPrimaryKeyColumn(column)) continue; + + const normalizedValue = normalizeHttpUrlForComparison(String(value)); + if (!normalizedValue) { + return `Primary key "${column.name}" must be a valid HTTP URL.`; + } + + const supportingSources = [ + ...(cellSources?.[column.name] ?? []), + ...rowSources, + ]; + const hasExactSource = supportingSources.some( + (source) => normalizeHttpUrlForComparison(source) === normalizedValue, + ); + const hasCellSource = (cellSources?.[column.name] ?? []).some( + (source) => normalizeHttpUrlForComparison(source) === normalizedValue, + ); + + if (!hasExactSource || !hasCellSource) { + return `URL primary key "${column.name}" must have a cell_sources entry containing the exact verified URL. If the URL 404s, redirects to another entity, or only appears in an unverified candidate, do not insert the row.`; + } + } + + return undefined; +} + +function isUrlPrimaryKeyColumn(column: PopulateColumn): boolean { + const haystack = `${column.name} ${column.description ?? ""}`.toLowerCase(); + return column.type === "url" || /\burl\b|https?:/.test(haystack); +} + +function membershipHostsFromHint(value: string | undefined): string[] { + if (!value) return []; + const urls = [...value.matchAll(/https?:\/\/[^\s)>"']+/gi)].map((match) => match[0]); + return [...new Set(urls.map(normalizeHost).filter(Boolean))]; +} + +function normalizeHost(value: string | undefined): string { + if (!value) return ""; + try { + return new URL(value.trim()).hostname.toLowerCase().replace(/^www\./, ""); + } catch { + return ""; + } +} + +function normalizeHttpUrlForComparison(value: string): string | undefined { + try { + const parsed = new URL(value.trim()); + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") return undefined; + parsed.hash = ""; + parsed.pathname = parsed.pathname.replace(/\/+$/, "") || "/"; + return parsed.toString(); + } catch { + return undefined; + } +} + /** * One place that records a refused capability check, so the log line + * PostHog event always carry the same fields. Called from update_row / @@ -102,6 +306,7 @@ function recordCapabilityViolation(params: { export function buildPopulateTools( authorizedDatasetId: string, authContext: AuthContext, + options: PopulateToolOptions = {}, ) { if (!authorizedDatasetId) { // Fail loud at construction time — never silently fall back to an @@ -120,17 +325,32 @@ export function buildPopulateTools( // Short prefix used in every tool's structured log line so a run's // entries can be grep'd together in the backend logs without parsing. const logCtx = `user=${authContext.authorizedUserId} run=${authContext.workflowRunId} dataset=${authorizedDatasetId}`; + const throwIfStopped = () => { + throwIfDatasetRunAborted(authorizedDatasetId); + throwIfAbortSignalAborted(options.abortSignal); + }; const insertRowTool = createTool({ id: "insert_row", description: "Insert a single row into the dataset you are populating. Call this each time you have a row ready — don't wait to batch them.", inputSchema: z.object({ - data: z.record(z.string(), z.any()), + data: z + .array(rowDataCellSchema) + .min(1) + .describe( + 'Row values as {"column": "column_name", "value": "cell value"} entries. Use an empty string for unknown values.', + ), sources: z .array(z.string()) .optional() .describe("URLs you visited or used to gather data for this row"), + cell_sources: z + .array(cellSourcesSchema) + .optional() + .describe( + 'Per-cell source URLs as {"column": "column_name", "sources": ["https://..."]}. Only include URLs that justify that exact cell value.', + ), row_summary: z .string() .optional() @@ -141,28 +361,60 @@ export function buildPopulateTools( .describe("Brief description of how you found and verified this data"), }), outputSchema: writeResultSchema, - execute: async ({ data, sources, row_summary, how_found }) => { - if (!data || Object.keys(data).length === 0) + execute: async ({ data, sources, cell_sources, row_summary, how_found }) => { + throwIfStopped(); + if (!data || data.length === 0) return { success: false, error: - 'data is required and must have at least one key. Pass an object like { "Column Name": value }.', + 'data is required and must include at least one entry like { "column": "column_name", "value": "cell value" }.', }; - const cleanedData = cleanDataKeys(data); + const cleanedData = mergeInsertData( + cleanDataKeys(rowDataCellsToRecord(data)), + options.insertDefaults, + ); + const mergedSources = uniqueStrings([ + ...(options.insertDefaults?.sources ?? []), + ...(sources ?? []), + ]); + const mergedCellSources = mergeCellSources( + options.insertDefaults?.cellSources, + cellSourcesToRecord(cell_sources), + ); + const primaryKeySourceError = validatePrimaryKeySources( + cleanedData, + mergedSources, + mergedCellSources, + options.columns, + options.enforcePrimaryKeySources, + options.membershipSourceHint, + ); + if (primaryKeySourceError) { + return { success: false, error: primaryKeySourceError }; + } + const mergedRowSummary = + row_summary ?? options.insertDefaults?.rowSummary; + const mergedHowFound = uniqueStrings([ + options.insertDefaults?.howFoundPrefix, + how_found, + ]).join("\n"); console.log( - `[insert_row] ${logCtx} cols=${Object.keys(cleanedData).length} sources=${sources?.length ?? 0}`, + `[insert_row] ${logCtx} cols=${Object.keys(cleanedData).length} sources=${mergedSources.length}`, ); try { + throwIfStopped(); await convex.mutation(internal.datasetRows.insert, { datasetId: authorizedDatasetId, data: cleanedData, - ...(sources !== undefined ? { sources } : {}), - ...(row_summary !== undefined ? { rowSummary: row_summary } : {}), - ...(how_found !== undefined ? { howFound: how_found } : {}), + ...(mergedSources.length > 0 ? { sources: mergedSources } : {}), + ...(mergedCellSources ? { cellSources: mergedCellSources } : {}), + ...(mergedRowSummary !== undefined ? { rowSummary: mergedRowSummary } : {}), + ...(mergedHowFound ? { howFound: mergedHowFound } : {}), }); return { success: true }; } catch (err) { + if (isAbortLikeError(err)) throw err; const msg = err instanceof Error ? err.message : String(err); console.error(`[insert_row] Failed: ${logCtx} err=${msg}`); if (/duplicate/i.test(msg)) @@ -195,13 +447,16 @@ export function buildPopulateTools( error: z.string().optional(), }), execute: async () => { + throwIfStopped(); console.log(`[list_rows] ${logCtx}`); try { + throwIfStopped(); const rows = await convex.query(internal.datasetRows.listInternal, { datasetId: authorizedDatasetId, }); return { rows }; } catch (err) { + if (isAbortLikeError(err)) throw err; const msg = err instanceof Error ? err.message : String(err); console.error(`[list_rows] Failed: ${logCtx} err=${msg}`); return { error: `List rows failed: ${msg}` }; @@ -221,10 +476,12 @@ export function buildPopulateTools( error: z.string().optional(), }), execute: async ({ rowId }) => { + throwIfStopped(); if (!rowId) return { error: "rowId is required." }; console.log(`[get_row] ${logCtx} row=${rowId}`); try { + throwIfStopped(); const row = await convex.query(internal.datasetRows.get, { id: rowId }); // Existence + ownership are collapsed into ONE uniform error so // the LLM (or a prompt-injecting page) can't probe row ids across @@ -243,6 +500,7 @@ export function buildPopulateTools( } return { row }; } catch (err) { + if (isAbortLikeError(err)) throw err; const msg = err instanceof Error ? err.message : String(err); console.error(`[get_row] Failed: ${logCtx} row=${rowId} err=${msg}`); if (msg.includes("validator") || msg.includes("Invalid")) @@ -257,14 +515,25 @@ export function buildPopulateTools( const updateRowTool = createTool({ id: "update_row", description: - "Update an existing row by its ID. Pass the full updated data object. Changes are tracked in history.", + "Update an existing row by its ID. Pass the full updated row data. Changes are tracked in history.", inputSchema: z.object({ rowId: z.string(), - data: z.record(z.string(), z.any()), + data: z + .array(rowDataCellSchema) + .min(1) + .describe( + 'Full row values as {"column": "column_name", "value": "cell value"} entries. Use an empty string for unknown values.', + ), sources: z .array(z.string()) .optional() .describe("Updated source URLs where this data was verified"), + cell_sources: z + .array(cellSourcesSchema) + .optional() + .describe( + 'Updated per-cell source URLs as {"column": "column_name", "sources": ["https://..."]}. Only include URLs that justify that exact cell value.', + ), row_summary: z .string() .optional() @@ -275,29 +544,35 @@ export function buildPopulateTools( .describe("Brief description of how the updated data was found"), }), outputSchema: writeResultSchema, - execute: async ({ rowId, data, sources, row_summary, how_found }) => { + execute: async ({ rowId, data, sources, cell_sources, row_summary, how_found }) => { + throwIfStopped(); if (!rowId) return { success: false, error: "rowId is required." }; - if (!data || Object.keys(data).length === 0) + if (!data || data.length === 0) return { success: false, - error: "data is required. Pass the full updated row data object.", + error: "data is required. Pass the full updated row data entries.", }; - const cleanedData = cleanDataKeys(data); + const cleanedData = cleanDataKeys(rowDataCellsToRecord(data)); console.log( `[update_row] ${logCtx} row=${rowId} cols=${Object.keys(cleanedData).length}`, ); try { + throwIfStopped(); await convex.mutation(internal.datasetRows.update, { id: rowId, expectedDatasetId: authorizedDatasetId, data: cleanedData, ...(sources !== undefined ? { sources } : {}), + ...(cell_sources !== undefined + ? { cellSources: cellSourcesToRecord(cell_sources) ?? {} } + : {}), ...(row_summary !== undefined ? { rowSummary: row_summary } : {}), ...(how_found !== undefined ? { howFound: how_found } : {}), }); return { success: true }; } catch (err) { + if (isAbortLikeError(err)) throw err; const msg = err instanceof Error ? err.message : String(err); console.error(`[update_row] Failed: ${logCtx} row=${rowId} err=${msg}`); if (msg.includes("Row not found") || msg.includes("not found")) { @@ -336,16 +611,19 @@ export function buildPopulateTools( }), outputSchema: writeResultSchema, execute: async ({ rowId }) => { + throwIfStopped(); if (!rowId) return { success: false, error: "rowId is required." }; console.log(`[delete_row] ${logCtx} row=${rowId}`); try { + throwIfStopped(); await convex.mutation(internal.datasetRows.remove, { id: rowId, expectedDatasetId: authorizedDatasetId, }); return { success: true }; } catch (err) { + if (isAbortLikeError(err)) throw err; const msg = err instanceof Error ? err.message : String(err); console.error(`[delete_row] Failed: ${logCtx} row=${rowId} err=${msg}`); if (msg.includes("Row not found") || msg.includes("not found")) { diff --git a/backend/src/mastra/tools/investigate-tool.ts b/backend/src/mastra/tools/investigate-tool.ts index 0139aa4..53af39d 100644 --- a/backend/src/mastra/tools/investigate-tool.ts +++ b/backend/src/mastra/tools/investigate-tool.ts @@ -2,10 +2,27 @@ import { createTool } from "@mastra/core/tools"; import { z } from "zod"; import { convex, internal } from "../../convex.js"; import { buildInvestigateAgent } from "../agents/investigate.js"; +import { validatePrimaryKeySources } from "./dataset-tools.js"; import type { AuthContext } from "../workflows/populate.js"; -import type { PopulateColumn } from "../../pipeline/populate.js"; +import type { CodificationProfile, PopulateColumn } from "../../pipeline/populate.js"; import type { RunMetrics } from "../run-metrics.js"; -import { getSignal } from "../../abort-registry.js"; +import { + getSignal, + isAbortLikeError, + isDatasetRunAborted, + throwIfDatasetRunAborted, +} from "../../abort-registry.js"; +import { + tryRowExtractorDraft, + type RowExtractorDraftResult, +} from "../../row-extractors/try-row-extractor.js"; +import type { LlmProviderConfig } from "../../config/llm.js"; +import { AGENT_MAX_OUTPUT_TOKENS } from "../../config/agent-output-tokens.js"; + +const keyValueSchema = z.object({ + column: z.string().min(1), + value: z.string().min(1), +}); const investigateInputSchema = z.object({ entity_hint: z @@ -14,12 +31,10 @@ const investigateInputSchema = z.object({ "What entity to look for, e.g. 'head of GTM at Appcharge' or 'Starbucks coffee products on Amazon'", ), primary_keys: z - .record(z.string(), z.string()) - .refine((v) => Object.keys(v).length > 0, { - message: "primary_keys must include at least one primary-key value", - }) + .array(keyValueSchema) + .min(1, "primary_keys must include at least one primary-key value") .describe( - "REQUIRED: the primary key column value(s) for this entity. e.g. {\"Company Name\": \"Stripe\"} or {\"First Name\": \"John\", \"Last Name\": \"Doe\"}. You MUST provide at least the primary key values you have found.", + 'REQUIRED: primary key values as {"column": "column_name", "value": "value"} entries. e.g. [{"column": "company_name", "value": "Stripe"}]. You MUST provide at least the primary key values you have found.', ), context: z .string() @@ -45,6 +60,241 @@ const investigateOutputSchema = z.object({ reason: z.string(), }); +const queueSubagentsInputSchema = z.object({ + candidates: z + .array(investigateInputSchema) + .min(1) + .max(50) + .describe("Candidate rows to enqueue for background investigation."), +}); + +const queueSubagentsOutputSchema = z.object({ + queued: z.number(), + pending: z.number(), + active: z.number(), + completed: z.number(), + reason: z.string().optional(), +}); + +const drainSubagentsOutputSchema = z.object({ + completed: z.number(), + inserted: z.number(), + failed: z.number(), + pending: z.number(), + active: z.number(), + reasons: z.array(z.string()), +}); + +interface DatasetContextForExtractor { + datasetName: string; + description: string; + retrievalStrategy?: "search_fetch" | "browser" | "hybrid"; + sourceHint?: string; + codificationProfile?: CodificationProfile; +} + +type InvestigateLead = z.infer; +type InvestigateResult = z.infer; +type ProcessLead = ( + lead: InvestigateLead, + abortSignal?: AbortSignal, +) => Promise; + +interface QueueSummary { + completed: number; + inserted: number; + failed: number; + pending: number; + active: number; + reasons: string[]; +} + +class SubagentQueue { + private readonly controller = new AbortController(); + private readonly waiting: Array<{ + lead: InvestigateLead; + resolve: (result: InvestigateResult) => void; + }> = []; + private readonly promises: Promise[] = []; + private readonly completed: InvestigateResult[] = []; + private readonly abortListener?: () => void; + private active = 0; + private canceledReason: string | undefined; + + constructor( + private readonly concurrency: number, + private readonly processLead: ProcessLead, + private readonly abortSignal?: AbortSignal, + ) { + if (abortSignal) { + this.abortListener = () => this.cancel("Run was stopped"); + abortSignal.addEventListener("abort", this.abortListener, { once: true }); + if (abortSignal.aborted) this.cancel("Run was stopped"); + } + } + + enqueue(lead: InvestigateLead): void { + const promise = new Promise((resolve) => { + if (this.canceledReason) { + const result = this.canceledResult(); + this.completed.push(result); + resolve(result); + return; + } + this.waiting.push({ lead, resolve }); + this.pump(); + }); + this.promises.push(promise); + } + + enqueueMany(leads: InvestigateLead[]): void { + for (const lead of leads) this.enqueue(lead); + } + + snapshot(): QueueSummary { + return this.toSummary(); + } + + cancel(reason: string): void { + if (this.canceledReason) return; + this.canceledReason = reason; + if (!this.controller.signal.aborted) { + this.controller.abort(new DOMException(reason, "AbortError")); + } + const waiting = this.waiting.splice(0); + for (const pending of waiting) { + const result = this.canceledResult(); + this.completed.push(result); + pending.resolve(result); + } + } + + async drain(): Promise { + let observedCount = -1; + while (observedCount !== this.promises.length) { + observedCount = this.promises.length; + await Promise.allSettled(this.promises); + } + return this.toSummary(); + } + + dispose(): void { + if (this.abortSignal && this.abortListener) { + this.abortSignal.removeEventListener("abort", this.abortListener); + } + } + + private pump(): void { + if (this.canceledReason) return; + while (this.active < this.concurrency && this.waiting.length > 0) { + const next = this.waiting.shift(); + if (!next) continue; + this.active++; + void this.processLead(next.lead, this.controller.signal) + .then((result) => { + this.completed.push(result); + next.resolve(result); + }) + .catch((err) => { + const reason = err instanceof Error ? err.message : String(err); + const result = { + inserted: false, + reason: `Queued subagent failed: ${reason}`, + row_summary: undefined, + clues: undefined, + }; + this.completed.push(result); + next.resolve(result); + }) + .finally(() => { + this.active--; + if (!this.canceledReason) this.pump(); + }); + } + } + + private canceledResult(): InvestigateResult { + return { + inserted: false, + reason: this.canceledReason ?? "Queued subagent canceled", + row_summary: undefined, + clues: undefined, + }; + } + + private toSummary(): QueueSummary { + const failed = this.completed.filter((result) => !result.inserted).length; + return { + completed: this.completed.length, + inserted: this.completed.filter((result) => result.inserted).length, + failed, + pending: this.waiting.length, + active: this.active, + reasons: this.completed + .filter((result) => !result.inserted) + .map((result) => result.reason) + .slice(-10), + }; + } +} + +const queuedSubagentsByRun = new Map(); + +export async function drainQueuedSubagents(workflowRunId: string): Promise { + const queue = queuedSubagentsByRun.get(workflowRunId); + if (!queue) { + return { completed: 0, inserted: 0, failed: 0, pending: 0, active: 0, reasons: [] }; + } + try { + return await queue.drain(); + } finally { + queue.dispose(); + queuedSubagentsByRun.delete(workflowRunId); + } +} + +export async function clearQueuedSubagents(workflowRunId: string): Promise { + const queue = queuedSubagentsByRun.get(workflowRunId); + if (!queue) { + return { completed: 0, inserted: 0, failed: 0, pending: 0, active: 0, reasons: [] }; + } + queue.cancel("Queued subagents cleared"); + try { + return await queue.drain(); + } finally { + queue.dispose(); + queuedSubagentsByRun.delete(workflowRunId); + } +} + +function abortSignalMessage(signal: AbortSignal): string { + const reason = signal.reason; + if (reason instanceof Error) return reason.message; + return reason ? String(reason) : "Run was stopped"; +} + +function throwIfAbortSignalAborted(signal: AbortSignal | undefined): void { + if (!signal?.aborted) return; + const reason = signal.reason; + if (reason instanceof Error) throw reason; + throw new DOMException(abortSignalMessage(signal), "AbortError"); +} + +function isStopped( + authorizedDatasetId: string, + abortSignal: AbortSignal | undefined, +): boolean { + return isDatasetRunAborted(authorizedDatasetId) || abortSignal?.aborted === true; +} + +function throwIfStopped( + authorizedDatasetId: string, + abortSignal: AbortSignal | undefined, +): void { + throwIfDatasetRunAborted(authorizedDatasetId); + throwIfAbortSignalAborted(abortSignal); +} + function parseInvestigateResult( text: string, ): z.infer { @@ -61,66 +311,290 @@ function parseInvestigateResult( }; } +function hasMeaningfulValue(value: unknown): boolean { + if (value === undefined || value === null) return false; + if (typeof value === "string") return value.trim().length > 0; + return true; +} + +function filledColumnNames( + draft: RowExtractorDraftResult | undefined, + columns: PopulateColumn[], +): string[] { + if (!draft?.data) return []; + const primaryKeyColumns = new Set( + columns.filter((column) => column.isPrimaryKey).map((column) => column.name), + ); + return Object.entries(draft.data) + .filter(([column, value]) => { + if (!hasMeaningfulValue(value)) return false; + if (primaryKeyColumns.has(column)) return true; + return (draft.cellSources?.[column]?.length ?? 0) > 0; + }) + .map(([column]) => column); +} + +function formatRowData(data: Record | undefined): string { + if (!data) return "(none)"; + const lines = Object.entries(data) + .filter(([, value]) => hasMeaningfulValue(value)) + .map(([column, value]) => `- ${column}: ${JSON.stringify(value)}`); + return lines.length > 0 ? lines.join("\n") : "(none)"; +} + +function pickRowData( + data: Record | undefined, + columns: string[], +): Record | undefined { + if (!data) return undefined; + const picked: Record = {}; + for (const column of columns) { + const value = data[column]; + if (hasMeaningfulValue(value)) picked[column] = value; + } + return Object.keys(picked).length > 0 ? picked : undefined; +} + +function formatUnresolvedColumns( + columns: PopulateColumn[], + draft: RowExtractorDraftResult | undefined, + lockedColumns: string[] = [], +): string { + if (!draft) return "(none)"; + const locked = new Set(lockedColumns); + const fallbackNames = columns + .filter((column) => !column.isPrimaryKey) + .filter((column) => { + if (locked.has(column.name)) return false; + return draft.missingColumns?.includes(column.name) || hasMeaningfulValue(draft.data?.[column.name]); + }) + .map((column) => column.name); + if (fallbackNames.length === 0) return "(none)"; + + const columnByName = new Map(columns.map((column) => [column.name, column])); + return fallbackNames + .map((name) => { + const column = columnByName.get(name); + const requiredness = column?.nullable === false ? "required" : "optional"; + const status = draft.columnStatuses?.[name]; + const missingCellSource = + hasMeaningfulValue(draft.data?.[name]) && (draft.cellSources?.[name]?.length ?? 0) === 0; + const detail = [ + `${requiredness}`, + missingCellSource ? "browser_status=unverified_cell_source" : undefined, + status?.status ? `browser_status=${status.status}` : undefined, + status?.reason ? `reason=${JSON.stringify(status.reason)}` : undefined, + column?.normalizationHint + ? `normalization=${JSON.stringify(column.normalizationHint)}` + : undefined, + ] + .filter(Boolean) + .join("; "); + return `- ${name} (${detail})`; + }) + .join("\n"); +} + /** - * Build the run_subagent tool scoped to one dataset. + * Build row-investigation tools scoped to one dataset. * - * The orchestrator calls this to hand off a lead to a fresh subagent. - * The subagent does deep research, inserts at most one row, and returns - * structured feedback including clues for finding more rows. + * `run_subagent` is the original synchronous path. `queue_subagents` lets the + * orchestrator keep enumerating while row workers build/reuse extractors and + * investigate queued candidates in the background. The workflow drains the queue + * before completing, so queued work is still part of the populate run. * * authorizedDatasetId and authContext are captured by closure — not * exposed in the tool schema, never visible to the orchestrator LLM. */ -export function buildSubagentTool( +export function buildSubagentTools( authorizedDatasetId: string, authContext: AuthContext, columns: PopulateColumn[], - openRouterApiKey: string, + llmConfig: LlmProviderConfig, maxRowCount: number, + datasetContext: DatasetContextForExtractor, metrics?: RunMetrics, ) { - return createTool({ - id: "run_subagent", - description: - "Hand off a lead to a subagent that will research it deeply and insert a single row if it finds real, verified data. You MUST pass the primary key values (primary_keys) for the entity — the subagent will fill in the remaining columns. Also pass any URLs and context you have found.", - inputSchema: investigateInputSchema, - outputSchema: investigateOutputSchema, - execute: async ({ entity_hint, primary_keys, context, urls, notes }) => { - try { - const rowCount = await convex.query(internal.datasetRows.countByDataset, { - datasetId: authorizedDatasetId, - }); - if (rowCount >= maxRowCount) { - return { - inserted: false, - reason: `ROW_LIMIT_REACHED: this BigSet dataset is capped at ${maxRowCount} rows. Stop calling run_subagent and finish the run.`, - row_summary: undefined, - clues: undefined, - }; + const throwIfStoppedForLead = (abortSignal?: AbortSignal) => + throwIfStopped(authorizedDatasetId, abortSignal); + + const processLead: ProcessLead = async ( + { entity_hint, primary_keys, context, urls, notes }, + leadAbortSignal, + ): Promise => { + try { + throwIfStoppedForLead(leadAbortSignal); + const rowCount = await convex.query(internal.datasetRows.countByDataset, { + datasetId: authorizedDatasetId, + }); + throwIfStoppedForLead(leadAbortSignal); + if (rowCount >= maxRowCount) { + return { + inserted: false, + reason: `ROW_LIMIT_REACHED: this BigSet dataset is capped at ${maxRowCount} rows. Stop calling run_subagent and finish the run.`, + row_summary: undefined, + clues: undefined, + }; + } + + if (metrics) metrics.investigateCalls++; + const primaryKeyRecord = Object.fromEntries( + primary_keys.map(({ column, value }) => [column, value]), + ); + + throwIfStoppedForLead(leadAbortSignal); + const extractorResult = await tryRowExtractorDraft({ + datasetId: authorizedDatasetId, + columns, + primaryKeys: primaryKeyRecord, + urls, + context, + datasetName: datasetContext.datasetName, + description: datasetContext.description, + retrievalStrategy: datasetContext.retrievalStrategy, + sourceHint: datasetContext.sourceHint, + codificationProfile: datasetContext.codificationProfile, + browserAttempts: authContext.modelConfig.rowExtractorBrowserAttempts, + extractorBuilderModel: authContext.modelConfig.extractorBuilder, + abortSignal: leadAbortSignal, + }); + throwIfStoppedForLead(leadAbortSignal); + if (/duplicate/i.test(extractorResult.reason)) { + return { + inserted: false, + reason: extractorResult.reason, + row_summary: undefined, + clues: undefined, + }; + } + + if (extractorResult.status === "extracted") { + const missingColumns = extractorResult.missingColumns ?? []; + const primaryKeyIssue = validatePrimaryKeySources( + extractorResult.data ?? {}, + extractorResult.sources ?? [], + extractorResult.cellSources, + columns, + true, + datasetContext.sourceHint, + ); + if (missingColumns.length === 0 && !primaryKeyIssue) { + try { + throwIfStoppedForLead(leadAbortSignal); + await convex.mutation(internal.datasetRows.insert, { + datasetId: authorizedDatasetId, + data: extractorResult.data ?? {}, + sources: extractorResult.sources, + cellSources: extractorResult.cellSources, + rowSummary: extractorResult.rowSummary, + howFound: + "Opened the row target with TinyFish Browser and ran the dataset's generated Playwright extractor. No fallback columns were unresolved.", + }); + if (metrics) metrics.rowsInserted++; + console.log( + `[run_subagent] row extractor inserted complete row entity="${entity_hint}" reason="${extractorResult.reason}"`, + ); + return { + inserted: true, + reason: extractorResult.reason, + row_summary: extractorResult.rowSummary, + clues: undefined, + }; + } catch (err) { + if (isAbortLikeError(err) && isStopped(authorizedDatasetId, leadAbortSignal)) { + throw err; + } + const msg = err instanceof Error ? err.message : String(err); + if (/duplicate/i.test(msg)) { + return { + inserted: false, + reason: `${msg} Move on to the next entity.`, + row_summary: undefined, + clues: undefined, + }; + } + throw err; + } } - if (metrics) metrics.investigateCalls++; + if (primaryKeyIssue) { + console.warn( + `[run_subagent] row extractor primary-key evidence insufficient entity="${entity_hint}" reason="${primaryKeyIssue}"`, + ); + } else { + console.log( + `[run_subagent] row extractor drafted entity="${entity_hint}" filled=${extractorResult.extractedColumns?.length ?? 0} unresolved=${missingColumns.length}`, + ); + } + } else if (extractorResult.status === "failed") { + console.warn( + `[run_subagent] row extractor failed entity="${entity_hint}" reason="${extractorResult.reason}"`, + ); + } else if (extractorResult.status === "miss") { console.log( - `[run_subagent] spawning subagent user=${authContext.authorizedUserId} run=${authContext.workflowRunId} dataset=${authorizedDatasetId} entity="${entity_hint}" pk=${JSON.stringify(primary_keys)}`, + `[run_subagent] row extractor missed entity="${entity_hint}" reason="${extractorResult.reason}"`, ); + } - const agent = buildInvestigateAgent( - authorizedDatasetId, - authContext, - columns, - openRouterApiKey, - ); + throwIfStoppedForLead(leadAbortSignal); + console.log( + `[run_subagent] spawning subagent user=${authContext.authorizedUserId} run=${authContext.workflowRunId} dataset=${authorizedDatasetId} entity="${entity_hint}" pk=${JSON.stringify(primary_keys)}`, + ); + + const browserFilledValues = + extractorResult.status === "extracted" ? extractorResult.data : undefined; + const browserFilledColumns = + extractorResult.status === "extracted" + ? filledColumnNames(extractorResult, columns) + : []; + const browserCandidateValues = + extractorResult.status === "extracted" + ? pickRowData(browserFilledValues, browserFilledColumns) + : undefined; + const browserSources = + extractorResult.status === "extracted" ? extractorResult.sources : undefined; + + const agent = buildInvestigateAgent( + authorizedDatasetId, + authContext, + columns, + llmConfig, + { + membershipSourceHint: datasetContext.sourceHint, + abortSignal: leadAbortSignal, + }, + ); + + const pkBlock = primary_keys + .map(({ column, value }) => `- ${column}: ${value}`) + .join("\n"); + const urlsBlock = + urls && urls.length > 0 + ? `\nUseful URLs to start from:\n${urls.map((u) => `- ${u}`).join("\n")}` + : ""; + const notesBlock = notes ? `\nAdditional notes: ${notes}` : ""; + const browserDraftBlock = + extractorResult.status === "extracted" + ? ` - const pkBlock = Object.entries(primary_keys) - .map(([k, v]) => `- ${k}: ${v}`) - .join("\n"); - const urlsBlock = - urls && urls.length > 0 - ? `\nUseful URLs to start from:\n${urls.map((u) => `- ${u}`).join("\n")}` - : ""; - const notesBlock = notes ? `\nAdditional notes: ${notes}` : ""; +Browser extraction produced these candidate values, but this row still needs fallback verification. Treat them as hints, not locked facts. Re-verify all primary key values and any non-empty candidate before insert_row. If a URL primary key 404s, redirects to a different entity, or cannot be justified by source-backed evidence, do not insert the row: +${formatRowData(browserCandidateValues)} - const prompt = `Research this entity and insert a row if you find real, verified data. +Unresolved columns to research now. Try every listed column, including optional ones. If a value still cannot be verified, insert "" for that column and explain why: +${formatUnresolvedColumns(columns, extractorResult, browserFilledColumns)} + +Browser sources already used: +${(browserSources ?? []).map((u) => `- ${u}`).join("\n") || "(none)"} +` + : ` + +Browser extraction did not produce a usable draft for this row: +- status: ${extractorResult.status} +- reason: ${extractorResult.reason} + +Research all non-primary columns normally.`; + + const prompt = `Research this entity and insert a row if you find real, verified data. Entity: ${entity_hint} @@ -128,47 +602,119 @@ Primary key values (MUST be included in insert_row): ${pkBlock} Context (partial data already found): -${context}${urlsBlock}${notesBlock}`; - - const abortSignal = getSignal(authorizedDatasetId); - const result = await agent.generate(prompt, { abortSignal, maxSteps: 25 }); - if (metrics) { - // Use result.toolCalls (the flat accumulated list across all steps) rather - // than iterating result.steps[n].toolCalls. The per-step arrays are snapshots - // captured at step-finish time; tool-call chunks that arrive after their - // step-finish event end up attributed to the wrong step, causing systematic - // miscounts. result.toolCalls is the authoritative list maintained by Mastra's - // stream processor as chunks arrive. - metrics.countToolCalls(result.toolCalls ?? []); - metrics.addInvestigateResult(result); - } +${context}${urlsBlock}${notesBlock}${browserDraftBlock}`; - const parsed = parseInvestigateResult(result.text); - if (metrics && parsed.inserted) metrics.rowsInserted++; + const agentAbortSignal = leadAbortSignal ?? getSignal(authorizedDatasetId); + throwIfStoppedForLead(leadAbortSignal); + const result = await agent.generate(prompt, { + abortSignal: agentAbortSignal, + maxSteps: 25, + modelSettings: { + maxOutputTokens: AGENT_MAX_OUTPUT_TOKENS.INVESTIGATE_SUBAGENT, + }, + }); + if (metrics) { + // Use result.toolCalls (the flat accumulated list across all steps) rather + // than iterating result.steps[n].toolCalls. The per-step arrays are snapshots + // captured at step-finish time; tool-call chunks that arrive after their + // step-finish event end up attributed to the wrong step, causing systematic + // miscounts. result.toolCalls is the authoritative list maintained by Mastra's + // stream processor as chunks arrive. + metrics.countToolCalls(result.toolCalls ?? []); + metrics.addInvestigateResult(result); + } - console.log( - `[run_subagent] done entity="${entity_hint}" inserted=${parsed.inserted} steps=${result.steps?.length ?? "?"} toolCalls=${result.toolCalls?.length ?? "?"}` + - (parsed.row_summary ? `\n summary: ${parsed.row_summary}` : "") + - (parsed.reason ? `\n reason: ${parsed.reason}` : "") + - (parsed.clues ? `\n clues: ${parsed.clues}` : ""), - ); - return parsed; - } catch (err) { - // Only propagate an AbortError if OUR signal was actually fired (i.e. - // the user pressed Stop). Network errors in Node.js can also surface as - // AbortError — re-throwing those would cause the orchestrator's - // agent.generate() to exit early and return a graceful empty result, - // producing a "0 rows" run without any user action. - if (err instanceof Error && err.name === "AbortError" && getSignal(authorizedDatasetId)?.aborted) throw err; - const msg = err instanceof Error ? err.message : String(err); - console.error(`[run_subagent] subagent error entity="${entity_hint}" err=${msg}`); - return { - inserted: false, - reason: `Subagent failed: ${msg}`, - row_summary: undefined, - clues: undefined, - }; + const parsed = parseInvestigateResult(result.text); + if (metrics && parsed.inserted) metrics.rowsInserted++; + + console.log( + `[run_subagent] done entity="${entity_hint}" inserted=${parsed.inserted} steps=${result.steps?.length ?? "?"} toolCalls=${result.toolCalls?.length ?? "?"}` + + (parsed.row_summary ? `\n summary: ${parsed.row_summary}` : "") + + (parsed.reason ? `\n reason: ${parsed.reason}` : "") + + (parsed.clues ? `\n clues: ${parsed.clues}` : ""), + ); + return parsed; + } catch (err) { + // Only propagate an AbortError if OUR signal was actually fired (i.e. + // the user pressed Stop). Network errors in Node.js can also surface as + // AbortError — re-throwing those would cause the orchestrator's + // agent.generate() to exit early and return a graceful empty result, + // producing a "0 rows" run without any user action. + if (isAbortLikeError(err) && isStopped(authorizedDatasetId, leadAbortSignal)) { + throw err; } + const msg = err instanceof Error ? err.message : String(err); + console.error(`[run_subagent] subagent error entity="${entity_hint}" err=${msg}`); + return { + inserted: false, + reason: `Subagent failed: ${msg}`, + row_summary: undefined, + clues: undefined, + }; + } + }; + + const queue = new SubagentQueue( + Math.min(20, Math.max(1, authContext.modelConfig.rowExtractorConcurrency)), + processLead, + getSignal(authorizedDatasetId), + ); + queuedSubagentsByRun.set(authContext.workflowRunId, queue); + + const runSubagentTool = createTool({ + id: "run_subagent", + description: + "Hand off a lead to a subagent that will research it deeply and insert a single row if it finds real, verified data. You MUST pass the primary key values (primary_keys) for the entity — the subagent will fill in the remaining columns. Also pass any URLs and context you have found.", + inputSchema: investigateInputSchema, + outputSchema: investigateOutputSchema, + execute: async (lead) => processLead(lead), + }); + + const queueSubagentsTool = createTool({ + id: "queue_subagents", + description: + "Queue a batch of candidate rows for background investigation. Use this when you have multiple leads so enumeration can continue while extractor builds and row research run in parallel. The workflow drains queued work before finishing.", + inputSchema: queueSubagentsInputSchema, + outputSchema: queueSubagentsOutputSchema, + execute: async ({ candidates }) => { + throwIfStoppedForLead(); + queue.enqueueMany(candidates); + const summary = queue.snapshot(); + console.log( + `[queue_subagents] user=${authContext.authorizedUserId} run=${authContext.workflowRunId} dataset=${authorizedDatasetId} queued=${candidates.length} pending=${summary.pending} active=${summary.active} completed=${summary.completed}`, + ); + return { + queued: candidates.length, + pending: summary.pending, + active: summary.active, + completed: summary.completed, + reason: + "Queued. Continue enumerating more candidates; call drain_subagents only when you need feedback or before finishing.", + }; }, }); + + const drainSubagentsTool = createTool({ + id: "drain_subagents", + description: + "Wait for queued candidate investigations to finish and return a summary. Use this when you need feedback from queued rows or before you finish.", + inputSchema: z.object({}), + outputSchema: drainSubagentsOutputSchema, + execute: async () => { + if (isDatasetRunAborted(authorizedDatasetId)) { + queue.cancel("Run was stopped"); + } + const summary = await queue.drain(); + console.log( + `[drain_subagents] user=${authContext.authorizedUserId} run=${authContext.workflowRunId} dataset=${authorizedDatasetId} completed=${summary.completed} inserted=${summary.inserted} failed=${summary.failed}`, + ); + return summary; + }, + }); + + return { + run_subagent: runSubagentTool, + queue_subagents: queueSubagentsTool, + drain_subagents: drainSubagentsTool, + }; } diff --git a/backend/src/mastra/tools/web-tools.ts b/backend/src/mastra/tools/web-tools.ts index 245ee43..f83d992 100644 --- a/backend/src/mastra/tools/web-tools.ts +++ b/backend/src/mastra/tools/web-tools.ts @@ -1,5 +1,11 @@ import { createTool } from "@mastra/core/tools"; import { z } from "zod"; +import { + datasetAbortError, + getSignal, + isAbortLikeError, + throwIfDatasetRunAborted, +} from "../../abort-registry.js"; import { FETCH_TIMEOUT_MS } from "../../fetch-timeout.js"; import { getTinyFishApiKey, tinyFishHeaders } from "../../local-credentials.js"; @@ -9,7 +15,147 @@ const searchResultSchema = z.object({ url: z.string(), }); -export const searchWebTool = createTool({ +const FETCH_FAILURE_CACHE_TTL_MS = 10 * 60 * 1000; +const recentFetchFailures = new Map< + string, + { at: number; code: string; status?: string; message: string } +>(); + +interface WebToolOptions { + datasetId?: string; + abortSignal?: AbortSignal; +} + +function abortSignalMessage(signal: AbortSignal): string { + const reason = signal.reason; + if (reason instanceof Error) return reason.message; + return reason ? String(reason) : "Run was stopped"; +} + +function throwIfAbortSignalAborted(signal: AbortSignal | undefined): void { + if (!signal?.aborted) return; + const reason = signal.reason; + if (reason instanceof Error) throw reason; + throw new DOMException(abortSignalMessage(signal), "AbortError"); +} + +function throwIfStopped( + datasetId: string | undefined, + abortSignal: AbortSignal | undefined, +): void { + if (datasetId) throwIfDatasetRunAborted(datasetId); + throwIfAbortSignalAborted(abortSignal); +} + +function timeoutSignalForDataset( + datasetId: string | undefined, + abortSignal: AbortSignal | undefined, +): { + signal: AbortSignal; + cleanup: () => void; + timedOut: () => boolean; +} { + throwIfStopped(datasetId, abortSignal); + + const controller = new AbortController(); + const runSignal = datasetId ? getSignal(datasetId) : undefined; + const abortListeners: Array<{ signal: AbortSignal; listener: () => void }> = []; + let timedOut = false; + + const addAbortListener = (signal: AbortSignal | undefined) => { + if (!signal) return; + const listener = () => { + controller.abort(signal.reason ?? datasetAbortError()); + }; + signal.addEventListener("abort", listener, { once: true }); + abortListeners.push({ signal, listener }); + if (signal.aborted) listener(); + }; + addAbortListener(runSignal); + addAbortListener(abortSignal); + + const timeout = setTimeout(() => { + timedOut = true; + controller.abort(new DOMException("Operation timed out", "TimeoutError")); + }, FETCH_TIMEOUT_MS); + + return { + signal: controller.signal, + cleanup: () => { + clearTimeout(timeout); + for (const { signal, listener } of abortListeners) { + signal.removeEventListener("abort", listener); + } + }, + timedOut: () => timedOut, + }; +} + +function throwIfStoppedAbort( + err: unknown, + datasetId: string | undefined, + abortSignal: AbortSignal | undefined, +): void { + const abortLike = + isAbortLikeError(err) || + (err instanceof Error && err.name === "TimeoutError"); + if (!abortLike) return; + if (datasetId && getSignal(datasetId)?.aborted === true) { + throw datasetAbortError(); + } + throwIfAbortSignalAborted(abortSignal); +} + +function cachedFetchFailure(url: string): string | undefined { + const cached = recentFetchFailures.get(url); + if (!cached) return undefined; + if (Date.now() - cached.at > FETCH_FAILURE_CACHE_TTL_MS) { + recentFetchFailures.delete(url); + return undefined; + } + console.log( + `[fetch_page] Skipping recently failed URL: url=${url} error=${cached.code} status=${cached.status ?? "n/a"}`, + ); + return cached.message; +} + +function rememberFetchFailure( + url: string, + code: string, + status: string | undefined, + message: string, +): void { + if (!isCacheableFetchFailure(code, status)) return; + recentFetchFailures.set(url, { + at: Date.now(), + code, + status, + message, + }); +} + +function isCacheableFetchFailure(code: string, status: string | undefined): boolean { + if (code === "page_not_found" || code === "bot_blocked") return true; + if (code !== "target_http_error" || !status) return false; + const statusCode = Number(status); + return Number.isFinite(statusCode) && statusCode >= 400 && statusCode < 500; +} + +function fetchPageErrorMessage(code: string, status: string | undefined): string { + const hints: Record = { + bot_blocked: "This site blocks automated access. Use the search snippet data instead.", + timeout: "Page took too long to load. Try a different URL.", + target_unreachable: "Could not connect to this site. Try a different URL.", + page_not_found: "Page not found (404). The URL may be outdated. Try a different one.", + target_http_error: `Site returned HTTP ${status ?? "error"}. Try a different URL.`, + }; + return hints[code] ?? `Fetch failed: ${code}. Try a different URL.`; +} + +function buildSearchWebTool(options: WebToolOptions = {}) { + const datasetId = options.datasetId; + const abortSignal = options.abortSignal; + return createTool({ id: "search_web", description: 'Search the web for information. Returns a list of results with titles, snippets, and URLs. Call with: {"query": "your search terms"}', @@ -21,8 +167,10 @@ export const searchWebTool = createTool({ error: z.string().optional(), }), execute: async ({ query }) => { - if (!query?.trim()) + query = query?.trim(); + if (!query) return { error: "query is required and cannot be empty." }; + throwIfStopped(datasetId, abortSignal); const apiKey = await getTinyFishApiKey(); if (!apiKey) @@ -31,14 +179,13 @@ export const searchWebTool = createTool({ const url = `https://api.search.tinyfish.ai?query=${encodeURIComponent(query)}`; console.log(`[search_web] Searching: "${query}"`); - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS); + const request = timeoutSignalForDataset(datasetId, abortSignal); try { const res = await fetch(url, { headers: tinyFishHeaders(apiKey), - signal: controller.signal, + signal: request.signal, }); - clearTimeout(timeout); + throwIfStopped(datasetId, abortSignal); if (!res.ok) { const body = await res.text(); @@ -51,6 +198,7 @@ export const searchWebTool = createTool({ } const data = await res.json(); + throwIfStopped(datasetId, abortSignal); const results = (data.results ?? []).map((r: Record) => ({ title: r.title as string, snippet: r.snippet as string, @@ -62,17 +210,26 @@ export const searchWebTool = createTool({ return { results: [], error: "No results found for this query. Try a broader search or use synthetic data." }; return { results }; } catch (err) { - clearTimeout(timeout); - if (err instanceof Error && err.name === "AbortError") + throwIfStoppedAbort(err, datasetId, abortSignal); + if ( + request.timedOut() || + (err instanceof Error && (err.name === "AbortError" || err.name === "TimeoutError")) + ) return { error: "Search timed out. Skip web search and use synthetic data." }; const msg = err instanceof Error ? err.message : String(err); console.error(`[search_web] Failed:`, msg); return { error: `Search failed: ${msg}. Skip web search and use synthetic data.` }; + } finally { + request.cleanup(); } }, -}); + }); +} -export const fetchPageTool = createTool({ +function buildFetchPageTool(options: WebToolOptions = {}) { + const datasetId = options.datasetId; + const abortSignal = options.abortSignal; + return createTool({ id: "fetch_page", description: 'Fetch a web page and extract its content as clean markdown text. Call with: {"url": "https://example.com/page"}', @@ -85,19 +242,23 @@ export const fetchPageTool = createTool({ error: z.string().optional(), }), execute: async ({ url: targetUrl }) => { - if (!targetUrl?.trim()) + targetUrl = targetUrl?.trim(); + if (!targetUrl) return { error: "url is required and cannot be empty." }; if (!targetUrl.startsWith("http://") && !targetUrl.startsWith("https://")) return { error: `Invalid URL "${targetUrl}". Must start with http:// or https://.` }; + throwIfStopped(datasetId, abortSignal); const apiKey = await getTinyFishApiKey(); if (!apiKey) return { error: "TINYFISH_API_KEY is not configured. Page fetch is unavailable — use data from search snippets instead." }; + const cachedFailure = cachedFetchFailure(targetUrl); + if (cachedFailure) return { error: cachedFailure }; + console.log(`[fetch_page] Fetching: ${targetUrl}`); - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS); + const request = timeoutSignalForDataset(datasetId, abortSignal); try { const res = await fetch("https://api.fetch.tinyfish.ai", { method: "POST", @@ -106,9 +267,9 @@ export const fetchPageTool = createTool({ ...tinyFishHeaders(apiKey), }, body: JSON.stringify({ urls: [targetUrl], format: "markdown" }), - signal: controller.signal, + signal: request.signal, }); - clearTimeout(timeout); + throwIfStopped(datasetId, abortSignal); if (!res.ok) { const body = await res.text(); @@ -121,18 +282,18 @@ export const fetchPageTool = createTool({ } const data = await res.json(); + throwIfStopped(datasetId, abortSignal); if (data.errors?.length > 0) { const err = data.errors[0]; - console.log(`[fetch_page] Failed: ${err.error}`); - const hints: Record = { - bot_blocked: "This site blocks automated access. Use the search snippet data instead.", - timeout: "Page took too long to load. Try a different URL.", - target_unreachable: "Could not connect to this site. Try a different URL.", - page_not_found: "Page not found (404). The URL may be outdated. Try a different one.", - target_http_error: `Site returned HTTP ${err.status ?? "error"}. Try a different URL.`, - }; - return { error: hints[err.error] ?? `Fetch failed: ${err.error}. Try a different URL.` }; + const code = String(err.error ?? "unknown_error"); + const status = err.status === undefined ? undefined : String(err.status); + const message = fetchPageErrorMessage(code, status); + console.log( + `[fetch_page] Failed: url=${targetUrl} error=${code} status=${status ?? "n/a"}`, + ); + rememberFetchFailure(targetUrl, code, status, message); + return { error: message }; } const page = data.results?.[0]; @@ -151,12 +312,28 @@ export const fetchPageTool = createTool({ text, }; } catch (err) { - clearTimeout(timeout); - if (err instanceof Error && err.name === "AbortError") + throwIfStoppedAbort(err, datasetId, abortSignal); + if ( + request.timedOut() || + (err instanceof Error && (err.name === "AbortError" || err.name === "TimeoutError")) + ) return { error: "Page fetch timed out. Try a different URL or use search snippet data." }; const msg = err instanceof Error ? err.message : String(err); console.error(`[fetch_page] Failed:`, msg); return { error: `Fetch failed: ${msg}. Use data from search snippets instead.` }; + } finally { + request.cleanup(); } }, -}); + }); +} + +export function buildWebTools(options: WebToolOptions = {}) { + return { + search_web: buildSearchWebTool(options), + fetch_page: buildFetchPageTool(options), + }; +} + +export const searchWebTool = buildSearchWebTool(); +export const fetchPageTool = buildFetchPageTool(); diff --git a/backend/src/mastra/workflows/infer-schema.ts b/backend/src/mastra/workflows/infer-schema.ts index f14ec80..de1f001 100644 --- a/backend/src/mastra/workflows/infer-schema.ts +++ b/backend/src/mastra/workflows/infer-schema.ts @@ -3,23 +3,34 @@ import { z } from "zod"; import { inferSchema } from "../../pipeline/schema-inference.js"; import { datasetSchemaSchema } from "../../pipeline/types.js"; +const schemaInferenceModelConfigSchema = z.object({ + schemaInference: z.string().min(1), +}); + +const inferSchemaInputSchema = z.object({ + prompt: z.string().min(1), + modelConfig: schemaInferenceModelConfigSchema.optional(), + authContext: z.object({ + modelConfig: schemaInferenceModelConfigSchema, + }).optional(), +}); + const inferSchemaStep = createStep({ id: "infer-schema", - inputSchema: z.object({ - prompt: z.string().min(1), - }), + inputSchema: inferSchemaInputSchema, outputSchema: datasetSchemaSchema, execute: async ({ inputData }) => { - const schema = await inferSchema(inputData.prompt); + const modelSlug = + inputData.modelConfig?.schemaInference ?? + inputData.authContext?.modelConfig.schemaInference; + const schema = await inferSchema(inputData.prompt, modelSlug); return schema; }, }); export const inferSchemaWorkflow = createWorkflow({ id: "infer-schema-workflow", - inputSchema: z.object({ - prompt: z.string().min(1), - }), + inputSchema: inferSchemaInputSchema, outputSchema: datasetSchemaSchema, }) .then(inferSchemaStep) diff --git a/backend/src/mastra/workflows/populate.ts b/backend/src/mastra/workflows/populate.ts index e07ed8e..279ed82 100644 --- a/backend/src/mastra/workflows/populate.ts +++ b/backend/src/mastra/workflows/populate.ts @@ -1,15 +1,25 @@ import { createStep, createWorkflow } from "@mastra/core/workflows"; import { z } from "zod"; import { generateText } from "ai"; -import { createOpenRouter } from "@openrouter/ai-sdk-provider"; -import { datasetContextSchema, populateColumnSchema } from "../../pipeline/populate.js"; +import { + codificationProfileSchema, + datasetContextSchema, + populateColumnSchema, +} from "../../pipeline/populate.js"; import { convex, internal } from "../../convex.js"; import { DEFAULT_MODEL_IDS } from "../../config/models.js"; -import { requireOpenRouterApiKey } from "../../local-credentials.js"; +import { createLanguageModel } from "../../config/llm.js"; +import { AGENT_MAX_OUTPUT_TOKENS } from "../../config/agent-output-tokens.js"; +import { requireLlmProviderConfig } from "../../local-credentials.js"; import { buildPopulateAgent } from "../agents/populate.js"; +import { + clearQueuedSubagents, + drainQueuedSubagents, +} from "../tools/investigate-tool.js"; import { RunMetrics } from "../run-metrics.js"; import { saveRunMetrics } from "../save-run-metrics.js"; import { getSignal } from "../../abort-registry.js"; +import { env } from "../../env.js"; /** * Server-set auth/run context threaded through every step. @@ -37,6 +47,9 @@ export const authContextSchema = z.object({ schemaInference: z.string().min(1), populateOrchestrator: z.string().min(1), investigateSubagent: z.string().min(1), + extractorBuilder: z.string().min(1), + rowExtractorConcurrency: z.number().int().min(1).max(100).default(5), + rowExtractorBrowserAttempts: z.number().int().min(1).max(10).default(2), }), isBenchmark: z.boolean().optional(), }); @@ -85,7 +98,7 @@ const enumerateStep = createStep({ const columnsDesc = inputData.columns .map( (c) => - `- "${c.name}" (${c.type})${c.isPrimaryKey ? " [PK]" : ""}${c.description ? `: ${c.description}` : ""}`, + `- "${c.name}" (${c.type})${c.isPrimaryKey ? " [PK]" : ""}${c.nullable === false ? " [REQUIRED]" : c.nullable === true ? " [OPTIONAL]" : ""}${c.validationRegex ? ` validation_regex=${JSON.stringify(c.validationRegex)}` : ""}${c.normalizationHint ? ` normalization_hint=${JSON.stringify(c.normalizationHint)}` : ""}${c.description ? `: ${c.description}` : ""}`, ) .join("\n"); @@ -109,15 +122,11 @@ Respond with EXACTLY one word: scraper or search`; let classification: "scraper" | "search" = "search"; try { - const apiKey = await requireOpenRouterApiKey(); - const openrouter = createOpenRouter({ - apiKey, - baseURL: process.env.OPENROUTER_BASE_URL, - }); + const llmConfig = await requireLlmProviderConfig(); const modelSlug = - inputData.authContext?.modelConfig?.schemaInference ?? DEFAULT_MODEL_IDS.SCHEMA_INFERENCE; + inputData.authContext?.modelConfig?.schemaInference ?? llmConfig.defaultModel ?? DEFAULT_MODEL_IDS.SCHEMA_INFERENCE; const result = await generateText({ - model: openrouter(modelSlug), + model: createLanguageModel(llmConfig, modelSlug), prompt: classificationPrompt, maxOutputTokens: 10, abortSignal: getSignal(inputData.datasetId), @@ -161,6 +170,11 @@ const buildPromptOutputSchema = z.object({ authContext: authContextSchema, columns: z.array(populateColumnSchema), maxRowCount: z.number().int().min(1), + datasetName: z.string(), + description: z.string(), + retrievalStrategy: z.enum(["search_fetch", "browser", "hybrid"]).optional(), + sourceHint: z.string().optional(), + codificationProfile: z.optional(codificationProfileSchema), }); const buildPromptStep = createStep({ @@ -172,13 +186,13 @@ const buildPromptStep = createStep({ const columnsDesc = inputData.columns .map( (c) => - `- "${c.name}" (${c.type})${c.isPrimaryKey ? " [PRIMARY KEY]" : ""}${c.description ? `: ${c.description}` : ""}`, + `- "${c.name}" (${c.type})${c.isPrimaryKey ? " [PRIMARY KEY]" : ""}${c.nullable === false ? " [REQUIRED]" : c.nullable === true ? " [OPTIONAL]" : ""}${c.validationRegex ? ` validation_regex=${JSON.stringify(c.validationRegex)}` : ""}${c.normalizationHint ? ` normalization_hint=${JSON.stringify(c.normalizationHint)}` : ""}${c.description ? `: ${c.description}` : ""}`, ) .join("\n"); const pkNote = pkColumns.length > 0 - ? `\nPrimary key column(s): ${pkColumns.map((c) => `"${c.name}"`).join(", ")}. When calling run_subagent, you MUST pass these values in the primary_keys field. The subagent will research and fill in the remaining columns.` + ? `\nPrimary key column(s): ${pkColumns.map((c) => `"${c.name}"`).join(", ")}. When calling run_subagent, you MUST pass these values in the primary_keys field as an array of {"column": "column_name", "value": "value"} entries. The subagent will research and fill in the remaining columns.` : ""; let manifestNote = ""; @@ -205,7 +219,9 @@ Data fields to collect: ${columnsDesc}${pkNote}${manifestNote}${strategyNote} Search the web broadly to find real entities that fit this dataset topic. -For each lead you find, call run_subagent with the primary key values and any context/URLs you have found. +When you find multiple leads, call queue_subagents with a batch of candidates so discovery can continue while row workers run in the background. Use run_subagent only when you need immediate feedback for one lead. +Call drain_subagents before finishing if you want queued feedback; the workflow will also drain queued candidates automatically before completing. +Example primary_keys format: [{"column": "company_name", "value": "Stripe"}] If run_subagent returns ROW_LIMIT_REACHED, stop immediately and do not make any more tool calls. Stop the populate run as soon as the dataset reaches ${inputData.maxRowCount} rows.`; @@ -218,6 +234,11 @@ Stop the populate run as soon as the dataset reaches ${inputData.maxRowCount} ro authContext: inputData.authContext, columns: inputData.columns, maxRowCount: inputData.maxRowCount, + datasetName: inputData.datasetName, + description: inputData.description, + retrievalStrategy: inputData.retrievalStrategy, + sourceHint: inputData.sourceHint, + codificationProfile: inputData.codificationProfile, }; }, }); @@ -245,21 +266,45 @@ const agentStep = createStep({ const startedAt = Date.now(); let status: "success" | "error" = "success"; let errorMsg: string | undefined; + let drainedQueuedSubagents = false; try { const agent = buildPopulateAgent( inputData.authorizedDatasetId, inputData.authContext, inputData.columns, - await requireOpenRouterApiKey(), + await requireLlmProviderConfig(), inputData.maxRowCount, + { + datasetName: inputData.datasetName, + description: inputData.description, + retrievalStrategy: inputData.retrievalStrategy, + sourceHint: inputData.sourceHint, + codificationProfile: inputData.codificationProfile, + }, metrics, ); const abortSignal = getSignal(inputData.authorizedDatasetId); - const result = await agent.generate(inputData.prompt, { abortSignal, maxSteps: 80 }); + console.log( + `[populate-agent] Running orchestrator with maxSteps=${env.POPULATE_ORCHESTRATOR_MAX_STEPS}`, + ); + const result = await agent.generate(inputData.prompt, { + abortSignal, + maxSteps: env.POPULATE_ORCHESTRATOR_MAX_STEPS, + modelSettings: { + maxOutputTokens: AGENT_MAX_OUTPUT_TOKENS.POPULATE_ORCHESTRATOR, + }, + }); metrics.addOrchestratorResult(result); // Use result.toolCalls (flat accumulated list) — same reasoning as investigate-tool.ts. metrics.countToolCalls(result.toolCalls ?? []); + const queueSummary = await drainQueuedSubagents(inputData.authContext.workflowRunId); + drainedQueuedSubagents = true; + if (queueSummary.completed > 0) { + console.log( + `[populate-agent] Drained queued subagents completed=${queueSummary.completed} inserted=${queueSummary.inserted} failed=${queueSummary.failed}`, + ); + } return { text: result.text }; } catch (err) { status = "error"; @@ -274,6 +319,9 @@ const agentStep = createStep({ console.error(`[populate-agent] agent.generate failed: ${errorMsg}`); throw err; } finally { + if (!drainedQueuedSubagents) { + await clearQueuedSubagents(inputData.authContext.workflowRunId); + } const finishedAt = Date.now(); void saveRunMetrics({ workflowRunId: inputData.authContext.workflowRunId, diff --git a/backend/src/mastra/workflows/update.ts b/backend/src/mastra/workflows/update.ts index 28aadee..a6a841c 100644 --- a/backend/src/mastra/workflows/update.ts +++ b/backend/src/mastra/workflows/update.ts @@ -4,10 +4,17 @@ import { datasetContextSchema, populateColumnSchema } from "../../pipeline/popul import { convex, internal } from "../../convex.js"; import { buildRefreshAgent } from "../agents/refresh.js"; import { authContextSchema } from "./populate.js"; -import { requireOpenRouterApiKey } from "../../local-credentials.js"; +import { requireLlmProviderConfig } from "../../local-credentials.js"; import { RunMetrics } from "../run-metrics.js"; import { saveRunMetrics } from "../save-run-metrics.js"; -import { getSignal } from "../../abort-registry.js"; +import { + getSignal, + isAbortLikeError, + isDatasetRunAborted, + throwIfDatasetRunAborted, +} from "../../abort-registry.js"; +import { tryRefreshRowExtractor } from "../../row-extractors/try-row-extractor.js"; +import { AGENT_MAX_OUTPUT_TOKENS } from "../../config/agent-output-tokens.js"; export const updateInputSchema = datasetContextSchema.extend({ authContext: authContextSchema, @@ -31,6 +38,7 @@ const markAndFetchStep = createStep({ inputSchema: updateInputSchema, outputSchema: markAndFetchOutputSchema, execute: async ({ inputData }) => { + throwIfDatasetRunAborted(inputData.datasetId); const selective = inputData.rowIds && inputData.rowIds.length > 0; console.log( `[mark-and-fetch] Marking ${selective ? inputData.rowIds!.length : "all"} rows for dataset ${inputData.datasetId}`, @@ -40,10 +48,12 @@ const markAndFetchStep = createStep({ datasetId: inputData.datasetId, ...(selective ? { rowIds: inputData.rowIds } : {}), }); + throwIfDatasetRunAborted(inputData.datasetId); const rawRows = await convex.query(internal.datasetRows.listInternal, { datasetId: inputData.datasetId, }); + throwIfDatasetRunAborted(inputData.datasetId); let rows = (rawRows as Record[]).map((r) => ({ _id: String(r._id), @@ -69,8 +79,6 @@ const refreshOutputSchema = z.object({ errors: z.number(), }); -const MAX_CONCURRENT = 5; - async function processWithConcurrency( items: T[], handler: (item: T) => Promise, @@ -94,23 +102,92 @@ const refreshRowsStep = createStep({ inputSchema: markAndFetchOutputSchema, outputSchema: refreshOutputSchema, execute: async ({ inputData }) => { - const { datasetId, columns, authContext, rows } = inputData; + const { + datasetId, + columns, + authContext, + rows, + datasetName, + description, + retrievalStrategy, + sourceHint, + } = inputData; let updatedCount = 0; let errors = 0; const metrics = new RunMetrics(); const startedAt = Date.now(); - const openRouterApiKey = await requireOpenRouterApiKey(); + let llmConfigPromise: ReturnType | undefined; + const getLlmConfig = () => { + llmConfigPromise ??= requireLlmProviderConfig(); + return llmConfigPromise; + }; const pkColumns = columns.filter((c) => c.isPrimaryKey); + const maxConcurrent = authContext.modelConfig.rowExtractorConcurrency; async function processRow(row: z.infer) { try { + throwIfDatasetRunAborted(datasetId); + const primaryKeyRecord = Object.fromEntries( + pkColumns + .map((column) => [column.name, String(row.data[column.name] ?? "").trim()]) + .filter(([, value]) => value.length > 0), + ); + + const extractorResult = await tryRefreshRowExtractor({ + datasetId, + rowId: row._id, + columns, + primaryKeys: primaryKeyRecord, + existingData: row.data, + urls: row.sources, + context: [row.rowSummary, row.howFound].filter(Boolean).join("\n"), + datasetName, + description, + retrievalStrategy, + sourceHint, + codificationProfile: inputData.codificationProfile, + browserAttempts: authContext.modelConfig.rowExtractorBrowserAttempts, + extractorBuilderModel: authContext.modelConfig.extractorBuilder, + }); + throwIfDatasetRunAborted(datasetId); + + if (extractorResult.status === "updated") { + updatedCount++; + metrics.rowsUpdated++; + console.log( + `[refresh-rows] Row ${row._id}: updated=true via=row_extractor reason="${extractorResult.reason}"`, + ); + return; + } + + if (extractorResult.status === "unchanged") { + console.log( + `[refresh-rows] Row ${row._id}: updated=false via=row_extractor reason="${extractorResult.reason}"`, + ); + return; + } + + if (extractorResult.status === "miss") { + console.log( + `[refresh-rows] Row ${row._id}: row extractor missed; falling back to refresh agent: ${extractorResult.reason}`, + ); + } + + if (extractorResult.status === "failed") { + if (isDatasetRunAborted(datasetId)) throwAbortError(); + console.warn( + `[refresh-rows] Row ${row._id}: row extractor failed; falling back to refresh agent: ${extractorResult.reason}`, + ); + } + + throwIfDatasetRunAborted(datasetId); const agent = buildRefreshAgent( datasetId, authContext, columns, - openRouterApiKey, + await getLlmConfig(), ); const pkBlock = @@ -141,7 +218,14 @@ ${row.rowSummary ? `\nPrevious summary: ${row.rowSummary}` : ""} ${row.howFound ? `\nPreviously found via: ${row.howFound}` : ""}`; const abortSignal = getSignal(datasetId); - const result = await agent.generate(prompt, { abortSignal, maxSteps: 10 }); + throwIfDatasetRunAborted(datasetId); + const result = await agent.generate(prompt, { + abortSignal, + maxSteps: 10, + modelSettings: { + maxOutputTokens: AGENT_MAX_OUTPUT_TOKENS.REFRESH_AGENT, + }, + }); // Accumulate token usage into the investigate tier (refresh agents map // to the investigate tier so the runStats schema needs no new columns). @@ -168,7 +252,7 @@ ${row.howFound ? `\nPreviously found via: ${row.howFound}` : ""}`; // Only re-throw if OUR signal was actually fired. Spurious network // AbortErrors must not terminate a worker — they should be counted as // row errors so the rest of the dataset continues refreshing. - if (err instanceof Error && err.name === "AbortError" && getSignal(datasetId)?.aborted) throw err; + if (isAbortLikeError(err) && isDatasetRunAborted(datasetId)) throw err; const msg = err instanceof Error ? err.message : String(err); console.error( `[refresh-rows] Row ${row._id} failed: ${msg}`, @@ -191,9 +275,9 @@ ${row.howFound ? `\nPreviously found via: ${row.howFound}` : ""}`; } console.log( - `[refresh-rows] Processing ${rows.length} rows (max ${MAX_CONCURRENT} concurrent)`, + `[refresh-rows] Processing ${rows.length} rows (max ${maxConcurrent} concurrent)`, ); - await processWithConcurrency(rows, processRow, MAX_CONCURRENT); + await processWithConcurrency(rows, processRow, maxConcurrent); const finishedAt = Date.now(); // If the run was stopped mid-update, workers exited early via AbortError. @@ -215,6 +299,10 @@ ${row.howFound ? `\nPreviously found via: ${row.howFound}` : ""}`; `[refresh-rows] Done: ${updatedCount} updated, ${errors} errors, ${rows.length - updatedCount - errors} unchanged`, ); + const allRowsFailed = rows.length > 0 && errors === rows.length; + const refreshError = + errors > 0 ? `${errors} of ${rows.length} row(s) failed to refresh` : undefined; + // Persist metrics — fire-and-forget; never block the workflow return. void saveRunMetrics({ workflowRunId: authContext.workflowRunId, @@ -226,10 +314,8 @@ ${row.howFound ? `\nPreviously found via: ${row.howFound}` : ""}`; // Total failure: every row errored. Partial failure: some rows errored // but at least one succeeded — still "success" overall, but the error // field records how many failed so partial issues are visible in the data. - status: errors > 0 && updatedCount === 0 ? "error" : "success", - error: errors > 0 - ? `${errors} of ${rows.length} row(s) failed to refresh` - : undefined, + status: allRowsFailed ? "error" : "success", + error: refreshError, workflowType: "update", }).catch((err) => console.error( @@ -238,10 +324,20 @@ ${row.howFound ? `\nPreviously found via: ${row.howFound}` : ""}`; ), ); + if (allRowsFailed) { + throw new Error(refreshError); + } + return { updatedCount, totalCount: rows.length, errors }; }, }); +function throwAbortError(): never { + const err = new Error("Run was stopped"); + err.name = "AbortError"; + throw err; +} + export const updateWorkflow = createWorkflow({ id: "update-workflow", inputSchema: updateInputSchema, diff --git a/backend/src/pipeline/codification.ts b/backend/src/pipeline/codification.ts new file mode 100644 index 0000000..5ebf9a6 --- /dev/null +++ b/backend/src/pipeline/codification.ts @@ -0,0 +1,287 @@ +import type { + CodificationProfile, + PopulateColumn, +} from "./populate.js"; +import type { + CodificationProfile as SchemaCodificationProfile, +} from "./types.js"; + +interface CodificationClassificationInput { + datasetName?: string; + description?: string; + columns: PopulateColumn[]; + primaryKeys?: Record; + urls?: string[]; + context?: string; + retrievalStrategy?: "search_fetch" | "browser" | "hybrid"; + sourceHint?: string; +} + +const PROFILE_VERSION = 1; +const BROAD_RESEARCH_TEXT_PATTERN = + /\b(across|around|from)\s+the\s+web\b|\bsearch\s+the\s+web\b|\bany\s+source\b/; + +export function schemaCodificationProfileToRuntime( + profile: SchemaCodificationProfile, +): CodificationProfile { + return { + version: PROFILE_VERSION, + mode: profile.mode, + reason: profile.reason, + primaryKeyShape: profile.primary_key_shape, + families: profile.families.map((family) => ({ + label: family.label, + sourceHost: family.source_host, + sourcePathPrefix: family.source_path_prefix, + urlTemplate: family.url_template, + primaryKeyRegex: family.primary_key_regex, + })), + }; +} + +export function normalizeCodificationProfile( + profile: CodificationProfile | undefined, + input: CodificationClassificationInput, +): CodificationProfile { + return profile ?? classifyCodificationProfile(input); +} + +export function shouldAttemptCodification( + profile: CodificationProfile, + input?: CodificationClassificationInput, +): boolean { + if (profile.mode === "candidate" || profile.mode === "required") return true; + if (!input || (profile.mode !== "disabled" && profile.mode !== "unknown")) return false; + return hasConcreteCodificationRoute(profile, input); +} + +export function classifyCodificationProfile( + input: CodificationClassificationInput, +): CodificationProfile { + const primaryKeyShape = inferPrimaryKeyShape(input.columns); + const sourceUrl = firstHttpUrl(input.sourceHint); + const sourceFamily = sourceUrl ? familyFromUrl(sourceUrl) : undefined; + const retrievalStrategy = input.retrievalStrategy ?? "search_fetch"; + const broadResearch = isBroadResearchInput(input); + + if (!sourceUrl && broadResearch) { + return disabledProfile( + primaryKeyShape, + "Prompt describes broad web research rather than one stable page family.", + ); + } + + if (!sourceUrl && primaryKeyShape !== "url") { + return disabledProfile( + primaryKeyShape, + "No source URL or URL-shaped primary key; legacy metadata only supports broad investigation.", + ); + } + + if (primaryKeyShape === "url") { + return { + version: PROFILE_VERSION, + mode: "candidate", + reason: "Primary key is URL-shaped, so rows can be routed by page family.", + primaryKeyShape, + families: sourceFamily ? [sourceFamily] : [], + }; + } + + if (sourceFamily && (primaryKeyShape === "slug" || primaryKeyShape === "id")) { + return { + version: PROFILE_VERSION, + mode: retrievalStrategy === "browser" ? "required" : "candidate", + reason: "Dataset has a source URL and structured primary keys that may map to one page family.", + primaryKeyShape, + families: [sourceFamily], + }; + } + + if (sourceFamily && retrievalStrategy === "browser") { + return { + version: PROFILE_VERSION, + mode: "candidate", + reason: "Browser retrieval with an explicit source URL may support a reusable extractor.", + primaryKeyShape, + families: [sourceFamily], + }; + } + + return disabledProfile( + primaryKeyShape, + "Legacy metadata does not expose a stable URL, slug, ID, or browser source family.", + ); +} + +function disabledProfile( + primaryKeyShape: CodificationProfile["primaryKeyShape"], + reason: string, +): CodificationProfile { + return { + version: PROFILE_VERSION, + mode: "disabled", + reason, + primaryKeyShape, + families: [], + }; +} + +function inferPrimaryKeyShape( + columns: PopulateColumn[], +): CodificationProfile["primaryKeyShape"] { + const pkColumns = columns.filter((column) => column.isPrimaryKey); + if (pkColumns.length === 0) return "unknown"; + const shapes = new Set(pkColumns.map(inferColumnShape)); + if (shapes.size === 1) return [...shapes][0] ?? "unknown"; + return "mixed"; +} + +function inferColumnShape(column: PopulateColumn): CodificationProfile["primaryKeyShape"] { + const name = column.name.toLowerCase(); + const description = (column.description ?? "").toLowerCase(); + const regex = (column.validationRegex ?? "").toLowerCase(); + const text = `${name} ${description} ${regex}`; + + if (column.type === "url" || /\burl\b|https\?:|https?:/.test(text)) return "url"; + if (/\bslug\b|\bhandle\b|\bpath\b|\brepo\b|\bpackage\b|\busername\b/.test(text)) { + return "slug"; + } + if (/\bid\b|_id\b|\bidentifier\b|\buuid\b|\bisbn\b|\bsku\b/.test(text)) { + return "id"; + } + if (/\bname\b|\btitle\b/.test(text)) return "name"; + return "unknown"; +} + +function hasConcreteCodificationRoute( + profile: CodificationProfile, + input: CodificationClassificationInput, +): boolean { + const hasUsableTemplate = profile.families.some( + (family) => family.urlTemplate && hasTemplateValues(family.urlTemplate, input), + ); + if (hasUsableTemplate) return true; + + const broadResearch = + isBroadResearchDisableReason(profile.reason) || isBroadResearchInput(input); + if (broadResearch) return false; + + const rowUrl = firstHttpUrl( + [ + ...Object.values(input.primaryKeys ?? {}), + ...(input.urls ?? []), + input.context, + ].join(" "), + ); + if (rowUrl) return true; + + const sourceUrl = firstHttpUrl(input.sourceHint); + const primaryKeyShape = + profile.primaryKeyShape === "unknown" + ? inferPrimaryKeyShape(input.columns) + : profile.primaryKeyShape; + const structuredPrimaryKey = + primaryKeyShape === "id" || + primaryKeyShape === "slug" || + primaryKeyShape === "url"; + if (sourceUrl && structuredPrimaryKey) return true; + + const accessRiskOnly = /\b(block|blocked|captcha|bot|automation|browser|fetch|access|degrad)/i.test( + profile.reason, + ); + return accessRiskOnly && (structuredPrimaryKey || hasIdentifierLikePrimaryKey(input.primaryKeys)); +} + +function isBroadResearchDisableReason(reason: string): boolean { + return /\b(broad web|broad investigation|arbitrary unrelated|unrelated domains|snippet-only|search snippets)\b/i.test(reason); +} + +function isBroadResearchInput(input: CodificationClassificationInput): boolean { + return BROAD_RESEARCH_TEXT_PATTERN.test(codificationSearchableText(input)); +} + +function codificationSearchableText(input: CodificationClassificationInput): string { + return [ + input.datasetName, + input.description, + input.sourceHint, + ...input.columns.map((column) => `${column.name} ${column.description ?? ""}`), + ] + .join(" ") + .toLowerCase(); +} + +function hasTemplateValues( + template: string, + input: CodificationClassificationInput, +): boolean { + const placeholders = [...template.matchAll(/\{([a-zA-Z0-9_]+)\}/g)].map( + (match) => match[1], + ); + if (placeholders.length === 0) return false; + return placeholders.every((placeholder) => + Boolean(findPrimaryKeyValue(placeholder, input.primaryKeys ?? {})), + ); +} + +function hasIdentifierLikePrimaryKey( + primaryKeys: Record | undefined, +): boolean { + const values = Object.values(primaryKeys ?? {}) + .map((value) => value.trim()) + .filter(Boolean); + if (values.length === 0) return false; + + return values.every((value) => { + if (/\s/.test(value)) return false; + if (/^https?:\/\//i.test(value)) return true; + if (!/^[a-z0-9._~:/?#\[\]@!$&'()*+,;=%-]{6,}$/i.test(value)) return false; + return /[0-9._~:/?#\[\]@!$&'()*+,;=%-]/.test(value); + }); +} + +function firstHttpUrl(value: string | undefined): URL | undefined { + if (!value) return undefined; + const match = value.match(/https?:\/\/[^\s)>"']+/i); + if (!match) return undefined; + try { + return new URL(match[0].replace(/[.,;:]+$/, "")); + } catch { + return undefined; + } +} + +function familyFromUrl(url: URL): CodificationProfile["families"][number] { + const pathPrefix = url.pathname.split("/").filter(Boolean)[0]; + return { + label: sanitizeFamilyLabel([url.hostname, pathPrefix].filter(Boolean).join("_")), + sourceHost: url.hostname.toLowerCase(), + sourcePathPrefix: pathPrefix ? `/${pathPrefix}` : undefined, + }; +} + +function sanitizeFamilyLabel(value: string): string { + const label = value + .toLowerCase() + .replace(/[^a-z0-9]+/g, "_") + .replace(/^_+|_+$/g, ""); + return /^[a-z]/.test(label) ? label : `site_${label || "unknown"}`; +} + +function findPrimaryKeyValue( + columnName: string | undefined, + primaryKeys: Record, +): string | undefined { + if (!columnName) return undefined; + if (primaryKeys[columnName]) return primaryKeys[columnName]; + const normalizedColumn = normalizeFieldName(columnName); + const entry = Object.entries(primaryKeys).find( + ([key]) => normalizeFieldName(key) === normalizedColumn, + ); + return entry?.[1]; +} + +function normalizeFieldName(value: string): string { + return value.toLowerCase().replace(/[^a-z0-9]+/g, ""); +} diff --git a/backend/src/pipeline/populate.ts b/backend/src/pipeline/populate.ts index 589db1e..1a1bb37 100644 --- a/backend/src/pipeline/populate.ts +++ b/backend/src/pipeline/populate.ts @@ -7,9 +7,29 @@ export const populateColumnSchema = z.object({ type: z.enum(["text", "number", "boolean", "url", "date"]), description: z.optional(z.string()), isPrimaryKey: z.optional(z.boolean()), + nullable: z.optional(z.boolean()), + validationRegex: z.optional(z.string()), + normalizationHint: z.optional(z.string()), }); export type PopulateColumn = z.infer; +export const codificationProfileSchema = z.object({ + version: z.literal(1), + mode: z.enum(["disabled", "candidate", "required", "unknown"]), + reason: z.string(), + primaryKeyShape: z.enum(["url", "slug", "name", "id", "mixed", "unknown"]), + families: z.array( + z.object({ + label: z.string(), + sourceHost: z.optional(z.string()), + sourcePathPrefix: z.optional(z.string()), + urlTemplate: z.optional(z.string()), + primaryKeyRegex: z.optional(z.string()), + }), + ), +}); +export type CodificationProfile = z.infer; + export const datasetContextSchema = z.object({ datasetId: z.string().min(1), datasetName: z.string(), @@ -17,5 +37,8 @@ export const datasetContextSchema = z.object({ maxRowCount: z.number().int().min(1).max(FREE_TIER_MONTHLY_QUOTA).default(100), columns: z.array(populateColumnSchema).min(1), rowIds: z.array(z.string()).min(1).optional(), + retrievalStrategy: z.enum(["search_fetch", "browser", "hybrid"]).optional(), + sourceHint: z.string().optional(), + codificationProfile: z.optional(codificationProfileSchema), }); export type DatasetContext = z.infer; diff --git a/backend/src/pipeline/schema-inference.ts b/backend/src/pipeline/schema-inference.ts index 467a393..3f48cc9 100644 --- a/backend/src/pipeline/schema-inference.ts +++ b/backend/src/pipeline/schema-inference.ts @@ -1,16 +1,21 @@ import { generateText, Output, NoObjectGeneratedError } from "ai"; -import { createOpenRouter } from "@openrouter/ai-sdk-provider"; import { DEFAULT_MODEL_IDS } from "../config/models.js"; -import { requireOpenRouterApiKey } from "../local-credentials.js"; -import { datasetSchemaSchema, type DatasetSchema } from "./types.js"; +import { createLanguageModel } from "../config/llm.js"; +import { requireLlmProviderConfig } from "../local-credentials.js"; +import { + datasetSchemaSchema, + type ColumnDefinition, + type DatasetSchema, + type RetrievalStrategy, +} from "./types.js"; const SYSTEM_PROMPT = `You are a data engineering assistant that converts natural-language prompts into structured dataset schemas. Given a user prompt describing a dataset they want to build, you produce a precise schema definition. Your job is to: 1. Identify the universe of entities the user wants to collect. Each entity becomes one row in the dataset. -2. Pick primary key column(s) — one or more columns whose combined values uniquely identify each row (no two legitimate rows should share the same values across all primary key columns in any case). Refrain from names unless necessary, as they may not always be unqiue (unless this is guarenteed). Otherwise use thigns like URLs or IDs that have a 100% guarentee of being unique. Set \`is_primary_key: true\` on each primary key column. Set \`primary_key\` to the column name if there is one, or an array of column names if there are multiple. Every primary key column must have \`nullable: false\` and \`is_enumerable: true\`. Prefer a single column when one naturally uniquely identifies each row. +2. Pick primary key column(s) — one or more columns whose combined values uniquely identify each row (no two legitimate rows should share the same values across all primary key columns in any case). Refrain from names unless necessary, as they may not always be unqiue (unless this is guarenteed). Otherwise use thigns like URLs or IDs that have a 100% guarentee of being unique. Set \`is_primary_key: true\` on each primary key column. Set \`primary_key\` to an array of primary key column names; use a one-item array for a single primary key. Every primary key column must have \`nullable: false\` and \`is_enumerable: true\`. Prefer a single column when one naturally uniquely identifies each row. 3. Choose useful columns. Each column captures one fact about the entity. Use snake_case names. Mark \`is_enumerable: true\` only on columns whose values can be used to list all rows (typically just the primary key, and occasionally one or two others when a source page lists them alongside the primary key). 4. Set \`retrieval_strategy\`: - \`search_fetch\` — the data lives on a static page or sitemap that can be fetched as HTML. @@ -18,6 +23,14 @@ Your job is to: - \`hybrid\` — unclear; the pipeline will try search_fetch first and fall back to browser. 5. Set \`source_hint\` to a specific URL whenever possible (e.g. \`https://www.ycombinator.com/companies?industry=Fintech\`). Avoid vague descriptions. 6. Write a \`retrieval_hint\` for each column describing where/how the value can be found later. Downstream agents will use this to fill the column for each row. +7. For each column where a value has a known shape, include \`validation_regex\` and \`normalization_hint\`. These are extractor contracts, not UI decoration. Examples: ratings, prices, dates, URL/slug shapes, repository slugs, app package names, counts, currencies, availability labels. Omit \`validation_regex\` only when the value is genuinely free-form text. +8. Set \`codification_profile\`. This is a cheap schema-time decision about whether BigSet should attempt to compile a reusable Playwright extractor later. + - \`mode: "disabled"\` when rows will come from broad web search, arbitrary unrelated domains, or search snippets with no stable page family. + - \`mode: "candidate"\` when rows likely share one or more stable page families and a reusable browser script may work after seeing a representative row. + - Use \`mode: "candidate"\`, not \`disabled\`, when the only concern is that the stable source has anti-bot or automation-blocking reputation. BigSet uses TinyFish Browser for interactive browser access, and TinyFish can often get through surfaces that plain fetches or commodity browser automation cannot. The extractor builder will inspect a real page and decline if it is actually blocked. + - \`mode: "required"\` when the dataset is clearly tied to one authoritative browser-heavy source or directory where repeated extraction is the intended path. + - \`mode: "unknown"\` only when the prompt/schema gives too little evidence; prefer \`disabled\` over \`unknown\` for broad web datasets to avoid expensive extractor attempts. + Include \`primary_key_shape\` as one of \`url\`, \`slug\`, \`name\`, \`id\`, \`mixed\`, or \`unknown\`. Include \`families\` for known source/page families. Use snake_case labels. For URL templates, use column placeholders like \`https://github.com/{repo_slug}\`. Rules: @@ -25,20 +38,50 @@ Rules: - \`dataset_name\` must be snake_case. - All column \`name\` values must be snake_case and unique. - Prefer concrete column choices over speculative ones — better to omit a column than guess wildly. +- Validation regexes must validate the normalized final value, not raw page text. Keep them practical and anchored, e.g. "^[0-5](\\\\.\\\\d)?$" for a normalized rating or "^[^/\\\\s]+/[^/\\\\s]+$" for an owner/repo slug. +- \`normalization_hint\` should tell the extractor how to convert raw page text into the stored value, e.g. "Convert '4.6 out of 5 stars' to '4.6'" or "Strip commas and convert 1.2k to 1200". +- \`codification_profile\` should be conservative. Do not mark arbitrary company/person/place/product research as codifiable just because pages exist on the web. Mark it codifiable only when rows can be routed to stable page families from the primary key, source_hint, or obvious URL templates. +- For marketplace/catalog identifiers with deterministic product pages, prefer a URL-template family over disabling codification. Use the site/schema's actual identifier and route shape; do not hardcode a source-specific template unless it is implied by the user prompt or discovered source hint. - When a column is a scalar numeric rating (e.g. average score like 4.3/5 for restaurants, cafes, hotels, products, apps): name it generically (e.g. "rating" not "yelp_rating") and write a retrieval_hint explaining that review sites (Yelp, TripAdvisor, Google Maps) block direct page fetches, so the agent must extract ratings from **search result snippets**. The hint should say: "Search for \\" rating reviews\\" and include location terms only when location is part of the entity identity. Look for ratings in snippets from TripAdvisor (\\"rated X.X of 5\\"), Yelp search listings (\\"X.X (N reviews)\\"), or aggregator sites (Birdeye, joe.coffee, giftly, Uber Eats, menufyy). Do NOT try to fetch yelp.com or tripadvisor.com directly — they block automated access. Accept ratings from any reputable source." If including a rating column, also add a "rating_source" text column so the agent records where the rating came from. Do not rename review-count or review-text fields to "rating" — keep those as distinct columns (e.g. "review_count") when the user explicitly asks for them.`; +export interface FinalizeSchemaColumnInput { + name: string; + type: ColumnDefinition["type"]; + description?: string; + isPrimaryKey?: boolean; +} + +export interface FinalizeSchemaInput { + prompt: string; + datasetName?: string; + columns: FinalizeSchemaColumnInput[]; + retrievalStrategy?: RetrievalStrategy; + sourceHint?: string; +} + async function getModel(modelSlug?: string) { - const apiKey = await requireOpenRouterApiKey(); - const openrouter = createOpenRouter({ - apiKey, - baseURL: process.env.OPENROUTER_BASE_URL, - }); - const resolvedSlug = modelSlug ?? DEFAULT_MODEL_IDS.SCHEMA_INFERENCE; - return openrouter(resolvedSlug); + const config = await requireLlmProviderConfig(); + const resolvedSlug = modelSlug ?? config.defaultModel ?? DEFAULT_MODEL_IDS.SCHEMA_INFERENCE; + return createLanguageModel(config, resolvedSlug); } export async function inferSchema(prompt: string, modelSlug?: string): Promise { const model = await getModel(modelSlug); + return await generateSchema(model, prompt); +} + +export async function finalizeSchemaContracts( + input: FinalizeSchemaInput, + modelSlug?: string, +): Promise { + const model = await getModel(modelSlug); + return await generateSchema(model, buildFinalizePrompt(input)); +} + +async function generateSchema( + model: Parameters[0]["model"], + prompt: string, +): Promise { try { return await callOnce(model, prompt); } catch (error) { @@ -51,6 +94,41 @@ export async function inferSchema(prompt: string, modelSlug?: string): Promise { + const primaryKey = column.isPrimaryKey ? "yes" : "no"; + const description = column.description?.trim() || "(none)"; + return `${index + 1}. visible_name=${JSON.stringify(column.name)} type=${column.type} primary_key=${primaryKey} description=${JSON.stringify(description)}`; + }) + .join("\n"); + + return `Refresh the hidden extraction contracts for this final, user-reviewed dataset schema. + +Original user request: +${input.prompt} + +Final dataset display name: +${input.datasetName?.trim() || "(not provided)"} + +Final visible columns: +${columns} + +Current retrieval strategy: ${input.retrievalStrategy ?? "(not set)"} +Current source hint: ${input.sourceHint?.trim() || "(not set)"} + +Return a complete DatasetSchema for the final visible schema above. + +Rules for this refresh: +- Do not add, remove, split, or reorder columns. Return exactly ${input.columns.length} columns in the same order. +- Preserve each column's type and visible meaning. Use a snake_case "name" derived from visible_name only because DatasetSchema requires it. +- Keep primary_key flags as provided unless no column is marked primary_key=yes; in that case choose the best primary key from the final columns. +- Use each visible description as the basis for retrieval_hint, refining only to clarify how to extract that same value. +- Regenerate nullable, validation_regex, normalization_hint, source_hint, retrieval_strategy, and codification_profile from this final schema. +- Include validation_regex for shaped values where it adds real validation value. Omit validation_regex for genuinely free-form text instead of emitting a catch-all regex. +- validation_regex must validate the normalized final stored value, not raw page text.`; +} + async function callOnce( model: Parameters[0]["model"], prompt: string, @@ -59,7 +137,7 @@ async function callOnce( model, output: Output.object({ schema: datasetSchemaSchema }), system: SYSTEM_PROMPT, - maxOutputTokens: 4096, + maxOutputTokens: 5000, prompt, }); if (!output) throw new Error("Model did not generate a valid schema object"); diff --git a/backend/src/pipeline/types.ts b/backend/src/pipeline/types.ts index e0b95f9..9aa9e6a 100644 --- a/backend/src/pipeline/types.ts +++ b/backend/src/pipeline/types.ts @@ -27,17 +27,56 @@ export const columnDefinitionSchema = z.object({ is_enumerable: z.boolean(), retrieval_hint: z.string(), nullable: z.boolean(), + validation_regex: z.string().optional(), + normalization_hint: z.string().optional(), }); export type ColumnDefinition = z.infer; +export const codificationModeSchema = z.enum([ + "disabled", + "candidate", + "required", + "unknown", +]); +export type CodificationMode = z.infer; + +export const primaryKeyShapeSchema = z.enum([ + "url", + "slug", + "name", + "id", + "mixed", + "unknown", +]); +export type PrimaryKeyShape = z.infer; + +export const codificationFamilySchema = z.object({ + label: z.string().regex(snakeCase, "must be snake_case"), + source_host: z.string().optional(), + source_path_prefix: z.string().optional(), + url_template: z.string().optional(), + primary_key_regex: z.string().optional(), +}); +export type CodificationFamily = z.infer; + +export const codificationProfileSchema = z.object({ + version: z.literal(1), + mode: codificationModeSchema, + reason: z.string().min(1), + primary_key_shape: primaryKeyShapeSchema, + families: z.array(codificationFamilySchema), +}); +export type CodificationProfile = z.infer; + export const datasetSchemaSchema = z .object({ dataset_name: z.string().regex(snakeCase, "must be snake_case"), description: z.string().min(1), columns: z.array(columnDefinitionSchema).min(1), - primary_key: z.union([z.string(), z.array(z.string())]), + primary_key: z.array(z.string()).min(1), retrieval_strategy: retrievalStrategySchema, source_hint: z.string().min(1), + codification_profile: codificationProfileSchema, }) .superRefine((data, ctx) => { const names = data.columns.map((c) => c.name); @@ -61,9 +100,7 @@ export const datasetSchemaSchema = z } const pkNames = pkCols.map((c) => c.name); - const declaredPkRaw = Array.isArray(data.primary_key) - ? data.primary_key - : [data.primary_key]; + const declaredPkRaw = data.primary_key; const declaredPk = [...new Set(declaredPkRaw)]; if ( declaredPk.length !== declaredPkRaw.length || diff --git a/backend/src/row-extractors/try-row-extractor.ts b/backend/src/row-extractors/try-row-extractor.ts new file mode 100644 index 0000000..da75487 --- /dev/null +++ b/backend/src/row-extractors/try-row-extractor.ts @@ -0,0 +1,3082 @@ +import { chromium, type Browser, type Page } from "playwright-core"; +import { Agent } from "@mastra/core/agent"; +import { createTool } from "@mastra/core/tools"; +import { spawn } from "node:child_process"; +import { createHash } from "node:crypto"; +import { fileURLToPath } from "node:url"; +import { z } from "zod"; + +import { getSignal } from "../abort-registry.js"; +import { convex, internal } from "../convex.js"; +import { FETCH_TIMEOUT_MS } from "../fetch-timeout.js"; +import { + getTinyFishApiKey, + requireLlmProviderConfig, + tinyFishHeaders, +} from "../local-credentials.js"; +import { + createLanguageModel, +} from "../config/llm.js"; +import { AGENT_MAX_OUTPUT_TOKENS } from "../config/agent-output-tokens.js"; +import type { CodificationProfile, PopulateColumn } from "../pipeline/populate.js"; +import { + normalizeCodificationProfile, + shouldAttemptCodification, +} from "../pipeline/codification.js"; +import { DEFAULT_MODEL_IDS } from "../config/models.js"; + +type ExtractorStatus = "inserted" | "updated" | "unchanged" | "miss" | "failed"; +type ExtractedValue = string | number | boolean; +type ColumnExtractionStatus = + | "extracted" + | "derived" + | "not_present_on_page" + | "blocked" + | "ambiguous" + | "validation_failed" + | "fallback_needed" + | "missing"; + +interface ColumnStatus { + status: ColumnExtractionStatus; + reason?: string; +} + +export interface TryRowExtractorInput { + datasetId: string; + datasetName?: string; + description?: string; + columns: PopulateColumn[]; + primaryKeys: Record; + urls?: string[]; + context?: string; + retrievalStrategy?: "search_fetch" | "browser" | "hybrid"; + sourceHint?: string; + codificationProfile?: CodificationProfile; + browserAttempts?: number; + extractorBuilderModel?: string; + abortSignal?: AbortSignal; +} + +export interface TryRefreshRowExtractorInput extends TryRowExtractorInput { + rowId: string; + existingData: Record; +} + +export interface TryRowExtractorResult { + status: ExtractorStatus; + reason: string; + rowSummary?: string; + sources?: string[]; +} + +export interface RowExtractorDraftResult { + status: "extracted" | "miss" | "failed"; + reason: string; + data?: Record; + rowSummary?: string; + sources?: string[]; + cellSources?: Record; + extractedColumns?: string[]; + missingColumns?: string[]; + requiredMissingColumns?: string[]; + optionalMissingColumns?: string[]; + columnStatuses?: Record; +} + +interface TinyFishBrowserSession { + session_id: string; + cdp_url: string; + base_url: string; +} + +interface PageEvidence { + finalUrl: string; + title?: string; + description?: string; + candidates: Record; + bodyText: string; +} + +interface GenericExtraction { + data: Record; + sources: string[]; + rowSummary?: string; + cellSources: Record; + extractedColumns: string[]; + missingColumns: string[]; + requiredMissingColumns: string[]; + optionalMissingColumns: string[]; + columnStatuses: Record; +} + +interface GeneratedExtractorResult { + data?: Record; + sources?: unknown; + cell_sources?: unknown; + cellSources?: unknown; + row_summary?: unknown; + rowSummary?: unknown; + how_found?: unknown; + howFound?: unknown; + column_status?: unknown; + columnStatus?: unknown; +} + +interface DetailedExtractorTestResult { + ok: boolean; + reason: string; + extraction?: GenericExtraction; +} + +interface AgenticExtractorBuildResult { + script: string; + probeSummary: string; +} + +interface CachedExtractorScript { + script: string; + updatedAt: number; +} + +interface RuntimeRepairResult { + script: string; + cachedExtractor?: CachedExtractorScript; +} + +interface ExtractorSmokeTestCase { + input: TryRowExtractorInput; + url: string; + source: "memory" | "persisted"; +} + +const BROWSER_TIMEOUT_MS = 45_000; +const CDP_CONNECT_TIMEOUT_MS = 45_000; +const DEFAULT_BROWSER_ATTEMPTS = 2; +const TRANSIENT_BROWSER_ATTEMPTS = 2; +const EXTRACTOR_RUNNER_TIMEOUT_MS = 60_000; +const EXTRACTOR_SCRIPT_OUTPUT_LIMIT = 256_000; +const EXTRACTOR_BUILDER_MAX_STEPS = 80; +const EXTRACTOR_REPAIR_MAX_STEPS = 24; +const BACKEND_ROOT = fileURLToPath(new URL("../../", import.meta.url)); +const GENERIC_EXTRACTOR_HOW_FOUND = + "Opened the row target with TinyFish Browser and ran the dataset's sandboxed browser extractor."; +const GENERIC_REFRESH_HOW_FOUND = + "Refreshed the row target with TinyFish Browser and ran the dataset's sandboxed browser extractor."; +const inFlightExtractorBuilds = new Map>(); +const inFlightExtractorRepairs = new Map>(); +const extractorSmokeTests = new Map(); +const EXTRACTOR_RUNNER_SOURCE = ` +import { chromium } from "playwright-core"; +import { Bash, InMemoryFs } from "just-bash"; +import dns from "node:dns/promises"; +import net from "node:net"; + +const chunks = []; +for await (const chunk of process.stdin) chunks.push(chunk); +const payload = JSON.parse(Buffer.concat(chunks).toString("utf8")); + +const browser = await chromium.connectOverCDP(payload.cdpUrl, { timeout: 45000 }); +let timeout; +let abortController; + +const MAX_SELECTOR_LENGTH = 2000; +const MAX_COMMAND_OUTPUT_CHARS = 256000; +const MAX_HTTP_RESPONSE_BYTES = 20 * 1024 * 1024; +const PUBLIC_HTTP_TIMEOUT_MS = 30000; +const PUBLIC_HTTP_MAX_REDIRECTS = 10; +const HTTP_REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]); +const HTTP_METHODS = new Set(["GET", "HEAD"]); +const EXTRACTOR_COMMANDS = [ + "cat", + "printf", + "echo", + "grep", + "fgrep", + "egrep", + "rg", + "sed", + "awk", + "sort", + "uniq", + "comm", + "cut", + "paste", + "tr", + "rev", + "nl", + "fold", + "expand", + "unexpand", + "strings", + "split", + "column", + "join", + "head", + "tail", + "wc", + "xargs", + "jq", + "yq", + "xan", + "sqlite3", + "base64", + "gzip", + "gunzip", + "zcat", + "tar", + "file", + "find", + "ls", + "mkdir", + "rmdir", + "rm", + "cp", + "mv", + "touch", + "pwd", + "dirname", + "basename", + "tee", + "date", + "seq", + "expr", + "md5sum", + "sha1sum", + "sha256sum", + "true", + "false", + "sleep", + "timeout", + "env", + "printenv", + "which", + "html-to-markdown", +]; + +const cleanText = (value) => String(value ?? "").replace(/\\s+/g, " ").trim(); +const parseCompactNumber = (value) => { + const normalized = String(value ?? "").replace(/,/g, ""); + const match = normalized.match(/[$€£]?\\s*([\\d]+(?:\\.\\d+)?)\\s*([kmb])?/i) || normalized.match(/([\\d]+(?:\\.\\d+)?)/); + if (!match) return undefined; + const base = Number(match[1]); + if (!Number.isFinite(base)) return undefined; + const suffix = match[2]?.toLowerCase(); + const multiplier = suffix === "k" ? 1000 : suffix === "m" ? 1000000 : suffix === "b" ? 1000000000 : 1; + return Math.round(base * multiplier * 100) / 100; +}; + +const helpersPython = [ + "import json, re, sys, urllib.parse", + "", + "def clean_text(value):", + " return re.sub(r'\\\\s+', ' ', str(value or '')).strip()", + "", + "def parse_compact_number(value):", + " text = str(value or '').replace(',', '')", + " match = re.search(r'[$€£]?\\\\s*([0-9]+(?:\\\\.[0-9]+)?)\\\\s*([kmb])?', text, re.I) or re.search(r'([0-9]+(?:\\\\.[0-9]+)?)', text)", + " if not match:", + " return None", + " base = float(match.group(1))", + " suffix = (match.group(2) or '').lower() if len(match.groups()) > 1 else ''", + " mult = {'k': 1000, 'm': 1000000, 'b': 1000000000}.get(suffix, 1)", + " value = round(base * mult, 2)", + " return int(value) if value.is_integer() else value", + "", + "parse_number = parse_compact_number", + "parse_price = parse_compact_number", + "parse_rating = parse_compact_number", + "", + "def absolute_url(value, base=None):", + " return urllib.parse.urljoin(base or '', str(value or ''))", + "", +].join("\\n"); + +function ok(stdout = "") { + return { stdout, stderr: "", exitCode: 0 }; +} + +function fail(err) { + const message = err instanceof Error ? err.message : String(err); + return { stdout: "", stderr: message + "\\n", exitCode: 1 }; +} + +function trimOutput(value, limit = MAX_COMMAND_OUTPUT_CHARS) { + const text = String(value ?? ""); + return text.length > limit ? text.slice(0, limit) : text; +} + +function jsonOutput(value) { + return ok(trimOutput(JSON.stringify(value)) + "\\n"); +} + +function textOutput(value) { + return ok(trimOutput(value) + "\\n"); +} + +function arg(args, index, name) { + const value = args[index]; + if (typeof value !== "string" || value.length === 0) { + throw new Error(name + " is required"); + } + return value; +} + +function parseLimit(value, fallback, max) { + const parsed = Number(value); + if (!Number.isFinite(parsed) || parsed <= 0) return fallback; + return Math.min(Math.floor(parsed), max); +} + +function ensureSelector(selector) { + if (selector.length > MAX_SELECTOR_LENGTH) { + throw new Error("selector is too long"); + } + return selector; +} + +function command(name, execute) { + return { + name, + trusted: true, + execute: async (args, ctx) => { + try { + return await execute(args, ctx); + } catch (err) { + return fail(err); + } + }, + }; +} + +async function firstLocator(page, selector) { + return page.locator(ensureSelector(selector)).first(); +} + +function pageCommands(page) { + return [ + command("page-goto", async (args) => { + const target = coerceHttpUrl(args[0] || payload.url); + const response = await page.goto(target, { + waitUntil: "domcontentloaded", + timeout: 45000, + }); + const status = response?.status(); + if (typeof status === "number" && status >= 400) { + throw new Error("page-goto returned HTTP " + status + ": " + page.url()); + } + await page.waitForLoadState("networkidle", { timeout: 10000 }).catch(() => {}); + return textOutput(page.url()); + }), + command("page-url", async () => textOutput(page.url())), + command("page-title", async () => textOutput(cleanText(await page.title()))), + command("page-text", async (args) => { + const locator = await firstLocator(page, arg(args, 0, "selector")); + return textOutput(cleanText(await locator.textContent({ timeout: 10000 }).catch(() => ""))); + }), + command("page-texts", async (args) => { + const selector = ensureSelector(arg(args, 0, "selector")); + const limit = parseLimit(args[1], 50, 200); + const values = await page.locator(selector).evaluateAll( + (elements, max) => elements + .slice(0, max) + .map((element) => String(element.textContent ?? "").replace(/\\s+/g, " ").trim()) + .filter(Boolean), + limit, + ); + return jsonOutput(values); + }), + command("page-attr", async (args) => { + const locator = await firstLocator(page, arg(args, 0, "selector")); + const name = arg(args, 1, "attribute"); + return textOutput(cleanText(await locator.getAttribute(name, { timeout: 10000 }).catch(() => ""))); + }), + command("page-count", async (args) => { + const selector = ensureSelector(arg(args, 0, "selector")); + return textOutput(String(await page.locator(selector).count())); + }), + command("page-exists", async (args) => { + const selector = ensureSelector(arg(args, 0, "selector")); + return textOutput((await page.locator(selector).count()) > 0 ? "true" : "false"); + }), + command("page-wait", async (args) => { + const selector = ensureSelector(arg(args, 0, "selector")); + const timeoutMs = parseLimit(args[1], 10000, 30000); + await page.locator(selector).first().waitFor({ state: "visible", timeout: timeoutMs }); + return textOutput("ok"); + }), + command("page-click", async (args) => { + const locator = await firstLocator(page, arg(args, 0, "selector")); + await locator.click({ timeout: 10000 }); + await page.waitForLoadState("networkidle", { timeout: 10000 }).catch(() => {}); + return textOutput(page.url()); + }), + command("page-fill", async (args) => { + const locator = await firstLocator(page, arg(args, 0, "selector")); + await locator.fill(args[1] ?? "", { timeout: 10000 }); + return textOutput("ok"); + }), + command("page-meta", async (args) => { + const name = arg(args, 0, "name"); + const value = await page.evaluate((metaName) => { + const normalized = String(metaName || "").toLowerCase(); + if (normalized === "canonical") { + return document.querySelector('link[rel="canonical"]')?.getAttribute("href") || ""; + } + const wanted = String(metaName || "").toLowerCase(); + for (const element of Array.from(document.querySelectorAll("meta"))) { + const key = ( + element.getAttribute("name") || + element.getAttribute("property") || + "" + ).toLowerCase(); + if (key === wanted) return element.getAttribute("content") || ""; + } + return ""; + }, name); + return textOutput(cleanText(value)); + }), + command("page-jsonld", async () => { + const values = await page.evaluate(() => { + return Array.from(document.querySelectorAll('script[type="application/ld+json"]')) + .map((script) => { + try { + return JSON.parse(script.textContent || ""); + } catch { + return undefined; + } + }) + .filter((value) => value !== undefined); + }); + return jsonOutput(values); + }), + command("page-links", async (args) => { + const limit = parseLimit(args[0], 100, 500); + const values = await page.evaluate((max) => { + const clean = (value) => String(value || "").replace(/\\s+/g, " ").trim(); + return Array.from(document.querySelectorAll("a[href]")) + .slice(0, max) + .map((element) => ({ + text: clean(element.textContent || element.getAttribute("aria-label") || ""), + href: element.href, + })) + .filter((link) => /^https?:\\/\\//i.test(link.href)); + }, limit); + return jsonOutput(values); + }), + command("page-visible-text", async (args) => { + const limit = parseLimit(args[0], 60000, 200000); + const text = await page.evaluate((max) => { + return String(document.body?.innerText || "") + .replace(/\\s+/g, " ") + .trim() + .slice(0, max); + }, limit); + return textOutput(text); + }), + ]; +} + +function coerceHttpUrl(value) { + const url = new URL(String(value || ""), payload.url); + if (url.protocol !== "http:" && url.protocol !== "https:") { + throw new Error("only http(s) URLs are allowed"); + } + return url.toString(); +} + +async function assertPublicHttpUrl(value) { + const url = new URL(String(value || "")); + if (url.protocol !== "http:" && url.protocol !== "https:") { + throw new Error("curl only supports http(s) URLs"); + } + const host = normalizeHost(url.hostname); + if (!host || host === "localhost" || host.endsWith(".localhost") || host === "host.docker.internal" || host.endsWith(".internal")) { + throw new Error("private/internal host is not allowed: " + host); + } + if (net.isIP(host) === 0 && !host.includes(".")) { + throw new Error("single-label host is not allowed: " + host); + } + if (isPrivateAddress(host)) { + throw new Error("private/internal IP is not allowed: " + host); + } + if (net.isIP(host) === 0) { + const records = await dns.lookup(host, { all: true }); + for (const record of records) { + if (isPrivateAddress(record.address)) { + throw new Error("host resolves to a private/internal IP: " + host); + } + } + } + return url.toString(); +} + +function normalizeHost(hostname) { + return String(hostname || "").toLowerCase().replace(/^\\[(.*)\\]$/, "$1"); +} + +function isPrivateAddress(address) { + const normalized = normalizeHost(address); + const family = net.isIP(normalized); + if (family === 4) return isPrivateIpv4(normalized); + if (family === 6) return isPrivateIpv6(normalized); + return false; +} + +function isPrivateIpv4(address) { + const parts = address.split(".").map((part) => Number(part)); + if (parts.length !== 4 || parts.some((part) => !Number.isInteger(part) || part < 0 || part > 255)) { + return true; + } + const [a, b] = parts; + return ( + a === 0 || + a === 10 || + a === 127 || + (a === 100 && b >= 64 && b <= 127) || + (a === 169 && b === 254) || + (a === 172 && b >= 16 && b <= 31) || + (a === 192 && b === 168) || + (a === 198 && (b === 18 || b === 19)) || + a >= 224 + ); +} + +function isPrivateIpv6(address) { + const value = address.toLowerCase(); + if (value === "::" || value === "::1") return true; + if (value.startsWith("fc") || value.startsWith("fd")) return true; + if (/^fe[89ab]/.test(value)) return true; + if (value.startsWith("::ffff:")) { + const mapped = value.slice("::ffff:".length); + return isPrivateAddress(mapped); + } + return false; +} + +async function publicFetch(rawUrl, options = {}) { + const method = String(options.method ?? "GET").toUpperCase(); + if (!HTTP_METHODS.has(method)) { + throw new Error("HTTP method not allowed in extractor sandbox: " + method); + } + + let current = await assertPublicHttpUrl(rawUrl); + const followRedirects = options.followRedirects ?? true; + const timeoutMs = Math.min(Number(options.timeoutMs) || PUBLIC_HTTP_TIMEOUT_MS, PUBLIC_HTTP_TIMEOUT_MS); + + for (let redirects = 0; ; redirects++) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + try { + const response = await fetch(current, { + method, + headers: options.headers, + redirect: "manual", + signal: controller.signal, + }); + + if (followRedirects && HTTP_REDIRECT_STATUSES.has(response.status)) { + const location = response.headers.get("location"); + if (!location) return await responseToFetchResult(response, current, method); + if (redirects >= PUBLIC_HTTP_MAX_REDIRECTS) { + throw new Error("too many redirects"); + } + current = await assertPublicHttpUrl(new URL(location, current).toString()); + continue; + } + + return await responseToFetchResult(response, current, method); + } finally { + clearTimeout(timer); + } + } +} + +async function responseToFetchResult(response, url, method) { + const headers = Object.create(null); + response.headers.forEach((value, key) => { + headers[key.toLowerCase()] = value; + }); + const length = Number(response.headers.get("content-length") || "0"); + if (length > MAX_HTTP_RESPONSE_BYTES) { + throw new Error("HTTP response too large"); + } + const body = method === "HEAD" + ? new Uint8Array() + : new Uint8Array(await response.arrayBuffer()); + if (body.byteLength > MAX_HTTP_RESPONSE_BYTES) { + throw new Error("HTTP response too large"); + } + return { + status: response.status, + statusText: response.statusText, + headers, + body, + url, + }; +} + +function parseExtractorOutput(stdout) { + const trimmed = String(stdout || "").trim(); + if (!trimmed) throw new Error("extractor script produced no JSON output"); + try { + return JSON.parse(trimmed); + } catch {} + const lines = trimmed.split(/\\r?\\n/).map((line) => line.trim()).filter(Boolean); + for (let index = lines.length - 1; index >= 0; index--) { + const line = lines[index]; + if (!line.startsWith("{")) continue; + try { + return JSON.parse(line); + } catch {} + } + throw new Error("extractor script did not print a JSON object on stdout"); +} + +try { + const context = browser.contexts()[0] ?? await browser.newContext(); + const page = context.pages()[0] ?? await context.newPage(); + const response = await page.goto(payload.url, { waitUntil: "domcontentloaded", timeout: 45000 }); + const status = response?.status(); + if (typeof status === "number" && status >= 400) { + throw new Error("start URL returned HTTP " + status + ": " + page.url()); + } + await page.waitForLoadState("networkidle", { timeout: 10000 }).catch(() => {}); + + const sandbox = new Bash({ + fs: new InMemoryFs({ + "/input.json": JSON.stringify(payload.input, null, 2), + "/schema.json": JSON.stringify(payload.input.columns ?? [], null, 2), + "/helpers.py": helpersPython, + }), + cwd: "/home/user", + env: { + START_URL: payload.url, + INPUT_JSON: "/input.json", + SCHEMA_JSON: "/schema.json", + PYTHONPATH: "/", + }, + commands: EXTRACTOR_COMMANDS, + customCommands: pageCommands(page), + fetch: publicFetch, + python: true, + javascript: false, + defenseInDepth: true, + executionLimits: { + maxCallDepth: 100, + maxCommandCount: 20000, + maxLoopIterations: 100000, + maxAwkIterations: 100000, + maxSedIterations: 100000, + }, + }); + + abortController = new AbortController(); + const timeoutPromise = new Promise((_, reject) => { + timeout = setTimeout(() => { + abortController.abort(); + reject(new Error("extract timed out")); + }, payload.timeoutMs); + }); + const execResult = await Promise.race([ + sandbox.exec(payload.script, { + rawScript: true, + replaceEnv: true, + env: { + START_URL: payload.url, + INPUT_JSON: "/input.json", + SCHEMA_JSON: "/schema.json", + PYTHONPATH: "/", + }, + signal: abortController.signal, + }), + timeoutPromise, + ]); + if (execResult.exitCode !== 0) { + throw new Error((execResult.stderr || "extractor script failed").trim()); + } + const result = parseExtractorOutput(execResult.stdout); + process.stdout.write(JSON.stringify(result ?? {})); +} finally { + if (timeout) clearTimeout(timeout); + if (abortController) abortController.abort(); + await browser.close().catch(() => {}); +} +`; + +class NonRepairableExtractorFailure extends Error { + constructor(message: string) { + super(message); + this.name = "NonRepairableExtractorFailure"; + } +} + +function abortSignalReason(signal: AbortSignal): unknown { + return signal.reason ?? new DOMException("Run was stopped", "AbortError"); +} + +function runWasAborted(datasetId: string, abortSignal?: AbortSignal): boolean { + return getSignal(datasetId)?.aborted === true || abortSignal?.aborted === true; +} + +function isAbortLikeError(err: unknown): boolean { + if (err instanceof DOMException && err.name === "AbortError") return true; + if (!(err instanceof Error)) return false; + return err.name === "AbortError" || /Run was stopped|AbortError/i.test(err.message); +} + +function throwIfAbortSignalAborted(signal: AbortSignal | undefined): void { + if (!signal?.aborted) return; + const reason = abortSignalReason(signal); + if (reason instanceof Error) throw reason; + throw new DOMException(String(reason), "AbortError"); +} + +function throwIfRunAborted(datasetId: string, abortSignal?: AbortSignal): void { + const runSignal = getSignal(datasetId); + if (runSignal?.aborted) { + const reason = abortSignalReason(runSignal); + if (reason instanceof Error) throw reason; + throw new DOMException(String(reason), "AbortError"); + } + throwIfAbortSignalAborted(abortSignal); +} + +function isTransientBrowserFailure(reason: string): boolean { + return /browserType\.connectOverCDP|WebSocket error|getaddrinfo|EAI_AGAIN|ENOTFOUND|ECONNRESET|ECONNREFUSED|ETIMEDOUT|EHOSTUNREACH|net::ERR_TUNNEL_CONNECTION_FAILED|Target page, context or browser has been closed/i.test( + reason, + ); +} + +function isHttpSourceFailure(reason: string): boolean { + return /start URL returned HTTP \d{3}:|net::ERR_HTTP_RESPONSE_CODE_FAILURE|TinyFish Browser returned HTTP/i.test( + reason, + ); +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +export async function tryRowExtractor( + input: TryRowExtractorInput, +): Promise { + const draft = await tryRowExtractorDraft(input); + if (draft.status !== "extracted") { + return { + status: draft.status, + reason: draft.reason, + rowSummary: draft.rowSummary, + sources: draft.sources, + }; + } + + if ((draft.missingColumns ?? []).length > 0) { + return { + status: "miss", + reason: `browser extractor left unresolved columns for fallback: ${(draft.missingColumns ?? []).join(", ")}`, + rowSummary: draft.rowSummary, + sources: draft.sources, + }; + } + + try { + await convex.mutation(internal.datasetRows.insert, { + datasetId: input.datasetId, + data: draft.data ?? {}, + sources: draft.sources, + cellSources: draft.cellSources, + rowSummary: draft.rowSummary, + howFound: GENERIC_EXTRACTOR_HOW_FOUND, + }); + + return { + status: "inserted", + reason: `Inserted by generic browser extractor (${(draft.extractedColumns ?? []).join(", ")})`, + rowSummary: draft.rowSummary, + sources: draft.sources, + }; + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + if (err instanceof NonRepairableExtractorFailure) { + return { + status: "miss", + reason: `browser extractor skipped repair for row-level/source issue: ${msg}`, + }; + } + if (/duplicate/i.test(msg)) { + return { + status: "miss", + reason: `${msg} Move on to the next entity.`, + }; + } + return { status: "failed", reason: msg }; + } +} + +export async function tryRowExtractorDraft( + input: TryRowExtractorInput, +): Promise { + throwIfRunAborted(input.datasetId, input.abortSignal); + const codificationProfile = normalizeCodificationProfile(input.codificationProfile, input); + if (!shouldAttemptCodification(codificationProfile, input)) { + return { + status: "miss", + reason: `codification profile is ${codificationProfile.mode}: ${codificationProfile.reason}`, + }; + } + + const url = initialBrowserUrl(input); + if (!url) return { status: "miss", reason: "no browser start URL or row context" }; + + try { + const extraction = await extractGenericRow(input, url); + if (!hasExtractedNonPrimaryValue(extraction, input.columns)) { + return { + status: "miss", + reason: `TinyFish Browser did not extract non-primary fields from ${safeHost(url)}`, + }; + } + + return { + status: "extracted", + reason: + extraction.missingColumns.length > 0 + ? `Browser extractor filled ${extraction.extractedColumns.length} columns and left ${extraction.missingColumns.length} for fallback` + : `Browser extractor filled all ${extraction.extractedColumns.length} columns`, + data: extraction.data, + rowSummary: extraction.rowSummary, + sources: extraction.sources, + cellSources: extraction.cellSources, + extractedColumns: extraction.extractedColumns, + missingColumns: extraction.missingColumns, + requiredMissingColumns: extraction.requiredMissingColumns, + optionalMissingColumns: extraction.optionalMissingColumns, + columnStatuses: extraction.columnStatuses, + }; + } catch (err) { + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + const msg = err instanceof Error ? err.message : String(err); + if (err instanceof NonRepairableExtractorFailure) { + return { + status: "miss", + reason: `browser extractor skipped repair for row-level/source issue: ${msg}`, + }; + } + if (/duplicate/i.test(msg)) { + return { + status: "miss", + reason: `${msg} Move on to the next entity.`, + }; + } + return { status: "failed", reason: msg }; + } +} + +export async function tryRefreshRowExtractor( + input: TryRefreshRowExtractorInput, +): Promise { + throwIfRunAborted(input.datasetId, input.abortSignal); + const codificationProfile = normalizeCodificationProfile(input.codificationProfile, input); + if (!shouldAttemptCodification(codificationProfile, input)) { + return { + status: "miss", + reason: `codification profile is ${codificationProfile.mode}: ${codificationProfile.reason}`, + }; + } + + const url = initialBrowserUrl(input); + if (!url) return { status: "miss", reason: "no browser start URL or row context" }; + + try { + const extraction = await extractGenericRow(input, url, input.existingData); + if (!hasExtractedNonPrimaryValue(extraction, input.columns)) { + return { + status: "miss", + reason: `TinyFish Browser did not extract non-primary fields from ${safeHost(url)}`, + }; + } + + const changedColumns = changedColumnNames( + extraction.data, + input.existingData, + input.columns, + ); + if (extraction.missingColumns.length > 0) { + return { + status: "miss", + reason: `browser extractor left unresolved columns during refresh: ${extraction.missingColumns.join(", ")}`, + rowSummary: extraction.rowSummary, + sources: extraction.sources, + }; + } + if (changedColumns.length === 0) { + return { + status: "unchanged", + reason: "Verified unchanged by generic browser extractor", + rowSummary: extraction.rowSummary, + sources: extraction.sources, + }; + } + + await convex.mutation(internal.datasetRows.update, { + id: input.rowId, + expectedDatasetId: input.datasetId, + data: extraction.data, + sources: extraction.sources, + cellSources: extraction.cellSources, + rowSummary: extraction.rowSummary, + howFound: GENERIC_REFRESH_HOW_FOUND, + }); + + return { + status: "updated", + reason: `Updated by generic browser extractor (${changedColumns.join(", ")})`, + rowSummary: extraction.rowSummary, + sources: extraction.sources, + }; + } catch (err) { + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + const msg = err instanceof Error ? err.message : String(err); + if (err instanceof NonRepairableExtractorFailure) { + return { + status: "miss", + reason: `browser extractor skipped repair for row-level/source issue: ${msg}`, + }; + } + return { status: "failed", reason: msg }; + } +} + +function initialBrowserUrl(input: TryRowExtractorInput): string | undefined { + const explicitUrl = browserStartUrlCandidates(input) + .map(coerceHttpUrl) + .find((value): value is string => Boolean(value)); + if (explicitUrl) return explicitUrl; + + const searchQuery = [ + input.sourceHint, + input.datasetName, + input.description, + ...Object.values(input.primaryKeys), + ] + .map((value) => value?.trim()) + .filter((value): value is string => Boolean(value)) + .join(" ") + .slice(0, 500); + + if (!searchQuery) return undefined; + return `https://www.google.com/search?q=${encodeURIComponent(searchQuery)}`; +} + +function browserStartUrlCandidates(input: TryRowExtractorInput): Array { + return uniqueCandidateValues([ + ...Object.values(input.primaryKeys), + ...(input.urls ?? []), + ...renderCodificationTemplateUrls(input), + ...extractHttpUrls(input.sourceHint), + ...extractHttpUrls(input.context), + input.sourceHint, + ]); +} + +function renderCodificationTemplateUrls(input: TryRowExtractorInput): string[] { + const urls: string[] = []; + for (const family of input.codificationProfile?.families ?? []) { + if (!family.urlTemplate) continue; + const rendered = renderUrlTemplate(family.urlTemplate, input.primaryKeys); + if (rendered) urls.push(rendered); + } + return urls; +} + +function renderUrlTemplate( + template: string, + primaryKeys: Record, +): string | undefined { + let missing = false; + const rendered = template.replace( + /\{([a-zA-Z0-9_]+)\}/g, + (match: string, key: string, offset: number) => { + const value = findPrimaryKeyValue(key, primaryKeys)?.trim(); + if (!value) { + missing = true; + return ""; + } + return encodeTemplateValue(template, offset, value); + }, + ); + return missing ? undefined : rendered; +} + +function encodeTemplateValue(template: string, placeholderOffset: number, value: string): string { + if (placeholderIsInQueryOrHash(template, placeholderOffset)) { + return encodeURIComponent(value); + } + return value.split("/").map(encodeURIComponent).join("/"); +} + +function placeholderIsInQueryOrHash(template: string, placeholderOffset: number): boolean { + const queryIndex = template.indexOf("?"); + const hashIndex = template.indexOf("#"); + const boundaryIndexes = [queryIndex, hashIndex].filter((index) => index >= 0); + return boundaryIndexes.length > 0 && Math.min(...boundaryIndexes) < placeholderOffset; +} + +function extractHttpUrls(value: string | undefined): string[] { + if (!value) return []; + return [...value.matchAll(/https?:\/\/[^\s)>"']+/gi)].map((match) => normalizeUrl(match[0])); +} + +function uniqueCandidateValues(values: Array): Array { + const seen = new Set(); + const output: Array = []; + for (const value of values) { + const key = value?.trim(); + if (!key) continue; + if (seen.has(key)) continue; + seen.add(key); + output.push(value); + } + return output; +} + +function normalizeUrl(value: string): string { + return value.trim().replace(/[.,;:]+$/, ""); +} + +function coerceHttpUrl(value: string | undefined): string | undefined { + if (!value) return undefined; + const normalized = normalizeUrl(value); + if (isHttpUrl(normalized)) return normalized; + if (/^[a-z0-9.-]+\.[a-z]{2,}(?:\/[^\s]*)?$/i.test(normalized)) { + const withScheme = `https://${normalized}`; + return isHttpUrl(withScheme) ? withScheme : undefined; + } + return undefined; +} + +function isHttpUrl(value: string | undefined): value is string { + if (!value) return false; + try { + const parsed = new URL(normalizeUrl(value)); + return parsed.protocol === "http:" || parsed.protocol === "https:"; + } catch { + return false; + } +} + +function safeHost(value: string): string { + try { + return new URL(value).host; + } catch { + return "invalid-url"; + } +} + +async function extractGenericRow( + input: TryRowExtractorInput, + url: string, + existingData?: Record, +): Promise { + const apiKey = await getTinyFishApiKey(); + if (!apiKey) throw new Error("TINYFISH_API_KEY is not configured"); + + const siteKey = siteKeyForInput(input, url); + const cachedExtractor = await getOrBuildExtractorScript(apiKey, input, url); + const script = cachedExtractor.script; + throwIfRunAborted(input.datasetId, input.abortSignal); + const attempts = normalizedBrowserAttempts(input.browserAttempts); + let lastError: unknown; + for (let attempt = 1; attempt <= attempts; attempt++) { + try { + console.log( + `[row_extractor] running generated sandboxed browser extractor for ${siteKey} attempt=${attempt}/${attempts}`, + ); + const raw = await runGeneratedExtractorScript(apiKey, input, url, script); + const extraction = buildRowFromExtractorResult(input, url, raw, existingData); + const qualityIssue = extractorQualityIssue(extraction, input.columns); + if (qualityIssue && isRowLevelQualityIssue(qualityIssue)) { + console.warn( + `[row_extractor] generated extractor produced row-level partial result for ${siteKey}; skipping repair: ${qualityIssue}`, + ); + return extraction; + } + if (qualityIssue) throw new Error(qualityIssue); + rememberExtractorSmokeTest(input, url); + return extraction; + } catch (err) { + lastError = err; + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + if (runWasAborted(input.datasetId, input.abortSignal) || attempt === attempts) break; + const msg = err instanceof Error ? err.message : String(err); + console.warn( + `[row_extractor] Browser attempt ${attempt}/${attempts} failed; retrying: ${msg}`, + ); + } + } + + throwIfRunAborted(input.datasetId, input.abortSignal); + const failure = lastError instanceof Error ? lastError.message : String(lastError); + if (isNonRepairableRuntimeFailure(failure)) { + console.warn( + `[row_extractor] generated extractor failed with non-repairable browser/source issue for ${siteKey}; skipping repair: ${failure}`, + ); + throw new NonRepairableExtractorFailure(failure); + } + + let extractorToMarkFailed = cachedExtractor; + try { + console.warn( + `[row_extractor] generated extractor failed; requesting repair: ${failure}`, + ); + const repaired = await repairExtractorAfterRuntimeFailure( + apiKey, + input, + url, + script, + failure, + ); + if (repaired.cachedExtractor) { + extractorToMarkFailed = repaired.cachedExtractor; + } + console.log(`[row_extractor] running repaired sandboxed browser extractor for ${siteKey}`); + const raw = await runGeneratedExtractorScript(apiKey, input, url, repaired.script); + const extraction = buildRowFromExtractorResult(input, url, raw, existingData); + const qualityIssue = extractorQualityIssue(extraction, input.columns); + if (qualityIssue) throw new Error(qualityIssue); + rememberExtractorSmokeTest(input, url); + return extraction; + } catch (repairErr) { + if (isAbortLikeError(repairErr) && runWasAborted(input.datasetId, input.abortSignal)) { + throw repairErr; + } + const repairMsg = repairErr instanceof Error ? repairErr.message : String(repairErr); + if (!isTransientBrowserFailure(repairMsg) && !isHttpSourceFailure(repairMsg)) { + const markedFailed = await markExtractorFailed( + input, + url, + extractorToMarkFailed, + repairMsg, + ); + console.warn( + markedFailed + ? `[row_extractor] repair failed; marked cached extractor failed for ${siteKey}: ${repairMsg}` + : `[row_extractor] repair failed; skipped stale cached extractor failure for ${siteKey}: ${repairMsg}`, + ); + } else { + console.warn( + `[row_extractor] repair failed with transient/source issue; keeping cache state unchanged for ${siteKey}: ${repairMsg}`, + ); + } + throw repairErr instanceof Error ? repairErr : new Error(String(repairErr)); + } +} + +function normalizedBrowserAttempts(value: number | undefined): number { + if (typeof value !== "number" || !Number.isFinite(value)) { + return DEFAULT_BROWSER_ATTEMPTS; + } + return Math.min(10, Math.max(1, Math.trunc(value))); +} + +async function getOrBuildExtractorScript( + apiKey: string, + input: TryRowExtractorInput, + url: string, +): Promise { + const siteKey = siteKeyForInput(input, url); + const columnsHash = hashColumns(input.columns); + const buildKey = extractorBuildKey(input.datasetId, siteKey, columnsHash); + const existing = (await convex.query(internal.datasetExtractors.getActive, { + datasetId: input.datasetId, + siteKey, + columnsHash, + })) as { script?: string; updatedAt?: number } | null; + if (existing?.script && typeof existing.updatedAt === "number") { + console.log(`[row_extractor] using cached generated sandboxed browser extractor for ${siteKey}`); + return { script: existing.script, updatedAt: existing.updatedAt }; + } + + const inFlightBuild = inFlightExtractorBuilds.get(buildKey); + if (inFlightBuild) { + console.log(`[row_extractor] waiting for in-flight extractor build for ${siteKey}`); + return await inFlightBuild; + } + + const buildPromise = buildAndPersistExtractorScript(apiKey, input, url, siteKey, columnsHash) + .finally(() => { + inFlightExtractorBuilds.delete(buildKey); + }); + inFlightExtractorBuilds.set(buildKey, buildPromise); + return await buildPromise; +} + +async function markExtractorFailed( + input: TryRowExtractorInput, + url: string, + extractor: CachedExtractorScript, + error: string, +): Promise { + const siteKey = siteKeyForInput(input, url); + const columnsHash = hashColumns(input.columns); + try { + const markedId = await convex.mutation(internal.datasetExtractors.markFailed, { + datasetId: input.datasetId, + siteKey, + columnsHash, + script: extractor.script, + updatedAt: extractor.updatedAt, + error, + }); + return Boolean(markedId); + } catch (err) { + console.warn( + `[row_extractor] failed to mark cached extractor failed for ${siteKey}: ${ + err instanceof Error ? err.message : String(err) + }`, + ); + return false; + } +} + +async function buildAndPersistExtractorScript( + apiKey: string, + input: TryRowExtractorInput, + url: string, + siteKey: string, + columnsHash: string, +): Promise { + const modelSlug = input.extractorBuilderModel ?? DEFAULT_MODEL_IDS.EXTRACTOR_BUILDER; + const built = await buildExtractorScriptWithAgent(apiKey, input, url, modelSlug); + const script = built.script; + const validation = await validateExtractorScript(apiKey, input, url, script); + if (validation.ok) { + const stored = (await convex.mutation(internal.datasetExtractors.upsert, { + datasetId: input.datasetId, + siteKey, + columnsHash, + script, + model: modelSlug, + probeSummary: built.probeSummary, + })) as { updatedAt: number }; + return { script, updatedAt: stored.updatedAt }; + } + + if (isNonRepairableValidationFailure(validation.reason)) { + throw new NonRepairableExtractorFailure(validation.reason); + } + + const repaired = await buildExtractorScriptWithAgent( + apiKey, + input, + url, + modelSlug, + { + previousScript: script, + failure: validation.reason, + }, + ); + const repairedValidation = await validateExtractorScript(apiKey, input, url, repaired.script); + if (!repairedValidation.ok) { + if (isNonRepairableValidationFailure(repairedValidation.reason)) { + throw new NonRepairableExtractorFailure(repairedValidation.reason); + } + throw new Error(`generated extractor failed validation: ${repairedValidation.reason}`); + } + + const stored = (await convex.mutation(internal.datasetExtractors.upsert, { + datasetId: input.datasetId, + siteKey, + columnsHash, + script: repaired.script, + model: modelSlug, + probeSummary: repaired.probeSummary, + })) as { updatedAt: number }; + return { script: repaired.script, updatedAt: stored.updatedAt }; +} + +function isNonRepairableValidationFailure(reason: string | undefined): boolean { + if (!reason) return false; + return isTransientBrowserFailure(reason) || isHttpSourceFailure(reason); +} + +function extractorBuildKey( + datasetId: string, + siteKey: string, + columnsHash: string, +): string { + return `${datasetId}:${siteKey}:${columnsHash}`; +} + +function extractorBuildKeyForInput(input: TryRowExtractorInput, url: string): string { + return extractorBuildKey( + input.datasetId, + siteKeyForInput(input, url), + hashColumns(input.columns), + ); +} + +function rememberExtractorSmokeTest(input: TryRowExtractorInput, url: string): void { + extractorSmokeTests.set(extractorBuildKeyForInput(input, url), { + input: cloneSmokeTestInput(input), + url, + source: "memory", + }); +} + +async function getExtractorSmokeTest( + input: TryRowExtractorInput, + url: string, +): Promise { + const key = extractorBuildKeyForInput(input, url); + const memoryCase = extractorSmokeTests.get(key); + if (memoryCase && !samePrimaryKeys(memoryCase.input.primaryKeys, input.primaryKeys)) { + return memoryCase; + } + + const rows = (await convex.query(internal.datasetRows.listInternal, { + datasetId: input.datasetId, + })) as Array<{ + _id?: string; + data?: Record; + sources?: string[]; + rowSummary?: string; + howFound?: string; + }>; + + const currentRowId = "rowId" in input ? input.rowId : undefined; + for (const row of rows) { + if (currentRowId && row._id === currentRowId) continue; + if (!isBrowserExtractedRow(row.howFound)) continue; + if (!row.data || !storedRowHasCompleteValues(row.data, input.columns)) continue; + + const primaryKeys = primaryKeysFromStoredRow(row.data, input.columns); + if (Object.keys(primaryKeys).length === 0) continue; + if (samePrimaryKeys(primaryKeys, input.primaryKeys)) continue; + + const smokeInput = cloneSmokeTestInput({ + ...input, + primaryKeys, + urls: row.sources, + context: [row.rowSummary, row.howFound].filter(Boolean).join("\n"), + }); + const smokeUrl = initialBrowserUrl(smokeInput) ?? url; + return { input: smokeInput, url: smokeUrl, source: "persisted" }; + } + + return undefined; +} + +function cloneSmokeTestInput(input: TryRowExtractorInput): TryRowExtractorInput { + return { + datasetId: input.datasetId, + datasetName: input.datasetName, + description: input.description, + columns: input.columns, + primaryKeys: { ...input.primaryKeys }, + urls: input.urls ? [...input.urls] : undefined, + context: input.context, + retrievalStrategy: input.retrievalStrategy, + sourceHint: input.sourceHint, + codificationProfile: input.codificationProfile, + browserAttempts: input.browserAttempts, + extractorBuilderModel: input.extractorBuilderModel, + }; +} + +function isBrowserExtractedRow(howFound: string | undefined): boolean { + return Boolean(howFound && /generated (?:Playwright|sandboxed browser) extractor/i.test(howFound)); +} + +function storedRowHasCompleteValues( + data: Record, + columns: PopulateColumn[], +): boolean { + return columns.every((column) => { + const value = normalizeStoredValue(data[column.name]); + return hasCompleteColumnValue(value) && !validateColumnValue(value, column); + }); +} + +function primaryKeysFromStoredRow( + data: Record, + columns: PopulateColumn[], +): Record { + return Object.fromEntries( + columns + .filter((column) => column.isPrimaryKey) + .map((column) => [column.name, String(data[column.name] ?? "").trim()]) + .filter(([, value]) => value.length > 0), + ); +} + +function samePrimaryKeys( + left: Record, + right: Record, +): boolean { + const leftEntries = Object.entries(left).sort(([a], [b]) => a.localeCompare(b)); + const rightEntries = Object.entries(right).sort(([a], [b]) => a.localeCompare(b)); + if (leftEntries.length !== rightEntries.length) return false; + return leftEntries.every(([key, value], index) => { + const [rightKey, rightValue] = rightEntries[index]!; + return key === rightKey && value === rightValue; + }); +} + +async function probePage( + apiKey: string, + input: TryRowExtractorInput, + url: string, +): Promise { + let lastError: unknown; + for (let attempt = 1; attempt <= TRANSIENT_BROWSER_ATTEMPTS; attempt++) { + let browser: Browser | undefined; + try { + throwIfRunAborted(input.datasetId, input.abortSignal); + const session = await createTinyFishBrowserSession( + apiKey, + url, + input.datasetId, + input.abortSignal, + ); + browser = await chromium.connectOverCDP(session.cdp_url, { + timeout: CDP_CONNECT_TIMEOUT_MS, + }); + const context = browser.contexts()[0] ?? (await browser.newContext()); + const page = context.pages()[0] ?? (await context.newPage()); + await page.goto(url, { + waitUntil: "domcontentloaded", + timeout: BROWSER_TIMEOUT_MS, + }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => { + // Many modern sites keep long-lived requests open. DOMContentLoaded is enough. + }); + + return await readPageEvidence(page); + } catch (err) { + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + lastError = err; + const msg = err instanceof Error ? err.message : String(err); + if (!isTransientBrowserFailure(msg) || attempt === TRANSIENT_BROWSER_ATTEMPTS) { + throw err; + } + console.warn( + `[extractor_builder] probe browser attempt ${attempt}/${TRANSIENT_BROWSER_ATTEMPTS} failed; retrying: ${reasonForLog(msg)}`, + ); + await sleep(500 * attempt); + } finally { + await browser?.close().catch(() => undefined); + } + } + throw lastError instanceof Error ? lastError : new Error(String(lastError)); +} + +async function createTinyFishBrowserSession( + apiKey: string, + url: string, + datasetId: string, + abortSignal?: AbortSignal, +): Promise { + const response = await withRunTimeoutSignal( + datasetId, + FETCH_TIMEOUT_MS, + (signal) => + fetch("https://agent.tinyfish.ai/v1/browser", { + method: "POST", + headers: { + ...tinyFishHeaders(apiKey), + "Content-Type": "application/json", + Accept: "application/json", + }, + body: JSON.stringify({ url }), + signal, + }), + abortSignal, + ); + + if (!response.ok) { + const body = await response.text().catch(() => ""); + throw new Error( + `TinyFish Browser returned HTTP ${response.status}: ${body.slice(0, 200)}`, + ); + } + + const data = (await response.json()) as Partial; + if (!data.session_id || !data.cdp_url || !data.base_url) { + throw new Error("TinyFish Browser response did not include CDP connection details"); + } + + return { + session_id: data.session_id, + cdp_url: data.cdp_url, + base_url: data.base_url, + }; +} + +async function withRunTimeoutSignal( + datasetId: string, + timeoutMs: number, + operation: (signal: AbortSignal) => Promise, + abortSignal?: AbortSignal, +): Promise { + throwIfRunAborted(datasetId, abortSignal); + const runSignal = getSignal(datasetId); + + const controller = new AbortController(); + let timeout: ReturnType; + const timeoutPromise = new Promise((_resolve, reject) => { + timeout = setTimeout(() => { + const reason = new DOMException( + `Timed out after ${Math.round(timeoutMs / 1000)}s`, + "TimeoutError", + ); + controller.abort(reason); + reject(reason); + }, timeoutMs); + }); + const abortListeners: Array<{ signal: AbortSignal; listener: () => void }> = []; + const abortPromise = new Promise((_resolve, reject) => { + const addAbortListener = (signal: AbortSignal | undefined) => { + if (!signal) return; + const listener = () => { + const reason = abortSignalReason(signal); + controller.abort(reason); + reject(reason); + }; + signal.addEventListener("abort", listener, { once: true }); + abortListeners.push({ signal, listener }); + if (signal.aborted) listener(); + }; + addAbortListener(runSignal); + addAbortListener(abortSignal); + }); + try { + return await Promise.race([operation(controller.signal), timeoutPromise, abortPromise]); + } finally { + clearTimeout(timeout!); + for (const { signal, listener } of abortListeners) { + signal.removeEventListener("abort", listener); + } + } +} + +async function withRunAbortSignal( + datasetId: string, + operation: (signal: AbortSignal) => Promise, + abortSignal?: AbortSignal, +): Promise { + throwIfRunAborted(datasetId, abortSignal); + const runSignal = getSignal(datasetId); + + const controller = new AbortController(); + const abortListeners: Array<{ signal: AbortSignal; listener: () => void }> = []; + const abortPromise = new Promise((_resolve, reject) => { + const addAbortListener = (signal: AbortSignal | undefined) => { + if (!signal) return; + const listener = () => { + const reason = abortSignalReason(signal); + controller.abort(reason); + reject(reason); + }; + signal.addEventListener("abort", listener, { once: true }); + abortListeners.push({ signal, listener }); + if (signal.aborted) listener(); + }; + addAbortListener(runSignal); + addAbortListener(abortSignal); + }); + + try { + return await Promise.race([operation(controller.signal), abortPromise]); + } finally { + for (const { signal, listener } of abortListeners) { + signal.removeEventListener("abort", listener); + } + } +} + +async function readPageEvidence(page: Page): Promise { + const finalUrl = page.url(); + const evidence = await page.evaluate(() => { + const normalize = (value: string) => + value + .trim() + .toLowerCase() + .replace(/[^a-z0-9]+/g, "_") + .replace(/^_+|_+$/g, ""); + const clean = (value: unknown) => + String(value ?? "") + .replace(/\s+/g, " ") + .trim(); + const candidates: Record = {}; + const push = (key: unknown, value: unknown) => { + const normalizedKey = normalize(clean(key)); + const cleanedValue = clean(value); + if (!normalizedKey || !cleanedValue || cleanedValue.length > 2_000) return; + const list = candidates[normalizedKey] ?? []; + if (list.length >= 8 || list.includes(cleanedValue)) return; + candidates[normalizedKey] = [...list, cleanedValue]; + }; + const text = (selector: string) => + clean(document.querySelector(selector)?.textContent ?? ""); + const attr = (selector: string, name: string) => + clean(document.querySelector(selector)?.getAttribute(name) ?? ""); + + const title = + text("h1") || + attr("meta[property='og:title']", "content") || + attr("meta[name='twitter:title']", "content") || + clean(document.title); + const description = + attr("meta[name='description']", "content") || + attr("meta[property='og:description']", "content") || + attr("meta[name='twitter:description']", "content"); + + push("title", title); + push("name", title); + push("description", description); + push("summary", description); + push("url", location.href); + push("canonical", attr("link[rel='canonical']", "href")); + + document.querySelectorAll("meta[name][content], meta[property][content]").forEach((el) => { + const key = el.getAttribute("name") || el.getAttribute("property"); + push(key, el.getAttribute("content")); + }); + + const walkJson = (value: unknown, path: string[] = []) => { + if (value == null) return; + if (Array.isArray(value)) { + value.slice(0, 40).forEach((item, index) => walkJson(item, [...path, String(index)])); + return; + } + if (typeof value === "object") { + for (const [key, nested] of Object.entries(value as Record)) { + walkJson(nested, [...path, key]); + } + return; + } + const leaf = clean(value); + if (!leaf) return; + const key = path[path.length - 1] ?? ""; + push(key, leaf); + push(path.filter((part) => !/^\d+$/.test(part)).join("_"), leaf); + }; + + document.querySelectorAll("script[type*='ld+json']").forEach((script) => { + try { + walkJson(JSON.parse(script.textContent || "")); + } catch { + // Ignore malformed structured data. + } + }); + + document.querySelectorAll("table tr").forEach((row) => { + const cells = Array.from(row.querySelectorAll("th,td")) + .map((cell) => clean(cell.textContent)) + .filter(Boolean); + if (cells.length >= 2) push(cells[0], cells.slice(1).join(" ")); + }); + + document.querySelectorAll("dt").forEach((dt) => { + let sibling = dt.nextElementSibling; + while (sibling && sibling.tagName.toLowerCase() !== "dd") { + sibling = sibling.nextElementSibling; + } + if (sibling) push(dt.textContent, sibling.textContent); + }); + + document.querySelectorAll("li,p,div,span").forEach((el) => { + const value = clean(el.textContent); + if (value.length < 4 || value.length > 500) return; + const match = value.match(/^([^:|\n]{2,80})\s*[:|]\s*(.{1,350})$/); + if (match) push(match[1], match[2]); + }); + + document.querySelectorAll("a[href]").forEach((el) => { + const label = clean(el.textContent) || clean(el.getAttribute("aria-label")); + const href = clean((el as HTMLAnchorElement).href); + if (!href || !/^https?:\/\//i.test(href)) return; + push(label || "link", href); + if (/website|homepage|site/i.test(label)) push("website", href); + }); + + return { + title, + description, + candidates, + bodyText: clean(document.body?.innerText ?? "").slice(0, 60_000), + }; + }); + + return { ...evidence, finalUrl }; +} + +async function buildExtractorScriptWithAgent( + apiKey: string, + input: TryRowExtractorInput, + url: string, + modelSlug: string, + repairContext?: { + previousScript: string; + failure: string; + }, + options: { + maxSteps?: number; + phase?: string; + } = {}, +): Promise { + const llmConfig = await requireLlmProviderConfig(); + let testedScript: string | undefined; + let finishedScript: string | undefined; + let declinedReason: string | undefined; + let lastEvidence: PageEvidence | undefined; + let lastProbeSummary = `url=${url}`; + const siteKey = siteKeyForInput(input, url); + const phase = options.phase ?? (repairContext ? "repair" : "build"); + const maxSteps = options.maxSteps ?? EXTRACTOR_BUILDER_MAX_STEPS; + + console.log( + `[extractor_builder] ${phase} started for ${siteKey} maxSteps=${maxSteps}`, + ); + + const inspectBrowserPageTool = createTool({ + id: "inspect_browser_page", + description: + "Open a URL with TinyFish Browser and return page evidence: final URL, title, meta description, structured candidate fields, and visible text excerpt. Use this whenever page structure is unclear.", + inputSchema: z.object({ + url: z + .string() + .optional() + .describe("Optional URL to inspect. If omitted, inspects the browser start URL."), + }), + outputSchema: z.object({ + finalUrl: z.string().optional(), + title: z.string().optional(), + description: z.string().optional(), + candidateFields: z.array( + z.object({ + key: z.string(), + values: z.array(z.string()), + }), + ), + visibleTextExcerpt: z.string(), + error: z.string().optional(), + }), + execute: async ({ url: requestedUrl }) => { + try { + const targetUrl = coerceHttpUrl(requestedUrl) ?? url; + console.log(`[extractor_builder] ${phase} inspecting ${targetUrl}`); + const evidence = await probePage(apiKey, input, targetUrl); + lastEvidence = evidence; + lastProbeSummary = summarizeEvidenceForStorage(evidence); + return evidenceForTool(evidence); + } catch (err) { + return { + candidateFields: [], + visibleTextExcerpt: "", + error: err instanceof Error ? err.message : String(err), + }; + } + }, + }); + + const testExtractorTool = createTool({ + id: "test_extractor", + description: + "Run a candidate sandboxed Bash/Python extractor against the representative row and return validation feedback. Call this repeatedly while improving the script.", + inputSchema: z.object({ + script: z.string().describe("The full Bash extractor script. It may call BigSet page-* commands, curl, jq, and python3, and must print one JSON object."), + }), + outputSchema: z.object({ + ok: z.boolean(), + reason: z.string(), + extractedColumns: z.array(z.string()).optional(), + missingColumns: z.array(z.string()).optional(), + requiredMissingColumns: z.array(z.string()).optional(), + optionalMissingColumns: z.array(z.string()).optional(), + rowSummary: z.string().optional(), + sources: z.array(z.string()).optional(), + dataPreview: z.record(z.string(), z.union([z.string(), z.number(), z.boolean()])).optional(), + }), + execute: async ({ script }) => { + console.log(`[extractor_builder] ${phase} testing candidate for ${siteKey}`); + const result = await testExtractorScriptDetailed(apiKey, input, url, script); + if (result.ok) { + testedScript = sanitizeGeneratedScript(script); + } + console.log( + `[extractor_builder] ${phase} test ${result.ok ? "passed" : "failed"} for ${siteKey}: ${reasonForLog(result.reason)}`, + ); + return detailedTestResultForTool(result); + }, + }); + + const finishExtractorTool = createTool({ + id: "finish_extractor", + description: + "Finalize a candidate extractor. This runs the same sandbox validation as test_extractor and only accepts the script if it passes.", + inputSchema: z.object({ + script: z.string().describe("The full Bash extractor script to persist."), + notes: z.string().optional().describe("Brief notes about the page pattern this script supports."), + }), + outputSchema: z.object({ + ok: z.boolean(), + reason: z.string(), + extractedColumns: z.array(z.string()).optional(), + missingColumns: z.array(z.string()).optional(), + requiredMissingColumns: z.array(z.string()).optional(), + optionalMissingColumns: z.array(z.string()).optional(), + }), + execute: async ({ script }) => { + console.log(`[extractor_builder] ${phase} finishing candidate for ${siteKey}`); + const result = await testExtractorScriptDetailed(apiKey, input, url, script); + if (result.ok) { + finishedScript = sanitizeGeneratedScript(script); + } + console.log( + `[extractor_builder] ${phase} finish ${result.ok ? "accepted" : "rejected"} for ${siteKey}: ${reasonForLog(result.reason)}`, + ); + return { + ok: result.ok, + reason: result.reason, + extractedColumns: result.extraction?.extractedColumns, + missingColumns: result.extraction?.missingColumns, + requiredMissingColumns: result.extraction?.requiredMissingColumns, + optionalMissingColumns: result.extraction?.optionalMissingColumns, + }; + }, +}); + + const declineExtractorTool = createTool({ + id: "decline_extractor", + description: + "Use this when the representative row is not a good fit for a reusable sandboxed browser extractor, for example arbitrary unrelated URLs, CAPTCHA/blocking, or no stable page family.", + inputSchema: z.object({ + reason: z.string().describe("Concrete reason this row/site pattern should fall back to the normal investigate agent."), + category: z + .enum([ + "mixed_unrelated_urls", + "blocked_or_captcha", + "no_stable_page_pattern", + "insufficient_row_context", + "other", + ]) + .describe("Why codification is not appropriate."), + }), + outputSchema: z.object({ + ok: z.boolean(), + reason: z.string(), + category: z.string(), + }), + execute: async ({ reason, category }) => { + declinedReason = `${category}: ${reason}`; + return { ok: true, reason, category }; + }, + }); + + const agent = new Agent({ + id: "extractor-builder-agent", + name: "Extractor Builder Agent", + instructions: extractorBuilderAgentInstructions(), + model: createLanguageModel(llmConfig, modelSlug), + tools: { + inspect_browser_page: inspectBrowserPageTool, + test_extractor: testExtractorTool, + finish_extractor: finishExtractorTool, + decline_extractor: declineExtractorTool, + }, + }); + + const prompt = extractorBuilderAgentPrompt(input, url, repairContext); + let result: Awaited>; + try { + result = await withRunAbortSignal(input.datasetId, (abortSignal) => + agent.generate(prompt, { + abortSignal, + maxSteps, + modelSettings: { + maxOutputTokens: AGENT_MAX_OUTPUT_TOKENS.EXTRACTOR_BUILDER, + }, + }), + input.abortSignal, + ); + } catch (err) { + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + if (finishedScript) { + console.warn( + `[extractor_builder] ${phase} returning finished script for ${siteKey} after agent error: ${reasonForLog(err instanceof Error ? err.message : String(err))}`, + ); + return { + script: finishedScript, + probeSummary: lastEvidence ? summarizeEvidenceForStorage(lastEvidence) : lastProbeSummary, + }; + } + if (testedScript) { + console.warn( + `[extractor_builder] ${phase} had a tested script for ${siteKey}, but finish_extractor did not accept it before agent error: ${reasonForLog(err instanceof Error ? err.message : String(err))}`, + ); + } + throw err; + } + + throwIfRunAborted(input.datasetId, input.abortSignal); + console.log( + `[extractor_builder] ${phase} ended for ${siteKey} steps=${result.steps?.length ?? "?"}`, + ); + + if (finishedScript) { + console.log( + `[extractor_builder] accepted finished script after ${result.steps?.length ?? "?"} steps for ${siteKey}`, + ); + return { + script: finishedScript, + probeSummary: lastEvidence ? summarizeEvidenceForStorage(lastEvidence) : lastProbeSummary, + }; + } + + if (testedScript) { + console.warn( + `[extractor_builder] ${phase} produced a tested script for ${siteKey}, but did not call finish_extractor`, + ); + } + + if (declinedReason) { + throw new Error(`extractor builder declined codification: ${declinedReason}`); + } + + throw new Error( + `extractor builder did not finish a validated script within ${maxSteps} steps`, + ); +} + +async function repairExtractorAfterRuntimeFailure( + apiKey: string, + input: TryRowExtractorInput, + url: string, + script: string, + failure: string, +): Promise { + const repairKey = extractorBuildKeyForInput(input, url); + const siteKey = siteKeyForInput(input, url); + const inFlightRepair = inFlightExtractorRepairs.get(repairKey); + if (inFlightRepair) { + console.log(`[row_extractor] waiting for in-flight extractor repair for ${siteKey}`); + return await inFlightRepair; + } + + const repairPromise = repairExtractorAfterRuntimeFailureImpl( + apiKey, + input, + url, + script, + failure, + ).finally(() => { + inFlightExtractorRepairs.delete(repairKey); + }); + inFlightExtractorRepairs.set(repairKey, repairPromise); + return await repairPromise; +} + +async function repairExtractorAfterRuntimeFailureImpl( + apiKey: string, + input: TryRowExtractorInput, + url: string, + script: string, + failure: string, +): Promise { + const modelSlug = input.extractorBuilderModel ?? DEFAULT_MODEL_IDS.EXTRACTOR_BUILDER; + const repaired = await buildExtractorScriptWithAgent( + apiKey, + input, + url, + modelSlug, + { + previousScript: script, + failure, + }, + { + maxSteps: EXTRACTOR_REPAIR_MAX_STEPS, + phase: "repair", + }, + ); + const validation = await validateExtractorScript(apiKey, input, url, repaired.script); + if (!validation.ok) { + if (isNonRepairableValidationFailure(validation.reason)) { + throw new NonRepairableExtractorFailure(validation.reason); + } + throw new Error(`repaired extractor failed validation: ${validation.reason}`); + } + + const siteKey = siteKeyForInput(input, url); + const columnsHash = hashColumns(input.columns); + const smokeTest = await getExtractorSmokeTest(input, url); + if (smokeTest) { + const regression = await testExtractorScriptDetailed( + apiKey, + smokeTest.input, + smokeTest.url, + repaired.script, + ); + if (!regression.ok) { + if (isNonRepairableValidationFailure(regression.reason)) { + throw new NonRepairableExtractorFailure(regression.reason); + } + throw new Error( + `repaired extractor regressed ${smokeTest.source} known-good row: ${regression.reason}`, + ); + } + console.log( + `[row_extractor] repaired extractor passed ${smokeTest.source} regression test for ${siteKey}`, + ); + } else { + console.warn( + `[row_extractor] repaired extractor has no known-good regression row for ${siteKey}; using for this row without updating cache`, + ); + return { script: repaired.script }; + } + + const stored = (await convex.mutation(internal.datasetExtractors.upsert, { + datasetId: input.datasetId, + siteKey, + columnsHash, + script: repaired.script, + model: modelSlug, + probeSummary: repaired.probeSummary, + })) as { updatedAt: number }; + return { + script: repaired.script, + cachedExtractor: { script: repaired.script, updatedAt: stored.updatedAt }, + }; +} + +function isRowLevelQualityIssue(reason: string): boolean { + return ( + reason.startsWith("generated extractor missed required columns:") || + reason.startsWith("generated extractor coverage too low:") + ); +} + +function isNonRepairableRuntimeFailure(reason: string): boolean { + return ( + isRowLevelQualityIssue(reason) || + /^primary key ".+" failed validation:/.test(reason) || + isHttpSourceFailure(reason) || + isTransientBrowserFailure(reason) || + /TinyFish Browser returned HTTP|Run was stopped|AbortError|TimeoutError/i.test(reason) + ); +} + +function evidenceForTool(evidence: PageEvidence): { + finalUrl: string; + title?: string; + description?: string; + candidateFields: Array<{ key: string; values: string[] }>; + visibleTextExcerpt: string; +} { + return { + finalUrl: evidence.finalUrl, + title: evidence.title, + description: evidence.description, + candidateFields: Object.entries(evidence.candidates) + .slice(0, 100) + .map(([key, values]) => ({ + key, + values: values.slice(0, 6), + })), + visibleTextExcerpt: evidence.bodyText.slice(0, 16_000), + }; +} + +async function testExtractorScriptDetailed( + apiKey: string, + input: TryRowExtractorInput, + url: string, + script: string, +): Promise { + try { + const sanitized = sanitizeGeneratedScript(script); + const raw = await runGeneratedExtractorScriptForValidation(apiKey, input, url, sanitized); + const extraction = buildRowFromExtractorResult(input, url, raw); + if (!hasExtractedNonPrimaryValue(extraction, input.columns)) { + return { + ok: false, + reason: "extractor returned no non-primary values", + extraction, + }; + } + const qualityIssue = extractorQualityIssue(extraction, input.columns); + if (qualityIssue) { + return { + ok: false, + reason: qualityIssue, + extraction, + }; + } + const hardcodedValues = hardcodedExtractorValues(sanitized, extraction, input, url); + if (hardcodedValues.length > 0) { + return { + ok: false, + reason: `extractor appears to hardcode representative-row values: ${hardcodedValues.join(", ")}`, + extraction, + }; + } + return { ok: true, reason: "passed", extraction }; + } catch (err) { + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + return { + ok: false, + reason: err instanceof Error ? err.message : String(err), + }; + } +} + +function detailedTestResultForTool(result: DetailedExtractorTestResult): { + ok: boolean; + reason: string; + extractedColumns?: string[]; + missingColumns?: string[]; + requiredMissingColumns?: string[]; + optionalMissingColumns?: string[]; + rowSummary?: string; + sources?: string[]; + dataPreview?: Record; +} { + return { + ok: result.ok, + reason: result.reason, + extractedColumns: result.extraction?.extractedColumns, + missingColumns: result.extraction?.missingColumns, + requiredMissingColumns: result.extraction?.requiredMissingColumns, + optionalMissingColumns: result.extraction?.optionalMissingColumns, + rowSummary: result.extraction?.rowSummary, + sources: result.extraction?.sources, + dataPreview: result.extraction?.data, + }; +} + +function reasonForLog(reason: string): string { + const normalized = reason.replace(/\s+/g, " ").trim(); + return normalized.length > 500 ? `${normalized.slice(0, 500)}...` : normalized; +} + +function extractorBuilderAgentInstructions(): string { + return `You are BigSet's autonomous extractor builder agent. + +Your job is to build a reusable sandboxed extractor script for one dataset/page family. You have browser inspection and sandbox testing tools. Use whichever tools fit the situation; do not follow a rigid checklist. + +Critical behavior: +- Inspect pages when structure, source URLs, or primary key semantics are unclear. +- Do not decline because a source has an anti-bot reputation. BigSet uses TinyFish Browser for interactive browser access; inspect the representative page and judge the actual session result. +- Write and test candidate scripts with test_extractor. Iterate on validation errors. +- Finish only by calling finish_extractor with the final script. A final text response without finish_extractor is failure. +- If the dataset is not a good fit for reusable browser codification, call decline_extractor with a concrete reason. +- Treat this as a general page-family compiler. Do not optimize for any named site. The page family may be a marketplace listing, social group, product page, blog post archive, repository page, directory profile, forum thread, app-store listing, or any other accessible browser surface. + +When to decline: +- The representative primary keys/URLs point at arbitrary unrelated domains with no shared page pattern. +- The representative page remains blocked by login, paywall, CAPTCHA, bot detection, or inaccessible content after a TinyFish Browser inspection attempt. +- The row context is too ambiguous to construct or find a stable browser target. +- TinyFish Browser can access the page but the requested values require actions/data outside the accessible page family and cannot be reliably derived. + +Important modeling rules: +- Primary keys are arbitrary row identifiers. They do not have to be URLs. +- You may construct navigation URLs from input.primaryKeys, top-level primary key fields, input.urls, input.sourceHint, dataset name, description, retrieval strategy, and context. +- If the source/schema imply a URL template, build the row URL from the primary key or identifier. This applies to any source family: handles, slugs, product IDs, post paths, item IDs, listing IDs, package names, tickers, and similar identifiers. +- Search is a last resort when a deterministic source URL can be built. +- Multiple known page families are allowed. If rows consistently contain URLs from a small set of stable domains or path patterns, write one reusable script that branches based on the available URL/domain. If rows are arbitrary unrelated URLs, decline. +- Respect validation_regex exactly. Normalize values before returning them, using normalization_hint and column descriptions. +- Attempt every column, including nullable/optional columns. Nullable means the final row may survive if all methods fail, not that you should skip the field. +- You may derive values from page structure instead of finding literal labels. For example: count repeated cards in a relevant section, infer booleans from the presence of controls/badges, parse counts from aria labels, or normalize visible variants into the schema's final format. + +Script contract: +- Return only a full Bash script in tool arguments, never Markdown fences. +- Do not write JavaScript. Do not define async function extract. Do not use Playwright APIs directly. +- The script runs in a just-bash sandbox with a virtual filesystem. It can read /input.json, /schema.json, and /helpers.py. +- The script must print exactly one JSON object on stdout as its final non-empty line. +- The script may use Bash, jq, sed, awk, grep, rg, yq, xan, sqlite3, tar/gzip, curl, and python3. +- Python may import /helpers.py for clean_text, parse_compact_number, parse_number, parse_price, parse_rating, and absolute_url. +- JavaScript execution is unavailable. Network access is public HTTP(S) GET/HEAD only via curl; private/internal hosts are blocked. +- Browser access is only through these BigSet commands: + - page-goto [url]: navigate the TinyFish Browser page and print the final URL. + - page-url: print the current page URL. + - page-title: print the page title. + - page-text CSS_SELECTOR: print the first matching element's cleaned text. + - page-texts CSS_SELECTOR [limit]: print a JSON array of cleaned matching texts. + - page-attr CSS_SELECTOR ATTRIBUTE: print the first matching element attribute. + - page-count CSS_SELECTOR: print the match count. + - page-exists CSS_SELECTOR: print true or false. + - page-wait CSS_SELECTOR [timeout_ms]: wait for a visible element. + - page-click CSS_SELECTOR: click an element and print the resulting URL. + - page-fill CSS_SELECTOR VALUE: fill an input. + - page-meta NAME_OR_PROPERTY: print a meta tag content value, or canonical for canonical link. + - page-jsonld: print a JSON array of parsed application/ld+json objects. + - page-links [limit]: print a JSON array of {text, href} links. + - page-visible-text [limit]: print cleaned visible page text. +- Prefer stable DOM sources: JSON-LD, meta tags, tables, definition lists, visible labels, aria labels, canonical links, and semantically named selectors. +- Return { data, column_status, cell_sources, sources, row_summary, how_found }. +- data should include every dataset column that the sandboxed browser commands can extract or derive. Preserve primary key values from input.primaryKeys. +- column_status should include every dataset column. Use status "extracted" or "derived" for values in data. Use "not_present_on_page", "blocked", "ambiguous", "validation_failed", or "fallback_needed" for unresolved values, with a short reason. +- cell_sources should map each extracted/derived column to the exact HTTP URL(s) that justify that specific value. Do not put row-level sources here. +- Returned sources must be HTTP URLs you actually used. +- If any column cannot be extracted or normalized to satisfy validation_regex, keep investigating with the browser tools. If it still cannot be solved from the browser page family, mark it in column_status for fallback instead of fabricating a value. + +Starter-code reference only. This shape will not work on the target site until you inspect the actual page and replace the URL construction/selectors: +#!/usr/bin/env bash +set -e + +page-goto "$START_URL" >/tmp/page_url.txt +title="$(page-text 'h1')" +current_url="$(page-url)" + +TITLE="$title" CURRENT_URL="$current_url" python3 - <<'PY' +import json, os +from helpers import clean_text + +input_data = json.load(open("/input.json")) +title = clean_text(os.environ.get("TITLE", "")) +url = os.environ.get("CURRENT_URL", "") +item_id = input_data.get("primaryKeys", {}).get("item_id") or input_data.get("item_id", "") +result = { + "data": {"item_id": item_id, "title": title}, + "column_status": { + "item_id": {"status": "extracted"}, + "title": {"status": "extracted"} if title else {"status": "fallback_needed", "reason": "No h1/title equivalent found after inspection"}, + }, + "cell_sources": {"title": [url]} if title else {}, + "sources": [url], + "row_summary": title or item_id or url, + "how_found": "Opened the row target with TinyFish Browser and extracted values using sandboxed BigSet page commands.", +} +print(json.dumps(result, separators=(",", ":"))) +PY + +Never hardcode representative-row values in the script. Literal values from the first tested row, such as a specific product name, profile URL, rating, count, or homepage, make the extractor invalid unless they come from input primary keys or are source-family constants like a host prefix.`; +} + +function extractorBuilderAgentPrompt( + input: TryRowExtractorInput, + url: string, + repairContext?: { + previousScript: string; + failure: string; + }, +): string { + const columns = input.columns + .map((column) => columnPromptLine(column)) + .join("\n"); + const primaryKeys = Object.entries(input.primaryKeys) + .map(([key, value]) => `- ${key}: ${JSON.stringify(value)}`) + .join("\n") || "(none)"; + const urls = (input.urls ?? []).map((value) => `- ${value}`).join("\n") || "(none)"; + const repair = repairContext + ? ` +You are repairing a previous extractor. The previous script failed with: +${repairContext.failure} + +Previous script: +${repairContext.previousScript.slice(0, 20_000)} +` + : ""; + + return `Build a reusable sandboxed Bash/Python extractor for this BigSet dataset. + +Dataset: +- Name: ${input.datasetName ?? ""} +- Description: ${input.description ?? ""} +- Retrieval strategy: ${input.retrievalStrategy ?? ""} +- Source hint: ${input.sourceHint ?? ""} +- Context: ${input.context ?? ""} + +Dataset columns: +${columns} + +Primary key values for the representative row: +${primaryKeys} + +Candidate URLs from row context: +${urls} + +Representative browser start URL: +${url} + +Use the tools to inspect, test, revise, finish, or decline. The goal is a validated script, not a prose answer.${repair}`; +} + +function columnPromptLine(column: PopulateColumn): string { + const details = [ + column.isPrimaryKey ? "PRIMARY KEY" : undefined, + column.nullable === undefined ? undefined : `nullable=${column.nullable}`, + column.validationRegex + ? `validation_regex=${JSON.stringify(column.validationRegex)}` + : undefined, + column.normalizationHint + ? `normalization_hint=${JSON.stringify(column.normalizationHint)}` + : undefined, + column.description ? `retrieval_hint=${JSON.stringify(column.description)}` : undefined, + ].filter(Boolean); + return `- ${JSON.stringify(column.name)} (${column.type})${ + details.length > 0 ? ` [${details.join("; ")}]` : "" + }`; +} + +function summarizeEvidenceForStorage(evidence: PageEvidence): string { + return [ + `url=${evidence.finalUrl}`, + evidence.title ? `title=${evidence.title}` : undefined, + evidence.description ? `description=${evidence.description}` : undefined, + ] + .filter(Boolean) + .join("\n") + .slice(0, 2_000); +} + +function sanitizeGeneratedScript(text: string): string { + const fenced = text.match(/```(?:bash|sh|shell)?\s*([\s\S]*?)```/i); + const raw = (fenced?.[1] ?? text) + .trim(); + rejectUnsupportedGeneratedScript(raw); + if (!raw) { + throw new Error("generated extractor script was empty"); + } + return raw; +} + +function rejectUnsupportedGeneratedScript(script: string): void { + const legacyJavaScriptPatterns = [ + /^\s*(?:export\s+)?async\s+function\s+extract\s*\(/m, + /^\s*(?:export\s+)?function\s+extract\s*\(/m, + /\bpage\.locator\s*\(/, + /\bnew\s+vm\.Script\b/, + ]; + const match = legacyJavaScriptPatterns.find((pattern) => pattern.test(script)); + if (match) { + throw new Error( + "legacy JavaScript extractor scripts are disabled; generate a Bash/Python extractor instead", + ); + } +} + +async function validateExtractorScript( + apiKey: string, + input: TryRowExtractorInput, + url: string, + script: string, +): Promise<{ ok: true } | { ok: false; reason: string }> { + const result = await testExtractorScriptDetailed(apiKey, input, url, script); + return result.ok ? { ok: true } : { ok: false, reason: result.reason }; +} + +async function runGeneratedExtractorScriptForValidation( + apiKey: string, + input: TryRowExtractorInput, + url: string, + script: string, +): Promise { + let lastError: unknown; + for (let attempt = 1; attempt <= TRANSIENT_BROWSER_ATTEMPTS; attempt++) { + try { + throwIfRunAborted(input.datasetId, input.abortSignal); + return await runGeneratedExtractorScript(apiKey, input, url, script); + } catch (err) { + if (isAbortLikeError(err) && runWasAborted(input.datasetId, input.abortSignal)) { + throw err; + } + lastError = err; + const msg = err instanceof Error ? err.message : String(err); + if (!isTransientBrowserFailure(msg) || attempt === TRANSIENT_BROWSER_ATTEMPTS) { + throw err; + } + console.warn( + `[extractor_builder] validation browser attempt ${attempt}/${TRANSIENT_BROWSER_ATTEMPTS} failed; retrying: ${reasonForLog(msg)}`, + ); + await sleep(500 * attempt); + } + } + throw lastError instanceof Error ? lastError : new Error(String(lastError)); +} + +async function runGeneratedExtractorScript( + apiKey: string, + input: TryRowExtractorInput, + url: string, + script: string, +): Promise { + throwIfRunAborted(input.datasetId, input.abortSignal); + const sanitized = sanitizeGeneratedScript(script); + const session = await createTinyFishBrowserSession( + apiKey, + url, + input.datasetId, + input.abortSignal, + ); + const payload = JSON.stringify({ + cdpUrl: session.cdp_url, + url, + script: sanitized, + input: { + ...input.primaryKeys, + url, + startUrl: url, + primaryKeys: input.primaryKeys, + urls: input.urls ?? [], + columns: input.columns, + datasetName: input.datasetName ?? "", + description: input.description ?? "", + retrievalStrategy: input.retrievalStrategy ?? "", + sourceHint: input.sourceHint ?? "", + context: input.context ?? "", + }, + timeoutMs: EXTRACTOR_RUNNER_TIMEOUT_MS, + }); + + return await new Promise((resolve, reject) => { + const runSignal = getSignal(input.datasetId); + try { + throwIfRunAborted(input.datasetId, input.abortSignal); + } catch (err) { + reject(err); + return; + } + + const child = spawn(process.execPath, ["--input-type=module", "-e", EXTRACTOR_RUNNER_SOURCE], { + cwd: BACKEND_ROOT, + env: {}, + stdio: ["pipe", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + const abortListeners: Array<{ signal: AbortSignal; listener: () => void }> = []; + let timeout: ReturnType | undefined; + const cleanup = () => { + if (timeout) clearTimeout(timeout); + for (const { signal, listener } of abortListeners) { + signal.removeEventListener("abort", listener); + } + }; + const addAbortListener = (signal: AbortSignal | undefined) => { + if (!signal) return; + const listener = () => { + child.kill("SIGKILL"); + cleanup(); + reject(abortSignalReason(signal)); + }; + signal.addEventListener("abort", listener, { once: true }); + abortListeners.push({ signal, listener }); + if (signal.aborted) listener(); + }; + timeout = setTimeout(() => { + child.kill("SIGKILL"); + cleanup(); + reject(new Error("generated extractor timed out")); + }, EXTRACTOR_RUNNER_TIMEOUT_MS + 5_000); + addAbortListener(runSignal); + addAbortListener(input.abortSignal); + + child.stdout.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + if (stdout.length > EXTRACTOR_SCRIPT_OUTPUT_LIMIT) { + child.kill("SIGKILL"); + cleanup(); + reject(new Error("generated extractor output limit exceeded")); + } + }); + child.stderr.setEncoding("utf8"); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (err) => { + cleanup(); + reject(err); + }); + child.on("close", (code) => { + cleanup(); + if (code !== 0) { + reject(new Error(stderr.trim() || `generated extractor exited ${code}`)); + return; + } + try { + resolve(JSON.parse(stdout) as GeneratedExtractorResult); + } catch (err) { + reject(new Error(`generated extractor returned invalid JSON: ${err}`)); + } + }); + child.stdin.end(payload); + }); +} + +function buildRowFromExtractorResult( + input: TryRowExtractorInput, + url: string, + result: GeneratedExtractorResult, + existingData?: Record, +): GenericExtraction { + if (!result || typeof result !== "object" || !result.data || typeof result.data !== "object") { + throw new Error("generated extractor did not return a data object"); + } + + const data: Record = {}; + const extractedColumns: string[] = []; + const missingColumns: string[] = []; + const requiredMissingColumns: string[] = []; + const optionalMissingColumns: string[] = []; + const rawColumnStatuses = normalizeColumnStatuses( + result.column_status ?? result.columnStatus, + ); + const columnStatuses: Record = {}; + + for (const column of input.columns) { + const pkValue = findPrimaryKeyValue(column.name, input.primaryKeys); + if (pkValue !== undefined) { + const value = coerceColumnValue(pkValue, column, url) ?? pkValue; + const validationError = validateColumnValue(value, column); + if (validationError) { + throw new Error(`primary key "${column.name}" failed validation: ${validationError}`); + } + data[column.name] = value; + extractedColumns.push(column.name); + columnStatuses[column.name] = { + status: rawColumnStatuses[column.name]?.status ?? "extracted", + reason: rawColumnStatuses[column.name]?.reason, + }; + continue; + } + + const rawValue = result.data[column.name]; + const value = coerceColumnValue(rawValueToExtractedValue(rawValue), column, url); + if (value !== undefined && hasCompleteColumnValue(value)) { + const validationError = validateColumnValue(value, column); + if (!validationError) { + data[column.name] = value; + extractedColumns.push(column.name); + columnStatuses[column.name] = { + status: rawColumnStatuses[column.name]?.status ?? "extracted", + reason: rawColumnStatuses[column.name]?.reason, + }; + continue; + } + columnStatuses[column.name] = { + status: "validation_failed", + reason: validationError, + }; + } + + if (existingData && existingData[column.name] !== undefined) { + const storedValue = normalizeStoredValue(existingData[column.name]); + if (hasCompleteColumnValue(storedValue)) { + const validationError = validateColumnValue(storedValue, column); + if (!validationError) { + data[column.name] = storedValue; + columnStatuses[column.name] = { + status: rawColumnStatuses[column.name]?.status ?? "extracted", + reason: rawColumnStatuses[column.name]?.reason ?? "Preserved existing validated value.", + }; + continue; + } + } + } + + data[column.name] = ""; + missingColumns.push(column.name); + if (isRequiredColumn(column)) { + requiredMissingColumns.push(column.name); + } else { + optionalMissingColumns.push(column.name); + } + columnStatuses[column.name] = columnStatuses[column.name] ?? rawColumnStatuses[column.name] ?? { + status: "fallback_needed", + reason: "The browser extractor did not return a validated value.", + }; + } + + const sources = Array.isArray(result.sources) + ? result.sources.filter((source): source is string => typeof source === "string" && isHttpUrl(source)) + : []; + const cellSources = normalizeCellSources(result.cell_sources ?? result.cellSources); + const summary = String(result.row_summary ?? result.rowSummary ?? "").trim().slice(0, 500); + + return { + data, + sources: sources.length > 0 ? sources : [url], + rowSummary: summary || url, + cellSources, + extractedColumns, + missingColumns, + requiredMissingColumns, + optionalMissingColumns, + columnStatuses, + }; +} + +function hasExtractedNonPrimaryValue( + extraction: GenericExtraction, + columns: PopulateColumn[], +): boolean { + const pkColumns = new Set( + columns.filter((column) => column.isPrimaryKey).map((column) => column.name), + ); + return extraction.extractedColumns.some((columnName) => !pkColumns.has(columnName)); +} + +function extractorQualityIssue( + extraction: GenericExtraction, + columns: PopulateColumn[], +): string | undefined { + const nonPrimaryColumns = columns.filter((column) => !column.isPrimaryKey); + if (nonPrimaryColumns.length === 0) return undefined; + + const extracted = new Set(extraction.extractedColumns); + const missing = new Set(extraction.missingColumns); + const extractedNonPrimary = nonPrimaryColumns.filter((column) => extracted.has(column.name)); + const missingCellSources = extractedNonPrimary.filter( + (column) => (extraction.cellSources[column.name]?.length ?? 0) === 0, + ); + if (missingCellSources.length > 0) { + return `generated extractor omitted cell_sources for extracted columns: ${missingCellSources.map((column) => column.name).join(", ")}`; + } + + const missingRequired = nonPrimaryColumns.filter( + (column) => isRequiredColumn(column) && missing.has(column.name), + ); + if (missingRequired.length > 0) { + return `generated extractor missed required columns: ${missingRequired.map((column) => column.name).join(", ")}`; + } + + const minimum = minimumBrowserExtractedColumnCount(nonPrimaryColumns.length); + if (extractedNonPrimary.length < minimum) { + const missingColumns = nonPrimaryColumns + .filter((column) => !extracted.has(column.name)) + .map((column) => column.name); + return [ + `generated extractor coverage too low: extracted ${extractedNonPrimary.length}/${nonPrimaryColumns.length} non-primary columns`, + `minimum=${minimum}`, + `missing=${missingColumns.join(", ") || "(none)"}`, + ].join("; "); + } + + return undefined; +} + +function minimumBrowserExtractedColumnCount(nonPrimaryColumnCount: number): number { + if (nonPrimaryColumnCount <= 1) return nonPrimaryColumnCount; + return Math.floor(nonPrimaryColumnCount / 2) + 1; +} + +function hardcodedExtractorValues( + script: string, + extraction: GenericExtraction, + input: TryRowExtractorInput, + url: string, +): string[] { + const scriptText = script.toLowerCase(); + const pkColumns = new Set( + input.columns.filter((column) => column.isPrimaryKey).map((column) => column.name), + ); + const sourceUrls = new Set( + [url, ...(input.urls ?? []), input.sourceHint] + .filter((value): value is string => Boolean(value)) + .map((value) => value.toLowerCase()), + ); + const violations: string[] = []; + + for (const [columnName, value] of Object.entries(extraction.data)) { + if (pkColumns.has(columnName)) continue; + if (typeof value !== "string") continue; + const normalized = value.trim(); + if (normalized.length < 8) continue; + const lowerValue = normalized.toLowerCase(); + if (sourceUrls.has(lowerValue)) continue; + if (!scriptText.includes(lowerValue)) continue; + violations.push(`${columnName}=${JSON.stringify(normalized.slice(0, 120))}`); + } + + return violations.slice(0, 8); +} + +function isRequiredColumn(column: PopulateColumn): boolean { + return column.isPrimaryKey === true || column.nullable === false; +} + +function normalizeColumnStatuses(value: unknown): Record { + if (!value || typeof value !== "object" || Array.isArray(value)) return {}; + + const statuses: Record = {}; + for (const [column, rawStatus] of Object.entries(value as Record)) { + const normalized = + typeof rawStatus === "string" + ? { status: normalizeColumnStatus(rawStatus) } + : rawStatus && typeof rawStatus === "object" + ? { + status: normalizeColumnStatus( + String((rawStatus as { status?: unknown }).status ?? "fallback_needed"), + ), + reason: + typeof (rawStatus as { reason?: unknown }).reason === "string" + ? String((rawStatus as { reason?: unknown }).reason).slice(0, 300) + : undefined, + } + : { status: "fallback_needed" as const }; + statuses[column] = normalized; + } + return statuses; +} + +function normalizeColumnStatus(value: string): ColumnExtractionStatus { + const normalized = value.trim().toLowerCase().replace(/[^a-z0-9]+/g, "_"); + switch (normalized) { + case "extracted": + case "derived": + case "not_present_on_page": + case "blocked": + case "ambiguous": + case "validation_failed": + case "fallback_needed": + case "missing": + return normalized; + case "not_found": + case "unavailable": + case "not_visible": + return "not_present_on_page"; + default: + return "fallback_needed"; + } +} + +function normalizeCellSources(value: unknown): Record { + if (!value) return {}; + + const output: Record = {}; + const add = (column: string, sources: unknown) => { + const normalizedColumn = column.trim().replace(/^["`]+|["`]+$/g, ""); + if (!normalizedColumn) return; + const sourceValues = Array.isArray(sources) ? sources : [sources]; + const cleaned = [ + ...new Set( + sourceValues.filter((source): source is string => { + return typeof source === "string" && isHttpUrl(source); + }), + ), + ]; + if (cleaned.length > 0) output[normalizedColumn] = cleaned; + }; + + if (Array.isArray(value)) { + for (const entry of value) { + if (!entry || typeof entry !== "object") continue; + const column = (entry as { column?: unknown }).column; + if (typeof column !== "string") continue; + add(column, (entry as { sources?: unknown; source?: unknown }).sources ?? (entry as { source?: unknown }).source); + } + return output; + } + + if (typeof value === "object") { + for (const [column, sources] of Object.entries(value as Record)) { + add(column, sources); + } + } + + return output; +} + +function rawValueToExtractedValue(value: unknown): ExtractedValue | undefined { + if (typeof value === "number" && Number.isFinite(value)) return value; + if (typeof value === "boolean") return value; + if (typeof value === "string") return value; + if (value == null) return undefined; + return String(value); +} + +function validateColumnValue( + value: ExtractedValue, + column: PopulateColumn, +): string | undefined { + const pattern = column.validationRegex?.trim(); + if (!pattern) return undefined; + + const text = String(value).trim(); + if (!text) { + return column.nullable === true ? undefined : "empty value"; + } + + let regex: RegExp; + try { + regex = compileValidationRegex(pattern); + } catch (err) { + return `invalid schema validation regex ${JSON.stringify(pattern)}: ${ + err instanceof Error ? err.message : String(err) + }`; + } + + if (!regex.test(text)) { + return `value ${JSON.stringify(text).slice(0, 120)} does not match ${JSON.stringify(pattern)}`; + } + return undefined; +} + +function compileValidationRegex(pattern: string): RegExp { + const literal = pattern.match(/^\/([\s\S]*)\/([a-z]*)$/i); + if (!literal) return new RegExp(pattern); + const flags = Array.from(new Set(literal[2].replace(/[gy]/g, "").split(""))).join(""); + return new RegExp(literal[1], flags); +} + +function siteKeyForUrl( + value: string, + primaryKeys: Record = {}, +): string { + try { + const url = new URL(value); + const primaryKeySegments = siteKeyPrimaryValueSegments(primaryKeys); + const pathShape = url.pathname + .split("/") + .filter(Boolean) + .slice(0, 3) + .map((segment, index) => + normalizeSiteKeyPathSegment(segment, index, primaryKeySegments), + ) + .filter((segment): segment is string => Boolean(segment)); + return [url.hostname.toLowerCase(), ...pathShape].filter(Boolean).join("/"); + } catch { + return "invalid-url"; + } +} + +function siteKeyForInput(input: TryRowExtractorInput, browserStartUrl: string): string { + const sourceUrl = browserStartUrlCandidates(input) + .map(coerceHttpUrl) + .find((value): value is string => Boolean(value)); + return siteKeyForUrl(sourceUrl ?? browserStartUrl, input.primaryKeys); +} + +function siteKeyPrimaryValueSegments(primaryKeys: Record): Set { + const values = new Set(); + for (const value of Object.values(primaryKeys)) { + const normalized = normalizeSiteKeyLiteral(value); + const slug = slugifySiteKeyLiteral(value); + if (normalized) values.add(normalized); + if (slug) values.add(slug); + for (const segment of siteKeyUrlPathSegments(value)) { + values.add(segment); + } + } + return values; +} + +function siteKeyUrlPathSegments(value: string): string[] { + try { + const url = new URL(value); + return url.pathname + .split("/") + .filter(Boolean) + .map(normalizeSiteKeyLiteral) + .filter(Boolean); + } catch { + return []; + } +} + +function normalizeSiteKeyPathSegment( + value: string, + index: number, + primaryKeySegments: Set, +): string | undefined { + const normalized = normalizeSiteKeyLiteral(value); + if (!normalized) return undefined; + if ( + index > 0 && + (primaryKeySegments.has(normalized) || looksDynamicSiteKeySegment(normalized)) + ) { + return ":value"; + } + return normalized; +} + +function looksDynamicSiteKeySegment(value: string): boolean { + return ( + /^\d+$/.test(value) || + /^[0-9a-f]{12,}$/i.test(value) || + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value) || + (value.length >= 6 && /\d/.test(value)) + ); +} + +function normalizeSiteKeyLiteral(value: string): string { + try { + return decodeURIComponent(value).toLowerCase().replace(/[^a-z0-9._~-]+/g, "-").replace(/^-+|-+$/g, ""); + } catch { + return value.toLowerCase().replace(/[^a-z0-9._~-]+/g, "-").replace(/^-+|-+$/g, ""); + } +} + +function slugifySiteKeyLiteral(value: string): string { + return normalizeSiteKeyLiteral(value.replace(/&/g, " and ")); +} + +function hashColumns(columns: PopulateColumn[]): string { + const payload = columns.map((column) => ({ + name: column.name, + type: column.type, + description: column.description ?? "", + isPrimaryKey: Boolean(column.isPrimaryKey), + nullable: column.nullable, + validationRegex: column.validationRegex ?? "", + normalizationHint: column.normalizationHint ?? "", + })); + return createHash("sha256").update(JSON.stringify(payload)).digest("hex"); +} + +function changedColumnNames( + nextRow: Record, + existingData: Record, + columns: PopulateColumn[], +): string[] { + return columns + .filter((column) => !valuesEqualForColumn(nextRow[column.name], existingData[column.name], column)) + .map((column) => column.name); +} + +function valuesEqualForColumn( + nextValue: ExtractedValue | undefined, + existingValue: unknown, + column: PopulateColumn, +): boolean { + if (nextValue === undefined) return existingValue === undefined || existingValue === ""; + + switch (column.type) { + case "number": { + const nextNumber = + typeof nextValue === "number" + ? nextValue + : Number(String(nextValue ?? "").replace(/,/g, "")); + const existingNumber = + typeof existingValue === "number" + ? existingValue + : Number(String(existingValue ?? "").replace(/,/g, "")); + return ( + Number.isFinite(nextNumber) && + Number.isFinite(existingNumber) && + existingNumber === nextNumber + ); + } + case "boolean": + if (typeof existingValue === "boolean") return existingValue === nextValue; + if (/^(true|yes)$/i.test(String(existingValue))) return nextValue === true; + if (/^(false|no)$/i.test(String(existingValue))) return nextValue === false; + return String(existingValue ?? "").trim() === String(nextValue).trim(); + case "date": { + const nextDate = new Date(String(nextValue)); + const existingDate = new Date(String(existingValue ?? "")); + return ( + Number.isFinite(nextDate.getTime()) && + Number.isFinite(existingDate.getTime()) && + nextDate.getTime() === existingDate.getTime() + ); + } + case "url": + case "text": + return String(existingValue ?? "").trim() === String(nextValue).trim(); + } +} + +function findPrimaryKeyValue( + columnName: string, + primaryKeys: Record, +): string | undefined { + if (primaryKeys[columnName]) return primaryKeys[columnName]; + const normalizedColumn = normalizeFieldName(columnName); + const entry = Object.entries(primaryKeys).find( + ([key]) => normalizeFieldName(key) === normalizedColumn, + ); + return entry?.[1]; +} + +function coerceColumnValue( + value: string | number | boolean | undefined, + column: PopulateColumn, + baseUrl: string, +): ExtractedValue | undefined { + if (value === undefined || value === "") return undefined; + switch (column.type) { + case "number": { + if (typeof value === "number") return Number.isFinite(value) ? value : undefined; + return parseCompactNumber(String(value)); + } + case "boolean": + if (typeof value === "boolean") return value; + if (/^(true|yes|available|in stock|active)$/i.test(String(value).trim())) return true; + if (/^(false|no|unavailable|out of stock|inactive)$/i.test(String(value).trim())) { + return false; + } + return undefined; + case "url": + return normalizeMaybeRelativeUrl(String(value), baseUrl); + case "date": { + const date = new Date(String(value)); + return Number.isNaN(date.getTime()) ? undefined : date.toISOString(); + } + case "text": + return String(value).trim() || undefined; + } +} + +function hasCompleteColumnValue(value: ExtractedValue): boolean { + if (typeof value === "string") return value.trim().length > 0; + return true; +} + +function normalizeMaybeRelativeUrl( + value: string, + baseUrl: string, +): string | undefined { + try { + const url = new URL(value, baseUrl); + return url.protocol === "http:" || url.protocol === "https:" + ? url.toString() + : undefined; + } catch { + return undefined; + } +} + +function normalizeStoredValue(value: unknown): ExtractedValue { + if (typeof value === "number" && Number.isFinite(value)) return value; + if (typeof value === "boolean") return value; + return String(value ?? ""); +} + +function parseCompactNumber(value: string | undefined): number | undefined { + if (!value) return undefined; + const normalized = value.replace(/,/g, ""); + const match = + normalized.match(/[$€£]?\s*([\d]+(?:\.\d+)?)\s*([kmb])?/i) ?? + normalized.match(/([\d]+(?:\.\d+)?)/); + if (!match) return undefined; + const base = Number(match[1]); + if (!Number.isFinite(base)) return undefined; + const suffix = match[2]?.toLowerCase(); + const multiplier = suffix === "k" ? 1_000 : suffix === "m" ? 1_000_000 : suffix === "b" ? 1_000_000_000 : 1; + return Math.round(base * multiplier * 100) / 100; +} + +function normalizeFieldName(value: string): string { + return value + .trim() + .toLowerCase() + .replace(/[^a-z0-9]+/g, "_") + .replace(/^_+|_+$/g, ""); +} diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 3506ef9..ecd496d 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -48,6 +48,7 @@ services: # When unset, backend analytics module no-ops. POSTHOG_KEY: ${NEXT_PUBLIC_POSTHOG_KEY:-} POSTHOG_HOST: ${NEXT_PUBLIC_POSTHOG_HOST:-https://us.i.posthog.com} + POPULATE_ORCHESTRATOR_MAX_STEPS: ${POPULATE_ORCHESTRATOR_MAX_STEPS:-80} REFRESH_SCHEDULER_ENABLED: ${REFRESH_SCHEDULER_ENABLED:-true} REFRESH_SCHEDULER_POLL_MS: ${REFRESH_SCHEDULER_POLL_MS:-60000} REFRESH_SCHEDULER_BATCH_SIZE: ${REFRESH_SCHEDULER_BATCH_SIZE:-5} diff --git a/frontend/Dockerfile.dev b/frontend/Dockerfile.dev index daa1db8..3a07d32 100644 --- a/frontend/Dockerfile.dev +++ b/frontend/Dockerfile.dev @@ -10,4 +10,7 @@ RUN bun install COPY --chown=node:node . . -CMD ["bun", "dev", "--hostname", "0.0.0.0", "--port", "3500"] +# Run Next directly instead of through Bun's script runner, and force webpack +# for local Docker dev. Next 16 defaults to Turbopack, which can get OOM-killed +# in constrained Docker Desktop setups during the first page compile. +CMD ["node", "../scripts/with-root-env.mjs", "./node_modules/.bin/next", "dev", "--webpack", "--hostname", "0.0.0.0", "--port", "3500"] diff --git a/frontend/app/dashboard/settings/models/page.tsx b/frontend/app/dashboard/settings/models/page.tsx index 6a3acb4..81aac6f 100644 --- a/frontend/app/dashboard/settings/models/page.tsx +++ b/frontend/app/dashboard/settings/models/page.tsx @@ -1,9 +1,9 @@ "use client"; -import { useState, useEffect } from "react"; +import { useState, useEffect, useCallback, useRef } from "react"; import { useQuery } from "convex/react"; import { api } from "@/convex/_generated/api"; -import { getModelConfig, saveModelConfig, getOpenRouterModels, refreshOpenRouterModels, type EffectiveModelConfig, type OpenRouterModel } from "@/lib/backend"; +import { getLocalSetupStatus, getLlmProviderModels, getModelConfig, saveModelConfig, refreshOpenRouterModels, type EffectiveModelConfig, type LlmProviderType, type LocalSetupStatus, type OpenRouterModel } from "@/lib/backend"; import { SettingsPageLayout } from "@/components/settings/SettingsPageLayout"; import { SettingsHeader } from "@/components/settings/SettingsHeader"; import { SettingsTile } from "@/components/settings/SettingsTile"; @@ -12,6 +12,17 @@ import { ModelSideSheet } from "@/components/settings/ModelSideSheet"; import { MODEL_ROLES, type ModelRole } from "@/components/settings/types"; import { SkeletonList } from "@/components/settings/Skeleton"; import { useAppAuth } from "@/lib/app-auth"; +import { isLocalMode } from "@/lib/app-mode"; + +function modelListCacheKey(status: LocalSetupStatus): string { + const llm = status.services.llm; + return [ + llm.provider ?? "openrouter", + llm.baseUrl ?? "", + llm.defaultModel ?? "", + llm.verifiedAt ?? "", + ].join("|"); +} export default function ModelSettingsPage() { const { getToken } = useAppAuth(); @@ -21,21 +32,85 @@ export default function ModelSettingsPage() { const [isLoadingConfig, setIsLoadingConfig] = useState(true); const [refreshing, setRefreshing] = useState(false); const [sheetModels, setSheetModels] = useState([]); + const [sheetModelsCacheKey, setSheetModelsCacheKey] = useState(null); const [activeSheet, setActiveSheet] = useState<{ role: ModelRole } | null>(null); + const [llmProvider, setLlmProvider] = useState( + isLocalMode ? null : "openrouter", + ); + const [activeModelListCacheKey, setActiveModelListCacheKey] = useState( + isLocalMode ? "" : "openrouter|||", + ); + const activeModelListCacheKeyRef = useRef(activeModelListCacheKey); const [isSavingModel, setIsSavingModel] = useState(false); + const [isSavingExtractorSettings, setIsSavingExtractorSettings] = useState(false); + const [extractorConcurrency, setExtractorConcurrency] = useState(5); + const [extractorAttempts, setExtractorAttempts] = useState(2); + const [modelConfigReloadKey, setModelConfigReloadKey] = useState(0); + + const needsOpenRouterCache = !isLocalMode || llmProvider === "openrouter"; + const isLoading = + (needsOpenRouterCache && convexModels === undefined) || + isLoadingConfig || + (isLocalMode && llmProvider === null); - const isLoading = convexModels === undefined || isLoadingConfig; + const syncLlmProvider = useCallback((status: LocalSetupStatus) => { + const nextCacheKey = modelListCacheKey(status); + activeModelListCacheKeyRef.current = nextCacheKey; + setLlmProvider(status.services.llm.provider ?? "openrouter"); + setActiveModelListCacheKey(nextCacheKey); + }, []); + + const handleLocalCredentialsStatus = useCallback( + (status: LocalSetupStatus) => { + syncLlmProvider(status); + setSheetModels([]); + setSheetModelsCacheKey(null); + setModelConfigReloadKey((key) => key + 1); + }, + [syncLlmProvider], + ); useEffect(() => { + let active = true; + getToken() .then((token) => { if (!token) throw new Error("Not authenticated"); return getModelConfig(token); }) - .then((config) => setEffectiveConfig(config)) - .catch(() => setEffectiveConfig(null)) - .finally(() => setIsLoadingConfig(false)); - }, [getToken]); + .then((config) => { + if (active) { + setEffectiveConfig(config); + setExtractorConcurrency( + Number.isFinite(config.rowExtractorConcurrency) + ? config.rowExtractorConcurrency + : 5, + ); + setExtractorAttempts( + Number.isFinite(config.rowExtractorBrowserAttempts) + ? config.rowExtractorBrowserAttempts + : 2, + ); + } + }) + .catch(() => { + if (active) setEffectiveConfig(null); + }) + .finally(() => { + if (active) setIsLoadingConfig(false); + }); + + return () => { + active = false; + }; + }, [getToken, modelConfigReloadKey]); + + useEffect(() => { + if (!isLocalMode) return; + getLocalSetupStatus() + .then(syncLlmProvider) + .catch(() => setLlmProvider("openrouter")); + }, [syncLlmProvider]); const models: OpenRouterModel[] = convexModels ? convexModels.map((m) => ({ @@ -46,19 +121,28 @@ export default function ModelSettingsPage() { promptCost: m.promptCost, })) : []; + const sideSheetModels = + sheetModels.length > 0 + ? sheetModels + : llmProvider === "openrouter" + ? models + : []; function getSelectedModel(role: ModelRole): string { - return effectiveConfig?.[role.key as keyof typeof effectiveConfig] ?? ""; + return String(effectiveConfig?.[role.key as keyof typeof effectiveConfig] ?? ""); } - async function handleModelSelect(role: ModelRole, model: OpenRouterModel) { + async function saveModelForRole(role: ModelRole, modelId: string) { + const nextModelId = modelId.trim(); + if (!nextModelId) return; + setIsSavingModel(true); try { const token = await getToken(); if (!token) throw new Error("Not authenticated"); - await saveModelConfig({ [role.key]: model.canonicalSlug }, token); + await saveModelConfig({ [role.key]: nextModelId }, token); setEffectiveConfig((prev: EffectiveModelConfig | null) => - prev ? { ...prev, [role.key]: model.canonicalSlug } : null + prev ? { ...prev, [role.key]: nextModelId } : null ); setActiveSheet(null); } catch { @@ -68,10 +152,45 @@ export default function ModelSettingsPage() { } } + async function saveExtractorSettings() { + setIsSavingExtractorSettings(true); + try { + const token = await getToken(); + if (!token) throw new Error("Not authenticated"); + await saveModelConfig( + { + rowExtractorConcurrency: extractorConcurrency, + rowExtractorBrowserAttempts: extractorAttempts, + }, + token, + ); + setEffectiveConfig((prev: EffectiveModelConfig | null) => + prev + ? { + ...prev, + rowExtractorConcurrency: extractorConcurrency, + rowExtractorBrowserAttempts: extractorAttempts, + } + : prev, + ); + } catch { + // we will add toast later + } finally { + setIsSavingExtractorSettings(false); + } + } + function openSideSheet(role: ModelRole) { - if (sheetModels.length === 0) { - getOpenRouterModels() - .then((models) => setSheetModels(models)) + const cacheKey = activeModelListCacheKeyRef.current || activeModelListCacheKey; + if (sheetModels.length === 0 || sheetModelsCacheKey !== cacheKey) { + setSheetModels([]); + setSheetModelsCacheKey(cacheKey); + getLlmProviderModels() + .then((models) => { + if (activeModelListCacheKeyRef.current !== cacheKey) return; + setSheetModels(models); + setSheetModelsCacheKey(cacheKey); + }) .catch(() => { // we will add toast later }); @@ -81,7 +200,7 @@ export default function ModelSettingsPage() { const navItems = [ { - label: "Models", + label: "General", href: "/dashboard/settings/models", icon: ( @@ -117,11 +236,15 @@ export default function ModelSettingsPage() { return (
- +
@@ -139,6 +262,72 @@ export default function ModelSettingsPage() { )) )}
+ + {isLocalMode && !isLoading && ( +
+ + +
+ + + +
+ +
+ +
+
+ )}
{activeSheet && ( @@ -147,18 +336,23 @@ export default function ModelSettingsPage() { onClose={() => !isSavingModel && setActiveSheet(null)} title={`Select ${activeSheet.role.label} Model`} selectedModel={getSelectedModel(activeSheet.role)} - models={sheetModels.length > 0 ? sheetModels : models} - onSelect={(slug) => { - const sourceModels = sheetModels.length > 0 ? sheetModels : models; - const model = sourceModels.find((m) => m.canonicalSlug === slug); - if (model) handleModelSelect(activeSheet.role, model); - }} + models={sideSheetModels} + onSelect={(slug) => saveModelForRole(activeSheet.role, slug)} onRefresh={async () => { setRefreshing(true); try { - const token = await getToken(); - if (!token) throw new Error("Not authenticated"); - const models = await refreshOpenRouterModels(token); + const provider = llmProvider ?? "openrouter"; + const cacheKey = activeModelListCacheKeyRef.current || activeModelListCacheKey; + let models: OpenRouterModel[]; + if (provider === "openrouter") { + const token = await getToken(); + if (!token) throw new Error("Not authenticated"); + models = await refreshOpenRouterModels(token); + } else { + models = await getLlmProviderModels(); + } + if (activeModelListCacheKeyRef.current !== cacheKey) return; + setSheetModelsCacheKey(cacheKey); setSheetModels(models); } catch { // we will add toast later @@ -170,6 +364,8 @@ export default function ModelSettingsPage() { isSaving={isSavingModel} /> )} + + ); } diff --git a/frontend/app/dataset/[id]/page.tsx b/frontend/app/dataset/[id]/page.tsx index 697addd..37ca9c9 100644 --- a/frontend/app/dataset/[id]/page.tsx +++ b/frontend/app/dataset/[id]/page.tsx @@ -42,7 +42,8 @@ export default function DatasetPage() { const [cellDetail, setCellDetail] = useState<{ column: DatasetColumn; value: unknown; - sources?: string[]; + cellSources?: string[]; + rowSources?: string[]; } | null>(null); const datasetId = params.id as Id<"datasets">; @@ -103,7 +104,12 @@ export default function DatasetPage() { const col = dataset.columns.find((c) => c.name === columnName); if (!col) return; const row = rows.find((r) => r._id === rowId); - setCellDetail({ column: col, value, sources: row?.sources }); + setCellDetail({ + column: col, + value, + cellSources: row?.cellSources?.[columnName], + rowSources: row?.sources, + }); }, [dataset, rows]); const openedFired = useRef(null); @@ -461,7 +467,8 @@ export default function DatasetPage() { )} diff --git a/frontend/app/dataset/new/page.tsx b/frontend/app/dataset/new/page.tsx index 63b8ebf..651ae68 100644 --- a/frontend/app/dataset/new/page.tsx +++ b/frontend/app/dataset/new/page.tsx @@ -6,7 +6,13 @@ import Link from "next/link"; import { useMutation, useQuery } from "convex/react"; import { api } from "@/convex/_generated/api"; import { EVENTS, track } from "@/lib/analytics"; -import { inferSchema, type InferredColumn } from "@/lib/backend"; +import { + finalizeSchema, + inferSchema, + type CodificationProfile, + type InferredCodificationProfile, + type InferredColumn, +} from "@/lib/backend"; import { useAppAuth, useAppConvexAuth } from "@/lib/app-auth"; import { REFRESH_CADENCE_OPTIONS, @@ -48,13 +54,31 @@ const DEFAULT_MAX_ROW_COUNT = 100; function mapBackendColumn(col: InferredColumn, index: number): ProposedColumn { return { id: String(index + 1), - name: col.display_name, + name: col.name, type: BACKEND_TYPE_MAP[col.type], description: col.retrieval_hint, isPrimaryKey: col.is_primary_key, }; } +function mapCodificationProfile( + profile: InferredCodificationProfile, +): CodificationProfile { + return { + version: 1, + mode: profile.mode, + reason: profile.reason, + primaryKeyShape: profile.primary_key_shape, + families: profile.families.map((family) => ({ + label: family.label, + sourceHost: family.source_host, + sourcePathPrefix: family.source_path_prefix, + urlTemplate: family.url_template, + primaryKeyRegex: family.primary_key_regex, + })), + }; +} + function TypeSelector({ value, onChange }: { value: ColumnType; onChange: (v: ColumnType) => void }) { return (
@@ -187,19 +211,50 @@ export default function NewDatasetPage() { setError(null); let datasetId: string; try { + const token = await getToken(); + if (!token) throw new Error("Not authenticated"); + + const finalizedSchema = await finalizeSchema( + { + prompt: prompt.trim(), + datasetName, + retrievalStrategy: retrievalStrategy ?? undefined, + sourceHint: sourceHint || undefined, + columns: columns.map((column) => ({ + name: column.name, + type: column.type, + description: column.description || undefined, + isPrimaryKey: column.isPrimaryKey || undefined, + })), + }, + token, + ); + const finalizedColumns = columns.map((column, index) => { + const finalizedColumn = finalizedSchema.columns[index]; + if (!finalizedColumn) { + throw new Error("Schema finalization returned an incomplete schema."); + } + + return { + name: column.name, + type: column.type, + description: column.description || undefined, + isPrimaryKey: finalizedColumn.is_primary_key || undefined, + nullable: finalizedColumn.nullable, + validationRegex: finalizedColumn.validation_regex || undefined, + normalizationHint: finalizedColumn.normalization_hint || undefined, + }; + }); + datasetId = await createDataset({ name: datasetName, description: prompt, refreshCadence, maxRowCount, - columns: columns.map((c) => ({ - name: c.name, - type: c.type, - description: c.description || undefined, - isPrimaryKey: c.isPrimaryKey || undefined, - })), - retrievalStrategy: retrievalStrategy ?? undefined, - sourceHint: sourceHint || undefined, + columns: finalizedColumns, + retrievalStrategy: finalizedSchema.retrieval_strategy ?? retrievalStrategy ?? undefined, + sourceHint: finalizedSchema.source_hint || sourceHint || undefined, + codificationProfile: mapCodificationProfile(finalizedSchema.codification_profile), }); } catch (err) { const message = err instanceof Error ? err.message : "Failed to create dataset"; diff --git a/frontend/app/setup/page.tsx b/frontend/app/setup/page.tsx index 9e6bd71..79c0ec0 100644 --- a/frontend/app/setup/page.tsx +++ b/frontend/app/setup/page.tsx @@ -1,6 +1,6 @@ "use client"; -import { useEffect, useMemo, useState, type ReactNode } from "react"; +import { useCallback, useEffect, useMemo, useRef, useState, type ReactNode } from "react"; import { useRouter } from "next/navigation"; import { CheckCircle2, @@ -11,18 +11,77 @@ import { } from "lucide-react"; import { getLocalSetupStatus, - saveOpenRouterApiKey, + getLlmProviderModels, + getModelConfig, + saveLlmProviderConfig, + saveModelConfig, saveTinyFishApiKey, + type EffectiveModelConfig, + type LlmProviderType, type LocalSetupStatus, + type OpenRouterModel, type ServiceSetupStatus, } from "@/lib/backend"; import { isLocalMode } from "@/lib/app-mode"; +import { + LlmProviderBrand, + LlmProviderSelector, + displayBaseUrl, + llmProviderLabelForStatus, + llmProviderOption, + localLlmPresetForBaseUrl, + type LlmProviderOptionValue, +} from "@/components/settings/llm-providers"; +import { + beginOpenRouterOAuth, + useCanUseOpenRouterOAuth, +} from "@/lib/openrouter-oauth"; +import { LocalUtilityMenu } from "@/components/LocalUtilityMenu"; +import { ModelSideSheet } from "@/components/settings/ModelSideSheet"; +import { MODEL_ROLES, type ModelRole } from "@/components/settings/types"; +import { useAppAuth } from "@/lib/app-auth"; + +function modelListCacheKey(status: LocalSetupStatus | null): string { + const llm = status?.services.llm; + if (!llm) return ""; + return [ + llm.provider ?? "openrouter", + llm.baseUrl ?? "", + llm.defaultModel ?? "", + llm.verifiedAt ?? "", + ].join("|"); +} + +function emptyModelConfig(): EffectiveModelConfig { + return { + schemaInference: "", + populateOrchestrator: "", + investigateSubagent: "", + extractorBuilder: "", + rowExtractorConcurrency: 5, + rowExtractorBrowserAttempts: 2, + }; +} export default function SetupPage() { const router = useRouter(); + const { getToken } = useAppAuth(); const [status, setStatus] = useState(null); const [loading, setLoading] = useState(true); - const [modal, setModal] = useState<"tinyfish" | "openrouter" | null>(null); + const [modal, setModal] = useState<"tinyfish" | "llm" | null>(null); + const [modelConfig, setModelConfig] = useState( + null, + ); + const [loadingModelConfig, setLoadingModelConfig] = useState(false); + const [modelError, setModelError] = useState(null); + const [activeModelRole, setActiveModelRole] = useState(null); + const [modelOptions, setModelOptions] = useState([]); + const [modelOptionsCacheKey, setModelOptionsCacheKey] = useState< + string | null + >(null); + const [refreshingModels, setRefreshingModels] = useState(false); + const [savingModel, setSavingModel] = useState(false); + const activeModelListCacheKeyRef = useRef(""); useEffect(() => { if (!isLocalMode) { @@ -35,7 +94,123 @@ export default function SetupPage() { .finally(() => setLoading(false)); }, [router]); + const activeModelListCacheKey = modelListCacheKey(status); + + useEffect(() => { + activeModelListCacheKeyRef.current = activeModelListCacheKey; + }, [activeModelListCacheKey]); + + useEffect(() => { + let active = true; + + if (!status?.services.llm.configured) return; + + async function loadModelConfig() { + setLoadingModelConfig(true); + setModelError(null); + try { + const token = await getToken(); + if (!token) throw new Error("Not authenticated"); + const config = await getModelConfig(token); + if (active) setModelConfig(config); + } catch (err) { + if (!active) return; + setModelConfig(emptyModelConfig()); + setModelError( + err instanceof Error ? err.message : "Failed to load model settings", + ); + } finally { + if (active) setLoadingModelConfig(false); + } + } + + void loadModelConfig(); + + return () => { + active = false; + }; + }, [ + getToken, + status?.services.llm.baseUrl, + status?.services.llm.configured, + status?.services.llm.provider, + status?.services.llm.verifiedAt, + ]); + + const loadProviderModels = useCallback( + async (force = false) => { + const cacheKey = activeModelListCacheKeyRef.current; + if (!force && modelOptionsCacheKey === cacheKey && modelOptions.length > 0) { + return; + } + + setRefreshingModels(true); + setModelError(null); + try { + const models = await getLlmProviderModels(); + if (activeModelListCacheKeyRef.current !== cacheKey) return; + setModelOptions(models); + setModelOptionsCacheKey(cacheKey); + } catch (err) { + if (activeModelListCacheKeyRef.current !== cacheKey) return; + setModelOptions([]); + setModelOptionsCacheKey(cacheKey); + setModelError( + err instanceof Error ? err.message : "Failed to load models", + ); + } finally { + if (activeModelListCacheKeyRef.current === cacheKey) { + setRefreshingModels(false); + } + } + }, + [modelOptions.length, modelOptionsCacheKey], + ); + + function modelForRole(role: ModelRole): string { + const key = role.key as keyof EffectiveModelConfig; + return String(modelConfig?.[key] ?? ""); + } + + function openModelSheet(role: ModelRole) { + setActiveModelRole(role); + void loadProviderModels(); + } + + async function saveModelForRole(role: ModelRole, modelId: string) { + const nextModelId = modelId.trim(); + if (!nextModelId) return; + + const key = role.key as keyof EffectiveModelConfig; + setSavingModel(true); + setModelError(null); + try { + const token = await getToken(); + if (!token) throw new Error("Not authenticated"); + await saveModelConfig({ [key]: nextModelId }, token); + setModelConfig((prev) => ({ + ...emptyModelConfig(), + ...prev, + [key]: nextModelId, + })); + void getLocalSetupStatus().then(setStatus).catch(() => undefined); + setActiveModelRole(null); + } catch (err) { + setModelError( + err instanceof Error ? err.message : "Failed to save model", + ); + } finally { + setSavingModel(false); + } + } + const complete = status?.complete ?? false; + const modelSelectionRequired = + !!status?.services.llm.configured && !status.services.llm.defaultModel; + const modelsConfigured = + !modelSelectionRequired || + MODEL_ROLES.every((role) => modelForRole(role).trim().length > 0); + const canCompleteSetup = complete && modelsConfigured; if (loading) { return ( @@ -47,9 +222,12 @@ export default function SetupPage() { return (
-
- BigSet - BigSet +
+
+ BigSet + BigSet +
+
@@ -59,12 +237,11 @@ export default function SetupPage() { Connect your services

- Add TinyFish and OpenRouter access to start building live - datasets. + Add TinyFish and choose where BigSet should run model calls.

-
+
@@ -80,42 +257,103 @@ export default function SetupPage() { /> } - description="BigSet uses TinyFish's best-in-class search API to unlock real-time information." + description="Connect TinyFish for live search and source pages." status={status?.services.tinyfish} primaryLabel={ status?.services.tinyfish.configured ? "Update key" : "Add API key" } onPrimary={() => setModal("tinyfish")} helperHref="https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2" - helperLabel="Need a TinyFish key?" - helperDescription="Open the TinyFish API keys page" + helperLabel="Get your TinyFish API Key" /> } - description="BigSet uses OpenRouter's API to power BigSet with AI model access." - status={status?.services.openrouter} + brand={ + + } + description="Choose the provider BigSet uses for schema generation and agents." + status={status?.services.llm} primaryLabel={ - status?.services.openrouter.configured - ? "Update key" - : "Add API key" + status?.services.llm.configured + ? "Update provider" + : "Choose provider" } - onPrimary={() => setModal("openrouter")} - helperHref="https://openrouter.ai/settings/keys" - helperLabel="Need an OpenRouter key?" - helperDescription="Open the OpenRouter keys page" + onPrimary={() => setModal("llm")} />
+ {status?.services.llm.configured && ( +
+
+
+

+ Models +

+

+ {llmProviderLabelForStatus(status.services.llm) ?? + "Model provider"} +

+
+ {modelSelectionRequired && !modelsConfigured && ( + + Required + + )} +
+
+ {MODEL_ROLES.map((role) => { + const selectedModel = modelForRole(role); + return ( + + ); + })} +
+ {modelError && ( +
+ {modelError} +
+ )} +
+ )} +

- {complete + {complete && modelsConfigured ? "Everything is connected. You can start building datasets." + : complete && modelSelectionRequired + ? "Choose models to continue." : "Complete both connections to continue."}

); } @@ -159,20 +412,22 @@ function ServiceCard({ onPrimary: () => void; secondaryLabel?: string; onSecondary?: () => void; - helperHref: string; - helperLabel: string; - helperDescription: string; + helperHref?: string; + helperLabel?: string; + helperDescription?: string; }) { const connected = status?.configured ?? false; const detail = useMemo(() => { if (!connected) return "Not connected"; + const llmLabel = llmProviderLabelForStatus(status); + if (llmLabel) return llmLabel; if (status?.connectionMethod === "oauth") return "Connected through OAuth"; if (status?.source === "env") return "Connected through .env"; return "Connected through API key"; - }, [connected, status?.connectionMethod, status?.source]); + }, [connected, status]); return ( -
+
{brand}
@@ -190,7 +445,7 @@ function ServiceCard({ {description}

-
+
- - {helperLabel} {helperDescription} - - + {helperHref && helperLabel ? ( + + {helperLabel} + {helperDescription ? ` ${helperDescription}` : null} + + + ) : helperLabel ? ( +

+ {helperLabel}:{" "} + {helperDescription} +

+ ) : null}
); } -function OpenRouterBrand() { - return ( -
- - OpenRouter -
- ); -} - function ApiKeyModal({ service, + status, onClose, onSaved, }: { - service: "tinyfish" | "openrouter"; + service: "tinyfish" | "llm"; + status: LocalSetupStatus | null; onClose: () => void; onSaved: (status: LocalSetupStatus) => void; }) { + const initialProvider = initialLlmProviderSelection(status); const [apiKey, setApiKey] = useState(""); + const [provider, setProvider] = + useState(initialProvider); + const [baseUrl, setBaseUrl] = useState(() => + initialBaseUrl(status, initialProvider), + ); const [saving, setSaving] = useState(false); const [error, setError] = useState(null); const isTinyFish = service === "tinyfish"; + const providerCopy = llmProviderOption(provider); + const providerStatuses = status?.services.llmProviders; + const resolvedProvider = providerCopy.provider; + const selectedProviderStatus = providerStatuses?.[resolvedProvider]; + const selectedProviderConfigured = + selectedProviderStatus?.configured ?? + (status?.services.llm.configured && + status.services.llm.provider === resolvedProvider) ?? + false; + const selectedRequiresBaseUrl = providerCopy.requiresBaseUrl ?? false; + const selectedRequiresApiKey = providerCopy.requiresApiKey ?? resolvedProvider !== "custom"; + const selectedUsesPresetBaseUrl = !!providerCopy.defaultBaseUrl; + const showOpenRouterOAuth = useCanUseOpenRouterOAuth(); + const isCustomEndpoint = provider === "custom"; + + function handleProviderChange(next: LlmProviderOptionValue) { + setProvider(next); + setBaseUrl(initialBaseUrl(status, next)); + setApiKey(""); + setError(null); + } async function handleSubmit() { - if (!apiKey.trim() || saving) return; + if (saving) return; + if (isTinyFish && !apiKey.trim()) return; + if (!isTinyFish && selectedRequiresApiKey && !apiKey.trim() && !selectedProviderConfigured) { + return; + } + if (!isTinyFish && selectedRequiresBaseUrl && !baseUrl.trim() && !selectedProviderConfigured) { + setError("Custom providers require a base URL"); + return; + } + setSaving(true); setError(null); try { const next = isTinyFish ? await saveTinyFishApiKey(apiKey.trim()) - : await saveOpenRouterApiKey(apiKey.trim()); + : await saveLlmProviderConfig({ + provider: resolvedProvider, + apiKey: + selectedRequiresApiKey || provider === "custom" + ? apiKey.trim() + : "", + defaultModel: providerCopy.defaultModel, + baseUrl: + selectedRequiresBaseUrl && baseUrl.trim() + ? baseUrl.trim() + : undefined, + }); onSaved(next); } catch (err) { setError(err instanceof Error ? err.message : "Verification failed"); @@ -282,6 +566,37 @@ function ApiKeyModal({ } } + const helperHref = isTinyFish + ? "https://agent.tinyfish.ai/api-keys?utm_source=github&utm_medium=organic&utm_campaign=bigset-developer-2026q2" + : providerCopy.helperHref; + const helperLabel = isTinyFish + ? "Get your TinyFish API Key" + : isCustomEndpoint + ? "OpenAI API docs" + : selectedRequiresBaseUrl + ? "Provider docs" + : "Get a key"; + const showApiKeyHelper = + !!helperHref && (isTinyFish || selectedRequiresApiKey); + const canSubmit = + !saving && + (isTinyFish + ? !!apiKey.trim() + : selectedRequiresBaseUrl + ? !!baseUrl.trim() || selectedProviderConfigured + : !selectedRequiresApiKey || !!apiKey.trim() || selectedProviderConfigured); + const usingSavedProvider = + !isTinyFish && + selectedProviderConfigured && + !apiKey.trim() && + (!selectedRequiresBaseUrl || + !baseUrl.trim() || + baseUrl.trim() === displayBaseUrl(selectedProviderStatus?.baseUrl)); + const modalTitle = isTinyFish ? "TinyFish API key" : "Model provider"; + const modalDescription = isTinyFish + ? "BigSet verifies the key and stores it in your OS keychain." + : "Select a provider. BigSet stores local credentials in your OS keychain."; + return (
-
- +
+ {!isTinyFish && ( +
+ + Provider + + +
+ )} + + {!isTinyFish && provider === "openrouter" && showOpenRouterOAuth && ( +
+
+

+ OpenRouter OAuth +

+

+ Connect through OpenRouter without pasting a key. +

+
+ +
+ )} + + {!isTinyFish && selectedRequiresBaseUrl && ( +
+
+ + {helperHref && ( + + {helperLabel} + + + )} +
+ setBaseUrl(event.target.value)} + className="w-full rounded-lg border border-border bg-background px-3 py-2.5 text-sm text-foreground outline-none transition-colors placeholder:text-muted/70 focus:border-foreground/30" + placeholder={providerCopy.defaultBaseUrl ?? "http://localhost:1234/v1"} + /> + + {selectedUsesPresetBaseUrl + ? "Default local endpoint. Change it only if your server uses another port." + : "Use the OpenAI-compatible `/v1` endpoint for local or experimental providers."} + +
+ )} + + {(isTinyFish || selectedRequiresApiKey || provider === "custom") && ( +
+
+ + {showApiKeyHelper && ( + + {helperLabel} + + + )} +
+ setApiKey(event.target.value)} + type="password" + autoFocus={isTinyFish} + disabled={!isTinyFish && !selectedRequiresApiKey && provider !== "custom"} + className="w-full rounded-lg border border-border bg-background px-3 py-2.5 text-sm text-foreground outline-none transition-colors placeholder:text-muted/70 focus:border-foreground/30 disabled:cursor-not-allowed disabled:opacity-60" + placeholder={isTinyFish ? "tf_..." : providerCopy.apiKeyPlaceholder} + /> +
+ )} {error && (
{error}
)} +
-
- - Get a key - - - -
+
+
); } + +function initialLlmProviderSelection( + status: LocalSetupStatus | null, +): LlmProviderOptionValue { + if (status?.services.llm.configured && status.services.llm.provider) { + if (status.services.llm.provider === "custom") { + return ( + localLlmPresetForBaseUrl(status.services.llm.baseUrl)?.value ?? "custom" + ); + } + return status.services.llm.provider; + } + + const savedProvider = (Object.entries(status?.services.llmProviders ?? {}) as [ + LlmProviderType, + ServiceSetupStatus, + ][]).find(([, providerStatus]) => providerStatus.configured)?.[0]; + + if (savedProvider === "custom") { + return ( + localLlmPresetForBaseUrl(status?.services.llmProviders?.custom?.baseUrl) + ?.value ?? "custom" + ); + } + + return savedProvider ?? "openrouter"; +} + +function initialBaseUrl( + status: LocalSetupStatus | null, + provider: LlmProviderOptionValue, +) { + const option = llmProviderOption(provider); + const savedBaseUrl = displayBaseUrl( + status?.services.llmProviders?.[option.provider]?.baseUrl, + ); + if (option.defaultBaseUrl) return savedBaseUrl || option.defaultBaseUrl; + if (option.provider === "custom") { + return savedBaseUrl; + } + return ""; +} diff --git a/frontend/components/SideSheet.tsx b/frontend/components/SideSheet.tsx index 215d477..09bb917 100644 --- a/frontend/components/SideSheet.tsx +++ b/frontend/components/SideSheet.tsx @@ -117,8 +117,8 @@ export function SideSheet({ open, onClose, children }: SideSheetProps) { interface CellDetailProps { column: DatasetColumn; value: unknown; - /** Row-level sources stored by the populate agent. */ - sources?: string[]; + cellSources?: string[]; + rowSources?: string[]; } function isValidHttpUrl(src: string): boolean { @@ -130,7 +130,7 @@ function isValidHttpUrl(src: string): boolean { } } -export function CellDetail({ column, value, sources }: CellDetailProps) { +export function CellDetail({ column, value, cellSources, rowSources }: CellDetailProps) { const [copied, setCopied] = useState(false); const copyTimerRef = useRef | null>(null); const displayValue = value == null || value === "" ? "—" : String(value); @@ -192,14 +192,49 @@ export function CellDetail({ column, value, sources }: CellDetailProps) {
- {/* Sources */} - {sources && sources.length > 0 && ( + {/* Cell sources */} + {cellSources && cellSources.length > 0 && (

Sources

    - {sources.map((src, i) => ( + {cellSources.map((src, i) => ( +
  • + {isValidHttpUrl(src) ? ( + + + {src} + + ) : ( + + + {src} + + )} +
    • + ))} +
+
+ )} + + {/* Row sources */} + {rowSources && rowSources.length > 0 && ( +
+

+ Row Sources +

+

+ These URLs were recorded for the row, not as proof for this specific cell. +

+
@@ -106,7 +125,7 @@ export function LocalCredentialsPanel() { {loadError} ) : ( -
+
setModal("tinyfish")} /> setModal("openrouter")} + onApiKey={() => setModal("llm")} />
)} @@ -125,9 +144,11 @@ export function LocalCredentialsPanel() { {modal && ( setModal(null)} onSaved={(next) => { setStatus(next); + onStatusChange?.(next); setModal(null); }} /> @@ -150,13 +171,25 @@ function CredentialCard({ const copy = SERVICE_COPY[service]; const connected = status?.configured ?? false; const detail = useCredentialDetail(status, loading); + const primaryLabel = + service === "llm" + ? connected + ? "Update provider" + : "Choose provider" + : connected + ? "Update key" + : "Add API key"; return ( -
+
); } -function ServiceBrand({ service }: { service: ServiceName }) { +function ServiceBrand({ + service, + provider, + baseUrl, +}: { + service: ServiceName; + provider?: LlmProviderType; + baseUrl?: string; +}) { if (service === "tinyfish") { return ( <> @@ -208,35 +259,7 @@ function ServiceBrand({ service }: { service: ServiceName }) { ); } - return ; -} - -function OpenRouterBrand() { - return ( -
- - OpenRouter -
- ); + return ; } function StatusLabel({ @@ -272,35 +295,86 @@ function useCredentialDetail( return useMemo(() => { if (loading) return "Checking connection..."; if (!status?.configured) return "Not connected"; + const llmLabel = llmProviderLabelForStatus(status); + if (llmLabel) return llmLabel; if (status.connectionMethod === "oauth") return "Connected through OAuth"; if (status.source === "env") return "Connected through .env"; return "Connected through API key"; - }, [loading, status?.configured, status?.connectionMethod, status?.source]); + }, [loading, status]); } function ApiKeyModal({ service, + status, onClose, onSaved, }: { service: ServiceName; + status: LocalSetupStatus | null; onClose: () => void; onSaved: (status: LocalSetupStatus) => void; }) { + const initialProvider = initialLlmProviderSelection(status); const [apiKey, setApiKey] = useState(""); + const [provider, setProvider] = + useState(initialProvider); + const [baseUrl, setBaseUrl] = useState(() => + initialBaseUrl(status, initialProvider), + ); const [saving, setSaving] = useState(false); const [error, setError] = useState(null); const copy = SERVICE_COPY[service]; const isTinyFish = service === "tinyfish"; + const providerCopy = llmProviderOption(provider); + const providerStatuses = status?.services.llmProviders; + const resolvedProvider = providerCopy.provider; + const selectedProviderStatus = providerStatuses?.[resolvedProvider]; + const selectedProviderConfigured = + selectedProviderStatus?.configured ?? + (status?.services.llm.configured && + status.services.llm.provider === resolvedProvider) ?? + false; + const selectedRequiresBaseUrl = providerCopy.requiresBaseUrl ?? false; + const selectedRequiresApiKey = providerCopy.requiresApiKey ?? resolvedProvider !== "custom"; + const selectedUsesPresetBaseUrl = !!providerCopy.defaultBaseUrl; + const showOpenRouterOAuth = useCanUseOpenRouterOAuth(); + const isCustomEndpoint = provider === "custom"; + + function handleProviderChange(next: LlmProviderOptionValue) { + setProvider(next); + setBaseUrl(initialBaseUrl(status, next)); + setApiKey(""); + setError(null); + } async function handleSubmit() { - if (!apiKey.trim() || saving) return; + if (saving) return; + if (isTinyFish && !apiKey.trim()) return; + if (!isTinyFish && selectedRequiresApiKey && !apiKey.trim() && !selectedProviderConfigured) { + return; + } + if (!isTinyFish && selectedRequiresBaseUrl && !baseUrl.trim() && !selectedProviderConfigured) { + setError("Custom providers require a base URL"); + return; + } + setSaving(true); setError(null); try { const next = isTinyFish ? await saveTinyFishApiKey(apiKey.trim()) - : await saveOpenRouterApiKey(apiKey.trim()); + : await saveLlmProviderConfig({ + provider: resolvedProvider, + apiKey: + selectedRequiresApiKey || provider === "custom" + ? apiKey.trim() + : "", + defaultModel: providerCopy.defaultModel, + baseUrl: + selectedRequiresBaseUrl && baseUrl.trim() + ? baseUrl.trim() + : undefined, + }); onSaved(next); } catch (err) { setError(err instanceof Error ? err.message : "Verification failed"); @@ -309,6 +383,31 @@ function ApiKeyModal({ } } + const helperHref = isTinyFish ? copy.helperHref : providerCopy.helperHref; + const helperLabel = isTinyFish + ? "Get your TinyFish API Key" + : isCustomEndpoint + ? "OpenAI API docs" + : selectedRequiresBaseUrl + ? "Provider docs" + : "Get a key"; + const showApiKeyHelper = + !!helperHref && (isTinyFish || selectedRequiresApiKey); + const canSubmit = + !saving && + (isTinyFish + ? !!apiKey.trim() + : selectedRequiresBaseUrl + ? !!baseUrl.trim() || selectedProviderConfigured + : !selectedRequiresApiKey || !!apiKey.trim() || selectedProviderConfigured); + const usingSavedProvider = + !isTinyFish && + selectedProviderConfigured && + !apiKey.trim() && + (!selectedRequiresBaseUrl || + !baseUrl.trim() || + baseUrl.trim() === displayBaseUrl(selectedProviderStatus?.baseUrl)); + return (
-
- +
+ {!isTinyFish && ( +
+ + Provider + + +
+ )} + + {!isTinyFish && provider === "openrouter" && showOpenRouterOAuth && ( +
+
+

+ OpenRouter OAuth +

+

+ Connect through OpenRouter without pasting a key. +

+
+ +
+ )} + + {!isTinyFish && selectedRequiresBaseUrl && ( +
+
+ + {helperHref && ( + + {helperLabel} + + + )} +
+ setBaseUrl(event.target.value)} + className="w-full rounded-lg border border-border bg-background px-3 py-2.5 text-sm text-foreground outline-none transition-colors placeholder:text-muted/70 focus:border-foreground/30" + placeholder={providerCopy.defaultBaseUrl ?? "http://localhost:1234/v1"} + /> + + {selectedUsesPresetBaseUrl + ? "Default local endpoint. Change it only if your server uses another port." + : "Use the OpenAI-compatible `/v1` endpoint for local or experimental providers."} + +
+ )} + + {(isTinyFish || selectedRequiresApiKey || provider === "custom") && ( +
+
+ + {showApiKeyHelper && ( + + {helperLabel} + + + )} +
+ setApiKey(event.target.value)} + type="password" + autoFocus={isTinyFish} + disabled={!isTinyFish && !selectedRequiresApiKey && provider !== "custom"} + className="w-full rounded-lg border border-border bg-background px-3 py-2.5 text-sm text-foreground outline-none transition-colors placeholder:text-muted/70 focus:border-foreground/30 disabled:cursor-not-allowed disabled:opacity-60" + placeholder={isTinyFish ? copy.inputPlaceholder : providerCopy.apiKeyPlaceholder} + /> +
+ )} {error && (
{error}
)} +
-
- - Get a key - - - -
+
+
); } + +function initialLlmProviderSelection( + status: LocalSetupStatus | null, +): LlmProviderOptionValue { + if (status?.services.llm.configured && status.services.llm.provider) { + if (status.services.llm.provider === "custom") { + return ( + localLlmPresetForBaseUrl(status.services.llm.baseUrl)?.value ?? "custom" + ); + } + return status.services.llm.provider; + } + + const savedProvider = (Object.entries(status?.services.llmProviders ?? {}) as [ + LlmProviderType, + ServiceSetupStatus, + ][]).find(([, providerStatus]) => providerStatus.configured)?.[0]; + + if (savedProvider === "custom") { + return ( + localLlmPresetForBaseUrl(status?.services.llmProviders?.custom?.baseUrl) + ?.value ?? "custom" + ); + } + + return savedProvider ?? "openrouter"; +} + +function initialBaseUrl( + status: LocalSetupStatus | null, + provider: LlmProviderOptionValue, +) { + const option = llmProviderOption(provider); + const savedBaseUrl = displayBaseUrl( + status?.services.llmProviders?.[option.provider]?.baseUrl, + ); + if (option.defaultBaseUrl) return savedBaseUrl || option.defaultBaseUrl; + if (option.provider === "custom") { + return savedBaseUrl; + } + return ""; +} + +function currentReturnPath() { + if (typeof window === "undefined") return "/setup"; + return `${window.location.pathname}${window.location.search}`; +} diff --git a/frontend/components/settings/ModelSideSheet.tsx b/frontend/components/settings/ModelSideSheet.tsx index 581103e..e775d89 100644 --- a/frontend/components/settings/ModelSideSheet.tsx +++ b/frontend/components/settings/ModelSideSheet.tsx @@ -1,7 +1,7 @@ "use client"; import { useEffect, useRef, useState } from "react"; -import { X, Search, RefreshCw } from "lucide-react"; +import { Check, X, Search, RefreshCw } from "lucide-react"; import type { OpenRouterModel } from "./types"; interface ModelSideSheetProps { @@ -70,8 +70,10 @@ export function ModelSideSheet({ isSaving, }: ModelSideSheetProps) { const [search, setSearch] = useState(""); + const [customSlug, setCustomSlug] = useState(selectedModel); const panelRef = useRef(null); const inputRef = useRef(null); + const customInputRef = useRef(null); const filteredModels = search.trim() ? models.filter( @@ -83,10 +85,17 @@ export function ModelSideSheet({ const groupedModels = groupModelsByProvider(filteredModels); const providers = Object.keys(groupedModels).sort(); + const customSlugValue = customSlug.trim(); useEffect(() => { if (open) { - setTimeout(() => inputRef.current?.focus(), 100); + const focusTimer = setTimeout(() => { + inputRef.current?.focus(); + if (!inputRef.current) { + customInputRef.current?.focus(); + } + }, 100); + return () => clearTimeout(focusTimer); } }, [open]); @@ -104,6 +113,12 @@ export function ModelSideSheet({ if (!open) return null; + function handleCustomSlugSubmit(e: React.FormEvent) { + e.preventDefault(); + if (!customSlugValue || isSaving) return; + onSelect(customSlugValue); + } + return (
-
-
- +
+ {models.length > 0 && ( +
+ + setSearch(e.target.value)} + placeholder="Search models..." + className="w-full rounded-lg border border-border bg-background pl-9 pr-3 py-2 text-sm text-foreground placeholder:text-muted outline-none focus:border-foreground/30 transition-colors" + /> +
+ )} +
setSearch(e.target.value)} - placeholder="Search models..." - className="w-full rounded-lg border border-border bg-background pl-9 pr-3 py-2 text-sm text-foreground placeholder:text-muted outline-none focus:border-foreground/30 transition-colors" + value={customSlug} + onChange={(e) => setCustomSlug(e.target.value)} + placeholder="Custom model slug" + aria-label="Custom model slug" + disabled={isSaving} + className="min-w-0 flex-1 rounded-lg border border-border bg-background px-3 py-2 text-sm font-mono text-foreground placeholder:font-sans placeholder:text-muted outline-none focus:border-foreground/30 transition-colors disabled:opacity-50" /> -
+ +
@@ -158,7 +195,6 @@ export function ModelSideSheet({ ) : providers.length === 0 ? (

No models found

-

Try a different search term

) : (
@@ -218,19 +254,21 @@ export function ModelSideSheet({

- {model.contextLength >= 1000 - ? `${(model.contextLength / 1000).toLocaleString()}K` - : model.contextLength.toLocaleString()} + {model.contextLength > 0 + ? model.contextLength >= 1000 + ? `${(model.contextLength / 1000).toLocaleString()}K` + : model.contextLength.toLocaleString() + : "—"}

- ${model.promptCost.toFixed(2)}/1M + {model.promptCost > 0 ? `$${model.promptCost.toFixed(2)}/1M` : "—"}

- ${model.completionCost.toFixed(2)}/1M + {model.completionCost > 0 ? `$${model.completionCost.toFixed(2)}/1M` : "—"}

diff --git a/frontend/components/settings/llm-providers.tsx b/frontend/components/settings/llm-providers.tsx new file mode 100644 index 0000000..31417de --- /dev/null +++ b/frontend/components/settings/llm-providers.tsx @@ -0,0 +1,575 @@ +"use client"; + +import { useState } from "react"; +import { CircleHelp, Plug, TriangleAlert } from "lucide-react"; +import type { LlmProviderType, ServiceSetupStatus } from "@/lib/backend"; + +type LlmProviderCategory = "direct" | "router" | "local" | "custom"; +export type LlmProviderOptionValue = LlmProviderType; + +export type LlmProviderOption = { + value: LlmProviderOptionValue; + provider: LlmProviderType; + label: string; + description: string; + category: LlmProviderCategory; + shortLabel: string; + capability: string; + authLabel: string; + defaultModel: string; + defaultBaseUrl?: string; + requiresBaseUrl?: boolean; + requiresApiKey?: boolean; + apiKeyPlaceholder: string; + helperHref: string; + iconSrc?: string; + wordmarkSrc?: string; +}; + +export const LLM_PROVIDER_GROUPS: { + categories: LlmProviderCategory[]; + label: string; +}[] = [ + { + categories: ["router"], + label: "Router", + }, + { + categories: ["direct"], + label: "Direct", + }, + { + categories: ["local"], + label: "Local", + }, + { + categories: ["custom"], + label: "Custom", + }, +]; + +export const LLM_PROVIDER_OPTIONS: LlmProviderOption[] = [ + { + value: "openai", + provider: "openai", + label: "OpenAI", + description: "Use OpenAI models directly with your API key.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "gpt-5.4-mini", + apiKeyPlaceholder: "sk-...", + helperHref: "https://platform.openai.com/api-keys", + iconSrc: "/logos/providers/openai-icon.svg", + wordmarkSrc: "/logos/providers/openai.svg", + }, + { + value: "anthropic", + provider: "anthropic", + label: "Anthropic", + description: "Use Claude models directly with your API key.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "claude-sonnet-4-6", + apiKeyPlaceholder: "sk-ant-...", + helperHref: "https://console.anthropic.com/settings/keys", + iconSrc: "/logos/providers/anthropic-icon.svg", + wordmarkSrc: "/logos/providers/anthropic.svg", + }, + { + value: "google", + provider: "google", + label: "Google Gemini", + description: "Use Gemini models directly with your Google AI Studio API key.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "gemini-3.5-flash", + apiKeyPlaceholder: "AIza...", + helperHref: "https://aistudio.google.com/app/apikey", + iconSrc: "/logos/providers/google-g.svg", + }, + { + value: "xai", + provider: "xai", + label: "xAI", + description: "Use Grok models directly with your xAI API key.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "grok-4.3", + apiKeyPlaceholder: "xai-...", + helperHref: "https://console.x.ai/", + iconSrc: "/logos/providers/xai.svg", + }, + { + value: "deepseek", + provider: "deepseek", + label: "DeepSeek", + description: "Use DeepSeek chat and reasoning models directly.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "deepseek-chat", + apiKeyPlaceholder: "sk-...", + helperHref: "https://platform.deepseek.com/api_keys", + iconSrc: "/logos/providers/deepseek.svg", + }, + { + value: "qwen", + provider: "qwen", + label: "Qwen", + description: "Use Qwen models through Alibaba Cloud Model Studio.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "qwen-plus", + apiKeyPlaceholder: "sk-...", + helperHref: "https://modelstudio.console.alibabacloud.com/", + iconSrc: "/logos/providers/qwen.svg", + }, + { + value: "mistral", + provider: "mistral", + label: "Mistral AI", + description: "Use Mistral chat and reasoning models directly.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "mistral-large-latest", + apiKeyPlaceholder: "sk-...", + helperHref: "https://console.mistral.ai/api-keys/", + iconSrc: "/logos/providers/mistral-ai.svg", + }, + { + value: "groq", + provider: "groq", + label: "Groq", + description: "Use fast hosted open-weight models through GroqCloud.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "openai/gpt-oss-120b", + apiKeyPlaceholder: "gsk_...", + helperHref: "https://console.groq.com/keys", + iconSrc: "/logos/providers/groq.svg", + }, + { + value: "togetherai", + provider: "togetherai", + label: "Together.ai", + description: "Use Together.ai serverless open-source model hosting.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "Qwen/Qwen3.5-397B-A17B", + apiKeyPlaceholder: "tok_...", + helperHref: "https://api.together.ai/settings/api-keys", + iconSrc: "/logos/providers/together-ai.svg", + }, + { + value: "deepinfra", + provider: "deepinfra", + label: "DeepInfra", + description: "Use DeepInfra's hosted open-source model catalog.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "meta-llama/Llama-3.3-70B-Instruct-Turbo", + apiKeyPlaceholder: "sk-...", + helperHref: "https://deepinfra.com/dash/api_keys", + iconSrc: "/logos/providers/deepinfra.svg", + }, + { + value: "fireworks", + provider: "fireworks", + label: "Fireworks AI", + description: "Use Fireworks-hosted open-weight and fine-tuned models.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "accounts/fireworks/models/kimi-k2p5", + apiKeyPlaceholder: "fw_...", + helperHref: "https://fireworks.ai/account/api-keys", + iconSrc: "/logos/providers/fireworks-ai.svg", + }, + { + value: "huggingface", + provider: "huggingface", + label: "Hugging Face", + description: "Use Hugging Face Inference Providers and routed models.", + category: "direct", + shortLabel: "Direct API", + capability: "Hosted", + authLabel: "API key", + defaultModel: "deepseek-ai/DeepSeek-V3-0324", + apiKeyPlaceholder: "hf_...", + helperHref: "https://huggingface.co/settings/tokens", + iconSrc: "/logos/providers/huggingface.svg", + }, + { + value: "openrouter", + provider: "openrouter", + label: "OpenRouter", + description: "Route across many hosted model families with one account.", + category: "router", + shortLabel: "Model router", + capability: "Multi-provider", + authLabel: "API key or OAuth", + defaultModel: "anthropic/claude-sonnet-4.6", + apiKeyPlaceholder: "sk-or-...", + helperHref: "https://openrouter.ai/settings/keys", + iconSrc: "/logos/providers/openrouter.svg", + wordmarkSrc: "/logos/providers/openrouter-wordmark.svg", + }, + { + value: "ollama", + provider: "ollama", + label: "Ollama", + description: "Use Ollama's local OpenAI-compatible endpoint.", + category: "local", + shortLabel: "Local", + capability: "Local", + authLabel: "No key", + defaultModel: "", + defaultBaseUrl: "http://localhost:11434/v1", + requiresBaseUrl: true, + requiresApiKey: false, + apiKeyPlaceholder: "No key required", + helperHref: "https://github.com/ollama/ollama/blob/main/docs/openai.md", + iconSrc: "/logos/providers/ollama.svg", + }, + { + value: "lmstudio", + provider: "lmstudio", + label: "LM Studio", + description: "Use LM Studio's local OpenAI-compatible server.", + category: "local", + shortLabel: "Local", + capability: "Local", + authLabel: "No key", + defaultModel: "", + defaultBaseUrl: "http://localhost:1234/v1", + requiresBaseUrl: true, + requiresApiKey: false, + apiKeyPlaceholder: "No key required", + helperHref: "https://lmstudio.ai/docs/app/api/endpoints/openai", + iconSrc: "/logos/providers/lmstudio.svg", + }, + { + value: "custom", + provider: "custom", + label: "Custom endpoint", + description: "Use another OpenAI-compatible base URL.", + category: "custom", + shortLabel: "OpenAI-compatible", + capability: "Local or hosted", + authLabel: "Optional key", + defaultModel: "", + requiresBaseUrl: true, + requiresApiKey: false, + apiKeyPlaceholder: "Optional for local endpoints", + helperHref: "https://platform.openai.com/docs/api-reference", + }, +]; + +const EXPERIMENTAL_PROVIDER_VALUES = new Set([ + "openai", + "anthropic", + "google", + "xai", + "deepseek", + "qwen", + "mistral", + "groq", + "togetherai", + "deepinfra", + "fireworks", + "huggingface", + "ollama", + "lmstudio", + "custom", +]); + +const LOCAL_MODEL_PROVIDER_VALUES = new Set([ + "ollama", + "lmstudio", +]); + +function isExperimentalProvider(value: LlmProviderOptionValue) { + return EXPERIMENTAL_PROVIDER_VALUES.has(value); +} + +function isLocalModelProvider(value: LlmProviderOptionValue) { + return LOCAL_MODEL_PROVIDER_VALUES.has(value); +} + +export function llmProviderOption(value: LlmProviderOptionValue) { + return ( + LLM_PROVIDER_OPTIONS.find((option) => option.value === value) ?? + LLM_PROVIDER_OPTIONS.find((option) => option.value === "openrouter") ?? + LLM_PROVIDER_OPTIONS[0] + ); +} + +export function displayBaseUrl(baseUrl?: string) { + return baseUrl?.replace("host.docker.internal", "localhost") ?? ""; +} + +export function localLlmPresetForBaseUrl(baseUrl?: string) { + const displayUrl = displayBaseUrl(baseUrl); + if (!displayUrl) return undefined; + + try { + const parsed = new URL(displayUrl); + if (parsed.port === "11434") { + return llmProviderOption("ollama"); + } + if (parsed.port === "1234") { + return llmProviderOption("lmstudio"); + } + } catch { + if (displayUrl.includes(":11434")) return llmProviderOption("ollama"); + if (displayUrl.includes(":1234")) return llmProviderOption("lmstudio"); + } + + return undefined; +} + +export function llmProviderLabelForStatus(status?: ServiceSetupStatus) { + if (status?.provider === "custom") { + return localLlmPresetForBaseUrl(status.baseUrl)?.label ?? "Custom endpoint"; + } + if (status?.provider) return llmProviderOption(status.provider).label; + return status?.providerLabel; +} + +export function LlmProviderLogo({ + provider, + variant = "wordmark", + className = "", +}: { + provider: LlmProviderOptionValue; + variant?: "icon" | "wordmark"; + className?: string; +}) { + const option = llmProviderOption(provider); + const src = option.iconSrc ?? option.wordmarkSrc; + + if (variant === "icon") { + if (!src) { + return ( + + ); + } + + return ( + {option.label} + ); + } + + if (!src) { + return ( + + + ); + } + + return ( + + + + {option.label} + + + ); +} + +export function LlmProviderBrand({ + provider, + baseUrl, +}: { + provider?: LlmProviderType; + baseUrl?: string; +}) { + if (provider) { + const option = + provider === "custom" + ? localLlmPresetForBaseUrl(baseUrl) ?? llmProviderOption("custom") + : llmProviderOption(provider); + + return ( +
+ +
+ ); + } + + return ( +
+ Model provider +
+ ); +} + +export function LlmProviderSelector({ + value, + onChange, +}: { + value: LlmProviderOptionValue; + onChange: (provider: LlmProviderOptionValue) => void; +}) { + const selectedProviderIsExperimental = isExperimentalProvider(value); + const [showExperimentalProviders, setShowExperimentalProviders] = + useState(false); + const shouldShowExperimentalProviders = + showExperimentalProviders || selectedProviderIsExperimental; + const orderedOptions = LLM_PROVIDER_GROUPS.flatMap((group) => + LLM_PROVIDER_OPTIONS.filter((option) => + group.categories.includes(option.category), + ), + ).filter( + (option) => + shouldShowExperimentalProviders || !isExperimentalProvider(option.value), + ); + + function handleExperimentalChange(checked: boolean) { + setShowExperimentalProviders(checked); + if (!checked && selectedProviderIsExperimental) { + onChange("openrouter"); + } + } + + return ( +
+
+ + + +
+
+
+ {orderedOptions.map((option) => { + const selected = option.value === value; + + return ( + + ); + })} +
+
+
+ ); +} diff --git a/frontend/components/settings/types.ts b/frontend/components/settings/types.ts index e04d2fc..3ef7d1b 100644 --- a/frontend/components/settings/types.ts +++ b/frontend/components/settings/types.ts @@ -16,4 +16,5 @@ export const MODEL_ROLES: ModelRole[] = [ { key: "schemaInference", label: "Schema Inference", description: "Used to generate dataset schema from natural language" }, { key: "populateOrchestrator", label: "Populate Orchestrator", description: "Coordinates row population workflow" }, { key: "investigateSubagent", label: "Investigate Subagent", description: "Researches individual entities" }, -]; \ No newline at end of file + { key: "extractorBuilder", label: "Extractor Builder", description: "Writes reusable Playwright extractors from browser probes" }, +]; diff --git a/frontend/components/table/types.ts b/frontend/components/table/types.ts index 6eff009..119156f 100644 --- a/frontend/components/table/types.ts +++ b/frontend/components/table/types.ts @@ -29,5 +29,6 @@ export interface DatasetRow { _creationTime: number; data: Record; sources?: string[]; + cellSources?: Record; updateStatus?: "pending"; } diff --git a/frontend/convex/_generated/api.d.ts b/frontend/convex/_generated/api.d.ts index bc8dc15..b09c911 100644 --- a/frontend/convex/_generated/api.d.ts +++ b/frontend/convex/_generated/api.d.ts @@ -8,6 +8,7 @@ * @module */ +import type * as datasetExtractors from "../datasetExtractors.js"; import type * as datasetRows from "../datasetRows.js"; import type * as datasets from "../datasets.js"; import type * as lib_authz from "../lib/authz.js"; @@ -27,6 +28,7 @@ import type { } from "convex/server"; declare const fullApi: ApiFromModules<{ + datasetExtractors: typeof datasetExtractors; datasetRows: typeof datasetRows; datasets: typeof datasets; "lib/authz": typeof lib_authz; diff --git a/frontend/convex/datasetExtractors.ts b/frontend/convex/datasetExtractors.ts new file mode 100644 index 0000000..e4a365a --- /dev/null +++ b/frontend/convex/datasetExtractors.ts @@ -0,0 +1,108 @@ +import { internalMutation, internalQuery } from "./_generated/server.js"; +import { v } from "convex/values"; + +export const getActive = internalQuery({ + args: { + datasetId: v.id("datasets"), + siteKey: v.string(), + columnsHash: v.string(), + }, + handler: async (ctx, args) => { + const extractor = await ctx.db + .query("datasetExtractors") + .withIndex("by_dataset_site", (q) => + q.eq("datasetId", args.datasetId).eq("siteKey", args.siteKey), + ) + .first(); + + if ( + !extractor || + extractor.columnsHash !== args.columnsHash || + extractor.status !== "active" + ) { + return null; + } + + return extractor; + }, +}); + +export const upsert = internalMutation({ + args: { + datasetId: v.id("datasets"), + siteKey: v.string(), + columnsHash: v.string(), + script: v.string(), + model: v.optional(v.string()), + probeSummary: v.optional(v.string()), + }, + handler: async (ctx, args) => { + const dataset = await ctx.db.get(args.datasetId); + if (!dataset) throw new Error("Dataset not found"); + + const now = Date.now(); + const existing = await ctx.db + .query("datasetExtractors") + .withIndex("by_dataset_site", (q) => + q.eq("datasetId", args.datasetId).eq("siteKey", args.siteKey), + ) + .first(); + + const patch = { + columnsHash: args.columnsHash, + script: args.script, + status: "active" as const, + model: args.model, + probeSummary: args.probeSummary, + lastError: undefined, + updatedAt: now, + }; + + if (existing) { + await ctx.db.patch(existing._id, patch); + return { id: existing._id, updatedAt: now }; + } + + const id = await ctx.db.insert("datasetExtractors", { + datasetId: args.datasetId, + siteKey: args.siteKey, + createdAt: now, + ...patch, + }); + return { id, updatedAt: now }; + }, +}); + +export const markFailed = internalMutation({ + args: { + datasetId: v.id("datasets"), + siteKey: v.string(), + columnsHash: v.string(), + script: v.string(), + updatedAt: v.number(), + error: v.string(), + }, + handler: async (ctx, args) => { + const existing = await ctx.db + .query("datasetExtractors") + .withIndex("by_dataset_site", (q) => + q.eq("datasetId", args.datasetId).eq("siteKey", args.siteKey), + ) + .first(); + if ( + !existing || + existing.columnsHash !== args.columnsHash || + existing.script !== args.script || + existing.updatedAt !== args.updatedAt + ) { + return null; + } + + await ctx.db.patch(existing._id, { + status: "failed", + lastError: args.error.slice(0, 1_000), + updatedAt: Date.now(), + }); + return existing._id; + }, +}); diff --git a/frontend/convex/datasetRows.ts b/frontend/convex/datasetRows.ts index a4a877c..29974a6 100644 --- a/frontend/convex/datasetRows.ts +++ b/frontend/convex/datasetRows.ts @@ -82,6 +82,7 @@ export const insert = internalMutation({ datasetId: v.id("datasets"), data: v.record(v.string(), v.any()), sources: v.optional(v.array(v.string())), + cellSources: v.optional(v.record(v.string(), v.array(v.string()))), rowSummary: v.optional(v.string()), howFound: v.optional(v.string()), }, @@ -167,6 +168,7 @@ export const update = internalMutation({ expectedDatasetId: v.id("datasets"), data: v.record(v.string(), v.any()), sources: v.optional(v.array(v.string())), + cellSources: v.optional(v.record(v.string(), v.array(v.string()))), rowSummary: v.optional(v.string()), howFound: v.optional(v.string()), }, @@ -199,6 +201,7 @@ export const update = internalMutation({ updateStatus: undefined, }; if (args.sources !== undefined) patch.sources = args.sources; + if (args.cellSources !== undefined) patch.cellSources = args.cellSources; if (args.rowSummary !== undefined) patch.rowSummary = args.rowSummary; if (args.howFound !== undefined) patch.howFound = args.howFound; await ctx.db.patch(args.id, patch); diff --git a/frontend/convex/datasets.ts b/frontend/convex/datasets.ts index 791671b..2de4b6b 100644 --- a/frontend/convex/datasets.ts +++ b/frontend/convex/datasets.ts @@ -31,6 +31,37 @@ const columnValidator = v.object({ ), description: v.optional(v.string()), isPrimaryKey: v.optional(v.boolean()), + nullable: v.optional(v.boolean()), + validationRegex: v.optional(v.string()), + normalizationHint: v.optional(v.string()), +}); + +const codificationProfileValidator = v.object({ + version: v.literal(1), + mode: v.union( + v.literal("disabled"), + v.literal("candidate"), + v.literal("required"), + v.literal("unknown"), + ), + reason: v.string(), + primaryKeyShape: v.union( + v.literal("url"), + v.literal("slug"), + v.literal("name"), + v.literal("id"), + v.literal("mixed"), + v.literal("unknown"), + ), + families: v.array( + v.object({ + label: v.string(), + sourceHost: v.optional(v.string()), + sourcePathPrefix: v.optional(v.string()), + urlTemplate: v.optional(v.string()), + primaryKeyRegex: v.optional(v.string()), + }), + ), }); function refreshCadenceFromLegacyLabel( @@ -282,6 +313,9 @@ export const claimScheduledRefreshInternal = internalMutation({ datasetName: dataset.name, description: dataset.description, columns: dataset.columns, + retrievalStrategy: dataset.retrievalStrategy, + sourceHint: dataset.sourceHint, + codificationProfile: dataset.codificationProfile, ownerId: dataset.ownerId, maxRowCount: dataset.maxRowCount ?? DEFAULT_MAX_ROW_COUNT, }, @@ -292,11 +326,15 @@ export const claimScheduledRefreshInternal = internalMutation({ export const completeScheduledRefreshInternal = internalMutation({ args: { id: v.id("datasets"), + runId: v.string(), now: v.number(), }, handler: async (ctx, args) => { const dataset = await ctx.db.get(args.id); if (!dataset) return { outcome: "not_found" as const }; + if (dataset.lastRefreshRunId !== args.runId) { + return { outcome: "stale_run" as const }; + } const refreshCadence = dataset.refreshCadence ?? "manual"; await ctx.db.patch(dataset._id, { @@ -304,6 +342,7 @@ export const completeScheduledRefreshInternal = internalMutation({ lastStatusError: undefined, lastRefreshAt: args.now, lastRefreshStartedAt: undefined, + lastRefreshRunId: undefined, nextRefreshAt: nextRefreshAtFor(refreshCadence, args.now), }); return { outcome: "completed" as const }; @@ -313,18 +352,23 @@ export const completeScheduledRefreshInternal = internalMutation({ export const failScheduledRefreshInternal = internalMutation({ args: { id: v.id("datasets"), + runId: v.string(), now: v.number(), lastStatusError: v.string(), }, handler: async (ctx, args) => { const dataset = await ctx.db.get(args.id); if (!dataset) return { outcome: "not_found" as const }; + if (dataset.lastRefreshRunId !== args.runId) { + return { outcome: "stale_run" as const }; + } const refreshCadence = dataset.refreshCadence ?? "manual"; await ctx.db.patch(dataset._id, { status: "failed", lastStatusError: args.lastStatusError, lastRefreshStartedAt: undefined, + lastRefreshRunId: undefined, nextRefreshAt: nextRefreshAtFor(refreshCadence, args.now), }); return { outcome: "failed" as const }; @@ -384,6 +428,7 @@ export const create = mutation({ ) ), sourceHint: v.optional(v.string()), + codificationProfile: v.optional(codificationProfileValidator), }, handler: async (ctx, args) => { const identity = await requireIdentity(ctx); @@ -422,6 +467,7 @@ export const createForOwnerInternal = internalMutation({ ) ), sourceHint: v.optional(v.string()), + codificationProfile: v.optional(codificationProfileValidator), }, handler: async (ctx, args) => { assertNotReservedOwner(args.ownerId); @@ -462,6 +508,21 @@ export const getOwnedInternal = internalQuery({ }, }); +export const setCodificationProfileInternal = internalMutation({ + args: { + id: v.id("datasets"), + codificationProfile: codificationProfileValidator, + }, + handler: async (ctx, args) => { + const dataset = await ctx.db.get(args.id); + if (!dataset) return { outcome: "not_found" as const }; + await ctx.db.patch(dataset._id, { + codificationProfile: args.codificationProfile, + }); + return { outcome: "updated" as const }; + }, +}); + export const updateRefreshSettings = mutation({ args: { id: v.id("datasets"), @@ -557,6 +618,15 @@ export const remove = mutation({ for (const row of rows) { await ctx.db.delete(row._id); } + + const extractors = await ctx.db + .query("datasetExtractors") + .withIndex("by_dataset_site", (q) => q.eq("datasetId", dataset._id)) + .collect(); + for (const extractor of extractors) { + await ctx.db.delete(extractor._id); + } + await ctx.db.delete(dataset._id); }, }); diff --git a/frontend/convex/localCredentials.ts b/frontend/convex/localCredentials.ts index d8e8c97..6c8f2db 100644 --- a/frontend/convex/localCredentials.ts +++ b/frontend/convex/localCredentials.ts @@ -3,7 +3,23 @@ import { v } from "convex/values"; const serviceValidator = v.union( v.literal("tinyfish"), + v.literal("llm"), v.literal("openrouter"), + v.literal("openai"), + v.literal("anthropic"), + v.literal("google"), + v.literal("xai"), + v.literal("deepseek"), + v.literal("qwen"), + v.literal("mistral"), + v.literal("groq"), + v.literal("togetherai"), + v.literal("deepinfra"), + v.literal("fireworks"), + v.literal("huggingface"), + v.literal("ollama"), + v.literal("lmstudio"), + v.literal("custom"), ); const connectionMethodValidator = v.union( @@ -11,6 +27,25 @@ const connectionMethodValidator = v.union( v.literal("oauth"), ); +const llmProviderValidator = v.union( + v.literal("openrouter"), + v.literal("openai"), + v.literal("anthropic"), + v.literal("google"), + v.literal("xai"), + v.literal("deepseek"), + v.literal("qwen"), + v.literal("mistral"), + v.literal("groq"), + v.literal("togetherai"), + v.literal("deepinfra"), + v.literal("fireworks"), + v.literal("huggingface"), + v.literal("ollama"), + v.literal("lmstudio"), + v.literal("custom"), +); + export const getInternal = internalQuery({ args: { service: serviceValidator }, handler: async (ctx, args) => { @@ -24,9 +59,12 @@ export const getInternal = internalQuery({ export const upsertInternal = internalMutation({ args: { service: serviceValidator, - keychainAccount: v.string(), + keychainAccount: v.optional(v.union(v.string(), v.null())), connectionMethod: connectionMethodValidator, verifiedAt: v.number(), + llmProvider: v.optional(llmProviderValidator), + llmBaseUrl: v.optional(v.string()), + llmDefaultModel: v.optional(v.string()), }, handler: async (ctx, args) => { const existing = await ctx.db @@ -34,21 +72,52 @@ export const upsertInternal = internalMutation({ .withIndex("by_service", (q) => q.eq("service", args.service)) .unique(); + const keychainAccountPatch = + args.keychainAccount !== undefined + ? { keychainAccount: args.keychainAccount ?? undefined } + : {}; + const keychainAccountInsert = + typeof args.keychainAccount === "string" + ? { keychainAccount: args.keychainAccount } + : {}; + const update = { - keychainAccount: args.keychainAccount, connectionMethod: args.connectionMethod, verifiedAt: args.verifiedAt, updatedAt: Date.now(), }; + const llmPatch = args.llmProvider !== undefined + ? { + llmProvider: args.llmProvider, + // Explicit undefined clears stale custom-provider values when the + // user switches back to OpenAI/Anthropic/OpenRouter. + llmBaseUrl: args.llmBaseUrl, + llmDefaultModel: args.llmDefaultModel, + } + : {}; + const llmInsert = args.llmProvider !== undefined + ? { + llmProvider: args.llmProvider, + ...(args.llmBaseUrl !== undefined ? { llmBaseUrl: args.llmBaseUrl } : {}), + ...(args.llmDefaultModel !== undefined ? { llmDefaultModel: args.llmDefaultModel } : {}), + } + : {}; if (existing) { - await ctx.db.patch(existing._id, { ...update, apiKey: undefined }); + await ctx.db.patch(existing._id, { + ...keychainAccountPatch, + ...update, + ...llmPatch, + apiKey: undefined, + }); return existing._id; } return await ctx.db.insert("localCredentials", { service: args.service, + ...keychainAccountInsert, ...update, + ...llmInsert, }); }, }); diff --git a/frontend/convex/modelConfig.ts b/frontend/convex/modelConfig.ts index e10fd3d..a6223ea 100644 --- a/frontend/convex/modelConfig.ts +++ b/frontend/convex/modelConfig.ts @@ -1,104 +1,180 @@ import { query, mutation, internalQuery, internalMutation } from "./_generated/server.js"; +import type { MutationCtx, QueryCtx } from "./_generated/server.js"; import { v } from "convex/values"; import { getIdentity } from "./lib/authz.js"; +type LlmProvider = + | "openrouter" + | "openai" + | "anthropic" + | "google" + | "xai" + | "deepseek" + | "qwen" + | "mistral" + | "groq" + | "togetherai" + | "deepinfra" + | "fireworks" + | "huggingface" + | "ollama" + | "lmstudio" + | "custom"; + +const providerValidator = v.union( + v.literal("openrouter"), + v.literal("openai"), + v.literal("anthropic"), + v.literal("google"), + v.literal("xai"), + v.literal("deepseek"), + v.literal("qwen"), + v.literal("mistral"), + v.literal("groq"), + v.literal("togetherai"), + v.literal("deepinfra"), + v.literal("fireworks"), + v.literal("huggingface"), + v.literal("ollama"), + v.literal("lmstudio"), + v.literal("custom"), +); + +async function findProviderConfig( + ctx: QueryCtx | MutationCtx, + userId: string, + provider: LlmProvider, +) { + const providerRow = await ctx.db + .query("modelConfig") + .withIndex("by_user_provider", (q) => + q.eq("userId", userId).eq("provider", provider), + ) + .first(); + + if (providerRow) return providerRow; + + if (provider === "openrouter") { + return await ctx.db + .query("modelConfig") + .withIndex("by_user", (q) => q.eq("userId", userId)) + .filter((q) => q.eq(q.field("provider"), undefined)) + .first(); + } + + return null; +} + export const get = query({ - args: {}, - handler: async (ctx) => { + args: { provider: v.optional(providerValidator) }, + handler: async (ctx, args) => { const identity = await getIdentity(ctx); if (!identity) return null; - const existing = await ctx.db - .query("modelConfig") - .withIndex("by_user", (q) => q.eq("userId", identity.subject)) - .first(); - return existing ?? null; + return await findProviderConfig( + ctx, + identity.subject, + args.provider ?? "openrouter", + ); }, }); /** - * Upsert one or more model preferences for the authenticated user. + * Upsert one or more model preferences for the authenticated user and provider. * * Only fields that are explicitly provided (not undefined) are updated. * Unset fields retain their existing database values. - * - * Example: sending only { schemaInference: "x" } will update schemaInference - * while leaving populateOrchestrator and investigateSubagent untouched. */ export const upsert = mutation({ args: { + provider: v.optional(providerValidator), schemaInference: v.optional(v.string()), populateOrchestrator: v.optional(v.string()), investigateSubagent: v.optional(v.string()), + extractorBuilder: v.optional(v.string()), + rowExtractorConcurrency: v.optional(v.number()), + rowExtractorBrowserAttempts: v.optional(v.number()), }, handler: async (ctx, args) => { const identity = await getIdentity(ctx); if (!identity) throw new Error("Not authenticated"); - const existing = await ctx.db - .query("modelConfig") - .withIndex("by_user", (q) => q.eq("userId", identity.subject)) - .first(); + const provider = args.provider ?? "openrouter"; + const existing = await findProviderConfig(ctx, identity.subject, provider); + + const patch: { + provider?: LlmProvider; + schemaInference?: string; + populateOrchestrator?: string; + investigateSubagent?: string; + extractorBuilder?: string; + rowExtractorConcurrency?: number; + rowExtractorBrowserAttempts?: number; + } = { provider }; + if (args.schemaInference !== undefined) patch.schemaInference = args.schemaInference; + if (args.populateOrchestrator !== undefined) patch.populateOrchestrator = args.populateOrchestrator; + if (args.investigateSubagent !== undefined) patch.investigateSubagent = args.investigateSubagent; + if (args.extractorBuilder !== undefined) patch.extractorBuilder = args.extractorBuilder; + if (args.rowExtractorConcurrency !== undefined) patch.rowExtractorConcurrency = args.rowExtractorConcurrency; + if (args.rowExtractorBrowserAttempts !== undefined) patch.rowExtractorBrowserAttempts = args.rowExtractorBrowserAttempts; if (existing) { - // Partial update — only touch fields that were explicitly provided. - // Omitting a field preserves its current database value. - const patch: Record = {}; - if (args.schemaInference !== undefined) patch.schemaInference = args.schemaInference; - if (args.populateOrchestrator !== undefined) patch.populateOrchestrator = args.populateOrchestrator; - if (args.investigateSubagent !== undefined) patch.investigateSubagent = args.investigateSubagent; await ctx.db.patch(existing._id, patch); } else { - // First-time save — build insert object from provided fields only. - // userId is always required and comes from the authenticated identity. - const insert: { - userId: string; - schemaInference?: string; - populateOrchestrator?: string; - investigateSubagent?: string; - } = { userId: identity.subject }; - if (args.schemaInference !== undefined) insert.schemaInference = args.schemaInference; - if (args.populateOrchestrator !== undefined) insert.populateOrchestrator = args.populateOrchestrator; - if (args.investigateSubagent !== undefined) insert.investigateSubagent = args.investigateSubagent; - await ctx.db.insert("modelConfig", insert); + await ctx.db.insert("modelConfig", { + userId: identity.subject, + ...patch, + }); } }, }); export const getInternal = internalQuery({ - args: { userId: v.string() }, + args: { userId: v.string(), provider: v.optional(providerValidator) }, handler: async (ctx, args) => { - const existing = await ctx.db - .query("modelConfig") - .withIndex("by_user", (q) => q.eq("userId", args.userId)) - .first(); - return existing ?? null; + return await findProviderConfig( + ctx, + args.userId, + args.provider ?? "openrouter", + ); }, }); /** - * Upsert model preferences for a specific user (internal, backend-only). + * Upsert model preferences for a specific user/provider (internal, backend-only). * * Only fields that are explicitly provided (not undefined) are updated. - * Unset fields are omitted from the insert, leaving the database unchanged. */ export const upsertInternal = internalMutation({ args: { userId: v.string(), + provider: v.optional(providerValidator), schemaInference: v.optional(v.string()), populateOrchestrator: v.optional(v.string()), investigateSubagent: v.optional(v.string()), + extractorBuilder: v.optional(v.string()), + rowExtractorConcurrency: v.optional(v.number()), + rowExtractorBrowserAttempts: v.optional(v.number()), }, handler: async (ctx, args) => { - const existing = await ctx.db - .query("modelConfig") - .withIndex("by_user", (q) => q.eq("userId", args.userId)) - .first(); + const provider = args.provider ?? "openrouter"; + const existing = await findProviderConfig(ctx, args.userId, provider); - const patch: Record = {}; + const patch: { + provider: LlmProvider; + schemaInference?: string; + populateOrchestrator?: string; + investigateSubagent?: string; + extractorBuilder?: string; + rowExtractorConcurrency?: number; + rowExtractorBrowserAttempts?: number; + } = { provider }; if (args.schemaInference !== undefined) patch.schemaInference = args.schemaInference; if (args.populateOrchestrator !== undefined) patch.populateOrchestrator = args.populateOrchestrator; if (args.investigateSubagent !== undefined) patch.investigateSubagent = args.investigateSubagent; + if (args.extractorBuilder !== undefined) patch.extractorBuilder = args.extractorBuilder; + if (args.rowExtractorConcurrency !== undefined) patch.rowExtractorConcurrency = args.rowExtractorConcurrency; + if (args.rowExtractorBrowserAttempts !== undefined) patch.rowExtractorBrowserAttempts = args.rowExtractorBrowserAttempts; if (existing) { await ctx.db.patch(existing._id, patch); diff --git a/frontend/convex/schema.ts b/frontend/convex/schema.ts index 83ea007..8588edc 100644 --- a/frontend/convex/schema.ts +++ b/frontend/convex/schema.ts @@ -65,6 +65,38 @@ export default defineSchema({ ), description: v.optional(v.string()), isPrimaryKey: v.optional(v.boolean()), + nullable: v.optional(v.boolean()), + validationRegex: v.optional(v.string()), + normalizationHint: v.optional(v.string()), + }) + ), + codificationProfile: v.optional( + v.object({ + version: v.literal(1), + mode: v.union( + v.literal("disabled"), + v.literal("candidate"), + v.literal("required"), + v.literal("unknown") + ), + reason: v.string(), + primaryKeyShape: v.union( + v.literal("url"), + v.literal("slug"), + v.literal("name"), + v.literal("id"), + v.literal("mixed"), + v.literal("unknown") + ), + families: v.array( + v.object({ + label: v.string(), + sourceHost: v.optional(v.string()), + sourcePathPrefix: v.optional(v.string()), + urlTemplate: v.optional(v.string()), + primaryKeyRegex: v.optional(v.string()), + }) + ), }) ), retrievalStrategy: v.optional( @@ -85,6 +117,7 @@ export default defineSchema({ datasetId: v.id("datasets"), data: v.record(v.string(), v.any()), sources: v.optional(v.array(v.string())), + cellSources: v.optional(v.record(v.string(), v.array(v.string()))), rowSummary: v.optional(v.string()), howFound: v.optional(v.string()), updateStatus: v.optional(v.literal("pending")), @@ -135,17 +168,100 @@ export default defineSchema({ modelConfig: defineTable({ userId: v.string(), + provider: v.optional( + v.union( + v.literal("openrouter"), + v.literal("openai"), + v.literal("anthropic"), + v.literal("google"), + v.literal("xai"), + v.literal("deepseek"), + v.literal("qwen"), + v.literal("mistral"), + v.literal("groq"), + v.literal("togetherai"), + v.literal("deepinfra"), + v.literal("fireworks"), + v.literal("huggingface"), + v.literal("ollama"), + v.literal("lmstudio"), + v.literal("custom") + ) + ), schemaInference: v.optional(v.string()), populateOrchestrator: v.optional(v.string()), investigateSubagent: v.optional(v.string()), - }).index("by_user", ["userId"]), + extractorBuilder: v.optional(v.string()), + rowExtractorConcurrency: v.optional(v.number()), + rowExtractorBrowserAttempts: v.optional(v.number()), + }) + .index("by_user", ["userId"]) + .index("by_user_provider", ["userId", "provider"]), + + datasetExtractors: defineTable({ + datasetId: v.id("datasets"), + siteKey: v.string(), + columnsHash: v.string(), + script: v.string(), + status: v.union(v.literal("active"), v.literal("failed")), + model: v.optional(v.string()), + probeSummary: v.optional(v.string()), + lastError: v.optional(v.string()), + createdAt: v.number(), + updatedAt: v.number(), + }).index("by_dataset_site", ["datasetId", "siteKey"]), localCredentials: defineTable({ - service: v.union(v.literal("tinyfish"), v.literal("openrouter")), + service: v.union( + v.literal("tinyfish"), + v.literal("llm"), + v.literal("openrouter"), + v.literal("openai"), + v.literal("anthropic"), + v.literal("google"), + v.literal("xai"), + v.literal("deepseek"), + v.literal("qwen"), + v.literal("mistral"), + v.literal("groq"), + v.literal("togetherai"), + v.literal("deepinfra"), + v.literal("fireworks"), + v.literal("huggingface"), + v.literal("ollama"), + v.literal("lmstudio"), + v.literal("custom") + ), keychainAccount: v.optional(v.string()), connectionMethod: v.union(v.literal("api_key"), v.literal("oauth")), verifiedAt: v.number(), updatedAt: v.number(), + // For service:"llm" this stores the active local LLM provider. + // Provider-specific rows store each provider's keychain account and + // optional custom base URL so users can switch providers without + // re-entering keys. + llmProvider: v.optional( + v.union( + v.literal("openrouter"), + v.literal("openai"), + v.literal("anthropic"), + v.literal("google"), + v.literal("xai"), + v.literal("deepseek"), + v.literal("qwen"), + v.literal("mistral"), + v.literal("groq"), + v.literal("togetherai"), + v.literal("deepinfra"), + v.literal("fireworks"), + v.literal("huggingface"), + v.literal("ollama"), + v.literal("lmstudio"), + v.literal("custom") + ) + ), + llmBaseUrl: v.optional(v.string()), + llmDefaultModel: v.optional(v.string()), // Legacy only: accepted so the migration can deploy, then cleared by the // backend startup purge. New code never writes this field. apiKey: v.optional(v.string()), diff --git a/frontend/lib/backend.ts b/frontend/lib/backend.ts index f5ea3e5..d0abf47 100644 --- a/frontend/lib/backend.ts +++ b/frontend/lib/backend.ts @@ -2,9 +2,42 @@ export interface InferredSchema { dataset_name: string; description: string; columns: InferredColumn[]; - primary_key: string; + primary_key: string[]; retrieval_strategy: "search_fetch" | "browser" | "hybrid"; source_hint: string; + codification_profile: InferredCodificationProfile; +} + +export interface InferredCodificationProfile { + version: 1; + mode: "disabled" | "candidate" | "required" | "unknown"; + reason: string; + primary_key_shape: "url" | "slug" | "name" | "id" | "mixed" | "unknown"; + families: InferredCodificationFamily[]; +} + +export interface InferredCodificationFamily { + label: string; + source_host?: string; + source_path_prefix?: string; + url_template?: string; + primary_key_regex?: string; +} + +export interface CodificationProfile { + version: 1; + mode: "disabled" | "candidate" | "required" | "unknown"; + reason: string; + primaryKeyShape: "url" | "slug" | "name" | "id" | "mixed" | "unknown"; + families: CodificationFamily[]; +} + +export interface CodificationFamily { + label: string; + sourceHost?: string; + sourcePathPrefix?: string; + urlTemplate?: string; + primaryKeyRegex?: string; } export interface InferredColumn { @@ -15,6 +48,8 @@ export interface InferredColumn { is_enumerable: boolean; retrieval_hint: string; nullable: boolean; + validation_regex?: string; + normalization_hint?: string; } export interface PopulateColumn { @@ -22,6 +57,24 @@ export interface PopulateColumn { type: "text" | "number" | "boolean" | "url" | "date"; description?: string; isPrimaryKey?: boolean; + nullable?: boolean; + validationRegex?: string; + normalizationHint?: string; +} + +export interface FinalizeSchemaColumn { + name: string; + type: PopulateColumn["type"]; + description?: string; + isPrimaryKey?: boolean; +} + +export interface FinalizeSchemaInput { + prompt: string; + datasetName?: string; + columns: FinalizeSchemaColumn[]; + retrievalStrategy?: InferredSchema["retrieval_strategy"]; + sourceHint?: string; } export interface PopulateStartResult { @@ -36,23 +89,29 @@ export interface WorkflowResult { /** * The effective model config — always complete, never null. - * schemaInference / populateOrchestrator / investigateSubagent are always strings + * schemaInference / populateOrchestrator / investigateSubagent / extractorBuilder are always strings * (user preference or system default from env). */ export interface EffectiveModelConfig { schemaInference: string; populateOrchestrator: string; investigateSubagent: string; + extractorBuilder: string; + rowExtractorConcurrency: number; + rowExtractorBrowserAttempts: number; } /** - * User's saved model preferences — stores the canonical slug (e.g. "anthropic/claude-sonnet-4.6") + * User's saved model preferences — stores the provider model id (e.g. "openai/gpt-5.4-mini" or "gpt-5.4-mini") * for each agent role. Null means no preference saved — backend will use the env default. */ export interface SavedModelConfig { schemaInference: string | null; populateOrchestrator: string | null; investigateSubagent: string | null; + extractorBuilder: string | null; + rowExtractorConcurrency: number | null; + rowExtractorBrowserAttempts: number | null; } export interface OpenRouterModel { @@ -63,11 +122,33 @@ export interface OpenRouterModel { promptCost: number; } +export type LlmProviderType = + | "openrouter" + | "openai" + | "anthropic" + | "google" + | "xai" + | "deepseek" + | "qwen" + | "mistral" + | "groq" + | "togetherai" + | "deepinfra" + | "fireworks" + | "huggingface" + | "ollama" + | "lmstudio" + | "custom"; + export interface ServiceSetupStatus { configured: boolean; source: "local" | "env" | null; connectionMethod: "api_key" | "oauth" | null; verifiedAt: number | null; + provider?: LlmProviderType; + providerLabel?: string; + baseUrl?: string; + defaultModel?: string; } export interface LocalSetupStatus { @@ -76,12 +157,58 @@ export interface LocalSetupStatus { complete: boolean; services: { tinyfish: ServiceSetupStatus; - openrouter: ServiceSetupStatus; + llm: ServiceSetupStatus; + llmProviders?: Record; + openrouter?: ServiceSetupStatus; }; } const BACKEND_URL = process.env.NEXT_PUBLIC_BACKEND_URL || "http://localhost:3501"; +const DEFAULT_ROW_EXTRACTOR_CONCURRENCY = 5; +const DEFAULT_ROW_EXTRACTOR_BROWSER_ATTEMPTS = 2; + +function normalizeIntegerSetting( + value: unknown, + fallback: number, + min: number, + max: number, +): number { + const parsed = typeof value === "number" ? value : Number(value); + if (!Number.isFinite(parsed)) return fallback; + return Math.min(max, Math.max(min, Math.trunc(parsed))); +} + +function normalizeEffectiveModelConfig( + config: Partial | null | undefined, +): EffectiveModelConfig { + return { + schemaInference: + typeof config?.schemaInference === "string" ? config.schemaInference : "", + populateOrchestrator: + typeof config?.populateOrchestrator === "string" + ? config.populateOrchestrator + : "", + investigateSubagent: + typeof config?.investigateSubagent === "string" + ? config.investigateSubagent + : "", + extractorBuilder: + typeof config?.extractorBuilder === "string" ? config.extractorBuilder : "", + rowExtractorConcurrency: normalizeIntegerSetting( + config?.rowExtractorConcurrency, + DEFAULT_ROW_EXTRACTOR_CONCURRENCY, + 1, + 100, + ), + rowExtractorBrowserAttempts: normalizeIntegerSetting( + config?.rowExtractorBrowserAttempts, + DEFAULT_ROW_EXTRACTOR_BROWSER_ATTEMPTS, + 1, + 10, + ), + }; +} async function errorMessage(res: Response): Promise { const body = await res.json().catch(() => null); @@ -116,13 +243,16 @@ export async function saveTinyFishApiKey( return res.json(); } -export async function saveOpenRouterApiKey( - apiKey: string, -): Promise { - const res = await fetch(`${BACKEND_URL}/local-setup/openrouter-key`, { +export async function saveLlmProviderConfig(config: { + provider: LlmProviderType; + apiKey?: string; + defaultModel: string; + baseUrl?: string; +}): Promise { + const res = await fetch(`${BACKEND_URL}/local-setup/llm-provider`, { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ apiKey }), + body: JSON.stringify(config), }); if (!res.ok) { @@ -132,6 +262,16 @@ export async function saveOpenRouterApiKey( return res.json(); } +export async function saveOpenRouterApiKey( + apiKey: string, +): Promise { + return saveLlmProviderConfig({ + provider: "openrouter", + apiKey, + defaultModel: "anthropic/claude-sonnet-4.6", + }); +} + export async function exchangeOpenRouterOAuth( code: string, codeVerifier: string, @@ -177,7 +317,7 @@ export async function getModelConfig(token: string): Promise { return data.models ?? []; } +export async function getLlmProviderModels(): Promise { + const res = await fetch(`${BACKEND_URL}/llm-provider/models`, { + method: "GET", + }); + + if (!res.ok) { + const body = await res.json().catch(() => null); + const message = body?.error || `Backend error (${res.status})`; + throw new Error(message); + } + + const data = await res.json(); + return data.models ?? []; +} + /** * Refresh the OpenRouter model cache by fetching the latest list from the * OpenRouter API and storing it in Convex. @@ -290,6 +445,28 @@ export async function inferSchema( return res.json(); } +export async function finalizeSchema( + input: FinalizeSchemaInput, + token: string, +): Promise { + const res = await fetch(`${BACKEND_URL}/finalize-schema`, { + method: "POST", + headers: { + "Content-Type": "application/json", + Authorization: `Bearer ${token}`, + }, + body: JSON.stringify(input), + }); + + if (!res.ok) { + const body = await res.json().catch(() => null); + const message = body?.error || `Backend error (${res.status})`; + throw new Error(message); + } + + return res.json(); +} + export async function populate( datasetId: string, datasetName: string, diff --git a/frontend/lib/openrouter-oauth.ts b/frontend/lib/openrouter-oauth.ts index 14dc14c..3d29980 100644 --- a/frontend/lib/openrouter-oauth.ts +++ b/frontend/lib/openrouter-oauth.ts @@ -1,6 +1,20 @@ +import { useSyncExternalStore } from "react"; + export const OPENROUTER_VERIFIER_KEY = "bigset:openrouter-code-verifier"; export const OPENROUTER_RETURN_TO_KEY = "bigset:openrouter-return-to"; +const LOCAL_HOST_SUFFIXES = [ + ".home", + ".home.arpa", + ".internal", + ".lan", + ".local", + ".localhost", + ".localdomain", +]; + +const LOCAL_HOSTNAMES = new Set(["localhost", "0.0.0.0"]); + function base64Url(bytes: ArrayBuffer): string { let binary = ""; for (const byte of new Uint8Array(bytes)) { @@ -30,7 +44,83 @@ function safeReturnTo(returnTo: string): string { return returnTo; } +function normalizedHostname(hostname: string): string { + return hostname + .trim() + .toLowerCase() + .replace(/^\[(.*)\]$/, "$1") + .replace(/\.$/, ""); +} + +function isLocalIpv4Hostname(hostname: string): boolean { + const parts = hostname.split("."); + if (parts.length !== 4) return false; + + const octets = parts.map((part) => { + if (!/^\d{1,3}$/.test(part)) return Number.NaN; + const value = Number(part); + return value >= 0 && value <= 255 ? value : Number.NaN; + }); + if (octets.some(Number.isNaN)) return false; + + const [first, second] = octets; + if (first === 0 || first === 10 || first === 127 || first === 192) { + return true; + } + if (first === 100 && second >= 64 && second <= 127) return true; + if (first === 169 && second === 254) return true; + if (first === 172 && second >= 16 && second <= 31) return true; + if (first === 198 && (second === 18 || second === 19)) return true; + + return false; +} + +function isLocalIpv6Hostname(hostname: string): boolean { + if (!hostname.includes(":")) return false; + if (hostname === "::1" || hostname === "0:0:0:0:0:0:0:1") return true; + + const firstSegment = Number.parseInt(hostname.split(":")[0] || "0", 16); + if (Number.isNaN(firstSegment)) return false; + + return (firstSegment & 0xfe00) === 0xfc00 || (firstSegment & 0xffc0) === 0xfe80; +} + +export function isLocalOpenRouterOAuthHostname(hostname: string): boolean { + const normalized = normalizedHostname(hostname); + if (!normalized) return true; + if (LOCAL_HOSTNAMES.has(normalized)) return true; + if (LOCAL_HOST_SUFFIXES.some((suffix) => normalized.endsWith(suffix))) { + return true; + } + if (!normalized.includes(".") && !normalized.includes(":")) return true; + + return isLocalIpv4Hostname(normalized) || isLocalIpv6Hostname(normalized); +} + +export function canUseOpenRouterOAuth(): boolean { + if (typeof window === "undefined") return false; + return !isLocalOpenRouterOAuthHostname(window.location.hostname); +} + +function subscribeToOpenRouterOAuthAvailability() { + return () => {}; +} + +function unavailableOnServer() { + return false; +} + +export function useCanUseOpenRouterOAuth(): boolean { + return useSyncExternalStore( + subscribeToOpenRouterOAuthAvailability, + canUseOpenRouterOAuth, + unavailableOnServer, + ); +} + export async function beginOpenRouterOAuth(returnTo = "/setup") { + if (!canUseOpenRouterOAuth()) return; + const verifier = randomVerifier(); const challenge = base64Url(await sha256(verifier)); sessionStorage.setItem(OPENROUTER_VERIFIER_KEY, verifier); diff --git a/frontend/public/logos/providers/anthropic-icon.svg b/frontend/public/logos/providers/anthropic-icon.svg new file mode 100644 index 0000000..88dc745 --- /dev/null +++ b/frontend/public/logos/providers/anthropic-icon.svg @@ -0,0 +1 @@ +Anthropic \ No newline at end of file diff --git a/frontend/public/logos/providers/anthropic.svg b/frontend/public/logos/providers/anthropic.svg new file mode 100644 index 0000000..cbcc39e --- /dev/null +++ b/frontend/public/logos/providers/anthropic.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/frontend/public/logos/providers/deepinfra.svg b/frontend/public/logos/providers/deepinfra.svg new file mode 100644 index 0000000..925c139 --- /dev/null +++ b/frontend/public/logos/providers/deepinfra.svg @@ -0,0 +1,29 @@ + + DeepInfra + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/frontend/public/logos/providers/deepseek.svg b/frontend/public/logos/providers/deepseek.svg new file mode 100644 index 0000000..117e952 --- /dev/null +++ b/frontend/public/logos/providers/deepseek.svg @@ -0,0 +1 @@ +DeepSeek diff --git a/frontend/public/logos/providers/fireworks-ai.svg b/frontend/public/logos/providers/fireworks-ai.svg new file mode 100644 index 0000000..06da2e7 --- /dev/null +++ b/frontend/public/logos/providers/fireworks-ai.svg @@ -0,0 +1 @@ +Fireworks AI diff --git a/frontend/public/logos/providers/google-g.svg b/frontend/public/logos/providers/google-g.svg new file mode 100644 index 0000000..40c9063 --- /dev/null +++ b/frontend/public/logos/providers/google-g.svg @@ -0,0 +1 @@ +Google Gemini diff --git a/frontend/public/logos/providers/groq.svg b/frontend/public/logos/providers/groq.svg new file mode 100644 index 0000000..49a43fa --- /dev/null +++ b/frontend/public/logos/providers/groq.svg @@ -0,0 +1 @@ +Groq diff --git a/frontend/public/logos/providers/huggingface.svg b/frontend/public/logos/providers/huggingface.svg new file mode 100644 index 0000000..5992e36 --- /dev/null +++ b/frontend/public/logos/providers/huggingface.svg @@ -0,0 +1 @@ +Hugging Face diff --git a/frontend/public/logos/providers/lmstudio.svg b/frontend/public/logos/providers/lmstudio.svg new file mode 100644 index 0000000..a2a179f --- /dev/null +++ b/frontend/public/logos/providers/lmstudio.svg @@ -0,0 +1 @@ +LM Studio \ No newline at end of file diff --git a/frontend/public/logos/providers/mistral-ai.svg b/frontend/public/logos/providers/mistral-ai.svg new file mode 100644 index 0000000..b0af170 --- /dev/null +++ b/frontend/public/logos/providers/mistral-ai.svg @@ -0,0 +1 @@ +Mistral AI diff --git a/frontend/public/logos/providers/ollama.svg b/frontend/public/logos/providers/ollama.svg new file mode 100644 index 0000000..432f73e --- /dev/null +++ b/frontend/public/logos/providers/ollama.svg @@ -0,0 +1 @@ +Ollama \ No newline at end of file diff --git a/frontend/public/logos/providers/openai-icon.svg b/frontend/public/logos/providers/openai-icon.svg new file mode 100644 index 0000000..ebbdab0 --- /dev/null +++ b/frontend/public/logos/providers/openai-icon.svg @@ -0,0 +1 @@ +OpenAI \ No newline at end of file diff --git a/frontend/public/logos/providers/openai.svg b/frontend/public/logos/providers/openai.svg new file mode 100644 index 0000000..367828b --- /dev/null +++ b/frontend/public/logos/providers/openai.svg @@ -0,0 +1,10 @@ + + + + + + + + + + \ No newline at end of file diff --git a/frontend/public/logos/providers/openrouter-wordmark.svg b/frontend/public/logos/providers/openrouter-wordmark.svg new file mode 100644 index 0000000..f4bfcbb --- /dev/null +++ b/frontend/public/logos/providers/openrouter-wordmark.svg @@ -0,0 +1,10 @@ + + OpenRouter + + + + + + + OpenRouter + diff --git a/frontend/public/logos/providers/openrouter.svg b/frontend/public/logos/providers/openrouter.svg new file mode 100644 index 0000000..d71ed64 --- /dev/null +++ b/frontend/public/logos/providers/openrouter.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/frontend/public/logos/providers/qwen.svg b/frontend/public/logos/providers/qwen.svg new file mode 100644 index 0000000..fbf9232 --- /dev/null +++ b/frontend/public/logos/providers/qwen.svg @@ -0,0 +1 @@ +Qwen diff --git a/frontend/public/logos/providers/together-ai.svg b/frontend/public/logos/providers/together-ai.svg new file mode 100644 index 0000000..8d0f75b --- /dev/null +++ b/frontend/public/logos/providers/together-ai.svg @@ -0,0 +1 @@ +Together.ai diff --git a/frontend/public/logos/providers/xai.svg b/frontend/public/logos/providers/xai.svg new file mode 100644 index 0000000..d8e9f54 --- /dev/null +++ b/frontend/public/logos/providers/xai.svg @@ -0,0 +1 @@ +xAI diff --git a/makefiles/Makefile b/makefiles/Makefile index cd7ef5e..1870a16 100644 --- a/makefiles/Makefile +++ b/makefiles/Makefile @@ -33,7 +33,9 @@ dev: validate-dev-env install-deps start-local-keychain @echo " Mastra Studio: http://localhost:4111" @echo " Convex Dashboard: http://localhost:6791" @echo "" - docker compose -f docker-compose.dev.yml logs -f + @echo "Streaming app logs (frontend, backend, mastra)." + @echo "For Convex logs: docker compose -f docker-compose.dev.yml logs -f convex" + docker compose -f docker-compose.dev.yml logs -f frontend backend mastra validate-dev-env: @if [ ! -f .env ]; then \ @@ -62,7 +64,7 @@ validate-dev-env: fi @prod="$$(grep '^PROD=' .env | cut -d= -f2-)"; \ if [[ "$$prod" != "1" ]]; then \ - echo "Local mode: Clerk, TinyFish, and OpenRouter env validation skipped."; \ + echo "Local mode: Clerk, TinyFish, and LLM provider env validation skipped."; \ fi @prod="$$(grep '^PROD=' .env | cut -d= -f2-)"; \ if [[ "$$prod" != "1" ]]; then exit 0; fi; \ @@ -198,7 +200,9 @@ convex-env: fi; \ if [[ "$$prod" != "1" ]]; then \ $(CONVEX_CLI_RUN) -e CONVEX_SELF_HOSTED_ADMIN_KEY="$$admin_key" frontend sh -lc \ - 'npx convex env set BIGSET_LOCAL_MODE 1 --url $(CONVEX_CLI_URL) --admin-key "$$CONVEX_SELF_HOSTED_ADMIN_KEY"'; \ + 'npx convex env set BIGSET_LOCAL_MODE 1 --url $(CONVEX_CLI_URL) --admin-key "$$CONVEX_SELF_HOSTED_ADMIN_KEY"' || exit 1; \ + $(CONVEX_CLI_RUN) -e CONVEX_SELF_HOSTED_ADMIN_KEY="$$admin_key" frontend sh -lc \ + 'npx convex env set CLERK_JWT_ISSUER_DOMAIN "https://bigset.local.invalid" --url $(CONVEX_CLI_URL) --admin-key "$$CONVEX_SELF_HOSTED_ADMIN_KEY"' || exit 1; \ exit 0; \ fi; \ issuer="$$(grep '^CLERK_JWT_ISSUER_DOMAIN=' .env | cut -d= -f2-)"; \