From 00cd57b611730ccc0f39a76c781d5084f638f7e4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Mon, 1 Jun 2026 23:49:23 +0200 Subject: [PATCH 01/24] docs(mcp): adaptive tool retrieval design spec Design for replacing all-MCP-tools-every-request (~95 tools, 41k tokens, proven cause of empty-array tool calls) with relevance-gated injection: local MiniLM embedding, per-session base set (cap K + threshold tau), on-demand find_tools(query) meta-tool, native tools always-on. --- ...026-06-01-mcp-adaptive-retrieval-design.md | 150 ++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-01-mcp-adaptive-retrieval-design.md diff --git a/docs/superpowers/specs/2026-06-01-mcp-adaptive-retrieval-design.md b/docs/superpowers/specs/2026-06-01-mcp-adaptive-retrieval-design.md new file mode 100644 index 00000000..d7a691a6 --- /dev/null +++ b/docs/superpowers/specs/2026-06-01-mcp-adaptive-retrieval-design.md @@ -0,0 +1,150 @@ +# Adaptive MCP Tool Retrieval — Design + +**Date:** 2026-06-01 +**Status:** Approved (design) — pending implementation plan +**Component:** `src/core/mcp/retrieval/` (new), `src/core/mcp/bootstrap.ts`, `src/core/prompts/system-prompt/registry/DiracToolSet.ts`, `cli/src/index.ts` + +## Problem + +ISAAC sends every registered tool schema to the worker on every request. With the +default Claude-Code plugin MCP set loaded, that is **~95 tool schemas** (25 native + +~70 MCP) ≈ **41k tokens per turn**. Measured this session: + +- 95 tools → the worker (Qwen3-Coder-30B Q4) intermittently streams a well-formed but + **empty required array** (`read_file {"paths":[]}`), failing validation and spinning + the agent into a retry loop. Reproduced deterministically: ~2 empty-array events per run. +- 25 tools (`--no-mcp`) → **0 empty-array events** across runs. + +Tool-list bloat is the proven primary cause (and the dominant per-turn latency driver). +Two non-fixes were ruled out empirically: forcing `temperature=0` (made retries a +deterministic lock — counterproductive) and a sharper retry message alone (the model +still repeated the empty call under 95-tool load). + +## Goal + +Replace "inject all ~70 MCP tools every request" with "inject only the MCP tools relevant +to the current need, automatically." Native tools stay always-on. The result keeps the +per-request tool count low (≤ K + 26) without the user manually curating an allowlist. + +## Decisions (locked) + +| Axis | Decision | +|------|----------| +| Granularity | **Hybrid** — per-session base set + on-demand expansion | +| Matching mechanism | **Local embedding in the CLI** — MiniLM `all-MiniLM-L6-v2` ONNX (transformers.js); no gateway hop, offline-capable | +| Expansion trigger | **Meta-tool `find_tools(query)`** the model calls when it needs a capability it lacks | +| Base set sizing | **Cap + threshold** — at most K MCP tools, only those with cosine similarity ≥ τ (may be **empty**) | + +Default knobs (tune empirically): **K = 8** (base cap), **K′ = 5** (find_tools cap), +**τ = 0.3** (cosine threshold). + +## Architecture + +Native tools (the 25 `DiracDefaultTool`) and `find_tools` are **always** emitted. MCP +tool specs are **gated** by an "active set" of qualified names, selected by relevance. + +### Components (`src/core/mcp/retrieval/`) + +1. **Embedder** — wraps a local MiniLM `all-MiniLM-L6-v2` ONNX model via transformers.js. + Lazy-loaded on first use; model weights pinned/vendored and cached in the user/config + dir. Interface: `embed(texts: string[]) → Float32Array[]` (384-d), plus a `cosine(a, b)` + helper. Mean-pooled + L2-normalized embeddings. + +2. **Tool index** — at bootstrap, after MCP specs are loaded, embed each MCP tool's text + (`name + "\n" + description`) once. Vectors are **persisted on disk**, keyed by + `qualifiedName` + a hash of the description, so the index is reused across sessions and + only re-embeds tools whose description changed. Native tools are NOT indexed. + +3. **Active-set manager** — session-scoped set of active MCP `qualifiedName`s. + - *Session start:* embed the first user task prompt → cosine vs the tool index → + select MCP tools with `sim ≥ τ`, capped at the top K → base active set (possibly empty). + - *`find_tools(query)`:* embed `query` → top matches (`sim ≥ τ`, capped at K′) → **add** + to the active set; the handler returns the newly-activated tool names + one-line + descriptions so the model knows what is now callable. + +4. **Assembly gate** — at the per-request seam in `DiracToolSet.getEnabledToolSpecs` + (already runs every request): emit all native specs + `find_tools` + only the MCP + specs whose `qualifiedName` is in the active set. Native filtering by + `contextRequirements` is unchanged. + +5. **`find_tools` native tool** — `name: "find_tools"`, param `query: string` + ("describe the capability you need"). Registered like other `DiracDefaultTool`s, + always-on (never gated). Its handler mutates the active-set manager and returns a + compact list of activated tools. + +### Data flow + +``` +bootstrap : load MCP specs → embed+cache tool vectors → register specs (gated) +task start : embed first user prompt → cosine top-K (≥τ) → base active set +each request (gate) : native specs + find_tools + active-set MCP specs [NO embedding here] +model calls find_tools(query) : embed query → top-K′ (≥τ) → grow active set → next turn includes them +``` + +The per-request path performs **no embedding** — the active set is precomputed; only +`find_tools` calls (and the one-shot session-start selection) embed text. + +### Flag composition + +- **default** = adaptive (base = cap+threshold, `find_tools` on). +- `--no-mcp` = fully off: no MCP servers, no embedder load, no `find_tools` (existing + `AILIANCE_NO_MCP` path, unchanged). +- `--mcp a,b` = restrict the **candidate pool** to servers `a,b`; retrieval and + `find_tools` only see tools from those servers (existing `AILIANCE_MCP_SERVERS` path, + reinterpreted as the pool the retriever ranks over). +- New optional tuning flags: `--mcp-top-k `, `--mcp-threshold <τ>` (sane defaults; env + fallbacks `AILIANCE_MCP_TOP_K`, `AILIANCE_MCP_THRESHOLD`). + +### Error handling / fallback + +- Embedder load or inference failure → **degrade to native-only with MCP off** (the + proven 25-tool state that produced 0 empty-array events) + a single warning log; + `find_tools` returns "tool retrieval unavailable." Never crash; never fall back to the + full 95-tool flood. +- Session-start embedding is best-effort: failure → empty base set (model can still + `find_tools`, which will also report unavailable if the embedder is down). +- Tool-vector cache entry is invalidated when the description hash changes. + +### Latency budget + +- Model download once (~88 MB, cached). Tool-index embedding: ~70 short texts once per + bootstrap, cached across sessions → tens of ms first time, ~0 afterward. +- Per request: **0 embedding** (active set precomputed). +- `find_tools`: embeds 1 query (~ms on CPU). + +## Testing + +- **Unit:** embedder wrapper (mocked model); cosine top-K selection with threshold + cap + edge cases (empty pool, all-below-threshold → `[]`, ties, cap boundary); active-set + manager (add, dedup, pool restriction); assembly gate (native always present, MCP gated + correctly, `find_tools` always present). +- **Integration:** bootstrap → index → base selection against a fixture MCP tool set; + `find_tools` expands the active set; `--no-mcp` disables everything; `--mcp a,b` scopes + the candidate pool. +- **Regression / e2e:** the repro task (read several files in sequence) now sends + ≤ K + 26 tools, and the empty-array event rate drops materially versus the 95-tool + baseline. + +## Supply-chain (HITL policy) + +- Pin the embedding library (transformers.js) by exact version + integrity hash in the + lockfile. +- Vendor/mirror the `all-MiniLM-L6-v2` ONNX weights into the `ailiance` org against a + frozen baseline; do not pull from the public HF CDN at runtime in production. +- Add both to the SBOM. Human-in-the-loop diff review before any upgrade. + +## Out of scope + +- Re-selecting tools automatically every turn (rejected in favor of the hybrid meta-tool + trigger — bounded latency, no churn). +- A gateway-side `/v1/embeddings` endpoint (rejected in favor of local embedding — no hop, + offline). +- Changing native-tool filtering or the `contextRequirements` mechanism. +- The separate `missingToolParameterError` wording improvement (already a small standalone + change; complementary but not part of this feature). + +## Open defaults to validate during implementation + +`K=8`, `K′=5`, `τ=0.3` are starting points; validate against the repro harness and adjust +so a pure-coding task yields a near-empty base set while genuinely tool-needing tasks +surface the right MCP tools. From 61d08ad6ffd3d98e42b853162c5c556034df881a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Mon, 1 Jun 2026 23:58:57 +0200 Subject: [PATCH 02/24] fix(tools): sharper corrective on missing/empty required param Generic 'retry with complete response' let the model repeat an empty required array; the message now explicitly names the empty-array mistake and instructs to re-issue with the value populated. --- src/core/prompts/responses.ts | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/src/core/prompts/responses.ts b/src/core/prompts/responses.ts index be421599..1605c756 100644 --- a/src/core/prompts/responses.ts +++ b/src/core/prompts/responses.ts @@ -2,9 +2,9 @@ import { Anthropic } from "@anthropic-ai/sdk" import { hashLines } from "@utils/line-hashing" import * as diff from "diff" import * as path from "path" +import type { FileInfo } from "../../services/glob/list-files" import { Mode } from "../../shared/storage/types" import { DiracIgnoreController, LOCK_TEXT_SYMBOL } from "../ignore/DiracIgnoreController" -import type { FileInfo } from "../../services/glob/list-files" const CONTEXT_WINDOW_WARNING_THRESHOLD_PERCENT = 50 @@ -24,7 +24,8 @@ export const formatResponse = { toolDenied: () => `The user denied this operation.`, - toolDeniedWithFeedback: (feedback: string) => `The user denied this operation and provided the following feedback:\n\n${feedback}\n`, + toolDeniedWithFeedback: (feedback: string) => + `The user denied this operation and provided the following feedback:\n\n${feedback}\n`, toolError: (error?: string) => `The tool execution failed with the following error:\n\n${error}\n`, @@ -37,8 +38,7 @@ export const formatResponse = { filePermissionError: (path: string, operation: string) => `Cannot ${operation} '${path}': Permission denied. You may need to ask the user to check file permissions or try a different path.`, - readOnlyError: (path: string) => - `Cannot write to '${path}': Read-only file system.`, + readOnlyError: (path: string) => `Cannot write to '${path}': Read-only file system.`, permissionDeniedError: (reason: string) => `Command execution blocked by DIRAC_COMMAND_PERMISSIONS: ${reason}. You must try a different approach or ask the user to update the permission settings.`, @@ -58,7 +58,11 @@ Otherwise, if you have not completed the task and do not need additional informa `You seem to be having trouble proceeding. The user has provided the following feedback to help guide you:\n\n${feedback}\n`, missingToolParameterError: (paramName: string, example?: string) => - `Missing value for required parameter '${paramName}'. Please retry with complete response.${example ? `\n\nExample of correct usage (arguments JSON):\n${example}` : ""}\n`, + `The required parameter '${paramName}' was missing or empty in your tool call. ` + + `You MUST provide a non-empty value for it. If '${paramName}' is an array (such as ` + + `paths or commands), it must contain at least one element — do NOT send an empty ` + + `array \`[]\`; put the actual value(s) inside it. Re-issue the SAME tool now with ` + + `'${paramName}' populated.${example ? `\n\nExample of correct usage (arguments JSON):\n${example}` : ""}\n`, /** * Specialized error for write_to_file when the 'content' parameter is missing. @@ -173,20 +177,14 @@ Otherwise, if you have not completed the task and do not need additional informa return aParts.length - bParts.length }) - const filtered = diracIgnoreController - ? sorted.filter((file) => diracIgnoreController.validateAccess(file.path)) - : sorted + const filtered = diracIgnoreController ? sorted.filter((file) => diracIgnoreController.validateAccess(file.path)) : sorted const formatted = filtered.map((file) => { let relativePath = path.relative(absolutePath, file.path).toPosix() if (relativePath === "" && !file.isDirectory) { relativePath = path.basename(file.path) } - const displayPath = file.isDirectory - ? relativePath.endsWith("/") - ? relativePath - : `${relativePath}/` - : relativePath + const displayPath = file.isDirectory ? (relativePath.endsWith("/") ? relativePath : `${relativePath}/`) : relativePath const lineCountSuffix = file.lineCount !== undefined ? ` ${file.lineCount} lines` : "" return `${displayPath}${lineCountSuffix}` }) From 5b292ac22ebaa20522ffa5da3d4296d01cbe5dc2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Mon, 1 Jun 2026 23:58:57 +0200 Subject: [PATCH 03/24] docs(mcp): adaptive tool retrieval implementation plan --- .../2026-06-01-mcp-adaptive-retrieval.md | 1187 +++++++++++++++++ 1 file changed, 1187 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-01-mcp-adaptive-retrieval.md diff --git a/docs/superpowers/plans/2026-06-01-mcp-adaptive-retrieval.md b/docs/superpowers/plans/2026-06-01-mcp-adaptive-retrieval.md new file mode 100644 index 00000000..cfd8e6d0 --- /dev/null +++ b/docs/superpowers/plans/2026-06-01-mcp-adaptive-retrieval.md @@ -0,0 +1,1187 @@ +# Adaptive MCP Tool Retrieval Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop sending all ~70 plugin MCP tool schemas on every request; inject only the MCP tools relevant to the current need (per-session base set + on-demand `find_tools`), keeping the per-request tool count at ≤ K + native, which removes the empty-array tool-call failures caused by 95-tool bloat. + +**Architecture:** Reuse the registry's existing `contextRequirements(ctx) => boolean` gate. MCP tool specs get a `contextRequirements` that checks `ctx.activeMcpTools` (a Set of active qualified names). A session-scoped `ActiveMcpToolSet` is seeded from the first user prompt via a local MiniLM embedder + cosine top-K (cap K, threshold τ), and grown when the model calls a new always-on `find_tools(query)` native tool. Embedder failure degrades to native-only (MCP off). + +**Tech Stack:** TypeScript, Node@22, biome, mocha (core unit) + vitest (cli), `@huggingface/transformers` (transformers.js, all-MiniLM-L6-v2 ONNX, pinned/vendored). + +**Build/test prelude (run once per shell):** +```bash +export PATH="/opt/homebrew/opt/node@22/bin:$PATH" +cd /Users/electron/code/ailiance-agent +``` +- Typecheck: `npm run check-types` +- Lint: `npm run lint` +- Core unit (one file): `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/.test.ts'` +- CLI build (deploys `cli/dist/cli.mjs`): `npm run cli:build` + +--- + +## File Structure + +- `src/core/mcp/retrieval/cosine.ts` — pure cosine similarity + top-K selection (cap + threshold). No deps. +- `src/core/mcp/retrieval/Embedder.ts` — lazy MiniLM ONNX wrapper; `embed(texts) → Float32Array[]`; throws on load/inference failure. +- `src/core/mcp/retrieval/ToolVectorIndex.ts` — embeds MCP tool texts once; on-disk cache keyed by `qualifiedName` + description hash. +- `src/core/mcp/retrieval/ActiveMcpToolSet.ts` — session state: base selection + `expand(query)`; holds the active Set; pure logic over an injected embedder+index. +- `src/core/mcp/retrieval/config.ts` — K/K′/τ defaults + env/flag overrides. +- `src/core/prompts/system-prompt/tools/find_tools.ts` — the `find_tools` native tool spec. +- `src/core/task/tools/handlers/FindToolsToolHandler.ts` — the `find_tools` handler. +- Modify: `src/shared/tools.ts` (enum), `src/core/prompts/system-prompt/tools/init.ts` (register spec), `src/core/prompts/system-prompt/registry/DiracToolSet.ts` (register handler list is elsewhere), `src/core/task/tools/ToolExecutorCoordinator.ts` (register handler), `src/core/mcp/bootstrap.ts` (gate via contextRequirements + build index/active-set), `src/core/prompts/system-prompt/types.ts` (`activeMcpTools` field), `cli/src/index.ts` (flags). + +--- + +## Task 1: Pure cosine + top-K selection + +**Files:** +- Create: `src/core/mcp/retrieval/cosine.ts` +- Test: `src/core/mcp/retrieval/__tests__/cosine.test.ts` + +- [ ] **Step 1: Write the failing test** + +```ts +import { describe, it } from "mocha" +import "should" +import { cosineSim, selectTopK } from "../cosine" + +describe("cosine retrieval", () => { + it("computes cosine similarity of normalized-ish vectors", () => { + cosineSim(new Float32Array([1, 0]), new Float32Array([1, 0])).should.be.approximately(1, 1e-6) + cosineSim(new Float32Array([1, 0]), new Float32Array([0, 1])).should.be.approximately(0, 1e-6) + cosineSim(new Float32Array([0, 0]), new Float32Array([1, 0])).should.equal(0) // zero-vector guard + }) + + it("selectTopK applies threshold then caps, sorted by score desc", () => { + const query = new Float32Array([1, 0]) + const items = [ + { id: "a", vec: new Float32Array([1, 0]) }, // 1.0 + { id: "b", vec: new Float32Array([0.9, 0.1]) }, // ~0.994 + { id: "c", vec: new Float32Array([0, 1]) }, // 0.0 (below τ) + ] + selectTopK(query, items, { k: 1, threshold: 0.3 }).should.deepEqual(["a"]) + selectTopK(query, items, { k: 5, threshold: 0.3 }).should.deepEqual(["a", "b"]) + selectTopK(query, items, { k: 5, threshold: 0.99 }).should.deepEqual(["a", "b"]) + }) + + it("returns [] when nothing clears the threshold or items is empty", () => { + const q = new Float32Array([1, 0]) + selectTopK(q, [{ id: "c", vec: new Float32Array([0, 1]) }], { k: 5, threshold: 0.3 }).should.deepEqual([]) + selectTopK(q, [], { k: 5, threshold: 0.3 }).should.deepEqual([]) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/cosine.test.ts'` +Expected: FAIL — `Cannot find module '../cosine'`. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/core/mcp/retrieval/cosine.ts +export function cosineSim(a: Float32Array, b: Float32Array): number { + let dot = 0 + let na = 0 + let nb = 0 + for (let i = 0; i < a.length; i++) { + dot += a[i] * b[i] + na += a[i] * a[i] + nb += b[i] * b[i] + } + if (na === 0 || nb === 0) { + return 0 + } + return dot / (Math.sqrt(na) * Math.sqrt(nb)) +} + +export interface ScoredItem { + id: string + vec: Float32Array +} + +export interface SelectOptions { + k: number + threshold: number +} + +/** Items with cosine(query, vec) >= threshold, sorted desc, capped at k. */ +export function selectTopK(query: Float32Array, items: ScoredItem[], opts: SelectOptions): string[] { + return items + .map((it) => ({ id: it.id, score: cosineSim(query, it.vec) })) + .filter((s) => s.score >= opts.threshold) + .sort((a, b) => b.score - a.score) + .slice(0, Math.max(0, opts.k)) + .map((s) => s.id) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/cosine.test.ts'` +Expected: PASS (3 passing). + +- [ ] **Step 5: Commit** + +```bash +git add src/core/mcp/retrieval/cosine.ts src/core/mcp/retrieval/__tests__/cosine.test.ts +git commit -m "feat(mcp): cosine similarity + top-K selection for tool retrieval" +``` + +--- + +## Task 2: Retrieval config (K/K′/τ defaults + overrides) + +**Files:** +- Create: `src/core/mcp/retrieval/config.ts` +- Test: `src/core/mcp/retrieval/__tests__/config.test.ts` + +- [ ] **Step 1: Write the failing test** + +```ts +import { describe, it, afterEach } from "mocha" +import "should" +import { getRetrievalConfig } from "../config" + +describe("getRetrievalConfig", () => { + afterEach(() => { + delete process.env.AILIANCE_MCP_TOP_K + delete process.env.AILIANCE_MCP_FIND_K + delete process.env.AILIANCE_MCP_THRESHOLD + }) + + it("returns sane defaults", () => { + getRetrievalConfig().should.deepEqual({ baseK: 8, findK: 5, threshold: 0.3 }) + }) + + it("honors env overrides and ignores invalid ones", () => { + process.env.AILIANCE_MCP_TOP_K = "12" + process.env.AILIANCE_MCP_THRESHOLD = "0.45" + process.env.AILIANCE_MCP_FIND_K = "not-a-number" + getRetrievalConfig().should.deepEqual({ baseK: 12, findK: 5, threshold: 0.45 }) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/config.test.ts'` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/core/mcp/retrieval/config.ts +export interface RetrievalConfig { + baseK: number + findK: number + threshold: number +} + +const DEFAULTS: RetrievalConfig = { baseK: 8, findK: 5, threshold: 0.3 } + +function numEnv(name: string, fallback: number): number { + const raw = process.env[name] + if (raw === undefined) { + return fallback + } + const n = Number(raw) + return Number.isFinite(n) ? n : fallback +} + +export function getRetrievalConfig(): RetrievalConfig { + return { + baseK: numEnv("AILIANCE_MCP_TOP_K", DEFAULTS.baseK), + findK: numEnv("AILIANCE_MCP_FIND_K", DEFAULTS.findK), + threshold: numEnv("AILIANCE_MCP_THRESHOLD", DEFAULTS.threshold), + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/config.test.ts'` +Expected: PASS (2 passing). + +- [ ] **Step 5: Commit** + +```bash +git add src/core/mcp/retrieval/config.ts src/core/mcp/retrieval/__tests__/config.test.ts +git commit -m "feat(mcp): retrieval config with env-overridable K/findK/threshold" +``` + +--- + +## Task 3: Embedder dependency (pin transformers.js) — supply-chain + +**Files:** +- Modify: `package.json` (root), lockfile + +- [ ] **Step 1: Add the pinned dependency** + +Run: +```bash +npm install --save-exact @huggingface/transformers@3.3.3 +``` +(Pin the exact version; verify the resolved integrity hash lands in `package-lock.json`.) + +- [ ] **Step 2: Verify install + hash present** + +Run: `node -e "console.log(require('@huggingface/transformers/package.json').version)"` +Expected: `3.3.3` +Run: `grep -c '"@huggingface/transformers"' package-lock.json` +Expected: ≥ 1 (entry with `integrity` sha512 present). + +- [ ] **Step 3: Record supply-chain note** + +Append to `docs/superpowers/plans/2026-06-01-mcp-adaptive-retrieval.md` a one-line SBOM note, or add to the existing dependency inventory: `@huggingface/transformers@3.3.3` + model `Xenova/all-MiniLM-L6-v2` (ONNX) must be mirrored into the `ailiance` org before production; runtime must load from the vendored/cached path, not the public HF CDN. + +- [ ] **Step 4: Commit** + +```bash +git add package.json package-lock.json docs/superpowers/plans/2026-06-01-mcp-adaptive-retrieval.md +git commit -m "build(mcp): pin @huggingface/transformers for local tool embedding" +``` + +--- + +## Task 4: Embedder wrapper (lazy MiniLM, fails loud) + +**Files:** +- Create: `src/core/mcp/retrieval/Embedder.ts` +- Test: `src/core/mcp/retrieval/__tests__/Embedder.test.ts` + +The Embedder takes an injectable "pipeline factory" so tests never load the real model. Production passes the transformers.js factory. + +- [ ] **Step 1: Write the failing test** + +```ts +import { describe, it } from "mocha" +import "should" +import { Embedder } from "../Embedder" + +describe("Embedder", () => { + it("embeds via the injected pipeline and returns Float32Array[]", async () => { + const fakePipeline = async (texts: string[]) => + texts.map((t) => new Float32Array([t.length, 0, 0])) + const e = new Embedder(async () => fakePipeline) + const [v] = await e.embed(["abc"]) + Array.from(v).should.deepEqual([3, 0, 0]) + }) + + it("loads the pipeline only once (lazy, memoized)", async () => { + let loads = 0 + const e = new Embedder(async () => { + loads++ + return async (texts: string[]) => texts.map(() => new Float32Array([1])) + }) + await e.embed(["a"]) + await e.embed(["b"]) + loads.should.equal(1) + }) + + it("propagates load failure (caller degrades)", async () => { + const e = new Embedder(async () => { + throw new Error("model load failed") + }) + let threw = false + try { + await e.embed(["a"]) + } catch { + threw = true + } + threw.should.equal(true) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/Embedder.test.ts'` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/core/mcp/retrieval/Embedder.ts +export type EmbedFn = (texts: string[]) => Promise +export type PipelineFactory = () => Promise + +/** + * Lazy local text embedder. The pipeline factory is injected so tests run + * without loading the real ONNX model. Production wires the transformers.js + * factory in `createDefaultEmbedder`. + */ +export class Embedder { + private pipelinePromise: Promise | undefined + + constructor(private readonly factory: PipelineFactory) {} + + async embed(texts: string[]): Promise { + if (!this.pipelinePromise) { + this.pipelinePromise = this.factory() + } + const fn = await this.pipelinePromise + return fn(texts) + } +} + +/** + * Production factory: all-MiniLM-L6-v2 ONNX via transformers.js, mean-pooled + + * normalized 384-d. Imported lazily so `--no-mcp` never pays the import cost. + */ +export function createDefaultEmbedder(): Embedder { + return new Embedder(async () => { + const { pipeline } = await import("@huggingface/transformers") + const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2") + return async (texts: string[]) => { + const out = await extractor(texts, { pooling: "mean", normalize: true }) + // out.tolist() → number[][]; convert each row to Float32Array + return (out.tolist() as number[][]).map((row) => Float32Array.from(row)) + } + }) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/Embedder.test.ts'` +Expected: PASS (3 passing). + +- [ ] **Step 5: Commit** + +```bash +git add src/core/mcp/retrieval/Embedder.ts src/core/mcp/retrieval/__tests__/Embedder.test.ts +git commit -m "feat(mcp): lazy injectable local embedder wrapper" +``` + +--- + +## Task 5: Tool-vector index with on-disk cache + +**Files:** +- Create: `src/core/mcp/retrieval/ToolVectorIndex.ts` +- Test: `src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts` + +The index embeds each MCP tool's `name + "\n" + description` once, caches vectors on disk keyed by `qualifiedName` + a hash of the embedded text, and re-embeds only changed/new tools. + +- [ ] **Step 1: Write the failing test** + +```ts +import { describe, it, beforeEach, afterEach } from "mocha" +import "should" +import * as fs from "fs" +import * as os from "os" +import * as path from "path" +import { Embedder } from "../Embedder" +import { ToolVectorIndex } from "../ToolVectorIndex" + +function tmpFile(): string { + return path.join(fs.mkdtempSync(path.join(os.tmpdir(), "tvi-")), "index.json") +} + +describe("ToolVectorIndex", () => { + let cachePath: string + beforeEach(() => { + cachePath = tmpFile() + }) + afterEach(() => { + fs.rmSync(path.dirname(cachePath), { recursive: true, force: true }) + }) + + it("embeds each tool once and returns vectors keyed by qualifiedName", async () => { + let embedCalls = 0 + const embedder = new Embedder(async () => async (texts: string[]) => { + embedCalls += texts.length + return texts.map((t) => new Float32Array([t.length])) + }) + const idx = new ToolVectorIndex(embedder, cachePath) + const tools = [ + { qualifiedName: "mcp__p_s__a", text: "alpha" }, + { qualifiedName: "mcp__p_s__b", text: "beta!" }, + ] + const vecs = await idx.build(tools) + embedCalls.should.equal(2) + Array.from(vecs.get("mcp__p_s__a")!).should.deepEqual([5]) // "alpha".length + }) + + it("re-uses the disk cache and only embeds new/changed tools", async () => { + let embedCalls = 0 + const embedder = new Embedder(async () => async (texts: string[]) => { + embedCalls += texts.length + return texts.map((t) => new Float32Array([t.length])) + }) + const tools = [{ qualifiedName: "mcp__p_s__a", text: "alpha" }] + await new ToolVectorIndex(embedder, cachePath).build(tools) // embeds 1 + await new ToolVectorIndex(embedder, cachePath).build(tools) // cache hit, embeds 0 + embedCalls.should.equal(1) + // changed text → re-embed + await new ToolVectorIndex(embedder, cachePath).build([{ qualifiedName: "mcp__p_s__a", text: "alpha2" }]) + embedCalls.should.equal(2) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts'` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/core/mcp/retrieval/ToolVectorIndex.ts +import * as crypto from "crypto" +import * as fs from "fs" +import * as path from "path" +import type { Embedder } from "./Embedder" + +export interface ToolText { + qualifiedName: string + text: string +} + +interface CacheEntry { + hash: string + vec: number[] +} + +function hashText(text: string): string { + return crypto.createHash("sha256").update(text).digest("hex").slice(0, 16) +} + +export class ToolVectorIndex { + constructor( + private readonly embedder: Embedder, + private readonly cachePath: string, + ) {} + + private readCache(): Record { + try { + return JSON.parse(fs.readFileSync(this.cachePath, "utf8")) + } catch { + return {} + } + } + + private writeCache(cache: Record): void { + fs.mkdirSync(path.dirname(this.cachePath), { recursive: true }) + fs.writeFileSync(this.cachePath, JSON.stringify(cache)) + } + + /** Returns a Map, embedding only new/changed tools. */ + async build(tools: ToolText[]): Promise> { + const cache = this.readCache() + const stale = tools.filter((t) => cache[t.qualifiedName]?.hash !== hashText(t.text)) + if (stale.length > 0) { + const vecs = await this.embedder.embed(stale.map((t) => t.text)) + stale.forEach((t, i) => { + cache[t.qualifiedName] = { hash: hashText(t.text), vec: Array.from(vecs[i]) } + }) + this.writeCache(cache) + } + const result = new Map() + for (const t of tools) { + result.set(t.qualifiedName, Float32Array.from(cache[t.qualifiedName].vec)) + } + return result + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts'` +Expected: PASS (2 passing). + +- [ ] **Step 5: Commit** + +```bash +git add src/core/mcp/retrieval/ToolVectorIndex.ts src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts +git commit -m "feat(mcp): on-disk tool-vector index with hash-keyed cache" +``` + +--- + +## Task 6: ActiveMcpToolSet (base selection + expand) + +**Files:** +- Create: `src/core/mcp/retrieval/ActiveMcpToolSet.ts` +- Test: `src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts` + +Session state. Built from a vector map (from Task 5) + embedder (Task 4) + config (Task 2). `seed(prompt)` sets the base set; `expand(query)` adds; `snapshot()` returns a `ReadonlySet`. Embedder failure → empty set (degrade), surfaced via `available()`. + +- [ ] **Step 1: Write the failing test** + +```ts +import { describe, it } from "mocha" +import "should" +import { Embedder } from "../Embedder" +import { ActiveMcpToolSet } from "../ActiveMcpToolSet" + +// query "alpha" embeds to [5]; tools embed to their length → cosine of 1-d +// vectors is always 1 for same-sign, so threshold gates by presence, and we +// instead use distinct directions to exercise ranking. +function makeEmbedder(map: Record) { + return new Embedder(async () => async (texts: string[]) => + texts.map((t) => Float32Array.from(map[t] ?? [0, 0])), + ) +} + +describe("ActiveMcpToolSet", () => { + const vectors = new Map([ + ["mcp__git__issues", Float32Array.from([1, 0])], + ["mcp__fs__read", Float32Array.from([0, 1])], + ]) + + it("seeds the base set from the prompt (cap + threshold)", async () => { + const embedder = makeEmbedder({ "find github issues": [1, 0] }) + const set = new ActiveMcpToolSet(embedder, vectors, { baseK: 8, findK: 5, threshold: 0.3 }) + await set.seed("find github issues") + set.snapshot().has("mcp__git__issues").should.equal(true) + set.snapshot().has("mcp__fs__read").should.equal(false) // cosine 0 < τ + }) + + it("expand() adds matching tools and is idempotent", async () => { + const embedder = makeEmbedder({ "read a file": [0, 1], seed: [1, 0] }) + const set = new ActiveMcpToolSet(embedder, vectors, { baseK: 8, findK: 5, threshold: 0.3 }) + await set.seed("seed") + const added = await set.expand("read a file") + added.should.deepEqual(["mcp__fs__read"]) + set.snapshot().has("mcp__fs__read").should.equal(true) + ;(await set.expand("read a file")).should.deepEqual([]) // already active + }) + + it("degrades to empty set + available()=false when embedder throws", async () => { + const embedder = new Embedder(async () => { + throw new Error("no model") + }) + const set = new ActiveMcpToolSet(embedder, vectors, { baseK: 8, findK: 5, threshold: 0.3 }) + await set.seed("anything") + set.snapshot().size.should.equal(0) + set.available().should.equal(false) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts'` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/core/mcp/retrieval/ActiveMcpToolSet.ts +import type { RetrievalConfig } from "./config" +import { type ScoredItem, selectTopK } from "./cosine" +import type { Embedder } from "./Embedder" + +export class ActiveMcpToolSet { + private readonly active = new Set() + private readonly items: ScoredItem[] + private embedderOk = true + + constructor( + private readonly embedder: Embedder, + vectors: Map, + private readonly config: RetrievalConfig, + ) { + this.items = Array.from(vectors.entries()).map(([id, vec]) => ({ id, vec })) + } + + available(): boolean { + return this.embedderOk + } + + snapshot(): ReadonlySet { + return this.active + } + + private async select(text: string, k: number): Promise { + const [q] = await this.embedder.embed([text]) + return selectTopK(q, this.items, { k, threshold: this.config.threshold }) + } + + /** Seed the base set from the first user prompt. Failure → empty set. */ + async seed(prompt: string): Promise { + try { + for (const id of await this.select(prompt, this.config.baseK)) { + this.active.add(id) + } + } catch { + this.embedderOk = false + } + } + + /** Grow the set on a find_tools query. Returns newly-added ids. */ + async expand(query: string): Promise { + try { + const added: string[] = [] + for (const id of await this.select(query, this.config.findK)) { + if (!this.active.has(id)) { + this.active.add(id) + added.push(id) + } + } + return added + } catch { + this.embedderOk = false + return [] + } + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts'` +Expected: PASS (3 passing). + +- [ ] **Step 5: Commit** + +```bash +git add src/core/mcp/retrieval/ActiveMcpToolSet.ts src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts +git commit -m "feat(mcp): session active-tool set with seed + expand" +``` + +--- + +## Task 7: Session singleton + context field for the gate + +**Files:** +- Modify: `src/core/prompts/system-prompt/types.ts` (add `activeMcpTools`) +- Create: `src/core/mcp/retrieval/session.ts` (process-wide current ActiveMcpToolSet accessor) +- Test: `src/core/mcp/retrieval/__tests__/session.test.ts` + +- [ ] **Step 1: Add the context field** + +In `src/core/prompts/system-prompt/types.ts`, inside `interface SystemPromptContext`, add after `readonly subagentsEnabled?: boolean`: + +```ts + /** Active MCP tool qualified names for adaptive retrieval. When set, only + * these MCP tools are emitted; undefined means "no gating" (legacy: all). */ + readonly activeMcpTools?: ReadonlySet +``` + +- [ ] **Step 2: Write the failing test for the session accessor** + +```ts +// src/core/mcp/retrieval/__tests__/session.test.ts +import { describe, it } from "mocha" +import "should" +import { getActiveMcpToolSet, setActiveMcpToolSet } from "../session" + +describe("active mcp tool set session", () => { + it("stores and returns the current set; undefined by default after clear", () => { + setActiveMcpToolSet(undefined) + ;(getActiveMcpToolSet() === undefined).should.equal(true) + const fake = { snapshot: () => new Set(["mcp__x__y"]) } as any + setActiveMcpToolSet(fake) + getActiveMcpToolSet()!.snapshot().has("mcp__x__y").should.equal(true) + }) +}) +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/session.test.ts'` +Expected: FAIL — module not found. + +- [ ] **Step 4: Write minimal implementation** + +```ts +// src/core/mcp/retrieval/session.ts +import type { ActiveMcpToolSet } from "./ActiveMcpToolSet" + +let current: ActiveMcpToolSet | undefined + +export function setActiveMcpToolSet(set: ActiveMcpToolSet | undefined): void { + current = set +} + +export function getActiveMcpToolSet(): ActiveMcpToolSet | undefined { + return current +} +``` + +- [ ] **Step 5: Run test + typecheck** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/session.test.ts'` +Expected: PASS (1 passing). +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/core/prompts/system-prompt/types.ts src/core/mcp/retrieval/session.ts src/core/mcp/retrieval/__tests__/session.test.ts +git commit -m "feat(mcp): session-scoped active tool set + context field" +``` + +--- + +## Task 8: Gate MCP specs via contextRequirements + +**Files:** +- Modify: `src/core/mcp/bootstrap.ts` (`mcpToolToSpec`, ~line 69-78) +- Test: `src/core/mcp/retrieval/__tests__/gate.test.ts` + +MCP specs get a `contextRequirements` that returns true only when the qualified name is in `ctx.activeMcpTools` (or when gating is disabled, i.e. `activeMcpTools === undefined` → legacy "all", preserving `--no-mcp`/no-retrieval behavior). + +- [ ] **Step 1: Write the failing test** + +```ts +// src/core/mcp/retrieval/__tests__/gate.test.ts +import { describe, it } from "mocha" +import "should" +import { mcpToolToSpec } from "../../bootstrap" + +const meta = { + qualifiedName: "mcp__p_s__a", + serverId: "s", + pluginName: "p", + rawName: "a", + description: "does A", + inputSchema: { type: "object", properties: {} }, +} + +describe("mcpToolToSpec gating", () => { + it("is enabled when no gating set (undefined activeMcpTools)", () => { + const spec = mcpToolToSpec(meta as any) + spec.contextRequirements!({ activeMcpTools: undefined } as any).should.equal(true) + }) + it("is enabled only when present in the active set", () => { + const spec = mcpToolToSpec(meta as any) + spec.contextRequirements!({ activeMcpTools: new Set(["mcp__p_s__a"]) } as any).should.equal(true) + spec.contextRequirements!({ activeMcpTools: new Set(["other"]) } as any).should.equal(false) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/gate.test.ts'` +Expected: FAIL — `spec.contextRequirements` is undefined. + +- [ ] **Step 3: Implement the gate in `mcpToolToSpec`** + +In `src/core/mcp/bootstrap.ts`, replace the return of `mcpToolToSpec`: + +```ts +export function mcpToolToSpec(tool: McpToolMetadata): DiracToolSpec { + const qualifiedName = tool.qualifiedName + return { + id: qualifiedName as DiracDefaultTool, + name: qualifiedName, + description: tool.description ?? `MCP tool from plugin ${tool.pluginName}`, + parameters: convertJsonSchemaToParams(tool.inputSchema), + // Adaptive retrieval gate: when ctx.activeMcpTools is set, only emit this + // MCP tool if its qualified name is active. undefined → legacy "all". + contextRequirements: (ctx) => ctx.activeMcpTools === undefined || ctx.activeMcpTools.has(qualifiedName), + } +} +``` + +- [ ] **Step 4: Run test + typecheck** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/mcp/retrieval/__tests__/gate.test.ts'` +Expected: PASS (2 passing). +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/core/mcp/bootstrap.ts src/core/mcp/retrieval/__tests__/gate.test.ts +git commit -m "feat(mcp): gate MCP tool specs by active set via contextRequirements" +``` + +--- + +## Task 9: `find_tools` native tool — spec + enum + register + +**Files:** +- Modify: `src/shared/tools.ts` (enum) +- Create: `src/core/prompts/system-prompt/tools/find_tools.ts` +- Modify: `src/core/prompts/system-prompt/tools/init.ts` (import + add to allTools) + +- [ ] **Step 1: Add enum member** + +In `src/shared/tools.ts`, inside `enum DiracDefaultTool`, add: + +```ts + FIND_TOOLS = "find_tools", +``` + +- [ ] **Step 2: Create the spec** + +```ts +// src/core/prompts/system-prompt/tools/find_tools.ts +import { DiracDefaultTool } from "@/shared/tools" +import type { DiracToolSpec } from "../spec" + +export const find_tools: DiracToolSpec = { + id: DiracDefaultTool.FIND_TOOLS, + name: "find_tools", + description: + "Discover and activate additional MCP tools that are not currently available to you. " + + "Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. " + + "If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), " + + "call find_tools with a short natural-language description of the capability you need. The matching " + + "tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + parameters: [ + { + name: "query", + required: true, + type: "string", + instruction: "A short natural-language description of the capability/tool you need.", + usage: "search and comment on GitHub issues", + }, + ], +} +``` + +- [ ] **Step 3: Register the spec** + +In `src/core/prompts/system-prompt/tools/init.ts`: add `import { find_tools } from "./find_tools"` with the other imports, and add `find_tools,` into the `allTools` array. + +- [ ] **Step 4: Typecheck** + +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/shared/tools.ts src/core/prompts/system-prompt/tools/find_tools.ts src/core/prompts/system-prompt/tools/init.ts +git commit -m "feat(mcp): add find_tools native tool spec" +``` + +--- + +## Task 10: `find_tools` handler + dispatch + +**Files:** +- Create: `src/core/task/tools/handlers/FindToolsToolHandler.ts` +- Modify: `src/core/task/tools/ToolExecutorCoordinator.ts` (register, near where native handlers are registered — see existing `registerByName` calls) +- Test: `src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts` + +First read how an existing simple handler (e.g. `ListSkillsHandler` / `list_skills`) is registered in `ToolExecutorCoordinator` so this follows the same pattern (`coordinator.registerByName(DiracDefaultTool.X, validator)` or a handler-instance registration — match what the file actually does). + +- [ ] **Step 1: Write the failing test** + +```ts +import { describe, it, afterEach } from "mocha" +import "should" +import { setActiveMcpToolSet } from "@core/mcp/retrieval/session" +import { FindToolsToolHandler } from "../FindToolsToolHandler" + +describe("FindToolsToolHandler", () => { + afterEach(() => setActiveMcpToolSet(undefined)) + + it("expands the active set and reports activated tools", async () => { + const calls: string[] = [] + setActiveMcpToolSet({ + expand: async (q: string) => { + calls.push(q) + return ["mcp__git__issues"] + }, + available: () => true, + } as any) + const handler = new FindToolsToolHandler() + const res = await handler.execute({ params: { query: "github issues" } } as any) + calls.should.deepEqual(["github issues"]) + String(res).should.match(/mcp__git__issues/) + }) + + it("reports unavailability when retrieval is disabled", async () => { + setActiveMcpToolSet(undefined) + const handler = new FindToolsToolHandler() + String(await handler.execute({ params: { query: "x" } } as any)).should.match(/unavailable|not available/i) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts'` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write the handler** + +Model the class shape on a sibling handler (read `src/core/task/tools/handlers/ListSkillsToolHandler.ts` for the exact `IFullyManagedTool`/`IToolHandler` interface, `name`, `getDescription`, `handlePartialBlock`, `execute(config, block)` signatures) and adapt: + +```ts +// src/core/task/tools/handlers/FindToolsToolHandler.ts +import type { ToolUse } from "@core/assistant-message" +import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" +import { DiracDefaultTool } from "@/shared/tools" +import type { ToolResponse } from "../../index" +import type { IFullyManagedTool } from "../ToolExecutorCoordinator" +import type { StronglyTypedUIHelpers } from "../types/UIHelpers" +import type { TaskConfig } from "../types/TaskConfig" + +export class FindToolsToolHandler implements IFullyManagedTool { + readonly name = DiracDefaultTool.FIND_TOOLS + + getDescription(block: ToolUse): string { + const q = (block.params.query as string) || "" + return `[find_tools for '${q}']` + } + + async handlePartialBlock(_block: ToolUse, _uiHelpers: StronglyTypedUIHelpers): Promise { + // no-op: find_tools has no streamed side effects to preview + } + + async execute(_config: TaskConfig, block: ToolUse): Promise { + const query = ((block.params.query as string) || "").trim() + const set = getActiveMcpToolSet() + if (!set || !set.available()) { + return "Tool retrieval is unavailable in this run; the default tool set is all that is available." + } + if (!query) { + return "find_tools requires a non-empty 'query' describing the capability you need." + } + const added = await set.expand(query) + if (added.length === 0) { + return `No additional tools matched "${query}". The relevant tools may already be available, or none exist for this need.` + } + return `Activated ${added.length} tool(s) for "${query}" (available next turn):\n${added.map((n) => `- ${n}`).join("\n")}` + } +} +``` + +NOTE: adjust the import paths / interface to exactly match `ListSkillsToolHandler.ts`. If `execute` there returns via `formatResponse.toolResult(...)`, wrap the strings the same way. + +- [ ] **Step 4: Register the handler** + +In `src/core/task/tools/ToolExecutorCoordinator.ts`, register `FindToolsToolHandler` next to the other native handlers (follow the exact registration call used for `list_skills`). + +- [ ] **Step 5: Run test + typecheck** + +Run: `npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha 'src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts'` +Expected: PASS (2 passing). +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/core/task/tools/handlers/FindToolsToolHandler.ts src/core/task/tools/ToolExecutorCoordinator.ts src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts +git commit -m "feat(mcp): find_tools handler wired to active set" +``` + +--- + +## Task 11: Wire bootstrap — build index, seed active set, publish to session + +**Files:** +- Modify: `src/core/mcp/bootstrap.ts` (`initializeMcpForTask`, after `listAllTools`) + +After MCP tools are listed and specs registered, build the vector index, create the `ActiveMcpToolSet`, and publish it via `setActiveMcpToolSet`. Seeding from the first user prompt happens in Task 12 (where the task text is available); here we publish an unseeded set so the gate has a Set (empty until seeded). When `--no-mcp` (returns early) or retrieval disabled, leave the session set `undefined` (legacy "all" — but with zero MCP tools registered, "all" = none, which is correct). + +- [ ] **Step 1: Extend `initializeMcpForTask`** + +After the `for (const tool of tools) { ... }` registration loop and before `return tools`, insert this block. On success it publishes a real active set; on ANY failure it publishes an active set backed by a throwing embedder + empty index, so the gate emits ZERO MCP tools (never the legacy "all" flood) and `available()` reports false: + +```ts + // Adaptive retrieval: build the vector index and publish a session active set. + const { ActiveMcpToolSet } = await import("./retrieval/ActiveMcpToolSet") + const { getRetrievalConfig } = await import("./retrieval/config") + const { setActiveMcpToolSet } = await import("./retrieval/session") + try { + const { createDefaultEmbedder } = await import("./retrieval/Embedder") + const { ToolVectorIndex } = await import("./retrieval/ToolVectorIndex") + const os = await import("os") + const path = await import("path") + const cachePath = path.join(os.homedir(), ".dirac", "mcp-tool-vectors.json") + const embedder = createDefaultEmbedder() + const index = await new ToolVectorIndex(embedder, cachePath).build( + tools.map((t) => ({ qualifiedName: t.qualifiedName, text: `${t.qualifiedName}\n${t.description ?? ""}` })), + ) + setActiveMcpToolSet(new ActiveMcpToolSet(embedder, index, getRetrievalConfig())) + } catch (err) { + Logger.warn("MCP adaptive retrieval unavailable; running native-only:", err) + const { Embedder } = await import("./retrieval/Embedder") + const dead = new Embedder(async () => { + throw new Error("embedder unavailable") + }) + setActiveMcpToolSet(new ActiveMcpToolSet(dead, new Map(), getRetrievalConfig())) + } +``` + +- [ ] **Step 2: Typecheck** + +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 3: Commit** + +```bash +git add src/core/mcp/bootstrap.ts +git commit -m "feat(mcp): build tool index + publish session active set at bootstrap" +``` + +--- + +## Task 12: Seed the base set from the first user prompt + +**Files:** +- Modify: the task-start path that has the initial user task text. Find it: `grep -rn "initiateTaskLoop\|startTask\|getActiveMcpToolSet" src/core/task` and the LifecycleManager (`src/core/task/LifecycleManager.ts startTask`). Seed right after MCP init completes and the first user message is known. + +- [ ] **Step 1: Locate the first-prompt seam** + +Run: `grep -rn "initializeMcpForTask\|startTask\|initiateTaskLoop" src/core/task` +Identify the call site where `initializeMcpForTask` is awaited and the initial user task string is in scope. + +- [ ] **Step 2: Seed after MCP init** + +At that site, after `await initializeMcpForTask(...)`, add: + +```ts + const _activeSet = getActiveMcpToolSet() + if (_activeSet) { + await _activeSet.seed(firstUserTaskText) + } +``` + +(Use the actual variable holding the first user task text at that location; import `getActiveMcpToolSet` from `@core/mcp/retrieval/session`.) + +- [ ] **Step 3: Wire `activeMcpTools` into SystemPromptContext construction** + +Find where `SystemPromptContext` is built per request: `grep -rn "providerInfo:" src/core/task | grep -i context` and the `getSystemPrompt(promptContext)` caller (`src/core/task/ApiRequestHandler.ts` and the CLI equivalent). In the object literal that builds the context, add: + +```ts + activeMcpTools: getActiveMcpToolSet()?.snapshot(), +``` + +- [ ] **Step 4: Typecheck + cli build** + +Run: `npm run check-types` +Expected: PASS. +Run: `npm run cli:build` +Expected: build OK. + +- [ ] **Step 5: Commit** + +```bash +git add src/core/task +git commit -m "feat(mcp): seed base active set from first prompt + expose to prompt context" +``` + +--- + +## Task 13: CLI flags `--mcp-top-k` / `--mcp-threshold` + +**Files:** +- Modify: `cli/src/index.ts` (two command definitions, near the existing `--mcp` / `--no-mcp` options at ~lines 64-72 and ~286-290) + +- [ ] **Step 1: Add the options + env wiring** + +For BOTH command definitions that currently declare `--mcp`/`--no-mcp`, add: + +```ts + .option("--mcp-top-k ", "Max number of MCP tools to preload by relevance (default 8)") + .option("--mcp-threshold <τ>", "Min cosine similarity to preload an MCP tool (default 0.3)") +``` + +In the action handler block where `AILIANCE_NO_MCP` / `AILIANCE_MCP_SERVERS` are set from options, add: + +```ts + if (typeof options.mcpTopK === "string") { + process.env.AILIANCE_MCP_TOP_K = options.mcpTopK + } + if (typeof options.mcpThreshold === "string") { + process.env.AILIANCE_MCP_THRESHOLD = options.mcpThreshold + } +``` + +- [ ] **Step 2: Typecheck + cli build** + +Run: `npm run check-types` +Expected: PASS. +Run: `npm run cli:build` +Expected: build OK. + +- [ ] **Step 3: Manual smoke** + +Run: `node cli/dist/cli.mjs --help` +Expected: `--mcp-top-k` and `--mcp-threshold` appear in the options list. + +- [ ] **Step 4: Commit** + +```bash +git add cli/src/index.ts +git commit -m "feat(cli): --mcp-top-k / --mcp-threshold tuning flags" +``` + +--- + +## Task 14: Full check + e2e regression + +**Files:** none (verification) + +- [ ] **Step 1: Full typecheck + lint + core tests + cli build** + +```bash +npm run check-types +npm run lint +npx cross-env TS_NODE_PROJECT=./tsconfig.unit-test.json mocha +npm run cli:build +``` +Expected: all green; new retrieval + find_tools tests included in the mocha count. + +- [ ] **Step 2: e2e — tool count + empty-array rate** + +Create a temp project and run the multi-file-read task that previously triggered 2 empty-array events/run with 95 tools. Confirm the request now carries ≤ K + ~26 tools and the task completes without an empty-array retry loop. + +```bash +mkdir -p /tmp/mcp-e2e/src && cd /tmp/mcp-e2e +printf '# Demo\nA small project.\n' > README.md +printf 'def a(): return 1\n' > src/a.py +printf 'def b(): return 2\n' > src/b.py +node /Users/electron/code/ailiance-agent/cli/dist/cli.mjs -y --json -t 90 \ + "Read the README, then read src/a.py and src/b.py one at a time, then summarize the project." \ + > /tmp/mcp-e2e.json 2>&1 +grep -c "without providing a value\|Missing value for required parameter" /tmp/mcp-e2e.json # expect 0 +grep -c '"completion_result"' /tmp/mcp-e2e.json # expect >=1 +``` +Expected: 0 empty-array events, task completes. (If non-zero, lower `--mcp-threshold` or raise `--mcp-top-k` and re-test; tune the defaults in `config.ts`.) + +- [ ] **Step 3: `find_tools` round-trip smoke** + +Run a task that needs an MCP capability NOT in the base set (e.g. a github action), and confirm the model calls `find_tools`, the response lists activated tools, and a subsequent turn can call one. + +- [ ] **Step 4: `--no-mcp` still fully off** + +```bash +node /Users/electron/code/ailiance-agent/cli/dist/cli.mjs -y --json --no-mcp -t 60 "List files here." > /tmp/nomcp.json 2>&1 +# confirm no find_tools / no MCP tools loaded, task runs +``` + +- [ ] **Step 5: Commit any default tuning** + +```bash +git add src/core/mcp/retrieval/config.ts +git commit -m "chore(mcp): tune retrieval defaults from e2e (K/findK/threshold)" +``` + +--- + +## Task 15: Ship — critic + PR + +- [ ] **Step 1: Run the pre-ship critic** (`/ship-critic`) on the branch diff vs `master`; address MAJOR findings. +- [ ] **Step 2: Open PR** `feat/mcp-adaptive-retrieval` → `master` with a body summarizing: problem (95-tool bloat → empty arrays), approach (relevance-gated MCP + find_tools), evidence (e2e tool-count + empty-array rate), supply-chain note (pinned/vendored model). +- [ ] **Step 3:** Do NOT merge until the model weights are mirrored into the `ailiance` org per the HITL supply-chain policy (call this out as a merge blocker in the PR). + +--- + +## Notes for the implementer + +- The `missingToolParameterError` wording change (sharper empty-array message) is an independent improvement already in the working tree on this branch; keep it. +- Native tools are NEVER gated — only specs whose `id` is an `mcp__…` qualified name carry the active-set `contextRequirements`. `find_tools` has no `contextRequirements`, so it is always emitted. +- `~/.dirac` is the existing CLI storage dir (see `src/core/storage`); the vector cache lives alongside it. +- If `getSystemPrompt` is also called from the VS Code extension host path, `getActiveMcpToolSet()` returns `undefined` there (no CLI bootstrap) → MCP specs fall back to legacy "all". That is acceptable for v1 (the empty-array problem is CLI/worker-specific); a follow-up can wire the extension host similarly. From 8a46ebac3fe2555b82cedad08fa4c0e8030a5b4c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:04:43 +0200 Subject: [PATCH 04/24] feat(mcp): cosine similarity + top-K selection --- .../mcp/retrieval/__tests__/cosine.test.ts | 29 ++++++++++++++++ src/core/mcp/retrieval/cosine.ts | 33 +++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 src/core/mcp/retrieval/__tests__/cosine.test.ts create mode 100644 src/core/mcp/retrieval/cosine.ts diff --git a/src/core/mcp/retrieval/__tests__/cosine.test.ts b/src/core/mcp/retrieval/__tests__/cosine.test.ts new file mode 100644 index 00000000..3e7fc8b5 --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/cosine.test.ts @@ -0,0 +1,29 @@ +import { describe, it } from "mocha" +import "should" +import { cosineSim, selectTopK } from "../cosine" + +describe("cosine retrieval", () => { + it("computes cosine similarity of normalized-ish vectors", () => { + cosineSim(new Float32Array([1, 0]), new Float32Array([1, 0])).should.be.approximately(1, 1e-6) + cosineSim(new Float32Array([1, 0]), new Float32Array([0, 1])).should.be.approximately(0, 1e-6) + cosineSim(new Float32Array([0, 0]), new Float32Array([1, 0])).should.equal(0) + }) + + it("selectTopK applies threshold then caps, sorted by score desc", () => { + const query = new Float32Array([1, 0]) + const items = [ + { id: "a", vec: new Float32Array([1, 0]) }, + { id: "b", vec: new Float32Array([0.9, 0.1]) }, + { id: "c", vec: new Float32Array([0, 1]) }, + ] + selectTopK(query, items, { k: 1, threshold: 0.3 }).should.deepEqual(["a"]) + selectTopK(query, items, { k: 5, threshold: 0.3 }).should.deepEqual(["a", "b"]) + selectTopK(query, items, { k: 5, threshold: 0.99 }).should.deepEqual(["a", "b"]) + }) + + it("returns [] when nothing clears the threshold or items is empty", () => { + const q = new Float32Array([1, 0]) + selectTopK(q, [{ id: "c", vec: new Float32Array([0, 1]) }], { k: 5, threshold: 0.3 }).should.deepEqual([]) + selectTopK(q, [], { k: 5, threshold: 0.3 }).should.deepEqual([]) + }) +}) diff --git a/src/core/mcp/retrieval/cosine.ts b/src/core/mcp/retrieval/cosine.ts new file mode 100644 index 00000000..8b6bae07 --- /dev/null +++ b/src/core/mcp/retrieval/cosine.ts @@ -0,0 +1,33 @@ +export function cosineSim(a: Float32Array, b: Float32Array): number { + let dot = 0 + let na = 0 + let nb = 0 + for (let i = 0; i < a.length; i++) { + dot += a[i] * b[i] + na += a[i] * a[i] + nb += b[i] * b[i] + } + if (na === 0 || nb === 0) { + return 0 + } + return dot / (Math.sqrt(na) * Math.sqrt(nb)) +} + +export interface ScoredItem { + id: string + vec: Float32Array +} + +export interface SelectOptions { + k: number + threshold: number +} + +export function selectTopK(query: Float32Array, items: ScoredItem[], opts: SelectOptions): string[] { + return items + .map((it) => ({ id: it.id, score: cosineSim(query, it.vec) })) + .filter((s) => s.score >= opts.threshold) + .sort((a, b) => b.score - a.score) + .slice(0, Math.max(0, opts.k)) + .map((s) => s.id) +} From ef8d58bcc5c73101171950491b1d40848a3dda89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:08:52 +0200 Subject: [PATCH 05/24] feat(mcp): retrieval config with env overrides --- .../mcp/retrieval/__tests__/config.test.ts | 22 +++++++++++++++++ src/core/mcp/retrieval/config.ts | 24 +++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 src/core/mcp/retrieval/__tests__/config.test.ts create mode 100644 src/core/mcp/retrieval/config.ts diff --git a/src/core/mcp/retrieval/__tests__/config.test.ts b/src/core/mcp/retrieval/__tests__/config.test.ts new file mode 100644 index 00000000..3038505f --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/config.test.ts @@ -0,0 +1,22 @@ +import { describe, it, afterEach } from "mocha" +import "should" +import { getRetrievalConfig } from "../config" + +describe("getRetrievalConfig", () => { + afterEach(() => { + delete process.env.AILIANCE_MCP_TOP_K + delete process.env.AILIANCE_MCP_FIND_K + delete process.env.AILIANCE_MCP_THRESHOLD + }) + + it("returns sane defaults", () => { + getRetrievalConfig().should.deepEqual({ baseK: 8, findK: 5, threshold: 0.3 }) + }) + + it("honors env overrides and ignores invalid ones", () => { + process.env.AILIANCE_MCP_TOP_K = "12" + process.env.AILIANCE_MCP_THRESHOLD = "0.45" + process.env.AILIANCE_MCP_FIND_K = "not-a-number" + getRetrievalConfig().should.deepEqual({ baseK: 12, findK: 5, threshold: 0.45 }) + }) +}) diff --git a/src/core/mcp/retrieval/config.ts b/src/core/mcp/retrieval/config.ts new file mode 100644 index 00000000..1de3fe75 --- /dev/null +++ b/src/core/mcp/retrieval/config.ts @@ -0,0 +1,24 @@ +export interface RetrievalConfig { + baseK: number + findK: number + threshold: number +} + +const DEFAULTS: RetrievalConfig = { baseK: 8, findK: 5, threshold: 0.3 } + +function numEnv(name: string, fallback: number): number { + const raw = process.env[name] + if (raw === undefined) { + return fallback + } + const n = Number(raw) + return Number.isFinite(n) ? n : fallback +} + +export function getRetrievalConfig(): RetrievalConfig { + return { + baseK: numEnv("AILIANCE_MCP_TOP_K", DEFAULTS.baseK), + findK: numEnv("AILIANCE_MCP_FIND_K", DEFAULTS.findK), + threshold: numEnv("AILIANCE_MCP_THRESHOLD", DEFAULTS.threshold), + } +} From b80edd3a1d1a6812ee7e8488078001180432b06d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:11:57 +0200 Subject: [PATCH 06/24] build(mcp): pin transformers.js for tool embedding --- package-lock.json | 544 +++++++++++++++++++++++++++++++++++++++++++++- package.json | 1 + 2 files changed, 539 insertions(+), 6 deletions(-) diff --git a/package-lock.json b/package-lock.json index acf39c61..6dcdb3a5 100644 --- a/package-lock.json +++ b/package-lock.json @@ -22,6 +22,7 @@ "@google/genai": "^1.49.0", "@grpc/grpc-js": "^1.9.15", "@grpc/reflection": "^1.0.4", + "@huggingface/transformers": "3.3.3", "@mistralai/mistralai": "^2.1.2", "@modelcontextprotocol/sdk": "^1.25.1", "@opentelemetry/api": "^1.9.0", @@ -3203,6 +3204,427 @@ "hono": "^4" } }, + "node_modules/@huggingface/jinja": { + "version": "0.3.4", + "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.3.4.tgz", + "integrity": "sha512-kFFQWJiWwvxezKQnvH1X7GjsECcMljFx+UZK9hx6P26aVHwwidJVTB0ptLfRVZQvVkOGHoMmTGvo4nT0X9hHOA==", + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/@huggingface/transformers": { + "version": "3.3.3", + "resolved": "https://registry.npmjs.org/@huggingface/transformers/-/transformers-3.3.3.tgz", + "integrity": "sha512-OcMubhBjW6u1xnp0zSt5SvCxdGHuhP2k+w2Vlm3i0vNcTJhJTZWxxYQmPBfcb7PX+Q6c43lGSzWD6tsJFwka4Q==", + "license": "Apache-2.0", + "dependencies": { + "@huggingface/jinja": "^0.3.3", + "onnxruntime-node": "1.20.1", + "onnxruntime-web": "1.21.0-dev.20250206-d981b153d3", + "sharp": "^0.33.5" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-darwin-arm64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.33.5.tgz", + "integrity": "sha512-UT4p+iz/2H4twwAoLCqfA9UH5pI6DggwKEGuaPy7nCVQ8ZsiY5PIcrRvD1DzuY3qYL07NtIQcWnBSY/heikIFQ==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-arm64": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-darwin-x64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.33.5.tgz", + "integrity": "sha512-fyHac4jIc1ANYGRDxtiqelIbdWkIuQaI84Mv45KvGRRxSAa7o7d1ZKAOBaYbnepLC1WqxfpimdeWfvqqSGwR2Q==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-x64": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-darwin-arm64": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.0.4.tgz", + "integrity": "sha512-XblONe153h0O2zuFfTAbQYAX2JhYmDHeWikp1LM9Hul9gVPjFY427k6dFEcOL72O01QxQsWi761svJ/ev9xEDg==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-darwin-x64": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.0.4.tgz", + "integrity": "sha512-xnGR8YuZYfJGmWPvmlunFaWJsb9T/AO2ykoP3Fz/0X5XV2aoYBPkX6xqCQvUTKKiLddarLaxpzNe+b1hjeWHAQ==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-linux-arm": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.0.5.tgz", + "integrity": "sha512-gvcC4ACAOPRNATg/ov8/MnbxFDJqf/pDePbBnuBDcjsI8PssmjoKMAz4LtLaVi+OnSb5FK/yIOamqDwGmXW32g==", + "cpu": [ + "arm" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-linux-arm64": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.0.4.tgz", + "integrity": "sha512-9B+taZ8DlyyqzZQnoeIvDVR/2F4EbMepXMc/NdVbkzsJbzkUjhXv/70GQJ7tdLA4YJgNP25zukcxpX2/SueNrA==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-linux-s390x": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.0.4.tgz", + "integrity": "sha512-u7Wz6ntiSSgGSGcjZ55im6uvTrOxSIS8/dgoVMoiGE9I6JAfU50yH5BoDlYA1tcuGS7g/QNtetJnxA6QEsCVTA==", + "cpu": [ + "s390x" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-linux-x64": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.0.4.tgz", + "integrity": "sha512-MmWmQ3iPFZr0Iev+BAgVMb3ZyC4KeFc3jFxnNbEPas60e1cIfevbtuyf9nDGIzOaW9PdnDciJm+wFFaTlj5xYw==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-linuxmusl-arm64": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.0.4.tgz", + "integrity": "sha512-9Ti+BbTYDcsbp4wfYib8Ctm1ilkugkA/uscUn6UXK1ldpC1JjiXbLfFZtRlBhjPZ5o1NCLiDbg8fhUPKStHoTA==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-libvips-linuxmusl-x64": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.0.4.tgz", + "integrity": "sha512-viYN1KX9m+/hGkJtvYYp+CCLgnJXwiQB39damAO7WMdKWlIhmYTfHjwSbQeUK/20vY154mwezd9HflVFM1wVSw==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-linux-arm": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.33.5.tgz", + "integrity": "sha512-JTS1eldqZbJxjvKaAkxhZmBqPRGmxgu+qFKSInv8moZ2AmT5Yib3EQ1c6gp493HvrvV8QgdOXdyaIBrhvFhBMQ==", + "cpu": [ + "arm" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm": "1.0.5" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-linux-arm64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.33.5.tgz", + "integrity": "sha512-JMVv+AMRyGOHtO1RFBiJy/MBsgz0x4AWrT6QoEVVTyh1E39TrCUpTRI7mx9VksGX4awWASxqCYLCV4wBZHAYxA==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm64": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-linux-s390x": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.33.5.tgz", + "integrity": "sha512-y/5PCd+mP4CA/sPDKl2961b+C9d+vPAveS33s6Z3zfASk2j5upL6fXVPZi7ztePZ5CuH+1kW8JtvxgbuXHRa4Q==", + "cpu": [ + "s390x" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-s390x": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-linux-x64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.33.5.tgz", + "integrity": "sha512-opC+Ok5pRNAzuvq1AG0ar+1owsu842/Ab+4qvU879ippJBHvyY5n2mxF1izXqkPYlGuP/M556uh53jRLJmzTWA==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-x64": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-linuxmusl-arm64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.33.5.tgz", + "integrity": "sha512-XrHMZwGQGvJg2V/oRSUfSAfjfPxO+4DkiRh6p2AFjLQztWUuY/o8Mq0eMQVIY7HJ1CDQUJlxGGZRw1a5bqmd1g==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-arm64": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-linuxmusl-x64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.33.5.tgz", + "integrity": "sha512-WT+d/cgqKkkKySYmqoZ8y3pxx7lx9vVejxW/W4DOFMYVSkErR+w7mf2u8m/y4+xHe7yY9DAXQMWQhpnMuFfScw==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-x64": "1.0.4" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-wasm32": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.33.5.tgz", + "integrity": "sha512-ykUW4LVGaMcU9lu9thv85CbRMAwfeadCJHRsg2GmeRa/cJxsVY9Rbd57JcMxBkKHag5U/x7TSBpScF4U8ElVzg==", + "cpu": [ + "wasm32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT", + "optional": true, + "dependencies": { + "@emnapi/runtime": "^1.2.0" + }, + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-win32-ia32": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.33.5.tgz", + "integrity": "sha512-T36PblLaTwuVJ/zw/LaH0PdZkRz5rd3SmMHX8GSmR7vtNSP5Z6bQkExdSK7xGWyxLw4sUknBuugTelgw2faBbQ==", + "cpu": [ + "ia32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/@img/sharp-win32-x64": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.33.5.tgz", + "integrity": "sha512-MpY/o8/8kj+EcnxwvrP4aTJSWw/aZ7JIGR4aBeZkZw5B7/Jn+tY9/VNwtcoGmdT7GfggGIU4kygOMSbYnOrAbg==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@huggingface/transformers/node_modules/sharp": { + "version": "0.33.5", + "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.33.5.tgz", + "integrity": "sha512-haPVm1EkS9pgvHrQ/F3Xy+hgcuMV0Wm9vfIBSiwZ05k+xgb0PkBQpGsAA/oWdDobNaZTH5ppvHtzCFbnSEwHVw==", + "hasInstallScript": true, + "license": "Apache-2.0", + "dependencies": { + "color": "^4.2.3", + "detect-libc": "^2.0.3", + "semver": "^7.6.3" + }, + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-darwin-arm64": "0.33.5", + "@img/sharp-darwin-x64": "0.33.5", + "@img/sharp-libvips-darwin-arm64": "1.0.4", + "@img/sharp-libvips-darwin-x64": "1.0.4", + "@img/sharp-libvips-linux-arm": "1.0.5", + "@img/sharp-libvips-linux-arm64": "1.0.4", + "@img/sharp-libvips-linux-s390x": "1.0.4", + "@img/sharp-libvips-linux-x64": "1.0.4", + "@img/sharp-libvips-linuxmusl-arm64": "1.0.4", + "@img/sharp-libvips-linuxmusl-x64": "1.0.4", + "@img/sharp-linux-arm": "0.33.5", + "@img/sharp-linux-arm64": "0.33.5", + "@img/sharp-linux-s390x": "0.33.5", + "@img/sharp-linux-x64": "0.33.5", + "@img/sharp-linuxmusl-arm64": "0.33.5", + "@img/sharp-linuxmusl-x64": "0.33.5", + "@img/sharp-wasm32": "0.33.5", + "@img/sharp-win32-ia32": "0.33.5", + "@img/sharp-win32-x64": "0.33.5" + } + }, "node_modules/@img/colour": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.0.0.tgz", @@ -3712,7 +4134,6 @@ "version": "4.0.1", "resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz", "integrity": "sha512-wgm9Ehl2jpeqP3zw/7mo3kRHFp5MEDhqAdwy1fTGkHAwnkGOVsgpvQhL8B5n1qlb01jV3n/bI0ZfZp5lWA1k4w==", - "dev": true, "license": "ISC", "dependencies": { "minipass": "^7.0.4" @@ -10173,7 +10594,6 @@ "version": "3.0.0", "resolved": "https://registry.npmjs.org/chownr/-/chownr-3.0.0.tgz", "integrity": "sha512-+IxzY9BZOQd/XuYPRmrvEVjF/nqj5kgT4kEq7VofrDoM1MxoRjEWkrCC3EtLi59TVawxTAn+orJwFQcrqEN1+g==", - "dev": true, "license": "BlueOak-1.0.0", "engines": { "node": ">=18" @@ -10451,6 +10871,19 @@ "node": "^12.20.0 || ^14.13.1 || >=16.0.0" } }, + "node_modules/color": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz", + "integrity": "sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==", + "license": "MIT", + "dependencies": { + "color-convert": "^2.0.1", + "color-string": "^1.9.0" + }, + "engines": { + "node": ">=12.5.0" + } + }, "node_modules/color-convert": { "version": "1.9.3", "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-1.9.3.tgz", @@ -10465,9 +10898,18 @@ "version": "1.1.3", "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.3.tgz", "integrity": "sha512-72fSenhMw2HZMTVHeCA9KCmpEIbzWiQsjN+BHcBbS9vr1mtt+vJjPdksIBNUmKAW8TFUDPJK5SUU3QhE9NEXDw==", - "dev": true, "license": "MIT" }, + "node_modules/color-string": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz", + "integrity": "sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==", + "license": "MIT", + "dependencies": { + "color-name": "^1.0.0", + "simple-swizzle": "^0.2.2" + } + }, "node_modules/color-support": { "version": "1.1.3", "resolved": "https://registry.npmjs.org/color-support/-/color-support-1.1.3.tgz", @@ -10477,6 +10919,24 @@ "color-support": "bin.js" } }, + "node_modules/color/node_modules/color-convert": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", + "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", + "license": "MIT", + "dependencies": { + "color-name": "~1.1.4" + }, + "engines": { + "node": ">=7.0.0" + } + }, + "node_modules/color/node_modules/color-name": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", + "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", + "license": "MIT" + }, "node_modules/colorette": { "version": "2.0.20", "resolved": "https://registry.npmjs.org/colorette/-/colorette-2.0.20.tgz", @@ -12400,6 +12860,12 @@ "flat": "cli.js" } }, + "node_modules/flatbuffers": { + "version": "25.9.23", + "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-25.9.23.tgz", + "integrity": "sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ==", + "license": "Apache-2.0" + }, "node_modules/follow-redirects": { "version": "1.16.0", "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz", @@ -13151,6 +13617,12 @@ "node": ">=14.0.0" } }, + "node_modules/guid-typescript": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", + "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", + "license": "ISC" + }, "node_modules/has-bigints": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/has-bigints/-/has-bigints-1.1.0.tgz", @@ -16573,7 +17045,6 @@ "version": "3.1.0", "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.1.0.tgz", "integrity": "sha512-KZxYo1BUkWD2TVFLr0MQoM8vUUigWD3LlD83a/75BqC+4qE0Hb1Vo5v1FgcfaNXvfXzr+5EhQ6ing/CaBijTlw==", - "dev": true, "license": "MIT", "dependencies": { "minipass": "^7.1.2" @@ -17709,6 +18180,48 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/onnxruntime-common": { + "version": "1.20.1", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.20.1.tgz", + "integrity": "sha512-YiU0s0IzYYC+gWvqD1HzLc46Du1sXpSiwzKb63PACIJr6LfL27VsXSXQvt68EzD3V0D5Bc0vyJTjmMxp0ylQiw==", + "license": "MIT" + }, + "node_modules/onnxruntime-node": { + "version": "1.20.1", + "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.20.1.tgz", + "integrity": "sha512-di/I4HDXRw+FLgq+TyHmQEDd3cEp9iFFZm0r4uJ1Wd7b/WE1VXtKWo8yemex347c6GNF/3Pv86ZfPhIWxORr0w==", + "hasInstallScript": true, + "license": "MIT", + "os": [ + "win32", + "darwin", + "linux" + ], + "dependencies": { + "onnxruntime-common": "1.20.1", + "tar": "^7.0.1" + } + }, + "node_modules/onnxruntime-web": { + "version": "1.21.0-dev.20250206-d981b153d3", + "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.21.0-dev.20250206-d981b153d3.tgz", + "integrity": "sha512-esDVQdRic6J44VBMFLumYvcGfioMh80ceLmzF1yheJyuLKq/Th8VT2aj42XWQst+2bcWnAhw4IKmRQaqzU8ugg==", + "license": "MIT", + "dependencies": { + "flatbuffers": "^25.1.24", + "guid-typescript": "^1.0.9", + "long": "^5.2.3", + "onnxruntime-common": "1.21.0-dev.20250206-d981b153d3", + "platform": "^1.3.6", + "protobufjs": "^7.2.4" + } + }, + "node_modules/onnxruntime-web/node_modules/onnxruntime-common": { + "version": "1.21.0-dev.20250206-d981b153d3", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.21.0-dev.20250206-d981b153d3.tgz", + "integrity": "sha512-TwaE51xV9q2y8pM61q73rbywJnusw9ivTEHAJ39GVWNZqxCoDBpe/tQkh/w9S+o/g+zS7YeeL0I/2mEWd+dgyA==", + "license": "MIT" + }, "node_modules/open": { "version": "10.2.0", "resolved": "https://registry.npmjs.org/open/-/open-10.2.0.tgz", @@ -18421,6 +18934,12 @@ "node": ">=8" } }, + "node_modules/platform": { + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", + "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", + "license": "MIT" + }, "node_modules/playwright": { "version": "1.58.1", "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.58.1.tgz", @@ -20226,6 +20745,21 @@ "url": "https://github.com/steveukx/git-js?sponsor=1" } }, + "node_modules/simple-swizzle": { + "version": "0.2.4", + "resolved": "https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.4.tgz", + "integrity": "sha512-nAu1WFPQSMNr2Zn9PGSZK9AGn4t/y97lEm+MXTtUDwfP0ksAIX4nO+6ruD9Jwut4C49SB1Ws+fbXsm/yScWOHw==", + "license": "MIT", + "dependencies": { + "is-arrayish": "^0.3.1" + } + }, + "node_modules/simple-swizzle/node_modules/is-arrayish": { + "version": "0.3.4", + "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.4.tgz", + "integrity": "sha512-m6UrgzFVUYawGBh1dUsWR5M2Clqic9RVXC/9f8ceNlv2IcO9j9J/z8UoCLPqtsPBFNzEpfR3xftohbfqDx8EQA==", + "license": "MIT" + }, "node_modules/sinon": { "version": "19.0.5", "resolved": "https://registry.npmjs.org/sinon/-/sinon-19.0.5.tgz", @@ -21089,7 +21623,6 @@ "version": "7.5.13", "resolved": "https://registry.npmjs.org/tar/-/tar-7.5.13.tgz", "integrity": "sha512-tOG/7GyXpFevhXVh8jOPJrmtRpOTsYqUIkVdVooZYJS/z8WhfQUX8RJILmeuJNinGAMSu1veBr4asSHFt5/hng==", - "dev": true, "license": "BlueOak-1.0.0", "dependencies": { "@isaacs/fs-minipass": "^4.0.0", @@ -21131,7 +21664,6 @@ "version": "5.0.0", "resolved": "https://registry.npmjs.org/yallist/-/yallist-5.0.0.tgz", "integrity": "sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw==", - "dev": true, "license": "BlueOak-1.0.0", "engines": { "node": ">=18" diff --git a/package.json b/package.json index e02c82a6..2615ea28 100644 --- a/package.json +++ b/package.json @@ -520,6 +520,7 @@ "@google/genai": "^1.49.0", "@grpc/grpc-js": "^1.9.15", "@grpc/reflection": "^1.0.4", + "@huggingface/transformers": "3.3.3", "@mistralai/mistralai": "^2.1.2", "@modelcontextprotocol/sdk": "^1.25.1", "@opentelemetry/api": "^1.9.0", From df423a7c163b6c88bd9ed18ac18ecc07b5c90778 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:15:37 +0200 Subject: [PATCH 07/24] feat(mcp): lazy injectable local embedder --- src/core/mcp/retrieval/Embedder.ts | 36 ++++++++++++++++++ .../mcp/retrieval/__tests__/Embedder.test.ts | 37 +++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 src/core/mcp/retrieval/Embedder.ts create mode 100644 src/core/mcp/retrieval/__tests__/Embedder.test.ts diff --git a/src/core/mcp/retrieval/Embedder.ts b/src/core/mcp/retrieval/Embedder.ts new file mode 100644 index 00000000..24f91013 --- /dev/null +++ b/src/core/mcp/retrieval/Embedder.ts @@ -0,0 +1,36 @@ +export type EmbedFn = (texts: string[]) => Promise +export type PipelineFactory = () => Promise + +/** + * Lazy local text embedder. The pipeline factory is injected so tests run + * without loading the real ONNX model. Production wires the transformers.js + * factory in `createDefaultEmbedder`. + */ +export class Embedder { + private pipelinePromise: Promise | undefined + + constructor(private readonly factory: PipelineFactory) {} + + async embed(texts: string[]): Promise { + if (!this.pipelinePromise) { + this.pipelinePromise = this.factory() + } + const fn = await this.pipelinePromise + return fn(texts) + } +} + +/** + * Production factory: all-MiniLM-L6-v2 ONNX via transformers.js, mean-pooled + + * normalized 384-d. Imported lazily so `--no-mcp` never pays the import cost. + */ +export function createDefaultEmbedder(): Embedder { + return new Embedder(async () => { + const { pipeline } = await import("@huggingface/transformers") + const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2") + return async (texts: string[]) => { + const out = await (extractor as any)(texts, { pooling: "mean", normalize: true }) + return ((out as { tolist(): number[][] }).tolist()).map((row) => Float32Array.from(row)) + } + }) +} diff --git a/src/core/mcp/retrieval/__tests__/Embedder.test.ts b/src/core/mcp/retrieval/__tests__/Embedder.test.ts new file mode 100644 index 00000000..1afb1f78 --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/Embedder.test.ts @@ -0,0 +1,37 @@ +import { describe, it } from "mocha" +import "should" +import { Embedder } from "../Embedder" + +describe("Embedder", () => { + it("embeds via the injected pipeline and returns Float32Array[]", async () => { + const fakePipeline = async (texts: string[]) => + texts.map((t) => new Float32Array([t.length, 0, 0])) + const e = new Embedder(async () => fakePipeline) + const [v] = await e.embed(["abc"]) + Array.from(v).should.deepEqual([3, 0, 0]) + }) + + it("loads the pipeline only once (lazy, memoized)", async () => { + let loads = 0 + const e = new Embedder(async () => { + loads++ + return async (texts: string[]) => texts.map(() => new Float32Array([1])) + }) + await e.embed(["a"]) + await e.embed(["b"]) + loads.should.equal(1) + }) + + it("propagates load failure (caller degrades)", async () => { + const e = new Embedder(async () => { + throw new Error("model load failed") + }) + let threw = false + try { + await e.embed(["a"]) + } catch { + threw = true + } + threw.should.equal(true) + }) +}) From a7fe5d037b336f8e73829f5a6036a5522d769245 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:19:54 +0200 Subject: [PATCH 08/24] feat(mcp): tool-vector index with hash-keyed cache --- src/core/mcp/retrieval/ToolVectorIndex.ts | 55 +++++++++++++++++++ .../__tests__/ToolVectorIndex.test.ts | 51 +++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 src/core/mcp/retrieval/ToolVectorIndex.ts create mode 100644 src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts diff --git a/src/core/mcp/retrieval/ToolVectorIndex.ts b/src/core/mcp/retrieval/ToolVectorIndex.ts new file mode 100644 index 00000000..9056abb5 --- /dev/null +++ b/src/core/mcp/retrieval/ToolVectorIndex.ts @@ -0,0 +1,55 @@ +import * as crypto from "crypto" +import * as fs from "fs" +import * as path from "path" +import type { Embedder } from "./Embedder" + +export interface ToolText { + qualifiedName: string + text: string +} + +interface CacheEntry { + hash: string + vec: number[] +} + +function hashText(text: string): string { + return crypto.createHash("sha256").update(text).digest("hex").slice(0, 16) +} + +export class ToolVectorIndex { + constructor( + private readonly embedder: Embedder, + private readonly cachePath: string, + ) {} + + private readCache(): Record { + try { + return JSON.parse(fs.readFileSync(this.cachePath, "utf8")) + } catch { + return {} + } + } + + private writeCache(cache: Record): void { + fs.mkdirSync(path.dirname(this.cachePath), { recursive: true }) + fs.writeFileSync(this.cachePath, JSON.stringify(cache)) + } + + async build(tools: ToolText[]): Promise> { + const cache = this.readCache() + const stale = tools.filter((t) => cache[t.qualifiedName]?.hash !== hashText(t.text)) + if (stale.length > 0) { + const vecs = await this.embedder.embed(stale.map((t) => t.text)) + stale.forEach((t, i) => { + cache[t.qualifiedName] = { hash: hashText(t.text), vec: Array.from(vecs[i]) } + }) + this.writeCache(cache) + } + const result = new Map() + for (const t of tools) { + result.set(t.qualifiedName, Float32Array.from(cache[t.qualifiedName].vec)) + } + return result + } +} diff --git a/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts b/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts new file mode 100644 index 00000000..2e9594ec --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts @@ -0,0 +1,51 @@ +import { describe, it, beforeEach, afterEach } from "mocha" +import "should" +import * as fs from "fs" +import * as os from "os" +import * as path from "path" +import { Embedder } from "../Embedder" +import { ToolVectorIndex } from "../ToolVectorIndex" + +function tmpFile(): string { + return path.join(fs.mkdtempSync(path.join(os.tmpdir(), "tvi-")), "index.json") +} + +describe("ToolVectorIndex", () => { + let cachePath: string + beforeEach(() => { + cachePath = tmpFile() + }) + afterEach(() => { + fs.rmSync(path.dirname(cachePath), { recursive: true, force: true }) + }) + + it("embeds each tool once and returns vectors keyed by qualifiedName", async () => { + let embedCalls = 0 + const embedder = new Embedder(async () => async (texts: string[]) => { + embedCalls += texts.length + return texts.map((t) => new Float32Array([t.length])) + }) + const idx = new ToolVectorIndex(embedder, cachePath) + const tools = [ + { qualifiedName: "mcp__p_s__a", text: "alpha" }, + { qualifiedName: "mcp__p_s__b", text: "beta!" }, + ] + const vecs = await idx.build(tools) + embedCalls.should.equal(2) + Array.from(vecs.get("mcp__p_s__a")!).should.deepEqual([5]) + }) + + it("re-uses the disk cache and only embeds new/changed tools", async () => { + let embedCalls = 0 + const embedder = new Embedder(async () => async (texts: string[]) => { + embedCalls += texts.length + return texts.map((t) => new Float32Array([t.length])) + }) + const tools = [{ qualifiedName: "mcp__p_s__a", text: "alpha" }] + await new ToolVectorIndex(embedder, cachePath).build(tools) + await new ToolVectorIndex(embedder, cachePath).build(tools) + embedCalls.should.equal(1) + await new ToolVectorIndex(embedder, cachePath).build([{ qualifiedName: "mcp__p_s__a", text: "alpha2" }]) + embedCalls.should.equal(2) + }) +}) From dcf72c106bd822182b903ccb164c687da6d62793 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:23:55 +0200 Subject: [PATCH 09/24] feat(mcp): session active-tool set seed + expand --- src/core/mcp/retrieval/ActiveMcpToolSet.ts | 56 +++++++++++++++++++ .../__tests__/ActiveMcpToolSet.test.ts | 45 +++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 src/core/mcp/retrieval/ActiveMcpToolSet.ts create mode 100644 src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts diff --git a/src/core/mcp/retrieval/ActiveMcpToolSet.ts b/src/core/mcp/retrieval/ActiveMcpToolSet.ts new file mode 100644 index 00000000..8488f108 --- /dev/null +++ b/src/core/mcp/retrieval/ActiveMcpToolSet.ts @@ -0,0 +1,56 @@ +import type { RetrievalConfig } from "./config" +import { type ScoredItem, selectTopK } from "./cosine" +import type { Embedder } from "./Embedder" + +export class ActiveMcpToolSet { + private readonly active = new Set() + private readonly items: ScoredItem[] + private embedderOk = true + + constructor( + private readonly embedder: Embedder, + vectors: Map, + private readonly config: RetrievalConfig, + ) { + this.items = Array.from(vectors.entries()).map(([id, vec]) => ({ id, vec })) + } + + available(): boolean { + return this.embedderOk + } + + snapshot(): ReadonlySet { + return this.active + } + + private async select(text: string, k: number): Promise { + const [q] = await this.embedder.embed([text]) + return selectTopK(q, this.items, { k, threshold: this.config.threshold }) + } + + async seed(prompt: string): Promise { + try { + for (const id of await this.select(prompt, this.config.baseK)) { + this.active.add(id) + } + } catch { + this.embedderOk = false + } + } + + async expand(query: string): Promise { + try { + const added: string[] = [] + for (const id of await this.select(query, this.config.findK)) { + if (!this.active.has(id)) { + this.active.add(id) + added.push(id) + } + } + return added + } catch { + this.embedderOk = false + return [] + } + } +} diff --git a/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts b/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts new file mode 100644 index 00000000..22d0ec59 --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts @@ -0,0 +1,45 @@ +import { describe, it } from "mocha" +import "should" +import { Embedder } from "../Embedder" +import { ActiveMcpToolSet } from "../ActiveMcpToolSet" + +function makeEmbedder(map: Record) { + return new Embedder(async () => async (texts: string[]) => + texts.map((t) => Float32Array.from(map[t] ?? [0, 0])), + ) +} + +describe("ActiveMcpToolSet", () => { + const vectors = new Map([ + ["mcp__git__issues", Float32Array.from([1, 0])], + ["mcp__fs__read", Float32Array.from([0, 1])], + ]) + + it("seeds the base set from the prompt (cap + threshold)", async () => { + const embedder = makeEmbedder({ "find github issues": [1, 0] }) + const set = new ActiveMcpToolSet(embedder, vectors, { baseK: 8, findK: 5, threshold: 0.3 }) + await set.seed("find github issues") + set.snapshot().has("mcp__git__issues").should.equal(true) + set.snapshot().has("mcp__fs__read").should.equal(false) + }) + + it("expand() adds matching tools and is idempotent", async () => { + const embedder = makeEmbedder({ "read a file": [0, 1], seed: [1, 0] }) + const set = new ActiveMcpToolSet(embedder, vectors, { baseK: 8, findK: 5, threshold: 0.3 }) + await set.seed("seed") + const added = await set.expand("read a file") + added.should.deepEqual(["mcp__fs__read"]) + set.snapshot().has("mcp__fs__read").should.equal(true) + ;(await set.expand("read a file")).should.deepEqual([]) + }) + + it("degrades to empty set + available()=false when embedder throws", async () => { + const embedder = new Embedder(async () => { + throw new Error("no model") + }) + const set = new ActiveMcpToolSet(embedder, vectors, { baseK: 8, findK: 5, threshold: 0.3 }) + await set.seed("anything") + set.snapshot().size.should.equal(0) + set.available().should.equal(false) + }) +}) From 238601367248bc8985c33ff15c8d7ea43ca3dfcc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:27:41 +0200 Subject: [PATCH 10/24] feat(mcp): session active set + context field --- src/core/mcp/retrieval/__tests__/session.test.ts | 13 +++++++++++++ src/core/mcp/retrieval/session.ts | 11 +++++++++++ src/core/prompts/system-prompt/types.ts | 3 +++ 3 files changed, 27 insertions(+) create mode 100644 src/core/mcp/retrieval/__tests__/session.test.ts create mode 100644 src/core/mcp/retrieval/session.ts diff --git a/src/core/mcp/retrieval/__tests__/session.test.ts b/src/core/mcp/retrieval/__tests__/session.test.ts new file mode 100644 index 00000000..15e2f327 --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/session.test.ts @@ -0,0 +1,13 @@ +import { describe, it } from "mocha" +import "should" +import { getActiveMcpToolSet, setActiveMcpToolSet } from "../session" + +describe("active mcp tool set session", () => { + it("stores and returns the current set; undefined by default after clear", () => { + setActiveMcpToolSet(undefined) + ;(getActiveMcpToolSet() === undefined).should.equal(true) + const fake = { snapshot: () => new Set(["mcp__x__y"]) } as any + setActiveMcpToolSet(fake) + getActiveMcpToolSet()!.snapshot().has("mcp__x__y").should.equal(true) + }) +}) diff --git a/src/core/mcp/retrieval/session.ts b/src/core/mcp/retrieval/session.ts new file mode 100644 index 00000000..62399dd2 --- /dev/null +++ b/src/core/mcp/retrieval/session.ts @@ -0,0 +1,11 @@ +import type { ActiveMcpToolSet } from "./ActiveMcpToolSet" + +let current: ActiveMcpToolSet | undefined + +export function setActiveMcpToolSet(set: ActiveMcpToolSet | undefined): void { + current = set +} + +export function getActiveMcpToolSet(): ActiveMcpToolSet | undefined { + return current +} diff --git a/src/core/prompts/system-prompt/types.ts b/src/core/prompts/system-prompt/types.ts index 543cac6d..678067c7 100644 --- a/src/core/prompts/system-prompt/types.ts +++ b/src/core/prompts/system-prompt/types.ts @@ -37,6 +37,9 @@ export interface SystemPromptContext { readonly runtimePlaceholders?: Readonly> readonly yoloModeToggled?: boolean readonly subagentsEnabled?: boolean + /** Active MCP tool qualified names for adaptive retrieval. When set, only + * these MCP tools are emitted; undefined means "no gating" (legacy: all). */ + readonly activeMcpTools?: ReadonlySet readonly diracWebToolsEnabled?: boolean readonly isMultiRootEnabled?: boolean readonly workspaceRoots?: Array<{ path: string; name: string; vcs?: string }> From 14e6c7b337df3078dd89bb2c63522f13a547bb66 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:31:55 +0200 Subject: [PATCH 11/24] feat(mcp): gate MCP specs by active set --- src/core/mcp/bootstrap.ts | 6 +++-- src/core/mcp/retrieval/__tests__/gate.test.ts | 24 +++++++++++++++++++ 2 files changed, 28 insertions(+), 2 deletions(-) create mode 100644 src/core/mcp/retrieval/__tests__/gate.test.ts diff --git a/src/core/mcp/bootstrap.ts b/src/core/mcp/bootstrap.ts index 575d6128..72b65182 100644 --- a/src/core/mcp/bootstrap.ts +++ b/src/core/mcp/bootstrap.ts @@ -67,13 +67,15 @@ export function convertJsonSchemaToParams(inputSchema: object): NonNullable ctx.activeMcpTools === undefined || ctx.activeMcpTools.has(qualifiedName), } } diff --git a/src/core/mcp/retrieval/__tests__/gate.test.ts b/src/core/mcp/retrieval/__tests__/gate.test.ts new file mode 100644 index 00000000..44e5f219 --- /dev/null +++ b/src/core/mcp/retrieval/__tests__/gate.test.ts @@ -0,0 +1,24 @@ +import { describe, it } from "mocha" +import "should" +import { mcpToolToSpec } from "../../bootstrap" + +const meta = { + qualifiedName: "mcp__p_s__a", + serverId: "s", + pluginName: "p", + rawName: "a", + description: "does A", + inputSchema: { type: "object", properties: {} }, +} + +describe("mcpToolToSpec gating", () => { + it("is enabled when no gating set (undefined activeMcpTools)", () => { + const spec = mcpToolToSpec(meta as any) + spec.contextRequirements!({ activeMcpTools: undefined } as any).should.equal(true) + }) + it("is enabled only when present in the active set", () => { + const spec = mcpToolToSpec(meta as any) + spec.contextRequirements!({ activeMcpTools: new Set(["mcp__p_s__a"]) } as any).should.equal(true) + spec.contextRequirements!({ activeMcpTools: new Set(["other"]) } as any).should.equal(false) + }) +}) From 9f3e89c1471f64d17573eb8af0d1cc9b3a65460e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:34:08 +0200 Subject: [PATCH 12/24] feat(mcp): add find_tools native tool spec --- .../prompts/system-prompt/tools/find_tools.ts | 20 +++++++++++++++++++ src/core/prompts/system-prompt/tools/init.ts | 2 ++ .../task/tools/ToolExecutorCoordinator.ts | 3 +++ src/shared/tools.ts | 2 ++ 4 files changed, 27 insertions(+) create mode 100644 src/core/prompts/system-prompt/tools/find_tools.ts diff --git a/src/core/prompts/system-prompt/tools/find_tools.ts b/src/core/prompts/system-prompt/tools/find_tools.ts new file mode 100644 index 00000000..f9443c75 --- /dev/null +++ b/src/core/prompts/system-prompt/tools/find_tools.ts @@ -0,0 +1,20 @@ +import { DiracDefaultTool } from "@/shared/tools" +import type { DiracToolSpec } from "../spec" + +export const find_tools: DiracToolSpec = { + id: DiracDefaultTool.FIND_TOOLS, + name: "find_tools", + description: + "Discover and activate additional MCP tools that are not currently available to you. " + + "Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. " + + "If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), " + + "call find_tools with a short natural-language description of the capability you need. The matching " + + "tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + parameters: [ + { + name: "query", + required: true, + instruction: "A short natural-language description of the capability or tool you need.", + }, + ], +} diff --git a/src/core/prompts/system-prompt/tools/init.ts b/src/core/prompts/system-prompt/tools/init.ts index edfcdf9d..ccb133a5 100644 --- a/src/core/prompts/system-prompt/tools/init.ts +++ b/src/core/prompts/system-prompt/tools/init.ts @@ -21,6 +21,7 @@ import { subagent } from "./subagent" import { summarize_task } from "./summarize_task" import { use_skill } from "./use_skill" import { list_skills } from "./list_skills" +import { find_tools } from "./find_tools" import { write_to_file } from "./write_to_file" /** @@ -52,6 +53,7 @@ export function registerDiracToolSets(): void { subagent, use_skill, list_skills, + find_tools, write_to_file, ] diff --git a/src/core/task/tools/ToolExecutorCoordinator.ts b/src/core/task/tools/ToolExecutorCoordinator.ts index ffae46e7..13f072ce 100644 --- a/src/core/task/tools/ToolExecutorCoordinator.ts +++ b/src/core/task/tools/ToolExecutorCoordinator.ts @@ -108,6 +108,9 @@ export class ToolExecutorCoordinator { [DiracDefaultTool.LIST_SKILLS]: (_v: ToolValidator) => new ListSkillsToolHandler(), [DiracDefaultTool.USE_SUBAGENTS]: (_v: ToolValidator) => new UseSubagentsToolHandler(), [DiracDefaultTool.GET_TOOL_RESULT]: (_v: ToolValidator) => new GetToolResultToolHandler(), + + // Handler implemented in Task 10 (adaptive MCP retrieval) + [DiracDefaultTool.FIND_TOOLS]: (_v: ToolValidator) => undefined, } /** diff --git a/src/shared/tools.ts b/src/shared/tools.ts index d3beb921..1d110bd2 100644 --- a/src/shared/tools.ts +++ b/src/shared/tools.ts @@ -36,6 +36,8 @@ export enum DiracDefaultTool { // Sprint 2 — async tool result lookup. Fetches the result of a // previously-dispatched long-running tool by task_id. GET_TOOL_RESULT = "get_tool_result", + + FIND_TOOLS = "find_tools", } // Array of all tool names for compatibility From 12ee764226d019514efbeb6160f3f2125fc7ff3e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:41:49 +0200 Subject: [PATCH 13/24] feat(mcp): find_tools handler wired to active set --- .../task/tools/ToolExecutorCoordinator.ts | 4 +- .../tools/handlers/FindToolsToolHandler.ts | 37 +++++++++++++++++++ .../__tests__/FindToolsToolHandler.test.ts | 30 +++++++++++++++ 3 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 src/core/task/tools/handlers/FindToolsToolHandler.ts create mode 100644 src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts diff --git a/src/core/task/tools/ToolExecutorCoordinator.ts b/src/core/task/tools/ToolExecutorCoordinator.ts index 13f072ce..73c0960f 100644 --- a/src/core/task/tools/ToolExecutorCoordinator.ts +++ b/src/core/task/tools/ToolExecutorCoordinator.ts @@ -26,6 +26,7 @@ import { UseSubagentsToolHandler } from "./handlers/SubagentToolHandler" import { SummarizeTaskHandler } from "./handlers/SummarizeTaskHandler" import { UseSkillToolHandler } from "./handlers/UseSkillToolHandler" +import { FindToolsToolHandler } from "./handlers/FindToolsToolHandler" import { WriteToFileToolHandler } from "./handlers/WriteToFileToolHandler" import { AgentConfigLoader } from "./subagent/AgentConfigLoader" import { ToolValidator } from "./ToolValidator" @@ -109,8 +110,7 @@ export class ToolExecutorCoordinator { [DiracDefaultTool.USE_SUBAGENTS]: (_v: ToolValidator) => new UseSubagentsToolHandler(), [DiracDefaultTool.GET_TOOL_RESULT]: (_v: ToolValidator) => new GetToolResultToolHandler(), - // Handler implemented in Task 10 (adaptive MCP retrieval) - [DiracDefaultTool.FIND_TOOLS]: (_v: ToolValidator) => undefined, + [DiracDefaultTool.FIND_TOOLS]: (_v: ToolValidator) => new FindToolsToolHandler(), } /** diff --git a/src/core/task/tools/handlers/FindToolsToolHandler.ts b/src/core/task/tools/handlers/FindToolsToolHandler.ts new file mode 100644 index 00000000..2f6e2601 --- /dev/null +++ b/src/core/task/tools/handlers/FindToolsToolHandler.ts @@ -0,0 +1,37 @@ +import type { ToolUse } from "@core/assistant-message" +import { DiracDefaultTool } from "@/shared/tools" +import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" +import type { ToolResponse } from "../../index" +import type { IPartialBlockHandler, IToolHandler } from "../ToolExecutorCoordinator" +import type { TaskConfig } from "../types/TaskConfig" +import type { StronglyTypedUIHelpers } from "../types/UIHelpers" + +export class FindToolsToolHandler implements IToolHandler, IPartialBlockHandler { + readonly name = DiracDefaultTool.FIND_TOOLS + + constructor() {} + + getDescription(block: ToolUse): string { + return `[find_tools for '${(block.params?.query as string) || ""}']` + } + + async handlePartialBlock(_block: ToolUse, _uiHelpers: StronglyTypedUIHelpers): Promise { + // no-op: no streaming UI needed for find_tools + } + + async execute(_config: TaskConfig, block: ToolUse): Promise { + const query = ((block.params?.query as string) || "").trim() + const set = getActiveMcpToolSet() + if (!set || !set.available()) { + return "Tool retrieval is unavailable in this run; the default tool set is all that is available." + } + if (!query) { + return "find_tools requires a non-empty 'query' describing the capability you need." + } + const added = await set.expand(query) + if (added.length === 0) { + return `No additional tools matched "${query}". The relevant tools may already be available, or none exist for this need.` + } + return `Activated ${added.length} tool(s) for "${query}" (available next turn):\n${added.map((n) => `- ${n}`).join("\n")}` + } +} diff --git a/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts b/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts new file mode 100644 index 00000000..1ee067b0 --- /dev/null +++ b/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts @@ -0,0 +1,30 @@ +import { describe, it, afterEach } from "mocha" +import "should" +import { setActiveMcpToolSet } from "@core/mcp/retrieval/session" +import { FindToolsToolHandler } from "../FindToolsToolHandler" + +describe("FindToolsToolHandler", () => { + afterEach(() => setActiveMcpToolSet(undefined)) + + it("expands the active set and reports activated tools", async () => { + const calls: string[] = [] + setActiveMcpToolSet({ + expand: async (q: string) => { + calls.push(q) + return ["mcp__git__issues"] + }, + available: () => true, + } as any) + const handler = new FindToolsToolHandler() + const res = await handler.execute({} as any, { params: { query: "github issues" } } as any) + calls.should.deepEqual(["github issues"]) + JSON.stringify(res).should.match(/mcp__git__issues/) + }) + + it("reports unavailability when retrieval is disabled", async () => { + setActiveMcpToolSet(undefined) + const handler = new FindToolsToolHandler() + const res = await handler.execute({} as any, { params: { query: "x" } } as any) + JSON.stringify(res).should.match(/unavailable|not available/i) + }) +}) From 8651230f865a6c845e43e3c5443c9565ae105255 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:50:54 +0200 Subject: [PATCH 14/24] test(mcp): update tool snapshots for find_tools --- .../anthropic_claude_4_5_sonnet.tools.snap | 16 ++++++++++++++ .../__snapshots__/gemini_gemini_3.tools.snap | 16 ++++++++++++++ .../__snapshots__/openai_gpt_5.tools.snap | 21 +++++++++++++++++++ .../__snapshots__/vertex_gemini_3.tools.snap | 16 ++++++++++++++ 4 files changed, 69 insertions(+) diff --git a/src/core/prompts/system-prompt/__tests__/__snapshots__/anthropic_claude_4_5_sonnet.tools.snap b/src/core/prompts/system-prompt/__tests__/__snapshots__/anthropic_claude_4_5_sonnet.tools.snap index eccecfe5..7d015deb 100644 --- a/src/core/prompts/system-prompt/__tests__/__snapshots__/anthropic_claude_4_5_sonnet.tools.snap +++ b/src/core/prompts/system-prompt/__tests__/__snapshots__/anthropic_claude_4_5_sonnet.tools.snap @@ -516,6 +516,22 @@ ] } }, + { + "name": "find_tools", + "description": "Discover and activate additional MCP tools that are not currently available to you. Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), call find_tools with a short natural-language description of the capability you need. The matching tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + "input_schema": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": "A short natural-language description of the capability or tool you need." + } + }, + "required": [ + "query" + ] + } + }, { "name": "write_to_file", "description": "Creates a new file or completely overwrites an existing file. Automatically creates required directories.", diff --git a/src/core/prompts/system-prompt/__tests__/__snapshots__/gemini_gemini_3.tools.snap b/src/core/prompts/system-prompt/__tests__/__snapshots__/gemini_gemini_3.tools.snap index fb479653..17d89839 100644 --- a/src/core/prompts/system-prompt/__tests__/__snapshots__/gemini_gemini_3.tools.snap +++ b/src/core/prompts/system-prompt/__tests__/__snapshots__/gemini_gemini_3.tools.snap @@ -516,6 +516,22 @@ ] } }, + { + "name": "find_tools", + "description": "Discover and activate additional MCP tools that are not currently available to you. Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), call find_tools with a short natural-language description of the capability you need. The matching tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + "parameters": { + "type": "OBJECT", + "properties": { + "query": { + "type": "STRING", + "description": "A short natural-language description of the capability or tool you need." + } + }, + "required": [ + "query" + ] + } + }, { "name": "write_to_file", "description": "Creates a new file or completely overwrites an existing file. Automatically creates required directories.", diff --git a/src/core/prompts/system-prompt/__tests__/__snapshots__/openai_gpt_5.tools.snap b/src/core/prompts/system-prompt/__tests__/__snapshots__/openai_gpt_5.tools.snap index b73ef101..ab9a8e8a 100644 --- a/src/core/prompts/system-prompt/__tests__/__snapshots__/openai_gpt_5.tools.snap +++ b/src/core/prompts/system-prompt/__tests__/__snapshots__/openai_gpt_5.tools.snap @@ -606,6 +606,27 @@ } } }, + { + "type": "function", + "function": { + "name": "find_tools", + "strict": false, + "description": "Discover and activate additional MCP tools that are not currently available to you. Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), call find_tools with a short natural-language description of the capability you need. The matching tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + "parameters": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": "A short natural-language description of the capability or tool you need." + } + }, + "required": [ + "query" + ], + "additionalProperties": false + } + } + }, { "type": "function", "function": { diff --git a/src/core/prompts/system-prompt/__tests__/__snapshots__/vertex_gemini_3.tools.snap b/src/core/prompts/system-prompt/__tests__/__snapshots__/vertex_gemini_3.tools.snap index fb479653..17d89839 100644 --- a/src/core/prompts/system-prompt/__tests__/__snapshots__/vertex_gemini_3.tools.snap +++ b/src/core/prompts/system-prompt/__tests__/__snapshots__/vertex_gemini_3.tools.snap @@ -516,6 +516,22 @@ ] } }, + { + "name": "find_tools", + "description": "Discover and activate additional MCP tools that are not currently available to you. Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), call find_tools with a short natural-language description of the capability you need. The matching tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + "parameters": { + "type": "OBJECT", + "properties": { + "query": { + "type": "STRING", + "description": "A short natural-language description of the capability or tool you need." + } + }, + "required": [ + "query" + ] + } + }, { "name": "write_to_file", "description": "Creates a new file or completely overwrites an existing file. Automatically creates required directories.", From 53c159baa4d25dde114df9c8baa3518aed22a7cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 00:53:23 +0200 Subject: [PATCH 15/24] feat(mcp): build tool index + publish active set at bootstrap --- src/core/mcp/bootstrap.ts | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/core/mcp/bootstrap.ts b/src/core/mcp/bootstrap.ts index 72b65182..d0b60840 100644 --- a/src/core/mcp/bootstrap.ts +++ b/src/core/mcp/bootstrap.ts @@ -157,6 +157,30 @@ export async function initializeMcpForTask( Logger.info(`MCP: registered ${tools.length} tool(s) from plugins`) } + // Adaptive retrieval: build the vector index and publish a session active set. + const { ActiveMcpToolSet } = await import("./retrieval/ActiveMcpToolSet") + const { getRetrievalConfig } = await import("./retrieval/config") + const { setActiveMcpToolSet } = await import("./retrieval/session") + try { + const { createDefaultEmbedder } = await import("./retrieval/Embedder") + const { ToolVectorIndex } = await import("./retrieval/ToolVectorIndex") + const os = await import("os") + const path = await import("path") + const cachePath = path.join(os.homedir(), ".dirac", "mcp-tool-vectors.json") + const embedder = createDefaultEmbedder() + const index = await new ToolVectorIndex(embedder, cachePath).build( + tools.map((t) => ({ qualifiedName: t.qualifiedName, text: `${t.qualifiedName}\n${t.description ?? ""}` })), + ) + setActiveMcpToolSet(new ActiveMcpToolSet(embedder, index, getRetrievalConfig())) + } catch (err) { + Logger.warn("MCP adaptive retrieval unavailable; running native-only:", err) + const { Embedder } = await import("./retrieval/Embedder") + const dead = new Embedder(async () => { + throw new Error("embedder unavailable") + }) + setActiveMcpToolSet(new ActiveMcpToolSet(dead, new Map(), getRetrievalConfig())) + } + return tools } catch (err) { Logger.warn("MCP initialization failed (continuing without plugins):", err) From 05820987d5a90ce6114ee8701c8ddaa95eef4acc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 01:02:53 +0200 Subject: [PATCH 16/24] feat(mcp): seed active set + inject into prompt context --- cli/esbuild.mts | 6 ++++++ src/core/task/ApiRequestHandler.ts | 2 ++ src/core/task/index.ts | 8 ++++++++ src/core/task/tools/subagent/SubagentRunner.ts | 2 ++ 4 files changed, 18 insertions(+) diff --git a/cli/esbuild.mts b/cli/esbuild.mts index 6760c501..0f7d242d 100644 --- a/cli/esbuild.mts +++ b/cli/esbuild.mts @@ -277,6 +277,12 @@ const sharedOptions: Partial = { "pino", "pino-roll", "@vscode/ripgrep", // Uses __dirname to locate the binary + // Adaptive MCP retrieval embedder: lazily imported ONNX runtime + model. + // Native .node binaries cannot be bundled by esbuild; keep them external + // so they resolve from node_modules at runtime (only loaded when MCP + // retrieval is active — guarded by --no-mcp). + "@huggingface/transformers", + "onnxruntime-node", ], supported: { "top-level-await": true }, } diff --git a/src/core/task/ApiRequestHandler.ts b/src/core/task/ApiRequestHandler.ts index fc352ed6..4f457fc8 100644 --- a/src/core/task/ApiRequestHandler.ts +++ b/src/core/task/ApiRequestHandler.ts @@ -15,6 +15,7 @@ import { import { formatResponse } from "@core/prompts/responses" import type { SystemPromptContext } from "@core/prompts/system-prompt" import { getSystemPrompt } from "@core/prompts/system-prompt" +import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { ensureRulesDirectoryExists, ensureTaskDirectoryExists } from "@core/storage/disk" import { isMultiRootEnabled } from "@core/workspace/multi-root-utils" import { HostProvider } from "@hosts/host-provider" @@ -206,6 +207,7 @@ export class ApiRequestHandler { availableCores: getAvailableCores(), shouldCompact, userPromptText: firstUserPromptText, + activeMcpTools: getActiveMcpToolSet()?.snapshot(), } // Notify user if any conditional rules were applied for this request diff --git a/src/core/task/index.ts b/src/core/task/index.ts index df7f58bd..c82d631e 100644 --- a/src/core/task/index.ts +++ b/src/core/task/index.ts @@ -8,6 +8,7 @@ import { ModelContextTracker } from "@core/context/context-tracking/ModelContext import { DiracIgnoreController } from "@core/ignore/DiracIgnoreController" import { initializeMcpForTask } from "@core/mcp/bootstrap" import { mcpClientManager } from "@core/mcp/McpClientManager" +import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { CommandPermissionController } from "@core/permissions" import { WorkspaceRootManager } from "@core/workspace/WorkspaceRootManager" import { HostProvider } from "@hosts/host-provider" @@ -517,6 +518,13 @@ export class Task { // on its first request. Failures are swallowed — ailiance-agent must work // without plugins. await initializeMcpForTask(this.toolExecutor) + // Adaptive MCP retrieval: seed the session active set from the first + // user task text so the prompt-context gate can filter MCP tools down + // to the relevant subset. Best-effort — must not block task start. + const _mcpActiveSet = getActiveMcpToolSet() + if (_mcpActiveSet && typeof task === "string" && task.length > 0) { + await _mcpActiveSet.seed(task) + } return this.lifecycleManager.startTask(task, images, files) } diff --git a/src/core/task/tools/subagent/SubagentRunner.ts b/src/core/task/tools/subagent/SubagentRunner.ts index 4dd118f6..c912a8c0 100644 --- a/src/core/task/tools/subagent/SubagentRunner.ts +++ b/src/core/task/tools/subagent/SubagentRunner.ts @@ -3,6 +3,7 @@ const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) import type { ApiHandler, buildApiHandler } from "@core/api" import { parseAssistantMessageV2, ToolParamName, ToolUse } from "@core/assistant-message" import { getOrDiscoverSkills } from "@core/context/instructions/user-instructions/skills" +import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { formatResponse } from "@core/prompts/responses" import { PromptRegistry } from "@core/prompts/system-prompt" import type { SystemPromptContext } from "@core/prompts/system-prompt/types" @@ -368,6 +369,7 @@ export class SubagentRunner { yoloModeToggled: false, enableParallelToolCalling: false, isSubagentRun: true, + activeMcpTools: getActiveMcpToolSet()?.snapshot(), isMultiRootEnabled: this.baseConfig.isMultiRootEnabled, workspaceRoots: this.baseConfig.workspaceManager?.getRoots().map((root) => ({ path: root.path, From b19a5055368a2a6b3e0ab7b564dc2a733c76039d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 01:06:36 +0200 Subject: [PATCH 17/24] feat(cli): mcp-top-k and mcp-threshold tuning flags --- cli/src/index.ts | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/cli/src/index.ts b/cli/src/index.ts index 05b23393..6be951bf 100644 --- a/cli/src/index.ts +++ b/cli/src/index.ts @@ -66,6 +66,8 @@ program "Comma-separated allowlist of plugin MCP servers to load (e.g. github,context7); only these load this run", ) .option("--no-mcp", "Disable all plugin MCP servers for this run (smaller prompt, faster on big-context backends)") + .option("--mcp-top-k ", "Max MCP tools to preload by relevance (default 8)") + .option("--mcp-threshold ", "Min cosine similarity to preload an MCP tool (default 0.3)") .action(async (prompt, options) => { // Per-run MCP control. Commander sets options.mcp = false for --no-mcp, // or the string list for --mcp . Surfaced to the core MCP @@ -75,6 +77,12 @@ program } else if (typeof options.mcp === "string") { process.env.AILIANCE_MCP_SERVERS = options.mcp } + if (typeof options.mcpTopK === "string") { + process.env.AILIANCE_MCP_TOP_K = options.mcpTopK + } + if (typeof options.mcpThreshold === "string") { + process.env.AILIANCE_MCP_THRESHOLD = options.mcpThreshold + } const { runTask } = await import("./commands/task") const { resumeTask } = await import("./commands/resume") if (options.taskId) { @@ -288,6 +296,8 @@ program "Comma-separated allowlist of plugin MCP servers to load (e.g. github,context7); only these load this run", ) .option("--no-mcp", "Disable all plugin MCP servers for this run (smaller prompt, faster on big-context backends)") + .option("--mcp-top-k ", "Max MCP tools to preload by relevance (default 8)") + .option("--mcp-threshold ", "Min cosine similarity to preload an MCP tool (default 0.3)") .action(async (prompt, options) => { // Per-run MCP control (mirrors the `task` command): surfaced to the core // MCP bootstrap via env so no global config is touched. @@ -296,6 +306,12 @@ program } else if (typeof options.mcp === "string") { process.env.AILIANCE_MCP_SERVERS = options.mcp } + if (typeof options.mcpTopK === "string") { + process.env.AILIANCE_MCP_TOP_K = options.mcpTopK + } + if (typeof options.mcpThreshold === "string") { + process.env.AILIANCE_MCP_THRESHOLD = options.mcpThreshold + } // ailiance-agent fork: kanban path removed. const { printWarning } = await import("./utils/display") // Check for ACP mode first - this takes precedence over everything else From e90fa751056b7b44b1c5868e3bdbe0ea80bed1f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 01:30:43 +0200 Subject: [PATCH 18/24] fix(mcp): guard tool index against partial embed --- src/core/mcp/retrieval/ToolVectorIndex.ts | 12 ++++++++++-- .../mcp/retrieval/__tests__/ToolVectorIndex.test.ts | 7 +++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/src/core/mcp/retrieval/ToolVectorIndex.ts b/src/core/mcp/retrieval/ToolVectorIndex.ts index 9056abb5..b598ccd0 100644 --- a/src/core/mcp/retrieval/ToolVectorIndex.ts +++ b/src/core/mcp/retrieval/ToolVectorIndex.ts @@ -42,13 +42,21 @@ export class ToolVectorIndex { if (stale.length > 0) { const vecs = await this.embedder.embed(stale.map((t) => t.text)) stale.forEach((t, i) => { - cache[t.qualifiedName] = { hash: hashText(t.text), vec: Array.from(vecs[i]) } + const vec = vecs[i] + if (!vec) { + return + } + cache[t.qualifiedName] = { hash: hashText(t.text), vec: Array.from(vec) } }) this.writeCache(cache) } const result = new Map() for (const t of tools) { - result.set(t.qualifiedName, Float32Array.from(cache[t.qualifiedName].vec)) + const entry = cache[t.qualifiedName] + if (!entry) { + continue + } + result.set(t.qualifiedName, Float32Array.from(entry.vec)) } return result } diff --git a/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts b/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts index 2e9594ec..bad8c351 100644 --- a/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts +++ b/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts @@ -48,4 +48,11 @@ describe("ToolVectorIndex", () => { await new ToolVectorIndex(embedder, cachePath).build([{ qualifiedName: "mcp__p_s__a", text: "alpha2" }]) embedCalls.should.equal(2) }) + + it("does not crash when the embedder returns fewer vectors than requested", async () => { + const embedder = new Embedder(async () => async (_texts: string[]) => [] as Float32Array[]) + const idx = new ToolVectorIndex(embedder, cachePath) + const vecs = await idx.build([{ qualifiedName: "mcp__p_s__x", text: "x" }]) + vecs.has("mcp__p_s__x").should.equal(false) + }) }) From f7196908a932f1b2e2ebdcdd603d81e2ef02a3cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 01:30:47 +0200 Subject: [PATCH 19/24] fix(mcp): seed active set on task resume --- src/core/task/index.ts | 53 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/src/core/task/index.ts b/src/core/task/index.ts index c82d631e..1a77ec7c 100644 --- a/src/core/task/index.ts +++ b/src/core/task/index.ts @@ -10,6 +10,7 @@ import { initializeMcpForTask } from "@core/mcp/bootstrap" import { mcpClientManager } from "@core/mcp/McpClientManager" import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { CommandPermissionController } from "@core/permissions" +import { getSavedApiConversationHistory } from "@core/storage/disk" import { WorkspaceRootManager } from "@core/workspace/WorkspaceRootManager" import { HostProvider } from "@hosts/host-provider" import { ICheckpointManager } from "@integrations/checkpoints/types" @@ -529,9 +530,61 @@ export class Task { } public async resumeTaskFromHistory() { + // Mirror startTask: the resume path must (re)publish MCP tool specs + + // a fresh session active set, otherwise getActiveMcpToolSet() returns + // undefined (or a stale set from a prior task in a long-lived process) + // and the prompt-context gate floods the prompt with every MCP spec + // still registered in the process-global DiracToolSet. Re-init is cheap + // (re-lists + re-publishes; the vector index is disk-cached). + await initializeMcpForTask(this.toolExecutor) + // Seed the active set from the resumed conversation's first user message + // so retrieval filters MCP tools down to the relevant subset. Best-effort. + const _mcpActiveSet = getActiveMcpToolSet() + if (_mcpActiveSet) { + const firstUserText = await this.extractFirstUserTextFromHistory() + if (firstUserText) { + await _mcpActiveSet.seed(firstUserText) + } + } return this.lifecycleManager.resumeTaskFromHistory() } + /** + * Best-effort extraction of the first user message text from the saved API + * conversation history, used to seed the adaptive MCP retrieval active set + * on resume. Mirrors the extraction in ApiRequestHandler. Returns undefined + * if the history is unreadable or contains no user text. + */ + private async extractFirstUserTextFromHistory(): Promise { + try { + const history = await getSavedApiConversationHistory(this.taskId) + const firstUser = history.find((m) => m.role === "user") + if (!firstUser) { + return undefined + } + const content: unknown = firstUser.content + if (typeof content === "string") { + return content.length > 0 ? content : undefined + } + if (Array.isArray(content)) { + const parts: string[] = [] + for (const block of content) { + if (block && typeof block === "object" && (block as { type?: unknown }).type === "text") { + const text = (block as { text?: unknown }).text + if (typeof text === "string") { + parts.push(text) + } + } + } + return parts.length > 0 ? parts.join(" ") : undefined + } + return undefined + } catch { + // best-effort — resume must never break on a missing/corrupt history + return undefined + } + } + private async initiateTaskLoop(userContent: DiracContent[]): Promise { return this.agentLoopRunner.initiateLoop(userContent) } From 2801aa0018acc0cecd4c2d330b9f4cdc0cfa26a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 08:33:06 +0200 Subject: [PATCH 20/24] chore(mcp): clear retrieval set on teardown + add logs --- src/core/mcp/retrieval/ActiveMcpToolSet.ts | 3 +++ .../__tests__/ActiveMcpToolSet.test.ts | 6 ++---- .../mcp/retrieval/__tests__/Embedder.test.ts | 3 +-- .../retrieval/__tests__/ToolVectorIndex.test.ts | 2 +- src/core/mcp/retrieval/__tests__/config.test.ts | 2 +- src/core/mcp/retrieval/session.ts | 17 +++++++++++++++++ src/core/task/ApiRequestHandler.ts | 6 ++---- src/core/task/index.ts | 10 +++++++++- src/core/task/tools/ToolExecutorCoordinator.ts | 3 +-- .../task/tools/handlers/FindToolsToolHandler.ts | 2 +- .../__tests__/FindToolsToolHandler.test.ts | 2 +- src/core/task/tools/subagent/SubagentRunner.ts | 13 ++++++++----- 12 files changed, 47 insertions(+), 22 deletions(-) diff --git a/src/core/mcp/retrieval/ActiveMcpToolSet.ts b/src/core/mcp/retrieval/ActiveMcpToolSet.ts index 8488f108..3ceabaa3 100644 --- a/src/core/mcp/retrieval/ActiveMcpToolSet.ts +++ b/src/core/mcp/retrieval/ActiveMcpToolSet.ts @@ -1,3 +1,4 @@ +import { Logger } from "@/shared/services/Logger" import type { RetrievalConfig } from "./config" import { type ScoredItem, selectTopK } from "./cosine" import type { Embedder } from "./Embedder" @@ -33,6 +34,7 @@ export class ActiveMcpToolSet { for (const id of await this.select(prompt, this.config.baseK)) { this.active.add(id) } + Logger.debug(`[mcp-retrieval] seed selected ${this.active.size} tool(s): ${Array.from(this.active).join(", ")}`) } catch { this.embedderOk = false } @@ -47,6 +49,7 @@ export class ActiveMcpToolSet { added.push(id) } } + Logger.debug(`[mcp-retrieval] find_tools added ${added.length} tool(s): ${added.join(", ")}`) return added } catch { this.embedderOk = false diff --git a/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts b/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts index 22d0ec59..eb5945f6 100644 --- a/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts +++ b/src/core/mcp/retrieval/__tests__/ActiveMcpToolSet.test.ts @@ -1,12 +1,10 @@ import { describe, it } from "mocha" import "should" -import { Embedder } from "../Embedder" import { ActiveMcpToolSet } from "../ActiveMcpToolSet" +import { Embedder } from "../Embedder" function makeEmbedder(map: Record) { - return new Embedder(async () => async (texts: string[]) => - texts.map((t) => Float32Array.from(map[t] ?? [0, 0])), - ) + return new Embedder(async () => async (texts: string[]) => texts.map((t) => Float32Array.from(map[t] ?? [0, 0]))) } describe("ActiveMcpToolSet", () => { diff --git a/src/core/mcp/retrieval/__tests__/Embedder.test.ts b/src/core/mcp/retrieval/__tests__/Embedder.test.ts index 1afb1f78..5730c127 100644 --- a/src/core/mcp/retrieval/__tests__/Embedder.test.ts +++ b/src/core/mcp/retrieval/__tests__/Embedder.test.ts @@ -4,8 +4,7 @@ import { Embedder } from "../Embedder" describe("Embedder", () => { it("embeds via the injected pipeline and returns Float32Array[]", async () => { - const fakePipeline = async (texts: string[]) => - texts.map((t) => new Float32Array([t.length, 0, 0])) + const fakePipeline = async (texts: string[]) => texts.map((t) => new Float32Array([t.length, 0, 0])) const e = new Embedder(async () => fakePipeline) const [v] = await e.embed(["abc"]) Array.from(v).should.deepEqual([3, 0, 0]) diff --git a/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts b/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts index bad8c351..c7631c00 100644 --- a/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts +++ b/src/core/mcp/retrieval/__tests__/ToolVectorIndex.test.ts @@ -1,4 +1,4 @@ -import { describe, it, beforeEach, afterEach } from "mocha" +import { afterEach, beforeEach, describe, it } from "mocha" import "should" import * as fs from "fs" import * as os from "os" diff --git a/src/core/mcp/retrieval/__tests__/config.test.ts b/src/core/mcp/retrieval/__tests__/config.test.ts index 3038505f..0a57b7d9 100644 --- a/src/core/mcp/retrieval/__tests__/config.test.ts +++ b/src/core/mcp/retrieval/__tests__/config.test.ts @@ -1,4 +1,4 @@ -import { describe, it, afterEach } from "mocha" +import { afterEach, describe, it } from "mocha" import "should" import { getRetrievalConfig } from "../config" diff --git a/src/core/mcp/retrieval/session.ts b/src/core/mcp/retrieval/session.ts index 62399dd2..a6e20044 100644 --- a/src/core/mcp/retrieval/session.ts +++ b/src/core/mcp/retrieval/session.ts @@ -9,3 +9,20 @@ export function setActiveMcpToolSet(set: ActiveMcpToolSet | undefined): void { export function getActiveMcpToolSet(): ActiveMcpToolSet | undefined { return current } + +/** + * Publish an EMPTY, embedder-dead active set so the gate emits ZERO MCP tools. + * Called on task teardown to prevent the previous task's active set from + * leaking into the next task's first request (before that task republishes). + * Using an empty set (not `undefined`) is deliberate: `undefined` makes the + * gate emit ALL MCP tools, whereas an empty set is the safe between-tasks state. + */ +export async function clearActiveMcpToolSet(): Promise { + const { ActiveMcpToolSet } = await import("./ActiveMcpToolSet") + const { Embedder } = await import("./Embedder") + const { getRetrievalConfig } = await import("./config") + const dead = new Embedder(async () => { + throw new Error("retrieval cleared") + }) + current = new ActiveMcpToolSet(dead, new Map(), getRetrievalConfig()) +} diff --git a/src/core/task/ApiRequestHandler.ts b/src/core/task/ApiRequestHandler.ts index 4f457fc8..4897227b 100644 --- a/src/core/task/ApiRequestHandler.ts +++ b/src/core/task/ApiRequestHandler.ts @@ -12,10 +12,10 @@ import { getLocalWindsurfRules, refreshExternalRulesToggles, } from "@core/context/instructions/user-instructions/external-rules" +import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { formatResponse } from "@core/prompts/responses" import type { SystemPromptContext } from "@core/prompts/system-prompt" import { getSystemPrompt } from "@core/prompts/system-prompt" -import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { ensureRulesDirectoryExists, ensureTaskDirectoryExists } from "@core/storage/disk" import { isMultiRootEnabled } from "@core/workspace/multi-root-utils" import { HostProvider } from "@hosts/host-provider" @@ -427,9 +427,7 @@ ${notice}` // For permanent client errors, mark attempt=1 to signal // "fail-fast, no retries attempted" so the UI does not // claim 3 retries were tried. - const finalAttempt = isPermanentClientError - ? this.taskState.autoRetryAttempts + 1 - : 3 + const finalAttempt = isPermanentClientError ? this.taskState.autoRetryAttempts + 1 : 3 await this.ctx.say( "error_retry", JSON.stringify({ diff --git a/src/core/task/index.ts b/src/core/task/index.ts index 1a77ec7c..483bb1a8 100644 --- a/src/core/task/index.ts +++ b/src/core/task/index.ts @@ -8,7 +8,7 @@ import { ModelContextTracker } from "@core/context/context-tracking/ModelContext import { DiracIgnoreController } from "@core/ignore/DiracIgnoreController" import { initializeMcpForTask } from "@core/mcp/bootstrap" import { mcpClientManager } from "@core/mcp/McpClientManager" -import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" +import { clearActiveMcpToolSet, getActiveMcpToolSet } from "@core/mcp/retrieval/session" import { CommandPermissionController } from "@core/permissions" import { getSavedApiConversationHistory } from "@core/storage/disk" import { WorkspaceRootManager } from "@core/workspace/WorkspaceRootManager" @@ -612,6 +612,14 @@ export class Task { mcpClientManager.disconnectAll().catch((_err) => { // non-fatal — MCP cleanup must never block abort }) + // Clear the process-global adaptive-retrieval active set so the previous + // task's selection cannot leak into the next task's first request. We + // publish an EMPTY set (not undefined) so the gate emits zero MCP tools + // in the between-tasks window; fire-and-forget since it only constructs + // objects (no async work runs). + clearActiveMcpToolSet().catch((_err) => { + // non-fatal — retrieval cleanup must never block abort + }) return this.lifecycleManager.abortTask() } diff --git a/src/core/task/tools/ToolExecutorCoordinator.ts b/src/core/task/tools/ToolExecutorCoordinator.ts index 73c0960f..b9bddbfb 100644 --- a/src/core/task/tools/ToolExecutorCoordinator.ts +++ b/src/core/task/tools/ToolExecutorCoordinator.ts @@ -9,6 +9,7 @@ import { DiagnosticsScanToolHandler } from "./handlers/DiagnosticsScanToolHandle import { EditFileToolHandler } from "./handlers/EditFileToolHandler" import { ExecuteCommandToolHandler } from "./handlers/ExecuteCommandToolHandler" import { FindSymbolReferencesToolHandler } from "./handlers/FindSymbolReferencesToolHandler" +import { FindToolsToolHandler } from "./handlers/FindToolsToolHandler" import { GenerateExplanationToolHandler } from "./handlers/GenerateExplanationToolHandler" import { GetFileSkeletonToolHandler } from "./handlers/GetFileSkeletonToolHandler" import { GetFunctionToolHandler } from "./handlers/GetFunctionToolHandler" @@ -25,8 +26,6 @@ import { SearchFilesToolHandler } from "./handlers/SearchFilesToolHandler" import { UseSubagentsToolHandler } from "./handlers/SubagentToolHandler" import { SummarizeTaskHandler } from "./handlers/SummarizeTaskHandler" import { UseSkillToolHandler } from "./handlers/UseSkillToolHandler" - -import { FindToolsToolHandler } from "./handlers/FindToolsToolHandler" import { WriteToFileToolHandler } from "./handlers/WriteToFileToolHandler" import { AgentConfigLoader } from "./subagent/AgentConfigLoader" import { ToolValidator } from "./ToolValidator" diff --git a/src/core/task/tools/handlers/FindToolsToolHandler.ts b/src/core/task/tools/handlers/FindToolsToolHandler.ts index 2f6e2601..e510750e 100644 --- a/src/core/task/tools/handlers/FindToolsToolHandler.ts +++ b/src/core/task/tools/handlers/FindToolsToolHandler.ts @@ -1,6 +1,6 @@ import type { ToolUse } from "@core/assistant-message" -import { DiracDefaultTool } from "@/shared/tools" import { getActiveMcpToolSet } from "@core/mcp/retrieval/session" +import { DiracDefaultTool } from "@/shared/tools" import type { ToolResponse } from "../../index" import type { IPartialBlockHandler, IToolHandler } from "../ToolExecutorCoordinator" import type { TaskConfig } from "../types/TaskConfig" diff --git a/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts b/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts index 1ee067b0..6dec1fc6 100644 --- a/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts +++ b/src/core/task/tools/handlers/__tests__/FindToolsToolHandler.test.ts @@ -1,4 +1,4 @@ -import { describe, it, afterEach } from "mocha" +import { afterEach, describe, it } from "mocha" import "should" import { setActiveMcpToolSet } from "@core/mcp/retrieval/session" import { FindToolsToolHandler } from "../FindToolsToolHandler" diff --git a/src/core/task/tools/subagent/SubagentRunner.ts b/src/core/task/tools/subagent/SubagentRunner.ts index c912a8c0..6c3c16e5 100644 --- a/src/core/task/tools/subagent/SubagentRunner.ts +++ b/src/core/task/tools/subagent/SubagentRunner.ts @@ -1,5 +1,7 @@ import * as path from "node:path" + const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + import type { ApiHandler, buildApiHandler } from "@core/api" import { parseAssistantMessageV2, ToolParamName, ToolUse } from "@core/assistant-message" import { getOrDiscoverSkills } from "@core/context/instructions/user-instructions/skills" @@ -17,13 +19,13 @@ import { getContextWindowInfo } from "@/core/context/context-management/context- import { HostRegistryInfo } from "@/registry" import { DiracError, DiracErrorType } from "@/services/error" import { calculateApiCostAnthropic } from "@/utils/cost" +import { TOOL_EXAMPLES } from "../../../prompts/tool-examples" import { TaskState } from "../../TaskState" +import { excerpt } from "../../utils/excerpt" import { ToolExecutorCoordinator } from "../ToolExecutorCoordinator" import { ToolValidator } from "../ToolValidator" import type { TaskConfig } from "../types/TaskConfig" import { SubagentBuilder } from "./SubagentBuilder" -import { excerpt } from "../../utils/excerpt" -import { TOOL_EXAMPLES } from "../../../prompts/tool-examples" const MAX_EMPTY_ASSISTANT_RETRIES = 3 const MAX_INITIAL_STREAM_ATTEMPTS = 3 @@ -438,7 +440,7 @@ ${partialResult}` type: "text", text: workspaceMetadataEnvironmentBlock, } as DiracTextContentBlock, - ] + ] : []), ], }) @@ -865,7 +867,9 @@ ${partialResult}` ...baseCallbacks, say: async () => undefined, sayAndCreateMissingParamError: async (_toolName, paramName) => - formatResponse.toolError(formatResponse.missingToolParameterError(paramName, TOOL_EXAMPLES[_toolName as DiracDefaultTool])), + formatResponse.toolError( + formatResponse.missingToolParameterError(paramName, TOOL_EXAMPLES[_toolName as DiracDefaultTool]), + ), executeCommandTool: async (command: string, timeoutSeconds: number | undefined) => { this.activeCommandExecutions += 1 try { @@ -932,7 +936,6 @@ ${partialResult}` } } - private shouldCompactBeforeNextRequest( requestTotalTokens: number, api: ReturnType, From 0ffdf5da3a3f2387da6e0204296ee0b551e7db60 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 08:34:17 +0200 Subject: [PATCH 21/24] style(mcp): biome formatting --- src/core/mcp/bootstrap.ts | 5 ++++- src/core/mcp/retrieval/Embedder.ts | 2 +- src/core/prompts/system-prompt/tools/find_tools.ts | 2 +- src/core/prompts/system-prompt/tools/init.ts | 9 ++++----- src/core/prompts/system-prompt/types.ts | 1 - 5 files changed, 10 insertions(+), 9 deletions(-) diff --git a/src/core/mcp/bootstrap.ts b/src/core/mcp/bootstrap.ts index d0b60840..3e0c9820 100644 --- a/src/core/mcp/bootstrap.ts +++ b/src/core/mcp/bootstrap.ts @@ -107,7 +107,10 @@ function readMcpSettings(): { const envServersRaw = process.env.AILIANCE_MCP_SERVERS const envEnabled = envServersRaw !== undefined - ? envServersRaw.split(",").map((s) => s.trim()).filter(Boolean) + ? envServersRaw + .split(",") + .map((s) => s.trim()) + .filter(Boolean) : undefined const noMcp = ["1", "true", "yes"].includes((process.env.AILIANCE_NO_MCP ?? "").toLowerCase()) || diff --git a/src/core/mcp/retrieval/Embedder.ts b/src/core/mcp/retrieval/Embedder.ts index 24f91013..f1a77956 100644 --- a/src/core/mcp/retrieval/Embedder.ts +++ b/src/core/mcp/retrieval/Embedder.ts @@ -30,7 +30,7 @@ export function createDefaultEmbedder(): Embedder { const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2") return async (texts: string[]) => { const out = await (extractor as any)(texts, { pooling: "mean", normalize: true }) - return ((out as { tolist(): number[][] }).tolist()).map((row) => Float32Array.from(row)) + return (out as { tolist(): number[][] }).tolist().map((row) => Float32Array.from(row)) } }) } diff --git a/src/core/prompts/system-prompt/tools/find_tools.ts b/src/core/prompts/system-prompt/tools/find_tools.ts index f9443c75..3d390425 100644 --- a/src/core/prompts/system-prompt/tools/find_tools.ts +++ b/src/core/prompts/system-prompt/tools/find_tools.ts @@ -9,7 +9,7 @@ export const find_tools: DiracToolSpec = { "Only a relevant subset of external (MCP) tools is loaded by default to keep the tool list small. " + "If you need a capability you don't see (e.g. interacting with GitHub, a database, a browser, etc.), " + "call find_tools with a short natural-language description of the capability you need. The matching " + - "tools become available on your next turn. Example: { query: \"search and comment on GitHub issues\" }.", + 'tools become available on your next turn. Example: { query: "search and comment on GitHub issues" }.', parameters: [ { name: "query", diff --git a/src/core/prompts/system-prompt/tools/init.ts b/src/core/prompts/system-prompt/tools/init.ts index ccb133a5..d34d532e 100644 --- a/src/core/prompts/system-prompt/tools/init.ts +++ b/src/core/prompts/system-prompt/tools/init.ts @@ -1,27 +1,26 @@ import { DiracToolSet } from "../registry/DiracToolSet" import { ask_followup_question } from "./ask_followup_question" import { attempt_completion } from "./attempt_completion" -import { diagnostics_scan } from "./diagnostics_scan" import { browser_action } from "./browser_action" +import { diagnostics_scan } from "./diagnostics_scan" import { edit_file } from "./edit_file" import { execute_command } from "./execute_command" import { find_symbol_references } from "./find_symbol_references" - +import { find_tools } from "./find_tools" import { get_file_skeleton } from "./get_file_skeleton" import { get_function } from "./get_function" import { get_tool_result } from "./get_tool_result" import { list_files } from "./list_files" +import { list_skills } from "./list_skills" import { new_task } from "./new_task" import { plan_mode_respond } from "./plan_mode_respond" import { read_file } from "./read_file" -import { replace_symbol } from "./replace_symbol" import { rename_symbol } from "./rename_symbol" +import { replace_symbol } from "./replace_symbol" import { search_files } from "./search_files" import { subagent } from "./subagent" import { summarize_task } from "./summarize_task" import { use_skill } from "./use_skill" -import { list_skills } from "./list_skills" -import { find_tools } from "./find_tools" import { write_to_file } from "./write_to_file" /** diff --git a/src/core/prompts/system-prompt/types.ts b/src/core/prompts/system-prompt/types.ts index 678067c7..61f82200 100644 --- a/src/core/prompts/system-prompt/types.ts +++ b/src/core/prompts/system-prompt/types.ts @@ -63,7 +63,6 @@ export interface SystemPromptContext { readonly userPromptText?: string } - /** * Utility functions for validating prompt components */ From 1e98344dfa4bbb23d10eccc4072692f4edef5ab9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 08:51:41 +0200 Subject: [PATCH 22/24] feat(mcp): make embed model + offline mode configurable --- src/core/mcp/retrieval/Embedder.ts | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/src/core/mcp/retrieval/Embedder.ts b/src/core/mcp/retrieval/Embedder.ts index f1a77956..02c486cf 100644 --- a/src/core/mcp/retrieval/Embedder.ts +++ b/src/core/mcp/retrieval/Embedder.ts @@ -20,14 +20,27 @@ export class Embedder { } } +/** Upstream public model id. Supply-chain policy: production should mirror the + * weights into the `ailiance` org and point `AILIANCE_EMBED_MODEL` at the + * vendored repo/path (loaded offline) rather than the public HF CDN. */ +const DEFAULT_EMBED_MODEL = "Xenova/all-MiniLM-L6-v2" + /** * Production factory: all-MiniLM-L6-v2 ONNX via transformers.js, mean-pooled + * normalized 384-d. Imported lazily so `--no-mcp` never pays the import cost. + * The model id is overridable via `AILIANCE_EMBED_MODEL` (point it at the + * mirrored/vendored weights); `AILIANCE_EMBED_OFFLINE=1` forbids any network + * fetch (transformers.js loads only from the local cache). */ export function createDefaultEmbedder(): Embedder { + const modelId = process.env.AILIANCE_EMBED_MODEL || DEFAULT_EMBED_MODEL return new Embedder(async () => { - const { pipeline } = await import("@huggingface/transformers") - const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2") + const transformers = await import("@huggingface/transformers") + const { pipeline } = transformers + if (process.env.AILIANCE_EMBED_OFFLINE === "1" && transformers.env) { + transformers.env.allowRemoteModels = false + } + const extractor = await pipeline("feature-extraction", modelId) return async (texts: string[]) => { const out = await (extractor as any)(texts, { pooling: "mean", normalize: true }) return (out as { tolist(): number[][] }).tolist().map((row) => Float32Array.from(row)) From 34aefbedb3ca48921580ca410374868a61ac65f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 08:52:30 +0200 Subject: [PATCH 23/24] docs(mcp): supply-chain mirror runbook --- ...06-02-mcp-retrieval-supply-chain-mirror.md | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md diff --git a/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md b/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md new file mode 100644 index 00000000..64abf936 --- /dev/null +++ b/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md @@ -0,0 +1,85 @@ +# Runbook — Mirror the MCP-retrieval embedding dependency (HITL) + +**Date:** 2026-06-02 +**Status:** Prepared for human-in-the-loop execution (merge blocker for PR #50). +**Why:** Per the ailiance supply-chain policy (no blind upstream; pin + mirror; +HITL diff review), the adaptive MCP retrieval feature's embedding stack must be +vendored into the `ailiance` org before it ships to production. This runbook is +**not** auto-executed — forks/pushes require human review. + +## What must be mirrored + +1. **npm package** `@huggingface/transformers@3.3.3` (pinned, sha512 in + `package-lock.json`). +2. **Model weights** `Xenova/all-MiniLM-L6-v2` (ONNX, ~88 MB) — the + feature-extraction model loaded at runtime by `createDefaultEmbedder`. + +## Runtime is already mirror-ready (no code change needed after mirroring) + +`src/core/mcp/retrieval/Embedder.ts` reads two env vars: +- `AILIANCE_EMBED_MODEL` — model id/path (default `Xenova/all-MiniLM-L6-v2`). + Point it at the mirrored repo/local path. +- `AILIANCE_EMBED_OFFLINE=1` — sets `transformers.env.allowRemoteModels = false` + so transformers.js loads ONLY from the local cache (no HF CDN fetch). + +So production sets, e.g.: +``` +AILIANCE_EMBED_MODEL=Ailiance-fr/all-MiniLM-L6-v2 # or an absolute local dir +AILIANCE_EMBED_OFFLINE=1 +``` + +## Step 1 — Mirror the model weights into the ailiance HF org + +On a host with `huggingface_hub` + write token for `Ailiance-fr`: +```bash +python - <<'PY' +from huggingface_hub import snapshot_download, create_repo, upload_folder +# 1. pull the exact upstream snapshot (pin the revision after first fetch) +local = snapshot_download("Xenova/all-MiniLM-L6-v2") # capture the commit sha +# 2. create the mirror + upload +create_repo("Ailiance-fr/all-MiniLM-L6-v2", repo_type="model", private=False, exist_ok=True) +upload_folder(repo_id="Ailiance-fr/all-MiniLM-L6-v2", folder_path=local, repo_type="model") +PY +``` +- Record the upstream commit SHA in the SBOM (audit trail). +- Set the mirror's license/card to match upstream (Apache-2.0). +- Verify the ONNX file (`onnx/model.onnx` / `model_quantized.onnx`) is present + and byte-identical (hash) to upstream. + +## Step 2 — Vendor/mirror the npm package + +Follow `ailiance-gateway/docs/superpowers/specs/2026-05-29-dependency-fork-audit-strategy.md`. +Options (pick per the strategy): +- Mirror the exact tarball `@huggingface/transformers@3.3.3` to the ailiance npm + registry/mirror, OR vendor it under the org. Keep the sha512 from + `package-lock.json` as the integrity anchor. +- **Image-dep trim (resolves the `sharp` duplicate):** the package pulls + `sharp@^0.33.5` (→ `@img/sharp-libvips@1.0.4`), which collides with the repo's + `sharp@0.34.5` (→ libvips `1.2.4`) and is never used by text feature-extraction. + At vendor time, drop/patch the `sharp` dependency from the vendored package + (text inference does not need it). This eliminates the duplicate-libvips objc + warning that a lockfile `overrides` cannot fix cleanly (forcing it cross-major + leaves npm in an `invalid` state — verified 2026-06-02). + +## Step 3 — Point production at the mirror + offline + +- Set `AILIANCE_EMBED_MODEL` + `AILIANCE_EMBED_OFFLINE=1` in the isaac CLI / + extension runtime environment. +- Pre-seed the local model cache on the deploy host (so first run is offline). +- Smoke test: `createDefaultEmbedder().embed(["test"])` returns a 384-d vector + with no network egress (verify with the network disabled). + +## Step 4 — SBOM + audit + +- Add `@huggingface/transformers@3.3.3` (+ sha512) and + `Ailiance-fr/all-MiniLM-L6-v2` (+ upstream commit SHA) to the SBOM. +- Note the HITL review date + reviewer. + +## Verification checklist + +- [ ] Model mirror exists in `Ailiance-fr/`, ONNX hash matches upstream. +- [ ] npm package mirrored/vendored with sha512 anchor; `sharp` trimmed. +- [ ] `AILIANCE_EMBED_MODEL` + `AILIANCE_EMBED_OFFLINE=1` set in prod. +- [ ] Embed works offline (no HF CDN egress). +- [ ] SBOM updated; HITL review recorded. +- [ ] PR #50 merge unblocked. From b0117e76fdbad41cd0b99b14315b791509808e39 Mon Sep 17 00:00:00 2001 From: electron-rare <108685187+electron-rare@users.noreply.github.com> Date: Tue, 2 Jun 2026 09:22:00 +0200 Subject: [PATCH 24/24] docs(mcp): record supply-chain mirror execution Mirror Ailiance-fr/all-MiniLM-L6-v2 created from upstream Xenova snapshot (sha 751bff37 -> mirror 5cbc3683). npm transformers@3.3.3 pin-only per org strategy (no vendor). Embedder verified loading from the mirror. --- ...06-02-mcp-retrieval-supply-chain-mirror.md | 47 ++++++++++++++++--- 1 file changed, 41 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md b/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md index 64abf936..097b8cf9 100644 --- a/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md +++ b/docs/superpowers/runbooks/2026-06-02-mcp-retrieval-supply-chain-mirror.md @@ -77,9 +77,44 @@ Options (pick per the strategy): ## Verification checklist -- [ ] Model mirror exists in `Ailiance-fr/`, ONNX hash matches upstream. -- [ ] npm package mirrored/vendored with sha512 anchor; `sharp` trimmed. -- [ ] `AILIANCE_EMBED_MODEL` + `AILIANCE_EMBED_OFFLINE=1` set in prod. -- [ ] Embed works offline (no HF CDN egress). -- [ ] SBOM updated; HITL review recorded. -- [ ] PR #50 merge unblocked. +- [x] Model mirror exists in `Ailiance-fr/`, ONNX hash matches upstream. +- [x] npm package pinned with sha512 anchor (org strategy = pin-only, NOT vendor). +- [ ] `AILIANCE_EMBED_MODEL` + `AILIANCE_EMBED_OFFLINE=1` set in prod (deploy-time). +- [x] Embed works against the mirror (verified end-to-end, 384-d). +- [x] Audit recorded (below); HITL review done by user 2026-06-02. +- [ ] PR #50 merge unblocked (pending deploy-time env wiring). + +## Correction vs the org supply-chain strategy + +The org policy (`ailiance-gateway/docs/.../2026-05-29-dependency-fork-audit-strategy.md` +§11–§12, user decision 2026-05-29) is **pin-only + SBOM for every ecosystem; +vendoring is reserved for the single Tier-0 untrusted case (`omlx`)**. npm is +explicitly *pin-only* (lockfile + `npm audit`), NOT vendored. So Step 2's +"vendor the npm package" over-specified: `@huggingface/transformers@3.3.3` is +already correctly frozen by its sha512 in `package-lock.json` — no fork/vendor +needed. The `sharp` trim is a perf nicety, not a supply-chain requirement. + +The only genuine supply-chain action was mirroring the **model weights** into +the `ailiance` org for a sovereign offline runtime posture (matching the 20 +other `Ailiance-fr` models). + +## Execution record (2026-06-02, HITL-approved) + +- **Model mirror:** `Ailiance-fr/all-MiniLM-L6-v2` (public, Apache-2.0) created + and populated from the full upstream snapshot. + - upstream `Xenova/all-MiniLM-L6-v2` commit sha = `751bff37182d3f1213fa05d7196b954e230abad9` + - mirror commit sha = `5cbc3683c2dbc1a6372202305af9762e7973c73e` + - contents: `onnx/model.onnx` (90.39 MB) + 7 quantized variants + (`model_fp16/int8/quantized/uint8/q4/q4f16/bnb4.onnx`) + tokenizer/config/ + vocab/README — byte-identical to upstream. + - mirrored by `clemsail` (Ailiance-fr admin, write token) via + `huggingface_hub` `snapshot_download` → `create_repo` → `upload_folder`. + - verified: `AILIANCE_EMBED_MODEL=Ailiance-fr/all-MiniLM-L6-v2` loads through + `createDefaultEmbedder()` and returns a 384-d vector. +- **npm pin (audit anchor):** `@huggingface/transformers@3.3.3`, + `sha512-OcMubhBjW6u1xnp0zSt5SvCxdGHuhP2k+w2Vlm3i0vNcTJhJTZWxxYQmPBfcb7PX+Q6c43lGSzWD6tsJFwka4Q==`, + license Apache-2.0 (`package-lock.json`). Pin-only per org strategy. +- **Remaining (deploy-time, not a code change):** set + `AILIANCE_EMBED_MODEL=Ailiance-fr/all-MiniLM-L6-v2` + `AILIANCE_EMBED_OFFLINE=1` + in the isaac CLI/extension runtime env and pre-seed the model cache on the + deploy host so first run is offline.