diff --git a/skills/capture-evidence/SKILL.md b/skills/capture-evidence/SKILL.md index 31d58847..ad2b1794 100644 --- a/skills/capture-evidence/SKILL.md +++ b/skills/capture-evidence/SKILL.md @@ -18,6 +18,13 @@ The OSS loop does not require registration, auth, provider keys, an Understudy account, or hosted gateway access. Do a sufficient local pass that turns the workload into auditable artifacts and can answer the named decision. +When the developer names a workload already captured by Understudy and the +active credentials can read it, use the hosted-workload front door in +[`references/hosted-workload-eval.md`](references/hosted-workload-eval.md). +It freezes the exact source cohort, downloads payloads with approval, and runs +the same local trace foundry. A hosted eval workspace is not required to author +the verifier. + ## Safety Gates Default to the evidence plan most likely to resolve the decision under the @@ -181,6 +188,21 @@ instead of building a single-output harness; its state-mutating lens records reset/seed state, API schemas, policy docs, request logs, and final-state validators as part of the harness. +For a named hosted workload, prefer: + +```sh +understudy evals build \ + --project \ + --workload \ + --name +``` + +This is a local-authoring operation: the service selects and freezes the source +captures, while the coding agent owns lineage reconstruction, tool-call +interpretation, task and contract proposals, verifier generation, and review. +Do not silently substitute an older server-generated environment for the local +source-bound result. + Start from the real local workload: - app route, eval suite, trace export, benchmark fixture, prompt set, dataset, diff --git a/skills/capture-evidence/references/hosted-workload-eval.md b/skills/capture-evidence/references/hosted-workload-eval.md new file mode 100644 index 00000000..421890bd --- /dev/null +++ b/skills/capture-evidence/references/hosted-workload-eval.md @@ -0,0 +1,111 @@ +# Build a local eval from an Understudy workload + +Use this path when Understudy already has captures for a named project and +workload. It is the shortest route from production evidence to a verifier draft +the coding agent can inspect and improve. + +## Build + +```sh +understudy evals build \ + --project \ + --workload \ + --name +``` + +Before downloading, show the redacted cohort summary and ask for approval. +Non-interactive or JSON runs must pass `--yes`; otherwise fail before any hosted +read. The downloaded files can contain prompts, completions, and tool payloads. +If download or compilation fails after the cohort is frozen, rerun the same +command and destination. The CLI reuses the recorded cohort, builds in a fresh +private attempt directory, and publishes the project directory only after the +cohort and compiler counts agree. + +The CLI records an exact, redacted create checkpoint before freezing the +cohort. Its workload-scoped operation ID makes the backend create idempotent, +so a lost response is recovered by retrying that same request without creating +a second cohort or scanning a bounded list. Failed download and compiler +attempts retain that checkpoint but delete payload-bearing partial files. Capture downloads +accept only short-lived HTTPS URLs from Understudy's R2 origin (plus the exact +configured loopback origin in local development), reject redirects, refresh +URLs before expiry, and enforce 16 MiB per-capture and 256 MiB per-cohort +limits. The materialization manifest records verified hashes and byte counts. + +The command composes three narrow hosted primitives—catalog, immutable cohort, +and export—with the existing local trace foundry. It writes: + +```text +.understudy/evals// +├── captures/ +│ └── cohort-manifest.json +├── benchmark/ +│ ├── manifest.json +│ ├── source-dag.json +│ ├── tasks.jsonl +│ ├── benchmark.json +│ ├── environment/ +│ └── viewer/index.html +├── build-state.json +└── eval-project.json +``` + +`eval-project.json` binds the workload identity and immutable cohort hash to the +local foundry artifacts. The project starts as `local_draft`; the generated +benchmark remains `machine_compiled_review_pending`. + +Leakage-audit details remain in the private manifest. Terminal output reports +counts only and never prints customer-derived excerpts. + +## Ownership boundary + +The backend owns only what requires shared authority: + +- authenticate and scope the organization, project, and workload; +- return a redacted capture catalog; +- freeze immutable capture references and hashes; +- provide bounded export access; +- later, accept an explicitly published verifier package and run it in an + isolated hosted environment. + +The coding agent owns authoring: + +- reconstruct W3C lineage and the source DAG; +- interpret requests, responses, streaming events, and tool calls; +- propose task boundaries, success contracts, splits, and failure modes; +- generate the Verifiers environment, oracle, and negative sentinels; +- organize human feedback and revise the verifier locally. + +A server-generated eval workspace or verifier seed is advisory input, not the +source of truth. Preserve its provenance if used, but regenerate and validate +the runnable artifacts from the frozen local captures. + +## Review before promotion + +Serve the local viewer: + +```sh +understudy traces serve \ + --benchmark .understudy/evals//benchmark \ + --port 3003 +``` + +Inspect complete executions rather than treating historical outputs as gold. +Confirm task boundaries, tool lineage, outcome contracts, held-out semantics, +and representative failure cases. Export the review decisions and import them: + +```sh +understudy traces import-reviews \ + --benchmark .understudy/evals//benchmark \ + --reviews +``` + +The deeper deterministic compiler and promotion contract are documented in +[`../../ingest-traces/references/trace-foundry-cli.md`](../../ingest-traces/references/trace-foundry-cli.md). + +## Privacy and publication + +`evals build` performs no upload after the capture download and calls no model +provider. Keep the project private: it contains customer payloads. Publication, +model sweeps, prompt experiments, and hosted verifier execution are separate, +explicit later actions. Do not infer upload permission from permission to build +locally. diff --git a/skills/ingest-traces/references/trace-foundry-cli.md b/skills/ingest-traces/references/trace-foundry-cli.md index 24986ae2..18a9b08f 100644 --- a/skills/ingest-traces/references/trace-foundry-cli.md +++ b/skills/ingest-traces/references/trace-foundry-cli.md @@ -3,6 +3,21 @@ Use these helpers instead of rewriting normalization, DAG construction, benchmark manifests, review application, environment packages, or replay loops. +When the source is a named workload already captured by Understudy, the +one-command front door freezes and downloads an immutable cohort before invoking +this same compiler locally: + +```sh +understudy evals build \ + --project \ + --workload \ + --name +``` + +See +[`../../capture-evidence/references/hosted-workload-eval.md`](../../capture-evidence/references/hosted-workload-eval.md) +for the service-versus-agent ownership boundary and privacy gates. + ## Compile or resume ```sh diff --git a/skills/understudy/SKILL.md b/skills/understudy/SKILL.md index 1d44c19d..80ebe1b8 100644 --- a/skills/understudy/SKILL.md +++ b/skills/understudy/SKILL.md @@ -150,7 +150,9 @@ Identify the developer's current stage and load exactly one: - **Codebase / evidence not yet pinned down** — LLM call sites, current model/ harness, traces, metric, splits, or incumbent baseline are missing, ambiguous, or stale → [`../capture-evidence/SKILL.md`](../capture-evidence/SKILL.md) - (also owns repo inspection + eval-harness discovery/build). + (also owns repo inspection + eval-harness discovery/build). For a named + workload already captured by Understudy, this route uses `understudy evals + build` to freeze the source cohort and construct the verifier draft locally. - **App is running but no traces exist yet** — the developer wants capture flowing in minutes with no app-code changes ("instrument my app", "start capturing my LLM calls") → [`../instrument/SKILL.md`](../instrument/SKILL.md) diff --git a/src/commands/evals.ts b/src/commands/evals.ts index ef72edf0..6fe410f5 100644 --- a/src/commands/evals.ts +++ b/src/commands/evals.ts @@ -1,64 +1,45 @@ -import { createHash } from "node:crypto"; -import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; -import { dirname, join, resolve } from "node:path"; +import { chmodSync, mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { basename, dirname, join, resolve } from "node:path"; import { confirm } from "@inquirer/prompts"; import { Command } from "commander"; import kleur from "kleur"; -import { z } from "zod"; +import { buildEvalProject } from "../eval-project.js"; +import { + acquireEvalBuildLease, + assertBuildStateMatches, + buildState, + cohortFromResponse, + creatingBuildState, + initializeBuildCheckpoint, + pathExists, + readEvalBuildState, + replacePrivateJson, + writePrivateJson, +} from "../evals/build-state.js"; +import { + CatalogItemSchema, + CatalogResponseSchema, + CohortExportSchema, + CohortSchema, + type CatalogItem, + type Cohort, + type EvalBuildCreatingState, + type EvalBuildIdentity, + type EvalBuildSelection, + type FrozenCohort, +} from "../evals/contracts.js"; +import { + assertEquivalentExport, + assertExportLineage, + downloadExport, + EXPORT_EXPIRES_SECONDS, +} from "../evals/materialize.js"; import { request } from "../internal/http.js"; import { isJsonMode, runAction } from "../internal/output.js"; import { resolveProject, type ProjectResolutionOptions } from "../internal/projects.js"; import { resolveWorkload } from "../internal/workloads.js"; -const Sha256Schema = z.string().regex(/^[a-f0-9]{64}$/); -const CatalogItemSchema = z.object({ - capture_key: z.string(), - request_id: z.string(), - content_sha256: Sha256Schema, - captured_at: z.string(), - provider: z.string(), - requested_model: z.string(), - served_model: z.string(), - status_code: z.number().int(), - latency_ms: z.number().nonnegative(), - has_tools: z.boolean(), - has_structured_output: z.boolean(), -}); -const CatalogResponseSchema = z.object({ - captures: z.array(CatalogItemSchema), - selection: z.object({ - from: z.string(), - to: z.string(), - limit: z.number().int().positive(), - sample_seed: z.string(), - requested_model: z.string().nullable(), - served_model: z.string().nullable(), - status_code: z.number().int().nullable(), - requires_tools: z.boolean(), - requires_structured_output: z.boolean(), - }), -}); -const CohortSchema = z.object({ - id: z.string(), - workload_id: z.string(), - name: z.string(), - capture_count: z.number().int().positive(), - cohort_sha256: Sha256Schema, - created_at: z.string(), -}).passthrough(); -const CohortExportSchema = z.object({ - export_id: z.string(), - cohort_id: z.string(), - cohort_sha256: Sha256Schema, - expires_at: z.string(), - captures: z.array(z.object({ - request_id: z.string(), - content_sha256: Sha256Schema, - url: z.string().url(), - })).min(1).max(500), -}); - interface WorkloadOpts extends ProjectResolutionOptions { workload: string; } @@ -98,23 +79,19 @@ interface GuidedCreateOpts extends WorkloadOpts { download: boolean; yes?: boolean; } +interface BuildOpts extends Omit { + maxAgeDays?: string; + batchSize: string; +} export function registerEvalsCommand(program: Command): void { const evals = program.command("evals") .description("Select, freeze, and materialize workload-scoped evaluation cohorts."); - addWorkloadOptions(evals.command("create") - .description("Create a frozen eval set from a recent workload window.") - .requiredOption("--name ", "Cohort name.") - .option("--description ", "Why these captures were selected.") - .option("--last ", "Recent window, such as 14d or 12h (max 31d).", "14d") - .option("--limit ", "Candidate limit, max 100.", "50") - .option("--seed ", "Deterministic sample seed.", "understudy-eval-catalog-v1") - .option("--requested-model ", "Filter by requested model.") - .option("--served-model ", "Filter by served model.") - .option("--status-code ", "Filter by HTTP status code.") - .option("--requires-tools", "Require a trace containing tools.") - .option("--requires-structured-output", "Require structured output.") + addWorkloadOptions(addRecentSelectionOptions( + evals.command("create").description("Create a frozen eval set from a recent workload window."), + "Cohort name.", + ) .option("--out ", "Destination directory (default: .understudy/evals/).") .option("--no-download", "Freeze the cohort without downloading trace bodies.") .option("--yes", "Approve freezing and local trace download without prompting.")) @@ -122,6 +99,18 @@ export function registerEvalsCommand(program: Command): void { await runAction(this, () => runGuidedCreate(this, opts)); }); + addWorkloadOptions(addRecentSelectionOptions( + evals.command("build").description("Build a private local draft eval project from a frozen workload cohort."), + "Local eval project and cohort name.", + ) + .option("--out ", "Destination directory (default: .understudy/evals/).") + .option("--max-age-days ", "Override freshness cutoff (must cover --last; default: derive from --last).") + .option("--batch-size ", "Local trace-foundry processing batch size.", "10") + .option("--yes", "Approve freezing and downloading payload-bearing traces without prompting.")) + .action(async function (this: Command, opts: BuildOpts) { + await runAction(this, () => runBuild(this, opts)); + }); + addWorkloadOptions(evals.command("catalog") .description("List redacted capture candidates for one workload.") .requiredOption("--from ", "Inclusive ISO-8601 window start.") @@ -165,6 +154,20 @@ function addWorkloadOptions(command: Command): Command { .option("--org ", "Org id (default: local config or only credential org)."); } +function addRecentSelectionOptions(command: Command, nameDescription: string): Command { + return command + .requiredOption("--name ", nameDescription) + .option("--description ", "Why these captures were selected.") + .option("--last ", "Recent window, such as 14d or 12h (max 31d).", "14d") + .option("--limit ", "Candidate limit, max 100.", "50") + .option("--seed ", "Deterministic sample seed.", "understudy-eval-catalog-v1") + .option("--requested-model ", "Filter by requested model.") + .option("--served-model ", "Filter by served model.") + .option("--status-code ", "Filter by HTTP status code.") + .option("--requires-tools", "Require a trace containing tools.") + .option("--requires-structured-output", "Require structured output."); +} + async function resolveContext(opts: WorkloadOpts) { const project = await resolveProject(opts); const workload = await resolveWorkload(project, opts.workload); @@ -207,7 +210,13 @@ async function runGuidedCreate(cmd: Command, opts: GuidedCreateOpts): Promise { + const batchSize = parsePositiveInteger("--batch-size", opts.batchSize); + const windowMs = parseDuration(opts.last); + const selectionDays = Math.ceil(windowMs / 86_400_000); + const maxAgeDays = opts.maxAgeDays === undefined + ? selectionDays + : parsePositiveInteger("--max-age-days", opts.maxAgeDays); + if (maxAgeDays < selectionDays) { + throw new Error(`--max-age-days must cover --last (${selectionDays} day(s)).`); + } + const selection = buildSelectionFromOptions(opts); + if (isJsonMode(cmd) && !opts.yes) { + throw new Error("JSON mode cannot prompt. Re-run with --yes to approve freezing and local trace download."); + } + if (!opts.yes && !process.stdin.isTTY) { + throw new Error("Non-interactive eval builds cannot prompt. Re-run with --yes to approve freezing and local trace download."); + } + const output = resolve(opts.out ?? join(".understudy", "evals", safeFileStem(opts.name))); + if (pathExists(output)) { + throw new Error(`Eval build destination already exists: ${output}. Choose a fresh --out directory.`); + } + const releaseLease = acquireEvalBuildLease(output); + try { + await runBuildWithLease(cmd, opts, { batchSize, windowMs, maxAgeDays, output, selection }); + } finally { + releaseLease(); + } +} + +async function runBuildWithLease( + cmd: Command, + opts: BuildOpts, + build: { batchSize: number; windowMs: number; maxAgeDays: number; output: string; selection: EvalBuildSelection }, +): Promise { + const { batchSize, windowMs, maxAgeDays, output, selection } = build; + if (pathExists(output)) { + throw new Error(`Eval build destination already exists: ${output}. Choose a fresh --out directory.`); + } + const staging = join(dirname(output), `.${basename(output)}.eval-build`); + const pending = pathExists(staging) ? readEvalBuildState(staging) : null; + const to = new Date(); + let context: Awaited>; + let cohort: FrozenCohort; + let identity: EvalBuildIdentity; + + if (pending) { + context = await resolveContext(opts); + const currentIdentity = identityFromContext(context); + assertBuildStateMatches(pending, opts.name, currentIdentity, selection, maxAgeDays, batchSize); + identity = pending.identity; + if (pending.status === "cohort_creating") { + const created = await createOrRecoverBuildCohort(context, pending); + cohort = cohortFromResponse(created); + replacePrivateJson( + join(staging, "build-state.json"), + buildState("cohort_frozen", opts.name, identity, cohort, selection, maxAgeDays, batchSize, new Date(pending.created_at)), + ); + } else { + cohort = pending.cohort; + } + if (!isJsonMode(cmd)) { + process.stdout.write(`Resuming frozen cohort ${cohort.id} (${cohort.capture_count} captures).\n`); + } + if (!opts.yes) { + const approved = await confirm({ + message: `Resume downloading ${cohort.capture_count} payload-bearing captures for local draft “${opts.name}” (16 MiB each, 256 MiB total maximum)?`, + default: false, + }); + if (!approved) throw new Error("Eval build resume cancelled before payload download."); + } + } else { + const from = new Date(to.getTime() - windowMs); + const catalog = await fetchCatalog(opts, from.toISOString(), to.toISOString()); + if (catalog.response.captures.length === 0) { + throw new Error(`No eligible captures found for ${catalog.workload.name} in the last ${opts.last}.`); + } + if (!isJsonMode(cmd)) printCatalogSummary(catalog.workload.name, catalog.response.captures); + if (!opts.yes) { + const approved = await confirm({ + message: `Freeze these ${catalog.response.captures.length} captures and download their payloads to build local draft “${opts.name}”? The files may contain prompts, completions, and tool payloads (16 MiB each, 256 MiB total maximum).`, + default: false, + }); + if (!approved) throw new Error("Eval build cancelled before payload download."); + } + context = catalog; + identity = identityFromContext(context); + const creating = creatingBuildState(opts.name, opts.description, identity, catalog.response, selection, maxAgeDays, batchSize, to); + initializeBuildCheckpoint(staging, creating); + const created = await createOrRecoverBuildCohort(context, creating); + cohort = cohortFromResponse(created); + replacePrivateJson( + join(staging, "build-state.json"), + buildState("cohort_frozen", opts.name, identity, cohort, selection, maxAgeDays, batchSize, to), + ); + } + + const attempts = join(staging, "attempts"); + // An interrupted process may have left payload-bearing partial attempts. + // The frozen cohort state lives outside this directory, so retries can + // safely clear them instead of accumulating customer data. + rmSync(attempts, { recursive: true, force: true }); + mkdirSync(attempts, { recursive: true, mode: 0o700 }); + chmodSync(attempts, 0o700); + const attempt = mkdtempSync(join(attempts, "attempt-")); + chmodSync(attempt, 0o700); + const buildNow = pending ? new Date(pending.created_at) : to; + let published = false; + let project!: ReturnType; + try { + const materialized = await materializeCohort( + context, + cohort.id, + cohort.cohort_sha256, + join(attempt, "captures"), + cohort.capture_count, + ); + project = buildEvalProject({ + output: attempt, + identity: { + orgId: identity.org_id, + projectId: identity.project_id, + workloadId: identity.workload_id, + workloadName: identity.workload_name, + }, + cohort: { + id: cohort.id, + cohortSha256: cohort.cohort_sha256, + captureCount: cohort.capture_count, + materializationManifest: materialized.manifest, + }, + maxAgeDays, + batchSize, + now: buildNow, + }); + writePrivateJson(join(attempt, "build-state.json"), buildState("complete", opts.name, identity, cohort, selection, maxAgeDays, batchSize, buildNow)); + renameSync(attempt, output); + published = true; + } finally { + if (!published) rmSync(attempt, { recursive: true, force: true }); + } + rmSync(staging, { recursive: true, force: true }); + project.project_file = join(output, "eval-project.json"); + + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(project)}\n`); + } else { + process.stdout.write(`${kleur.green("✓")} Created local draft eval project at ${output}\n`); + process.stdout.write(`Project manifest: ${project.project_file}\n`); + process.stdout.write(`Review viewer: ${join(output, project.foundry.artifacts.viewer)}\n`); + process.stdout.write(`${kleur.yellow("warning")}: local files contain prompts, completions, or tool payloads; nothing was uploaded and no model provider was called\n`); + } +} + async function fetchCatalog(opts: Omit, from: string, to: string) { const limit = parseLimit(opts.limit); validateStatusCode(opts.statusCode); @@ -239,7 +401,7 @@ async function fetchCatalog(opts: Omit, from: string if (opts.requiresTools) params.set("requires_tools", "true"); if (opts.requiresStructuredOutput) params.set("requires_structured_output", "true"); const response = await request( - { url: `${base}/eval-capture-catalog?${params}`, orgId: project.auth.orgId }, + { url: `${base}/eval-capture-catalog?${params}`, orgId: project.auth.orgId, signal: AbortSignal.timeout(60_000) }, CatalogResponseSchema, ); return { project, workload, base, response: response.data }; @@ -252,6 +414,7 @@ async function createCohort( ) { const response = await request({ url: `${context.base}/eval-cohorts`, method: "POST", orgId: context.project.auth.orgId, + signal: AbortSignal.timeout(60_000), body: { name, selection: { source: "explicit_capture_references", description, sampling_seed: context.response.selection.sample_seed }, @@ -261,19 +424,68 @@ async function createCohort( return response.data; } +async function createOrRecoverBuildCohort( + context: Awaited>, + state: EvalBuildCreatingState, +): Promise { + let response: { data: Cohort } | null = null; + let createError: unknown; + for (let attempt = 0; attempt < 2 && response === null; attempt += 1) { + try { + response = await request({ + url: `${context.base}/eval-cohorts`, + method: "POST", + orgId: context.project.auth.orgId, + signal: AbortSignal.timeout(60_000), + body: state.create_request, + }, CohortSchema); + } catch (error) { + createError = error; + } + } + if (response === null) throw createError; + const cohort = response.data; + if ( + cohort.operation_id !== state.create_request.operation_id || + cohort.org_id !== context.project.auth.orgId || + cohort.project_id !== context.project.projectId || + cohort.workload_id !== context.workload.id || + cohort.capture_count !== state.create_request.captures.length + ) { + throw new Error(`Created cohort ${cohort.id} does not match the persisted eval build selection.`); + } + return cohort; +} + async function materializeCohort( - context: Awaited>, + context: Awaited>, cohortId: string, + expectedCohortSha256: string, out: string, + expectedCaptureCount: number, ) { - const response = await request( - { url: `${context.base}/eval-cohorts/${encodeURIComponent(cohortId)}/export`, method: "POST", body: {}, orgId: context.project.auth.orgId }, - CohortExportSchema, - ); - return downloadExport(response.data, context.workload.id, out); + const createExport = async () => { + const response = await request( + { + url: `${context.base}/eval-cohorts/${encodeURIComponent(cohortId)}/export`, + method: "POST", + body: { expires_seconds: EXPORT_EXPIRES_SECONDS }, + orgId: context.project.auth.orgId, + signal: AbortSignal.timeout(60_000), + }, + CohortExportSchema, + ); + assertExportLineage(response.data, cohortId, expectedCohortSha256); + if (response.data.captures.length !== expectedCaptureCount) { + throw new Error(`Cohort export count ${response.data.captures.length} does not match frozen cohort count ${expectedCaptureCount}.`); + } + return response.data; + }; + const exportData = await createExport(); + return downloadExport(exportData, context.workload.id, out, context.project.auth.gatewayUrl, createExport); } -function printCatalogSummary(workloadName: string, captures: z.infer[]): void { +function printCatalogSummary(workloadName: string, captures: CatalogItem[]): void { const models = new Set(captures.map((capture) => capture.served_model)); const errors = captures.filter((capture) => capture.status_code >= 400).length; const tools = captures.filter((capture) => capture.has_tools).length; @@ -337,11 +549,26 @@ async function runCohortExport(cmd: Command, cohortId: string, opts: CohortExpor throw new Error("Cohort files may contain prompts/completions. Re-run with --yes to download them locally."); } const { project, workload, base } = await resolveContext(opts); - const response = await request( - { url: `${base}/eval-cohorts/${encodeURIComponent(cohortId)}/export`, method: "POST", body: {}, orgId: project.auth.orgId }, - CohortExportSchema, - ); - const payload = await downloadExport(response.data, workload.id, opts.out); + const createExport = async () => { + const response = await request( + { + url: `${base}/eval-cohorts/${encodeURIComponent(cohortId)}/export`, + method: "POST", + body: { expires_seconds: EXPORT_EXPIRES_SECONDS }, + orgId: project.auth.orgId, + signal: AbortSignal.timeout(60_000), + }, + CohortExportSchema, + ); + if (response.data.cohort_id !== cohortId) throw new Error(`Cohort export lineage does not match requested cohort ${cohortId}.`); + return response.data; + }; + const firstExport = await createExport(); + const payload = await downloadExport(firstExport, workload.id, opts.out, project.auth.gatewayUrl, async () => { + const refreshed = await createExport(); + assertEquivalentExport(firstExport, refreshed); + return refreshed; + }); if (isJsonMode(cmd)) process.stdout.write(`${JSON.stringify({ ok: true, ...payload })}\n`); else { process.stdout.write(`${kleur.green("✓")} Materialized ${payload.count} frozen captures at ${payload.output}\n`); @@ -349,42 +576,20 @@ async function runCohortExport(cmd: Command, cohortId: string, opts: CohortExpor } } -async function downloadExport(exportData: z.infer, workloadId: string, out: string) { - const outputDir = resolve(out); - mkdirSync(outputDir, { recursive: true }); - const files: Array<{ request_id: string; path: string; content_sha256: string }> = []; - const fileNames = new Set(); - for (const capture of exportData.captures) { - const download = await fetch(capture.url, { headers: { Accept: "application/x-ndjson" } }); - if (!download.ok) throw new Error(`Capture ${capture.request_id} download failed with status ${download.status}.`); - const bytes = new Uint8Array(await download.arrayBuffer()); - const digest = createHash("sha256").update(bytes).digest("hex"); - if (digest !== capture.content_sha256) throw new Error(`Capture ${capture.request_id} failed SHA-256 verification.`); - const stem = safeFileStem(capture.request_id); - let fileName = `${stem}.jsonl`; - if (fileNames.has(fileName)) fileName = `${stem}-${capture.content_sha256.slice(0, 12)}.jsonl`; - if (fileNames.has(fileName)) throw new Error(`Capture ${capture.request_id} collides with another local filename.`); - fileNames.add(fileName); - writeFileSync(join(outputDir, fileName), bytes); - files.push({ request_id: capture.request_id, path: fileName, content_sha256: digest }); - } - const localManifest = join(outputDir, "cohort-manifest.json"); - writeJson(localManifest, { - schema_version: "understudy.eval-cohort-materialization.v1", - cohort_id: exportData.cohort_id, - cohort_sha256: exportData.cohort_sha256, - workload_id: workloadId, - capture_count: files.length, - privacy: { local_only: true, upload_performed: false }, - captures: files, - }); - return { output: outputDir, manifest: localManifest, count: files.length, cohort_sha256: exportData.cohort_sha256 }; -} - function writeJson(path: string, value: unknown): void { const absolute = resolve(path); - mkdirSync(dirname(absolute), { recursive: true }); - writeFileSync(absolute, `${JSON.stringify(value, null, 2)}\n`, "utf8"); + mkdirSync(dirname(absolute), { recursive: true, mode: 0o700 }); + writeFileSync(absolute, `${JSON.stringify(value, null, 2)}\n`, { encoding: "utf8", mode: 0o600 }); + chmodSync(absolute, 0o600); +} + +function identityFromContext(context: Awaited>) { + return { + org_id: context.project.auth.orgId, + project_id: context.project.projectId, + workload_id: context.workload.id, + workload_name: context.workload.name, + }; } function parseIsoOption(name: string, value: string): string { @@ -409,6 +614,22 @@ function validateStatusCode(value?: string): void { } } +function buildSelectionFromOptions(opts: BuildOpts): EvalBuildSelection { + const limit = parseLimit(opts.limit); + validateStatusCode(opts.statusCode); + return { + last: opts.last, + limit, + seed: opts.seed, + description: opts.description ?? null, + requested_model: opts.requestedModel ?? null, + served_model: opts.servedModel ?? null, + status_code: opts.statusCode === undefined ? null : Number(opts.statusCode), + requires_tools: opts.requiresTools ?? false, + requires_structured_output: opts.requiresStructuredOutput ?? false, + }; +} + function parseDuration(value: string): number { const match = /^(\d+)(h|d)$/.exec(value); if (!match) throw new Error("--last must be a duration such as 12h or 14d."); @@ -420,6 +641,14 @@ function parseDuration(value: string): number { return durationMs; } +function parsePositiveInteger(name: string, value: string): number { + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed <= 0) { + throw new Error(`${name} must be a positive integer.`); + } + return parsed; +} + function safeFileStem(value: string): string { const safe = value.replace(/[^a-zA-Z0-9._-]/g, "_"); if (!safe || safe === "." || safe === "..") throw new Error(`Unsafe request id: ${value}`); diff --git a/src/eval-project.ts b/src/eval-project.ts new file mode 100644 index 00000000..566b00cb --- /dev/null +++ b/src/eval-project.ts @@ -0,0 +1,139 @@ +import { mkdirSync, writeFileSync } from "node:fs"; +import { isAbsolute, join, relative, resolve, sep } from "node:path"; + +import { compileTraceFoundry, type FoundryResult } from "./trace-foundry.js"; + +export interface EvalProjectIdentity { + orgId: string; + projectId: string; + workloadId: string; + workloadName: string; +} + +export interface EvalProjectCohort { + id: string; + cohortSha256: string; + captureCount: number; + materializationManifest: string; +} + +export interface BuildEvalProjectOptions { + output: string; + identity: EvalProjectIdentity; + cohort: EvalProjectCohort; + maxAgeDays: number; + batchSize: number; + now?: Date; +} + +export interface EvalProjectManifest { + schema_version: "understudy.eval-project.v1"; + status: "local_draft"; + created_at: string; + identity: { + org_id: string; + project_id: string; + workload_id: string; + workload_name: string; + }; + cohort: { + id: string; + cohort_sha256: string; + capture_count: number; + materialization_manifest: string; + }; + foundry: { + status: FoundryResult["status"]; + output: string; + manifest: string; + artifacts: Record; + counts: FoundryResult["counts"]; + }; + privacy: { + local_only: true; + contains_customer_payloads: true; + upload_performed: false; + provider_called: false; + }; +} + +export interface EvalProjectBuildResult extends EvalProjectManifest { + project_file: string; +} + +function portableRelative(root: string, path: string): string { + const value = relative(root, path); + if (!value || value === ".." || value.startsWith(`..${sep}`)) { + throw new Error(`Eval project artifact must remain inside ${root}.`); + } + return value.split(sep).join("/"); +} + +function foundryArtifactPath(projectRoot: string, benchmarkRoot: string, value: string): string { + const absolute = isAbsolute(value) ? value : resolve(benchmarkRoot, value); + return portableRelative(projectRoot, absolute); +} + +/** + * Compile already-materialized, workload-scoped captures into the existing + * trace-foundry proposal and bind both artifacts in one small local manifest. + * This function performs no upload and no provider call. + */ +export function buildEvalProject(options: BuildEvalProjectOptions): EvalProjectBuildResult { + const now = options.now ?? new Date(); + const projectRoot = resolve(options.output); + const capturesRoot = join(projectRoot, "captures"); + const benchmarkRoot = join(projectRoot, "benchmark"); + mkdirSync(projectRoot, { recursive: true, mode: 0o700 }); + + const foundry = compileTraceFoundry( + capturesRoot, + benchmarkRoot, + options.maxAgeDays, + now, + { workload: options.identity.workloadId, batchSize: options.batchSize }, + ); + if (foundry.counts.captures !== options.cohort.captureCount || foundry.counts.stale_filtered !== 0) { + throw new Error( + `Compiled capture count ${foundry.counts.captures} does not match frozen cohort count ${options.cohort.captureCount}.`, + ); + } + const projectFile = join(projectRoot, "eval-project.json"); + const project: EvalProjectManifest = { + schema_version: "understudy.eval-project.v1", + status: "local_draft", + created_at: now.toISOString(), + identity: { + org_id: options.identity.orgId, + project_id: options.identity.projectId, + workload_id: options.identity.workloadId, + workload_name: options.identity.workloadName, + }, + cohort: { + id: options.cohort.id, + cohort_sha256: options.cohort.cohortSha256, + capture_count: options.cohort.captureCount, + materialization_manifest: portableRelative(projectRoot, resolve(options.cohort.materializationManifest)), + }, + foundry: { + status: foundry.status, + output: portableRelative(projectRoot, benchmarkRoot), + manifest: portableRelative(projectRoot, join(benchmarkRoot, "manifest.json")), + artifacts: Object.fromEntries( + Object.entries(foundry.artifacts).map(([name, path]) => [ + name, + foundryArtifactPath(projectRoot, benchmarkRoot, path), + ]), + ), + counts: foundry.counts, + }, + privacy: { + local_only: true, + contains_customer_payloads: true, + upload_performed: false, + provider_called: false, + }, + }; + writeFileSync(projectFile, `${JSON.stringify(project, null, 2)}\n`, { mode: 0o600, flag: "wx" }); + return { ...project, project_file: projectFile }; +} diff --git a/src/evals/build-state.ts b/src/evals/build-state.ts new file mode 100644 index 00000000..fba63bc0 --- /dev/null +++ b/src/evals/build-state.ts @@ -0,0 +1,254 @@ +import { randomUUID } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { chmodSync, lstatSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { basename, dirname, join } from "node:path"; + +import { + EvalBuildCreatingStateSchema, + EvalBuildStateSchema, + type CatalogResponse, + type Cohort, + type EvalBuildCreatingState, + type EvalBuildIdentity, + type EvalBuildSelection, + type EvalBuildState, + type FrozenCohort, +} from "./contracts.js"; + +export function pathExists(path: string): boolean { + try { + lstatSync(path); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; + throw error; + } +} + +export function createPrivateDirectory(path: string): void { + mkdirSync(dirname(path), { recursive: true, mode: 0o700 }); + mkdirSync(path, { mode: 0o700 }); + chmodSync(path, 0o700); +} + +export function writePrivateJson(path: string, value: unknown): void { + writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`, { + encoding: "utf8", + mode: 0o600, + flag: "wx", + }); +} + +export function replacePrivateJson(path: string, value: unknown): void { + const temporary = `${path}.tmp-${randomUUID()}`; + try { + writePrivateJson(temporary, value); + renameSync(temporary, path); + chmodSync(path, 0o600); + } finally { + rmSync(temporary, { force: true }); + } +} + +export function initializeBuildCheckpoint(staging: string, state: EvalBuildCreatingState): void { + if (pathExists(staging)) throw new Error(`Eval build checkpoint already exists: ${staging}`); + const temporary = join(dirname(staging), `.${basename(staging)}.init-${randomUUID()}`); + try { + createPrivateDirectory(temporary); + writePrivateJson(join(temporary, "build-state.json"), state); + renameSync(temporary, staging); + } finally { + rmSync(temporary, { recursive: true, force: true }); + } +} + +export function readEvalBuildState(staging: string): EvalBuildState { + const stagingStat = lstatSync(staging); + if (stagingStat.isSymbolicLink() || !stagingStat.isDirectory()) { + throw new Error(`Eval build staging path must be a real directory: ${staging}`); + } + chmodSync(staging, 0o700); + const statePath = join(staging, "build-state.json"); + const stateStat = lstatSync(statePath); + if (stateStat.isSymbolicLink() || !stateStat.isFile()) { + throw new Error(`Eval build state must be a real file: ${statePath}`); + } + chmodSync(statePath, 0o600); + return EvalBuildStateSchema.parse(JSON.parse(readFileSync(statePath, "utf8"))); +} + +export function creatingBuildState( + name: string, + description: string | undefined, + identity: EvalBuildIdentity, + catalog: CatalogResponse, + selection: EvalBuildSelection, + maxAgeDays: number, + batchSize: number, + now: Date, +): EvalBuildCreatingState { + const operationId = randomUUID(); + return EvalBuildCreatingStateSchema.parse({ + schema_version: "understudy.eval-build-state.v1", + status: "cohort_creating", + created_at: now.toISOString(), + name, + identity, + selection, + create_request: { + operation_id: operationId, + name, + selection: { + source: "explicit_capture_references", + ...(description === undefined ? {} : { description }), + sampling_seed: catalog.selection.sample_seed, + }, + captures: catalog.captures.map(({ capture_key, request_id, content_sha256 }) => ({ capture_key, request_id, content_sha256 })), + }, + compile: { max_age_days: maxAgeDays, batch_size: batchSize }, + }); +} + +export function buildState( + status: "cohort_frozen" | "complete", + name: string, + identity: EvalBuildIdentity, + cohort: FrozenCohort, + selection: EvalBuildSelection, + maxAgeDays: number, + batchSize: number, + now: Date, +) { + return { + schema_version: "understudy.eval-build-state.v1" as const, + status, + created_at: now.toISOString(), + name, + identity, + selection, + cohort, + compile: { max_age_days: maxAgeDays, batch_size: batchSize }, + }; +} + +export function assertBuildStateMatches( + state: EvalBuildState, + name: string, + identity: EvalBuildIdentity, + selection: EvalBuildSelection, + maxAgeDays: number, + batchSize: number, +): void { + if (state.status === "complete" || state.name !== name) { + throw new Error("Existing eval build state does not match this resumable build."); + } + for (const key of ["org_id", "project_id", "workload_id"] as const) { + if (state.identity[key] !== identity[key]) { + throw new Error(`Existing eval build state does not match ${key}.`); + } + } + if (state.compile.max_age_days !== maxAgeDays || state.compile.batch_size !== batchSize) { + throw new Error("Existing eval build state does not match the compile options."); + } + if (JSON.stringify(state.selection) !== JSON.stringify(selection)) { + throw new Error("Existing eval build state does not match the capture selection options."); + } +} + +export function cohortFromResponse(cohort: Cohort): FrozenCohort { + return { id: cohort.id, cohort_sha256: cohort.cohort_sha256, capture_count: cohort.capture_count }; +} + +interface LeaseOwner { + token: string; + pid: number; + process_instance_id?: string; + created_at: string; +} + +function processInstanceId(pid: number): string | null { + if (process.platform === "linux") { + try { + const bootId = readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim(); + const stat = readFileSync(`/proc/${pid}/stat`, "utf8"); + const commandEnd = stat.lastIndexOf(")"); + if (!bootId || commandEnd === -1) return null; + const fieldsAfterCommand = stat.slice(commandEnd + 1).trim().split(/\s+/); + const startTimeTicks = fieldsAfterCommand[19]; + if (!/^\d+$/.test(startTimeTicks ?? "")) return null; + return `linux-proc-v1:${pid}:${bootId}:${startTimeTicks}`; + } catch { + return null; + } + } + + if (process.platform === "darwin") { + const result = spawnSync("/bin/ps", ["-p", String(pid), "-o", "lstart="], { + encoding: "utf8", + env: { LC_ALL: "C", LANG: "C" }, + timeout: 1_000, + maxBuffer: 1_024, + }); + const startedAt = result.status === 0 ? result.stdout.trim().replace(/\s+/g, " ") : ""; + return startedAt ? `darwin-ps-v1:${pid}:${startedAt}` : null; + } + + return null; +} + +function processIsAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code !== "ESRCH"; + } +} + +export function acquireEvalBuildLease(output: string): () => void { + const leasePath = join(dirname(output), `.${basename(output)}.eval-build.lock`); + mkdirSync(dirname(leasePath), { recursive: true, mode: 0o700 }); + try { + mkdirSync(leasePath, { mode: 0o700 }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + let owner: LeaseOwner | null = null; + try { + owner = JSON.parse(readFileSync(join(leasePath, "owner.json"), "utf8")) as LeaseOwner; + } catch { + throw new Error(`Another eval build owns ${output}; lock metadata is incomplete at ${leasePath}.`); + } + if (Number.isInteger(owner.pid) && processIsAlive(owner.pid)) { + const observedProcessInstanceId = processInstanceId(owner.pid); + if ( + typeof owner.process_instance_id !== "string" || + observedProcessInstanceId === null || + owner.process_instance_id === observedProcessInstanceId + ) { + throw new Error(`Another eval build (pid ${owner.pid}) already owns ${output}.`); + } + } + throw new Error(`A stale eval build lock remains at ${leasePath} (owner pid ${owner.pid}). Remove that exact lock directory, then rerun to resume.`); + } + const instanceId = processInstanceId(process.pid); + const owner: LeaseOwner = { + token: randomUUID(), + pid: process.pid, + ...(instanceId === null ? {} : { process_instance_id: instanceId }), + created_at: new Date().toISOString(), + }; + try { + writePrivateJson(join(leasePath, "owner.json"), owner); + } catch (error) { + rmSync(leasePath, { recursive: true, force: true }); + throw error; + } + return () => { + try { + const current = JSON.parse(readFileSync(join(leasePath, "owner.json"), "utf8")) as LeaseOwner; + if (current.token === owner.token) rmSync(leasePath, { recursive: true, force: true }); + } catch { + // A missing lock is already released; a replaced lock belongs to another process. + } + }; +} diff --git a/src/evals/contracts.ts b/src/evals/contracts.ts new file mode 100644 index 00000000..d30e5dac --- /dev/null +++ b/src/evals/contracts.ts @@ -0,0 +1,132 @@ +import { z } from "zod"; + +export const Sha256Schema = z.string().regex(/^[a-f0-9]{64}$/); + +export const CatalogSelectionSchema = z.object({ + from: z.string(), + to: z.string(), + limit: z.number().int().positive(), + sample_seed: z.string(), + requested_model: z.string().nullable(), + served_model: z.string().nullable(), + status_code: z.number().int().nullable(), + requires_tools: z.boolean(), + requires_structured_output: z.boolean(), +}); + +export const CohortSelectionSchema = z.object({ + source: z.literal("explicit_capture_references"), + description: z.string().min(1).max(1000).optional(), + sampling_seed: z.string().min(1).max(200).optional(), +}); + +export const CatalogItemSchema = z.object({ + capture_key: z.string(), + request_id: z.string(), + content_sha256: Sha256Schema, + captured_at: z.string(), + provider: z.string(), + requested_model: z.string(), + served_model: z.string(), + status_code: z.number().int(), + latency_ms: z.number().nonnegative(), + has_tools: z.boolean(), + has_structured_output: z.boolean(), +}); + +export const CatalogResponseSchema = z.object({ + captures: z.array(CatalogItemSchema), + selection: CatalogSelectionSchema, +}); + +export const CohortSchema = z.object({ + id: z.string(), + org_id: z.string(), + project_id: z.string(), + workload_id: z.string(), + name: z.string().min(1).max(120), + operation_id: z.string().uuid().nullable().optional(), + selection: CohortSelectionSchema, + capture_count: z.number().int().positive(), + cohort_sha256: Sha256Schema, + created_at: z.string(), +}).passthrough(); + +export const CohortExportSchema = z.object({ + export_id: z.string(), + cohort_id: z.string(), + cohort_sha256: Sha256Schema, + expires_at: z.string().datetime(), + captures: z.array(z.object({ + request_id: z.string(), + content_sha256: Sha256Schema, + url: z.string().url(), + })).min(1).max(500), +}); + +export const EvalBuildStateBaseSchema = z.object({ + schema_version: z.literal("understudy.eval-build-state.v1"), + created_at: z.string().datetime(), + name: z.string().min(1).max(120), + identity: z.object({ + org_id: z.string(), + project_id: z.string(), + workload_id: z.string(), + workload_name: z.string(), + }), + compile: z.object({ + max_age_days: z.number().int().positive(), + batch_size: z.number().int().positive(), + }), + selection: z.object({ + last: z.string(), + limit: z.number().int().min(1).max(100), + seed: z.string(), + description: z.string().min(1).max(1000).nullable(), + requested_model: z.string().nullable(), + served_model: z.string().nullable(), + status_code: z.number().int().min(100).max(599).nullable(), + requires_tools: z.boolean(), + requires_structured_output: z.boolean(), + }), +}); + +export const FrozenCohortSchema = z.object({ + id: z.string(), + cohort_sha256: Sha256Schema, + capture_count: z.number().int().positive(), +}); + +export const EvalBuildCreatingStateSchema = EvalBuildStateBaseSchema.extend({ + status: z.literal("cohort_creating"), + create_request: z.object({ + operation_id: z.string().uuid(), + name: z.string().min(1).max(120), + selection: CohortSelectionSchema, + captures: z.array(z.object({ + capture_key: z.string(), + request_id: z.string(), + content_sha256: Sha256Schema, + })).min(1).max(500), + }), +}); + +export const EvalBuildFrozenStateSchema = EvalBuildStateBaseSchema.extend({ + status: z.enum(["cohort_frozen", "complete"]), + cohort: FrozenCohortSchema, +}); + +export const EvalBuildStateSchema = z.discriminatedUnion("status", [ + EvalBuildCreatingStateSchema, + EvalBuildFrozenStateSchema, +]); + +export type CatalogItem = z.infer; +export type CatalogResponse = z.infer; +export type Cohort = z.infer; +export type CohortExport = z.infer; +export type EvalBuildState = z.infer; +export type EvalBuildCreatingState = z.infer; +export type EvalBuildIdentity = EvalBuildState["identity"]; +export type EvalBuildSelection = EvalBuildState["selection"]; +export type FrozenCohort = z.infer; diff --git a/src/evals/materialize.ts b/src/evals/materialize.ts new file mode 100644 index 00000000..7b19c618 --- /dev/null +++ b/src/evals/materialize.ts @@ -0,0 +1,219 @@ +import { createHash } from "node:crypto"; +import { closeSync, openSync, renameSync, rmSync, writeSync } from "node:fs"; +import { join, resolve } from "node:path"; + +import { createPrivateDirectory, pathExists, writePrivateJson } from "./build-state.js"; +import type { CohortExport } from "./contracts.js"; + +export const EXPORT_EXPIRES_SECONDS = 3600; +const CAPTURE_DOWNLOAD_TIMEOUT_MS = 60_000; +const CAPTURE_DOWNLOAD_CONCURRENCY = 4; +export const MAX_CAPTURE_BYTES = 16 * 1024 * 1024; +export const MAX_COHORT_BYTES = 256 * 1024 * 1024; +const EXPORT_MIN_REMAINING_MS = 2 * 60_000; +const MAX_PORTABLE_FILE_NAME_BYTES = 240; +const CAPTURE_FILE_EXTENSION = ".jsonl"; +const WINDOWS_RESERVED_BASENAME = /^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/i; + +export async function downloadExport( + exportData: CohortExport, + workloadId: string, + out: string, + gatewayUrl: string, + refreshExport?: () => Promise, +) { + const outputDir = resolve(out); + if (pathExists(outputDir)) throw new Error(`Capture destination already exists: ${outputDir}. Choose a fresh directory.`); + const plans = localFilePlans(exportData); + createPrivateDirectory(outputDir); + const files: Array<{ request_id: string; path: string; content_sha256: string; size_bytes: number }> = []; + const aggregateBudget = { used: 0 }; + let currentExport = exportData; + try { + for (let offset = 0; offset < plans.length; offset += CAPTURE_DOWNLOAD_CONCURRENCY) { + currentExport = await freshExport(currentExport, refreshExport); + assertEquivalentExport(exportData, currentExport); + const batch = plans.slice(offset, offset + CAPTURE_DOWNLOAD_CONCURRENCY); + const settled = await Promise.allSettled(batch.map(async (plan, index) => { + const capture = currentExport.captures[offset + index]!; + const downloaded = await downloadCapture(capture, plan.fileName, outputDir, gatewayUrl, aggregateBudget); + return { request_id: capture.request_id, path: plan.fileName, content_sha256: downloaded.digest, size_bytes: downloaded.sizeBytes }; + })); + const failure = settled.find((result): result is PromiseRejectedResult => result.status === "rejected"); + if (failure) throw failure.reason; + files.push(...settled.map((result) => (result as PromiseFulfilledResult<(typeof files)[number]>).value)); + } + const totalBytes = files.reduce((sum, file) => sum + file.size_bytes, 0); + if (totalBytes !== aggregateBudget.used) throw new Error("Cohort download byte accounting does not match streamed payloads."); + const localManifest = join(outputDir, "cohort-manifest.json"); + writePrivateJson(localManifest, { + schema_version: "understudy.eval-cohort-materialization.v1", + cohort_id: exportData.cohort_id, + cohort_sha256: exportData.cohort_sha256, + workload_id: workloadId, + capture_count: files.length, + size_bytes: totalBytes, + privacy: { local_only: true, upload_performed: false }, + captures: files, + }); + return { output: outputDir, manifest: localManifest, count: files.length, cohort_sha256: exportData.cohort_sha256 }; + } catch (error) { + rmSync(outputDir, { recursive: true, force: true }); + throw error; + } +} + +function localFilePlans(exportData: CohortExport): Array<{ fileName: string }> { + const fileNameKeys = new Set(); + return exportData.captures.map((capture, index) => { + let fileName = portableCaptureFileName(capture.request_id); + if (fileNameKeys.has(portableFileNameKey(fileName))) { + fileName = portableCaptureFileName(capture.request_id, `-${capture.content_sha256.slice(0, 12)}`); + } + if (fileNameKeys.has(portableFileNameKey(fileName))) { + fileName = portableCaptureFileName(capture.request_id, `-${capture.content_sha256.slice(0, 12)}-${index}`); + } + const key = portableFileNameKey(fileName); + if (fileNameKeys.has(key)) throw new Error(`Capture ${capture.request_id} collides with another local filename.`); + fileNameKeys.add(key); + return { fileName }; + }); +} + +export function assertExportLineage(exportData: CohortExport, cohortId: string, cohortSha256: string): void { + if (exportData.cohort_id !== cohortId || exportData.cohort_sha256 !== cohortSha256) { + throw new Error(`Cohort export lineage does not match frozen cohort ${cohortId}.`); + } +} + +export function assertEquivalentExport(expected: CohortExport, candidate: CohortExport): void { + assertExportLineage(candidate, expected.cohort_id, expected.cohort_sha256); + if (candidate.captures.length !== expected.captures.length || candidate.captures.some((capture, index) => { + const original = expected.captures[index]!; + return capture.request_id !== original.request_id || capture.content_sha256 !== original.content_sha256; + })) { + throw new Error(`Refreshed export does not match frozen cohort ${expected.cohort_id}.`); + } +} + +async function freshExport(current: CohortExport, refresh?: () => Promise): Promise { + const expiresAt = Date.parse(current.expires_at); + if (expiresAt > Date.now() + EXPORT_MIN_REMAINING_MS) return current; + if (!refresh) throw new Error(`Cohort export ${current.export_id} expires too soon to download safely.`); + const refreshed = await refresh(); + assertEquivalentExport(current, refreshed); + if (Date.parse(refreshed.expires_at) <= Date.now() + EXPORT_MIN_REMAINING_MS) { + throw new Error(`Refreshed cohort export ${refreshed.export_id} expires too soon to download safely.`); + } + return refreshed; +} + +async function downloadCapture( + capture: CohortExport["captures"][number], + fileName: string, + outputDir: string, + gatewayUrl: string, + aggregateBudget: { used: number }, +): Promise<{ digest: string; sizeBytes: number }> { + const url = allowedCaptureUrl(capture.url, gatewayUrl); + const finalPath = join(outputDir, fileName); + const partialPath = `${finalPath}.partial`; + let descriptor: number | null = null; + let complete = false; + try { + const download = await fetch(url, { + headers: { Accept: "application/x-ndjson" }, + redirect: "error", + signal: AbortSignal.timeout(CAPTURE_DOWNLOAD_TIMEOUT_MS), + }); + if (!download.ok) throw new Error(`Capture ${capture.request_id} download failed with status ${download.status}.`); + const declaredLengthHeader = download.headers.get("content-length"); + const declaredLength = declaredLengthHeader === null ? null : Number(declaredLengthHeader); + if (declaredLength !== null && Number.isFinite(declaredLength) && declaredLength > MAX_CAPTURE_BYTES) { + throw new Error(`Capture ${capture.request_id} exceeds the ${MAX_CAPTURE_BYTES}-byte local download limit.`); + } + if (!download.body) throw new Error(`Capture ${capture.request_id} download returned no body.`); + descriptor = openSync(partialPath, "wx", 0o600); + const hash = createHash("sha256"); + const reader = download.body.getReader(); + let sizeBytes = 0; + while (true) { + const chunk = await reader.read(); + if (chunk.done) break; + try { + sizeBytes = reserveDownloadedChunk(capture.request_id, sizeBytes, chunk.value.byteLength, aggregateBudget); + } catch (error) { + await reader.cancel(); + throw error; + } + hash.update(chunk.value); + let written = 0; + while (written < chunk.value.byteLength) { + const count = writeSync(descriptor, chunk.value, written, chunk.value.byteLength - written); + if (count <= 0) throw new Error(`Capture ${capture.request_id} could not be written completely.`); + written += count; + } + } + closeSync(descriptor); + descriptor = null; + const digest = hash.digest("hex"); + if (digest !== capture.content_sha256) throw new Error(`Capture ${capture.request_id} failed SHA-256 verification.`); + renameSync(partialPath, finalPath); + complete = true; + return { digest, sizeBytes }; + } finally { + if (descriptor !== null) closeSync(descriptor); + if (!complete) rmSync(partialPath, { force: true }); + } +} + +export function reserveDownloadedChunk( + requestId: string, + captureBytes: number, + chunkBytes: number, + aggregateBudget: { used: number }, +): number { + if (captureBytes + chunkBytes > MAX_CAPTURE_BYTES) { + throw new Error(`Capture ${requestId} exceeds the ${MAX_CAPTURE_BYTES}-byte local download limit.`); + } + if (aggregateBudget.used + chunkBytes > MAX_COHORT_BYTES) { + throw new Error(`Cohort payloads exceed the ${MAX_COHORT_BYTES}-byte local download limit.`); + } + aggregateBudget.used += chunkBytes; + return captureBytes + chunkBytes; +} + +function allowedCaptureUrl(raw: string, gatewayUrl: string): string { + const url = new URL(raw); + if (url.username || url.password) throw new Error("Capture download URL must not contain credentials."); + const trustedR2 = url.protocol === "https:" && (url.port === "" || url.port === "443") && /^[a-z0-9-]+\.r2\.cloudflarestorage\.com$/i.test(url.hostname); + if (trustedR2) return url.toString(); + const gateway = new URL(gatewayUrl); + const loopback = ["127.0.0.1", "::1", "[::1]", "localhost"].includes(gateway.hostname); + if (loopback && url.origin === gateway.origin && ["http:", "https:"].includes(url.protocol)) return url.toString(); + throw new Error(`Refusing capture download from untrusted origin ${url.origin}.`); +} + +function portableCaptureFileName(requestId: string, suffix = ""): string { + const tail = `${suffix}${CAPTURE_FILE_EXTENSION}`; + const stemBytes = MAX_PORTABLE_FILE_NAME_BYTES - Buffer.byteLength(tail); + if (stemBytes < 1) throw new Error(`Capture filename suffix is too long for request ${requestId}.`); + return `${safeFileStem(requestId, stemBytes)}${tail}`; +} + +function portableFileNameKey(fileName: string): string { + return fileName.toLowerCase(); +} + +function safeFileStem(value: string, maxBytes: number): string { + let safe = value.replace(/[^a-zA-Z0-9._-]/g, "_").replace(/[. ]+$/g, ""); + if (!safe || safe === "." || safe === "..") safe = "request"; + if (WINDOWS_RESERVED_BASENAME.test(safe)) safe = `_${safe}`; + + // The replacement above guarantees ASCII, so code units and UTF-8 bytes + // have the same length. Trim again in case truncation lands on a dot. + safe = safe.slice(0, maxBytes).replace(/[. ]+$/g, ""); + if (!safe) safe = "request".slice(0, maxBytes); + if (WINDOWS_RESERVED_BASENAME.test(safe)) safe = `_${safe}`.slice(0, maxBytes); + return safe; +} diff --git a/src/internal/http.ts b/src/internal/http.ts index ea77f1ea..4f773adc 100644 --- a/src/internal/http.ts +++ b/src/internal/http.ts @@ -83,6 +83,8 @@ export interface RequestInput { headers?: Record; /** Anything JSON-serializable. Caller does not stringify. */ body?: unknown; + /** Optional caller-owned cancellation or timeout signal. */ + signal?: AbortSignal; /** Org id whose credential is used. Defaults to the only org if there's * exactly one in credentials; throws otherwise. */ orgId?: string; @@ -250,6 +252,7 @@ export async function request( method: input.method ?? "GET", headers, body, + signal: input.signal, }); const responseHeaders: Record = {}; diff --git a/src/trace-foundry.ts b/src/trace-foundry.ts index 90ce0729..0cd0cf59 100644 --- a/src/trace-foundry.ts +++ b/src/trace-foundry.ts @@ -9,9 +9,11 @@ import { buildRejectionGuidance, loadGuidanceFile } from "./rejection-guidance.j type J = null | boolean | number | string | J[] | { [key: string]: J }; type Obj = Record; +export const TRACE_FOUNDRY_DRAFT_STATUS = "machine_compiled_review_pending" as const; export type FoundryResult = { schema_version: typeof TRACE_FOUNDRY_SCHEMA; + status: typeof TRACE_FOUNDRY_DRAFT_STATUS; source: string; output_dir: string; freshness: { max_age_days: number; cutoff_utc: string; newest_capture_utc: string }; @@ -161,7 +163,12 @@ function parseTraceContext(envelope: Obj): Obj | null { } function normalize(envelope: Obj, pointer: string): Obj | null { - const version = Number(envelope.schema_version ?? 4); + // Hosted cohort exports use the public contract name while older local + // capture bundles carry the internal numeric capture version. They describe + // the same envelope fields consumed below. + const version = envelope.schema_version === "understudy.capture.v1" + ? 4 + : Number(envelope.schema_version ?? 4); if (![2, 3, 4].includes(version)) throw new Error(`Unsupported capture schema_version ${version}`); const requestRaw = envelope.customer_request_body ?? envelope.request_body ?? envelope.request; const upstreamRequestRaw = envelope.upstream_request_body ?? envelope.forwarded_request_body ?? null; @@ -193,7 +200,7 @@ function normalize(envelope: Obj, pointer: string): Obj | null { parent_event_id: envelope.parent_event_id ?? asObject(envelope.metadata).parent_event_id ?? null, }, }, - scope: { org_id: envelope.workos_org_id ?? envelope.org_id ?? null, project_id: envelope.project_id ?? null, workload_id: envelope.workload_id ?? null, workload_name: envelope.workload_name ?? null }, + scope: { org_id: envelope.workos_org_id ?? envelope.org_id ?? null, project_id: envelope.project_id ?? null, workload_id: envelope.workload_id ?? envelope.placement_id ?? null, workload_name: envelope.workload_name ?? null }, routing: { provider: envelope.provider ?? null, requested_model: envelope.requested_model ?? request.model ?? null, upstream_model: envelope.upstream_model ?? null }, transport: { endpoint: envelope.endpoint ?? null, status_code: envelope.status_code ?? null, latency_ms: envelope.latency_ms ?? null }, request: { system: request.system ?? null, messages, tools, settings: Object.fromEntries(Object.entries(request).filter(([k]) => !["system", "messages", "tools"].includes(k))) }, @@ -606,7 +613,34 @@ function readJsonl(path: string): Obj[] { return readJsonlFile(path).items.map(asObject); } function appendJsonl(path: string, rows: Obj[]): void { if (rows.length === 0) return; mkdirSync(resolve(path, ".."), { recursive: true }); appendFileSync(path, rows.map((row) => JSON.stringify(row)).join("\n") + "\n", { mode: 0o600 }); } -const pyName = (value: string): string => { const clean = value.replace(/[^A-Za-z0-9_]/g, "_"); return /^[A-Za-z_]/.test(clean) ? clean : `tool_${clean}`; }; +const PYTHON_KEYWORDS = new Set([ + "False", "None", "True", "and", "as", "assert", "async", "await", "break", "case", "class", "continue", + "def", "del", "elif", "else", "except", "finally", "for", "from", "global", "if", "import", "in", "is", + "lambda", "match", "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", "with", "yield", +]); + +function pythonIdentifier(value: string, fallback: string): string { + let clean = value.replace(/[^A-Za-z0-9_]/g, "_"); + if (!clean) clean = fallback; + if (!/^[A-Za-z_]/.test(clean)) clean = `${fallback}_${clean}`; + if (PYTHON_KEYWORDS.has(clean) || clean === "self") clean = `${clean}_`; + return clean; +} + +function allocatePythonIdentifiers(values: string[], options: { prefix?: string; reserved?: string[] } = {}): Map { + const used = new Set(options.reserved ?? []); + const allocated = new Map(); + for (const value of values) { + const rawBase = pythonIdentifier(value, options.prefix ?? "value"); + const base = options.prefix ? `${options.prefix}_${rawBase}` : rawBase; + let candidate = base; + let suffix = 2; + while (used.has(candidate) || PYTHON_KEYWORDS.has(candidate)) candidate = `${base}_${suffix++}`; + used.add(candidate); + allocated.set(value, candidate); + } + return allocated; +} /** * Semantic-outcome matching (the contract's advertised @@ -1555,17 +1589,13 @@ export function auditGoldLeakage(tasks: Obj[], taskRows: Obj[], fixtures: Obj[], } /** Build-time report: the audit is advisory, so it prints and moves on. */ -function printLeakageAudit(audit: LeakageAudit): void { +export function printLeakageAudit(audit: LeakageAudit): void { if (audit.status === "clean") { console.error(`[leakage-audit] clean — no verbatim or fuzzy contract targets found in candidate-readable surfaces (${audit.checked_tasks} task(s) checked)`); return; } const tierSummary = `${audit.tier_counts.verbatim} verbatim, ${audit.tier_counts.fuzzy} fuzzy (advisory)${audit.tier_counts.semantic > 0 ? `, ${audit.tier_counts.semantic} semantic` : ""}`; - console.error(`[leakage-audit] ${audit.findings.length} potential gold-leakage finding(s) [${tierSummary}] across ${audit.checked_tasks} task(s) — recorded in manifest.leakage_audit (report-only, nothing redacted):`); - for (const finding of audit.findings.slice(0, 8)) { - console.error(`[leakage-audit] ${finding.task_id} · ${finding.tier} (${finding.similarity}) · ${finding.kind} · ${finding.location} · "${finding.excerpt.slice(0, 80)}"`); - } - if (audit.findings.length > 8) console.error(`[leakage-audit] … ${audit.findings.length - 8} more (see manifest.json)`); + console.error(`[leakage-audit] ${audit.findings.length} potential gold-leakage finding(s) [${tierSummary}] across ${audit.checked_tasks} task(s) — details recorded only in the private manifest (report-only, nothing redacted)`); } /** @@ -1699,13 +1729,18 @@ export function writeVerifiersEnvironment(output: string, tasks: Obj[], sourceCo writeJson(join(servers, "guidance.json"), rejectionGuidance); const pyType = (value: unknown): string => typeof value === "boolean" ? "bool" : typeof value === "number" ? "float" : Array.isArray(value) ? "list" : value && typeof value === "object" ? "dict" : "str"; const schemaType = (schema: Obj, fallback: unknown): string => schema.type === "boolean" ? "bool" : ["number", "integer"].includes(schema.type) ? "float" : schema.type === "array" ? "list" : schema.type === "object" ? "dict" : schema.type === "string" ? "str" : pyType(fallback); + const methodNames = allocatePythonIdentifiers(toolNames, { + prefix: "tool", + reserved: ["setup_task", "_fixture_reply", "_accept", "run"], + }); const methods = toolNames.map((name) => { const observed = observedByTool.get(name) ?? {}, properties = asObject(schemaByTool.get(name)?.properties); const keys = [...new Set([...Object.keys(properties), ...Object.keys(observed)])]; - const parameters = keys.map((key) => `${pyName(key)}: ${schemaType(asObject(properties[key]), observed[key])} | None = None`).join(", "); - const args = keys.map((key) => `${JSON.stringify(key)}: ${pyName(key)}`).join(", "); + const parameterNames = allocatePythonIdentifiers(keys, { reserved: ["self"] }); + const parameters = keys.map((key) => `${parameterNames.get(key)!}: ${schemaType(asObject(properties[key]), observed[key])} | None = None`).join(", "); + const args = keys.map((key) => `${JSON.stringify(key)}: ${parameterNames.get(key)!}`).join(", "); const mutating = mutationPrefixes.some((prefix) => name.toLowerCase().startsWith(prefix)); - return ` @vf.tool(name=${JSON.stringify(name)})\n async def ${pyName(name)}(self${parameters ? `, ${parameters}` : ""}) -> str:\n \"\"\"Execute the trace-derived ${name} transition against per-rollout state.\"\"\"\n return self._accept({\"tool\": ${JSON.stringify(name)}, \"arguments\": {${args}}}, ${mutating ? "True" : "False"})`; + return ` @vf.tool(name=${JSON.stringify(name)})\n async def ${methodNames.get(name)!}(self${parameters ? `, ${parameters}` : ""}) -> str:\n \"\"\"Execute one trace-derived transition against per-rollout state.\"\"\"\n return self._accept({\"tool\": ${JSON.stringify(name)}, \"arguments\": {${args}}}, ${mutating ? "True" : "False"})`; }).join("\n\n"); // Candidate-readable vs scorer-only split (fixtures-state-split): fixtures // carry ONLY what the incumbent observed BEFORE its first gold write — @@ -2243,7 +2278,7 @@ function writeFoundryArtifacts(ctx: { source: string; output: string; files: str // "understudy.benchmark.v1" is reserved for the executable manifest written // by `traces promote` after human review (this resolves the known // foundry-vs-hub schema-name collision). Same content, honest name. - const benchmark = benchmarkManifestFrom(tasks, { schemaVersion: "understudy.benchmark_proposal.v1", benchmarkId: `trace-${hash({ source, workload: options.workload ?? null }).slice(0, 16)}`, name: options.workload ? `${options.workload} trace benchmark` : "Trace-derived benchmark", description: "Machine-compiled from a source-history DAG with human final judgment.", createdAt: now.toISOString(), sourceRefs: [relative(output, join(output, "capture-ledger.jsonl")), relative(output, join(output, "source-dag.json"))], packageSha256: environment.package_sha256, auditedCommit: environment.audited_commit, heldoutNovel, status: "machine_compiled_review_pending", executable: false, promotionBlockers }); + const benchmark = benchmarkManifestFrom(tasks, { schemaVersion: "understudy.benchmark_proposal.v1", benchmarkId: `trace-${hash({ source, workload: options.workload ?? null }).slice(0, 16)}`, name: options.workload ? `${options.workload} trace benchmark` : "Trace-derived benchmark", description: "Machine-compiled from a source-history DAG with human final judgment.", createdAt: now.toISOString(), sourceRefs: [relative(output, join(output, "capture-ledger.jsonl")), relative(output, join(output, "source-dag.json"))], packageSha256: environment.package_sha256, auditedCommit: environment.audited_commit, heldoutNovel, status: TRACE_FOUNDRY_DRAFT_STATUS, executable: false, promotionBlockers }); const manifestErrors = validateBenchmarkManifest({ ...benchmark, schema_version: "understudy.benchmark.v1" }); if (manifestErrors.length > 0) throw new Error(`Generated benchmark manifest is invalid: ${manifestErrors.join("; ")}`); writeJson(join(output, "benchmark.json"), benchmark); @@ -2256,7 +2291,7 @@ function writeFoundryArtifacts(ctx: { source: string; output: string; files: str finalGoal.updated_at = now.toISOString(); writeJson(join(output, "goal-state.json"), finalGoal); appendJsonl(join(output, "goal-events.jsonl"), [{ at: now.toISOString(), action: "finalize", input_hash: finalGoal.input_hash, validation: { dag_valid: dag.valid, oracle_pass: environment.oracle_pass, sentinel_pass: environment.sentinel_pass }, next_action: finalGoal.next_action }]); - const result: FoundryResult = { schema_version: TRACE_FOUNDRY_SCHEMA, source, output_dir: output, freshness: { max_age_days: ctx.maxAgeDays, cutoff_utc: cutoff.toISOString(), newest_capture_utc: rows.map((row) => row.captured_at).sort().at(-1) }, counts: { source_files: files.length, captures: rows.length, tasks: tasks.length, edges: dag.edges.length, stale_filtered: staleFiltered, invalid_timestamp_filtered: notNormalizableFiltered, not_normalizable_filtered: notNormalizableFiltered, filtered_reasons: filteredReasons }, artifacts: { normalized: "normalized-captures.jsonl", dag: "source-dag.json", tasks: "tasks.jsonl", benchmark: "benchmark.json", environment: toPortablePath(output, environment.path), ledger: "capture-ledger.jsonl", goal: "goal-state.json", viewer: toPortablePath(output, join(viewer, "index.html")) }, privacy: { local_only: true, contains_customer_payloads: true, upload_performed: false, provider_called: false }, source_scope: ctx.sourceScope, quarantines: dag.quarantines ?? [], self_check: selfCheck, leakage_audit: environment.leakage_audit, fixtures_split: environment.fixtures_split, versioning: { bumps: versionBumps } }; + const result: FoundryResult = { schema_version: TRACE_FOUNDRY_SCHEMA, status: TRACE_FOUNDRY_DRAFT_STATUS, source, output_dir: output, freshness: { max_age_days: ctx.maxAgeDays, cutoff_utc: cutoff.toISOString(), newest_capture_utc: rows.map((row) => row.captured_at).sort().at(-1) }, counts: { source_files: files.length, captures: rows.length, tasks: tasks.length, edges: dag.edges.length, stale_filtered: staleFiltered, invalid_timestamp_filtered: notNormalizableFiltered, not_normalizable_filtered: notNormalizableFiltered, filtered_reasons: filteredReasons }, artifacts: { normalized: "normalized-captures.jsonl", dag: "source-dag.json", tasks: "tasks.jsonl", benchmark: "benchmark.json", environment: toPortablePath(output, environment.path), ledger: "capture-ledger.jsonl", goal: "goal-state.json", viewer: toPortablePath(output, join(viewer, "index.html")) }, privacy: { local_only: true, contains_customer_payloads: true, upload_performed: false, provider_called: false }, source_scope: ctx.sourceScope, quarantines: dag.quarantines ?? [], self_check: selfCheck, leakage_audit: environment.leakage_audit, fixtures_split: environment.fixtures_split, versioning: { bumps: versionBumps } }; writeJson(join(output, "manifest.json"), result); return result; } diff --git a/tests/cli.test.mjs b/tests/cli.test.mjs index 2b7bc079..1313a0a7 100644 --- a/tests/cli.test.mjs +++ b/tests/cli.test.mjs @@ -1,10 +1,10 @@ import assert from "node:assert/strict"; import { spawn, spawnSync } from "node:child_process"; import { createHash } from "node:crypto"; -import { existsSync, mkdtempSync, mkdirSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { existsSync, mkdtempSync, mkdirSync, readFileSync, readdirSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs"; import { createServer } from "node:http"; import { tmpdir } from "node:os"; -import { join, resolve } from "node:path"; +import { dirname, join, resolve } from "node:path"; import { describe, it } from "node:test"; const cli = ["node", resolve("dist/bin.js")]; @@ -171,6 +171,14 @@ async function withHostedFixture(fn) { transientCaptureFailures: new Map([["req_retry", 1]]), unavailableCaptureIds: new Set(), captureAuthorizationFailure: false, + evalCaptureFailures: 0, + evalCaptureDelayMs: 0, + evalCohortDropResponses: 0, + evalCohorts: [], + evalCaptureUrl: null, + evalCaptureDeclaredLength: null, + evalExportCohortSha: "a".repeat(64), + evalExportExpiries: [], }; const server = createServer(async (req, res) => { @@ -540,29 +548,56 @@ async function withHostedFixture(fn) { }, }); } + if (req.method === "GET" && url.pathname === `${evalBase}/eval-cohorts`) { + return send(200, { eval_cohorts: state.evalCohorts }); + } if (req.method === "POST" && url.pathname === `${evalBase}/eval-cohorts`) { - return send(201, { + const existing = body.operation_id + ? state.evalCohorts.find((entry) => entry.operation_id === body.operation_id) + : null; + if (existing) return send(200, existing); + const cohort = { id: "evc_123", org_id: "org_1", project_id: "proj_1", workload_id: "usp_classify", + operation_id: body.operation_id ?? null, name: body.name, selection: body.selection, capture_count: body.captures.length, cohort_sha256: "a".repeat(64), created_at: "2026-06-07T01:00:00Z", - }); + }; + state.evalCohorts.push(cohort); + if (state.evalCohortDropResponses > 0) { + state.evalCohortDropResponses -= 1; + return res.destroy(); + } + return send(201, cohort); } if (req.method === "POST" && url.pathname === `${evalBase}/eval-cohorts/evc_123/export`) { return send(201, { export_id: "eve_123", cohort_id: "evc_123", - cohort_sha256: "a".repeat(64), - expires_at: "2026-06-07T02:00:00Z", - captures: [{ request_id: "req_123", content_sha256: rawCaptureSha, url: `${gatewayUrl}/eval-capture-req_123` }], + cohort_sha256: state.evalExportCohortSha, + expires_at: state.evalExportExpiries.shift() ?? new Date(Date.now() + 60 * 60 * 1000).toISOString(), + captures: [{ request_id: "req_123", content_sha256: rawCaptureSha, url: state.evalCaptureUrl ?? `${gatewayUrl}/eval-capture-req_123` }], }); } - if (req.method === "GET" && url.pathname === "/eval-capture-req_123") return sendBytes(200, rawCapture); + if (req.method === "GET" && url.pathname === "/eval-capture-req_123") { + if (state.evalCaptureDelayMs > 0) { + await new Promise((resolve) => setTimeout(resolve, state.evalCaptureDelayMs)); + } + if (state.evalCaptureFailures > 0) { + state.evalCaptureFailures -= 1; + return send(503, { message: "synthetic eval capture failure" }); + } + return sendBytes(200, rawCapture, state.evalCaptureDeclaredLength === null ? {} : { "content-length": String(state.evalCaptureDeclaredLength) }); + } + if (req.method === "GET" && url.pathname === "/eval-capture-redirect") { + res.writeHead(302, { location: `${gatewayUrl}/eval-capture-req_123` }); + return res.end(); + } if (req.method === "POST" && (url.pathname === "/v1/messages" || url.pathname === "/v1/chat/completions")) { if (body?.model === "fixture-error") { return send(503, { error: { type: "upstream_capacity", code: "deployment_unavailable", message: "SECRET_PROVIDER_DETAIL" } }, { @@ -4067,6 +4102,7 @@ class ScoreWithFeedback: assert.equal(JSON.parse(create.stdout).cohort.id, "evc_123"); const createRequest = requests.find((entry) => entry.path.endsWith("/eval-cohorts") && entry.method === "POST"); assert.equal(createRequest.body.captures[0].request_id, "req_123"); + assert.equal(createRequest.body.selection.sampling_seed, "cedar-july"); const blocked = await runWithEnvAsync([ "evals", "cohort", "export", "evc_123", "--project", "rehearsal", "--workload", "classify", @@ -4118,6 +4154,311 @@ class ScoreWithFeedback: }); }); + it("builds a private local eval project from a frozen workload cohort", async () => { + await withHostedFixture(async ({ home, repo, requests, state }) => { + const env = { HOME: home, USERPROFILE: home }; + const outputDir = join(repo, ".understudy", "evals", "local-builder"); + + const blocked = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--name", "local-builder", "--out", outputDir, "--max-age-days", "365", + ], env, repo); + assert.notEqual(blocked.status, 0); + assert.match(blocked.stderr, /JSON mode cannot prompt/); + assert.equal(existsSync(join(outputDir, "eval-project.json")), false); + assert.equal(requests.length, 0, "approval failure must happen before hosted reads"); + + const nonInteractive = await runWithEnvAsync([ + "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--name", "local-builder", "--out", outputDir, "--max-age-days", "365", + ], env, repo); + assert.notEqual(nonInteractive.status, 0); + assert.match(nonInteractive.stderr, /Non-interactive eval builds cannot prompt/); + assert.equal(requests.length, 0, "piped builds must fail before hosted reads"); + + const invalid = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--name", "local-builder", "--out", outputDir, "--max-age-days", "365", + "--batch-size", "0", "--yes", + ], env, repo); + assert.notEqual(invalid.status, 0); + assert.match(invalid.stderr, /--batch-size must be a positive integer/); + assert.equal(requests.length, 0, "numeric validation must happen before hosted reads"); + + const tooNarrow = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "local-builder", "--out", outputDir, + "--max-age-days", "7", "--yes", + ], env, repo); + assert.notEqual(tooNarrow.status, 0); + assert.match(tooNarrow.stderr, /--max-age-days must cover --last/); + assert.equal(requests.length, 0, "freshness validation must happen before hosted reads"); + + const resumableDir = join(repo, ".understudy", "evals", "resumable-builder"); + state.evalCaptureFailures = 1; + const failedAttempt = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "resumable-builder", "--out", resumableDir, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(failedAttempt.status, 0); + assert.equal(existsSync(resumableDir), false, "failed attempts never publish the final project"); + const resumableStateDir = join(dirname(resumableDir), ".resumable-builder.eval-build"); + assert.equal(existsSync(join(resumableStateDir, "build-state.json")), true); + assert.deepEqual(readdirSync(join(resumableStateDir, "attempts")), [], "failed attempts retain no payload-bearing partial data"); + + const changedSelection = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "resumable-builder", "--out", resumableDir, + "--requires-tools", "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(changedSelection.status, 0); + assert.match(changedSelection.stderr, /does not match the capture selection options/); + + state.workloads.find((workload) => workload.id === "usp_classify").name = "classify-renamed"; + + const resumed = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "usp_classify", + "--last", "31d", "--name", "resumable-builder", "--out", resumableDir, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.equal(resumed.status, 0, resumed.stderr); + assert.equal( + requests.filter((entry) => entry.path.endsWith("/eval-cohorts") && entry.method === "POST").length, + 1, + "resume must reuse the already frozen cohort", + ); + assert.equal(existsSync(join(resumableDir, "eval-project.json")), true); + assert.equal( + JSON.parse(readFileSync(join(resumableDir, "eval-project.json"), "utf8")).identity.workload_name, + "classify", + "resume preserves the freeze-time display name while matching stable ids", + ); + state.workloads.find((workload) => workload.id === "usp_classify").name = "classify"; + + const lostResponseDir = join(repo, ".understudy", "evals", "lost-response-builder"); + const postsBeforeLostResponse = requests.filter((entry) => entry.path.endsWith("/eval-cohorts") && entry.method === "POST").length; + const cohortsBeforeLostResponse = state.evalCohorts.length; + state.evalCohortDropResponses = 1; + const recovered = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "lost-response-builder", "--out", lostResponseDir, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.equal(recovered.status, 0, recovered.stderr); + assert.equal( + requests.filter((entry) => entry.path.endsWith("/eval-cohorts") && entry.method === "POST").length - postsBeforeLostResponse, + 2, + "a lost create response is recovered by retrying the same idempotent operation", + ); + assert.equal(state.evalCohorts.length - cohortsBeforeLostResponse, 1, "idempotent retries freeze only one cohort"); + assert.equal(existsSync(join(lostResponseDir, "eval-project.json")), true); + + const built = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "local-builder", "--out", outputDir, + "--max-age-days", "365", "--batch-size", "2", "--yes", + ], env, repo); + assert.equal(built.status, 0, built.stderr); + assert.doesNotMatch(built.stdout + built.stderr, /SECRET_PROMPT|SECRET_COMPLETION/); + + const payload = JSON.parse(built.stdout); + assert.equal(payload.status, "local_draft"); + assert.equal(payload.privacy.upload_performed, false); + assert.equal(payload.privacy.provider_called, false); + + const capturesDir = join(outputDir, "captures"); + const benchmarkDir = join(outputDir, "benchmark"); + assert.match(readFileSync(join(capturesDir, "req_123.jsonl"), "utf8"), /SECRET_PROMPT/); + assert.equal(existsSync(join(benchmarkDir, "manifest.json")), true); + assert.equal(existsSync(join(benchmarkDir, "viewer", "index.html")), true); + + const project = JSON.parse(readFileSync(join(outputDir, "eval-project.json"), "utf8")); + assert.equal(project.schema_version, "understudy.eval-project.v1"); + assert.equal(project.status, "local_draft"); + assert.deepEqual(project.identity, { + org_id: "org_1", + project_id: "proj_1", + workload_id: "usp_classify", + workload_name: "classify", + }); + assert.equal(project.cohort.id, "evc_123"); + assert.equal(project.cohort.cohort_sha256, "a".repeat(64)); + assert.equal(project.cohort.materialization_manifest, "captures/cohort-manifest.json"); + assert.equal(project.foundry.status, "machine_compiled_review_pending"); + assert.equal(project.foundry.manifest, "benchmark/manifest.json"); + assert.equal(project.foundry.artifacts.viewer, "benchmark/viewer/index.html"); + assert.deepEqual(project.privacy, { + local_only: true, + contains_customer_payloads: true, + upload_performed: false, + provider_called: false, + }); + assert.doesNotMatch(JSON.stringify(project), /SECRET_PROMPT|SECRET_COMPLETION|https?:\/\//); + if (process.platform !== "win32") { + assert.equal(statSync(join(capturesDir, "req_123.jsonl")).mode & 0o077, 0); + assert.equal(statSync(join(outputDir, "eval-project.json")).mode & 0o077, 0); + } + + const hostedPaths = requests.map((entry) => entry.path); + assert.ok(hostedPaths.some((path) => path.endsWith("/eval-capture-catalog"))); + assert.ok(hostedPaths.some((path) => path.endsWith("/eval-cohorts"))); + assert.ok(hostedPaths.some((path) => path.endsWith("/eval-cohorts/evc_123/export"))); + + const requestCount = requests.length; + const repeated = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "local-builder", "--out", outputDir, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(repeated.status, 0); + assert.match(repeated.stderr, /destination already exists/); + assert.equal(requests.length, requestCount, "destination reuse must fail before hosted reads"); + }); + }); + + it("publishes a validated checkpoint and excludes a concurrent builder from the same output", async () => { + await withHostedFixture(async ({ home, repo, requests, state }) => { + const env = { HOME: home, USERPROFILE: home }; + const invalidOutput = join(repo, ".understudy", "evals", "invalid-checkpoint"); + const invalidStaging = join(dirname(invalidOutput), ".invalid-checkpoint.eval-build"); + const invalid = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "invalid-checkpoint", "--out", invalidOutput, + "--description", "x".repeat(1001), "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(invalid.status, 0); + assert.equal(existsSync(invalidStaging), false, "invalid state is never published as a resumable checkpoint"); + + const corrected = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "invalid-checkpoint", "--out", invalidOutput, + "--description", "corrected", "--max-age-days", "365", "--yes", + ], env, repo); + assert.equal(corrected.status, 0, corrected.stderr); + + const staleOutput = join(repo, ".understudy", "evals", "stale-builder"); + const staleLock = join(dirname(staleOutput), ".stale-builder.eval-build.lock"); + mkdirSync(staleLock, { recursive: true }); + writeFileSync(join(staleLock, "owner.json"), JSON.stringify({ + token: "stale-owner", + pid: 2_147_483_647, + created_at: "2026-01-01T00:00:00.000Z", + })); + const requestsBeforeStale = requests.length; + const stale = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "stale-builder", "--out", staleOutput, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(stale.status, 0); + assert.match(stale.stderr, /stale eval build lock remains/); + assert.equal(requests.length, requestsBeforeStale, "stale locks fail before hosted reads"); + rmSync(staleLock, { recursive: true, force: true }); + + const concurrentOutput = join(repo, ".understudy", "evals", "concurrent-builder"); + state.evalCaptureDelayMs = 400; + const captureReadsBefore = requests.filter((entry) => entry.path === "/eval-capture-req_123").length; + const first = runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "concurrent-builder", "--out", concurrentOutput, + "--max-age-days", "365", "--yes", + ], env, repo); + const deadline = Date.now() + 5_000; + while (requests.filter((entry) => entry.path === "/eval-capture-req_123").length === captureReadsBefore) { + if (Date.now() > deadline) throw new Error("first eval builder did not reach capture download"); + await new Promise((resolve) => setTimeout(resolve, 10)); + } + const second = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "concurrent-builder", "--out", concurrentOutput, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(second.status, 0); + assert.match(second.stderr, /already owns/); + const completed = await first; + state.evalCaptureDelayMs = 0; + assert.equal(completed.status, 0, completed.stderr); + assert.equal(existsSync(join(concurrentOutput, "eval-project.json")), true); + }); + }); + + it("rejects unsafe eval exports and never leaves partial payloads", async () => { + await withHostedFixture(async ({ home, repo, requests, state, gatewayUrl }) => { + const env = { HOME: home, USERPROFILE: home }; + const exportArgs = (out) => [ + "--json", "evals", "cohort", "export", "evc_123", "--project", "rehearsal", "--workload", "classify", + "--out", out, "--yes", + ]; + + state.evalCaptureUrl = "http://169.254.169.254/latest/meta-data"; + const untrustedOut = join(repo, ".understudy", "evals", "untrusted-export"); + const untrusted = await runWithEnvAsync(exportArgs(untrustedOut), env, repo); + assert.notEqual(untrusted.status, 0); + assert.match(untrusted.stderr, /untrusted origin/); + assert.equal(existsSync(untrustedOut), false); + + state.evalCaptureUrl = `${gatewayUrl}/eval-capture-redirect`; + const redirectOut = join(repo, ".understudy", "evals", "redirect-export"); + const redirected = await runWithEnvAsync(exportArgs(redirectOut), env, repo); + assert.notEqual(redirected.status, 0); + assert.equal(existsSync(redirectOut), false, "redirect failures retain no partial files"); + + state.evalCaptureUrl = null; + state.evalCaptureDeclaredLength = 16 * 1024 * 1024 + 1; + const oversizedOut = join(repo, ".understudy", "evals", "oversized-export"); + const oversized = await runWithEnvAsync(exportArgs(oversizedOut), env, repo); + assert.notEqual(oversized.status, 0); + assert.match(oversized.stderr, /local download limit/); + assert.equal(existsSync(oversizedOut), false); + + state.evalCaptureDeclaredLength = null; + state.evalExportExpiries.push( + new Date(Date.now() + 30_000).toISOString(), + new Date(Date.now() + 60 * 60 * 1000).toISOString(), + ); + const refreshedOut = join(repo, ".understudy", "evals", "refreshed-export"); + const exportsBeforeRefresh = requests.filter((entry) => entry.path.endsWith("/export") && entry.method === "POST").length; + const refreshed = await runWithEnvAsync(exportArgs(refreshedOut), env, repo); + assert.equal(refreshed.status, 0, refreshed.stderr); + assert.equal( + requests.filter((entry) => entry.path.endsWith("/export") && entry.method === "POST").length - exportsBeforeRefresh, + 2, + "an export near expiry is refreshed for the same frozen cohort before download", + ); + + state.evalExportCohortSha = "b".repeat(64); + const mismatchedOut = join(repo, ".understudy", "evals", "lineage-mismatch"); + const captureReadsBefore = requests.filter((entry) => entry.path === "/eval-capture-req_123").length; + const mismatched = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "lineage-mismatch", "--out", mismatchedOut, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(mismatched.status, 0); + assert.match(mismatched.stderr, /lineage does not match frozen cohort/); + assert.equal(requests.filter((entry) => entry.path === "/eval-capture-req_123").length, captureReadsBefore, "lineage is rejected before payload fetch"); + assert.equal(existsSync(mismatchedOut), false); + }); + }); + + it("does not publish a destination when local eval compilation fails", async () => { + await withHostedFixture(async ({ home, repo, state }) => { + const env = { HOME: home, USERPROFILE: home }; + state.captures[0].workload_id = "usp_other"; + const outputDir = join(repo, ".understudy", "evals", "compile-failure"); + const failed = await runWithEnvAsync([ + "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", + "--last", "31d", "--name", "compile-failure", "--out", outputDir, + "--max-age-days", "365", "--yes", + ], env, repo); + assert.notEqual(failed.status, 0); + assert.equal(existsSync(outputDir), false); + const staging = join(dirname(outputDir), ".compile-failure.eval-build"); + assert.deepEqual(readdirSync(join(staging, "attempts")), [], "compiler failures keep only the redacted cohort checkpoint"); + }); + }); + it("runs gateway health and probes without printing secrets", async () => { await withHostedFixture(async ({ home, repo, gatewayUrl, requests }) => { const env = { HOME: home, USERPROFILE: home, UPSTREAM_TEST_KEY: "provider_secret_value" }; diff --git a/tests/eval-build-state.test.mjs b/tests/eval-build-state.test.mjs new file mode 100644 index 00000000..607367f2 --- /dev/null +++ b/tests/eval-build-state.test.mjs @@ -0,0 +1,60 @@ +import assert from "node:assert/strict"; +import childProcess from "node:child_process"; +import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { syncBuiltinESMExports } from "node:module"; +import { tmpdir } from "node:os"; +import { basename, dirname, join } from "node:path"; +import { test } from "node:test"; + +test("a recycled live pid is stale only when its process instance can be distinguished", async (t) => { + const root = mkdtempSync(join(tmpdir(), "understudy-eval-build-lease-")); + const originalSpawnSync = childProcess.spawnSync; + try { + if (process.platform === "darwin") { + childProcess.spawnSync = (command, args, options) => { + if (command === "/bin/ps") { + return { status: 0, stdout: "Sat Aug 29 20:30:00 2026\n", stderr: "" }; + } + return originalSpawnSync(command, args, options); + }; + syncBuiltinESMExports(); + } + const { acquireEvalBuildLease } = await import(`../dist/evals/build-state.js?pid-reuse=${Date.now()}`); + const probeOutput = join(root, "probe"); + const releaseProbe = acquireEvalBuildLease(probeOutput); + const probeLease = join(dirname(probeOutput), `.${basename(probeOutput)}.eval-build.lock`); + const probeOwner = JSON.parse(readFileSync(join(probeLease, "owner.json"), "utf8")); + releaseProbe(); + + if (typeof probeOwner.process_instance_id !== "string") { + t.skip("process-instance identity is unavailable on this platform"); + return; + } + + const output = join(root, "recycled-pid"); + const lease = join(dirname(output), `.${basename(output)}.eval-build.lock`); + const owner = { + token: "previous-builder", + pid: process.pid, + process_instance_id: `${probeOwner.process_instance_id}-different-start`, + created_at: "2026-01-01T00:00:00.000Z", + }; + mkdirSync(lease, { mode: 0o700 }); + writeFileSync(join(lease, "owner.json"), JSON.stringify(owner), { mode: 0o600 }); + + assert.throws( + () => acquireEvalBuildLease(output), + /stale eval build lock remains/, + ); + assert.equal(existsSync(lease), true, "stale detection must not delete the lock automatically"); + assert.deepEqual( + JSON.parse(readFileSync(join(lease, "owner.json"), "utf8")), + owner, + "a rejected builder must not replace another owner's lock metadata", + ); + } finally { + childProcess.spawnSync = originalSpawnSync; + syncBuiltinESMExports(); + rmSync(root, { recursive: true, force: true }); + } +}); diff --git a/tests/eval-materialize.test.mjs b/tests/eval-materialize.test.mjs new file mode 100644 index 00000000..f4adf681 --- /dev/null +++ b/tests/eval-materialize.test.mjs @@ -0,0 +1,91 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, it } from "node:test"; + +import { + downloadExport, + MAX_CAPTURE_BYTES, + MAX_COHORT_BYTES, + reserveDownloadedChunk, +} from "../dist/evals/materialize.js"; + +describe("eval materialization byte budgets", () => { + it("accepts exact limits and rejects the next byte before reserving it", () => { + const exactCapture = { used: MAX_COHORT_BYTES - MAX_CAPTURE_BYTES }; + assert.equal( + reserveDownloadedChunk("req_exact", 0, MAX_CAPTURE_BYTES, exactCapture), + MAX_CAPTURE_BYTES, + ); + assert.equal(exactCapture.used, MAX_COHORT_BYTES); + + const captureOverflow = { used: 0 }; + assert.throws( + () => reserveDownloadedChunk("req_capture", MAX_CAPTURE_BYTES, 1, captureOverflow), + /Capture req_capture exceeds/, + ); + assert.equal(captureOverflow.used, 0); + + const cohortOverflow = { used: MAX_COHORT_BYTES }; + assert.throws( + () => reserveDownloadedChunk("req_cohort", 0, 1, cohortOverflow), + /Cohort payloads exceed/, + ); + assert.equal(cohortOverflow.used, MAX_COHORT_BYTES); + }); +}); + +describe("eval materialization filenames", () => { + it("bounds long request IDs and avoids Windows device basenames", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-eval-materialize-")); + const asciiRequestId = "a".repeat(400); + const unicodeRequestId = `${"a".repeat(400)}${"界".repeat(200)}`; + const bodies = new Map([ + [asciiRequestId, '{"capture":"ascii"}\n'], + [unicodeRequestId, '{"capture":"unicode"}\n'], + ["CON", '{"capture":"reserved"}\n'], + ]); + const captures = [...bodies].map(([request_id, body], index) => ({ + request_id, + content_sha256: createHash("sha256").update(body).digest("hex"), + url: `http://localhost:8787/captures/${index}`, + })); + const exportData = { + export_id: "export_portable_names", + cohort_id: "cohort_portable_names", + cohort_sha256: "f".repeat(64), + expires_at: new Date(Date.now() + 10 * 60_000).toISOString(), + captures, + }; + const originalFetch = globalThis.fetch; + globalThis.fetch = async (url) => { + const capture = captures[Number(new URL(url).pathname.split("/").at(-1))]; + return new Response(bodies.get(capture.request_id), { + headers: { "content-length": String(Buffer.byteLength(bodies.get(capture.request_id))) }, + }); + }; + + try { + const first = join(root, "first"); + const second = join(root, "second"); + await downloadExport(exportData, "workload_portable_names", first, "http://localhost:8787"); + await downloadExport(exportData, "workload_portable_names", second, "http://localhost:8787"); + + const firstPaths = JSON.parse(readFileSync(join(first, "cohort-manifest.json"), "utf8")) + .captures.map((capture) => capture.path); + const secondPaths = JSON.parse(readFileSync(join(second, "cohort-manifest.json"), "utf8")) + .captures.map((capture) => capture.path); + + assert.deepEqual(firstPaths, secondPaths, "portable filenames must be deterministic"); + assert.equal(new Set(firstPaths.map((path) => path.toLowerCase())).size, firstPaths.length); + assert.ok(firstPaths.every((path) => Buffer.byteLength(path) <= 240)); + assert.equal(firstPaths[2], "_CON.jsonl"); + assert.match(firstPaths[1], new RegExp(`-${captures[1].content_sha256.slice(0, 12)}\\.jsonl$`)); + } finally { + globalThis.fetch = originalFetch; + rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/trace-foundry.test.mjs b/tests/trace-foundry.test.mjs index 286ab3ec..5cddbde4 100644 --- a/tests/trace-foundry.test.mjs +++ b/tests/trace-foundry.test.mjs @@ -74,6 +74,25 @@ test("fails closed when no trace is within the requested window", () => { assert.throws(() => compileTraceFoundry(source, join(root, ".understudy", "out"), 3, new Date("2026-07-21T00:00:00Z")), /Refusing to compile a stale benchmark/); }); +test("hosted v3 placement ids remain workload-scoped after normalization", () => { + const root = mkdtempSync(join(tmpdir(), "understudy-foundry-v3-scope-")); + const source = join(root, "captures"), output = join(root, "out"); + mkdirSync(source, { recursive: true }); + const legacy = capture("legacy-v3", "2026-07-20T00:00:00Z", [{ role: "user", content: "Update legacy record" }], { + content: [{ type: "tool_use", id: "x", name: "update-record", input: { id: 1 } }], + }); + legacy.schema_version = 3; + legacy.placement_id = "usp_legacy"; + delete legacy.workload_id; + delete legacy.workload_name; + writeFileSync(join(source, "legacy.json"), JSON.stringify(legacy)); + const result = compileTraceFoundry(source, output, 3, new Date("2026-07-21T00:00:00Z"), { workload: "usp_legacy" }); + assert.equal(result.counts.captures, 1); + const normalized = JSON.parse(readFileSync(join(output, "normalized-captures.jsonl"), "utf8")); + assert.equal(normalized.scope.workload_id, "usp_legacy"); + assert.ok(normalized.warnings.includes("legacy_placement_id")); +}); + test("scopes workloads, preserves upstream requests, emits a resumable v1 environment, and applies review decisions", () => { const root = mkdtempSync(join(tmpdir(), "understudy-foundry-lifecycle-")); const source = join(root, ".understudy", "captures"), output = join(root, ".understudy", "benchmarks", "automation"); mkdirSync(source, { recursive: true }); @@ -807,6 +826,28 @@ test("leakage audit: clean when contract targets never surface outside the input assert.equal(audit.checked_tasks, 1); }); +test("build-time leakage warnings keep customer-derived excerpts off stderr", async () => { + const { printLeakageAudit } = await import("../dist/trace-foundry.js"); + const customerValue = "confidential-customer-document-998877"; + const stderr = []; + const originalError = console.error; + console.error = (...parts) => stderr.push(parts.join(" ")); + try { + printLeakageAudit({ + schema_version: "understudy.leakage_audit.v1", + status: "findings", + checked_tasks: 1, + findings: [{ task_id: "task-private", location: "fixtures.json", kind: "state_effect_value", excerpt: customerValue, tier: "verbatim", similarity: 1, signal: "exact" }], + tier_counts: { verbatim: 1, fuzzy: 0, semantic: 0 }, + heuristic: "synthetic test fixture", + }); + } finally { + console.error = originalError; + } + assert.doesNotMatch(stderr.join("\n"), new RegExp(customerValue)); + assert.match(stderr.join("\n"), /details recorded only in the private manifest/); +}); + test("build-benchmark records the leakage audit additively in manifest.json", () => { const root = mkdtempSync(join(tmpdir(), "understudy-foundry-leakage-")); const source = join(root, "captures"), output = join(root, "out"); mkdirSync(source, { recursive: true }); @@ -921,6 +962,41 @@ const pythonBin = ["python3", "python3.13", "python3.12", "python3.11", "python3 (bin) => spawnSync(bin, ["-c", "import sys; sys.exit(0 if sys.version_info >= (3, 10) else 1)"], { encoding: "utf8" }).status === 0, ); +test("hostile observed tool names remain data in generated Python", { skip: !pythonBin }, () => { + const root = mkdtempSync(join(tmpdir(), "understudy-foundry-hostile-tool-")); + const source = join(root, "captures"), output = join(root, "out"); + mkdirSync(source, { recursive: true }); + const hostileName = 'update-record\"\"\"\nraise RuntimeError("trace-data-executed")\n#'; + writeFileSync(join(source, "one.json"), JSON.stringify(capture( + "hostile", + "2026-07-20T00:00:00Z", + [{ role: "user", content: "Apply the update" }], + { content: [ + { type: "tool_use", id: "x", name: hostileName, input: { class: 1, self: 2, "foo-bar": 3, foo_bar: 4 } }, + { type: "tool_use", id: "y", name: "foo-bar", input: { id: 1 } }, + { type: "tool_use", id: "z", name: "foo_bar", input: { id: 2 } }, + { type: "tool_use", id: "dunder-init", name: "__init__", input: {} }, + { type: "tool_use", id: "dunder-get", name: "__getattribute__", input: {} }, + { type: "tool_use", id: "inherited", name: "run", input: {} }, + ] }, + ))); + compileTraceFoundry(source, output, 3, new Date("2026-07-21T00:00:00Z")); + const worldPath = join(output, "environment", "understudy_trace_env", "servers", "world.py"); + const world = readFileSync(worldPath, "utf8"); + assert.match(world, /Execute one trace-derived transition against per-rollout state/); + assert.doesNotMatch(world, /Execute the trace-derived/); + assert.match(world, /async def tool_foo_bar\(/); + assert.match(world, /async def tool_foo_bar_2\(/, "colliding tool names receive distinct Python identifiers"); + assert.match(world, /async def tool___init__\(/); + assert.match(world, /async def tool___getattribute__\(/); + assert.match(world, /async def tool_run\(/, "captured names cannot override inherited runtime methods"); + assert.match(world, /class_: float \| None = None/); + assert.match(world, /self_: float \| None = None/); + assert.match(world, /foo_bar_2: float \| None = None/, "colliding argument names receive distinct Python identifiers"); + const parsed = spawnSync(pythonBin, ["-m", "py_compile", worldPath], { encoding: "utf8" }); + assert.equal(parsed.status, 0, parsed.stderr); +}); + test("two sequential rollouts: rollout 2 starts from the seeded initial state, no residue from rollout 1", { skip: !pythonBin }, () => { const root = mkdtempSync(join(tmpdir(), "understudy-foundry-isolation-")); const source = join(root, "captures"), output = join(root, "out"); mkdirSync(source, { recursive: true }); @@ -974,7 +1050,7 @@ test("two sequential rollouts: rollout 2 starts from the seeded initial state, n " state = world.WorldState()", " ts = world.WorldToolset(state)", " initial = state.model_dump()", - " reply = await ts.update_record(id=7, status='active')", + " reply = await ts.tool_update_record(id=7, status='active')", " return initial, state.model_dump(), reply", "i1, f1, r1 = asyncio.run(rollout())", "i2, f2, r2 = asyncio.run(rollout())",