diff --git a/README.md b/README.md index c6ad997b..46e01db3 100644 --- a/README.md +++ b/README.md @@ -291,6 +291,8 @@ understudy captures list --project rehearsal --workload classify understudy captures export --request-ids-file request-ids.txt --project rehearsal --out .understudy/capture-batch --include-payload --yes understudy traces export --project rehearsal --out .understudy/trace-exports --include-payload --yes understudy traces export --trace-ids-file trace-ids.txt --project rehearsal --out .understudy/trace-exports --include-payload --yes +understudy traces export --project rehearsal --workload classify --date 2026-08-29 --out .understudy/trace-exports/day --include-payload --yes +understudy evals build --project rehearsal --workload classify --name classification-day --out .understudy/evals/classification-day --yes understudy routes set classify --project rehearsal --model-id glm-5.1 --traffic-pct 10 understudy routes show classify --project rehearsal understudy routes clear classify --project rehearsal @@ -324,13 +326,15 @@ file-only, and requires `--include-payload --yes`. For a customer-owned batch, put one request id per line in a file and pass `--request-ids-file`; the CLI retries transient failures, resumes from completed files, and writes `failed-request-ids.txt`. Redacted batch files use `.summary.json`, keeping -them distinct from full-payload `.payload.json` files. `traces export` resolves -one explicit `trace_id` through the customer trace request-ID endpoint, then -reuses that same bounded request exporter for every returned ID. Use a -positional trace ID or an explicit `--trace-ids-file`; there is no unbounded -`--all` trace scan. Each trace writes a private `trace.json` membership manifest, -per-request summary or payload files, `failed-request-ids.txt`, and a batch-level -`failed-trace-ids.txt`. `models list` shows public Understudy model IDs and +them distinct from full-payload `.payload.json` files. `traces export` can +resolve explicit trace IDs through the customer trace request-ID endpoint, or +download exactly one raw workload day through `--workload`. Workload mode uses +the rolling latest 24 hours by default; `--date YYYY-MM-DD` selects a completed +UTC calendar day. There is no unbounded `--all` trace scan. Explicit traces +write a private `trace.json` membership manifest, per-request files, and batch +failure manifests. Workload days write only a final ordered +`source/index.jsonl` and `source/summary.json` after success. `models list` +shows public Understudy model IDs and display names only. If the coding agent has an approved native email connector, it may complete the diff --git a/docs/current-functionality.md b/docs/current-functionality.md index eb6d7fb2..3233cc2e 100644 --- a/docs/current-functionality.md +++ b/docs/current-functionality.md @@ -52,6 +52,8 @@ understudy captures export --out .understudy/captures/. understudy captures export --request-ids-file request-ids.txt --project rehearsal --out .understudy/capture-batch --include-payload --yes understudy traces export --project rehearsal --out .understudy/trace-exports --include-payload --yes understudy traces export --trace-ids-file trace-ids.txt --project rehearsal --out .understudy/trace-exports --include-payload --yes +understudy traces export --project rehearsal --workload classify --date 2026-08-29 --out .understudy/trace-exports/day --include-payload --yes +understudy evals build --project rehearsal --workload classify --name classification-day --out .understudy/evals/classification-day --yes understudy routes show classify --project rehearsal understudy routes set classify --project rehearsal --model-id glm-5.1 --traffic-pct 10 understudy routes clear classify --project rehearsal @@ -150,11 +152,14 @@ model and what remains passthrough/frontier. Hosted capture commands are metadata-first. `captures list` and `captures get` redact prompt/completion-bearing fields into presence booleans. Full capture export is opt-in with `--include-payload --yes`, writes only to a file, and never -prints raw payloads to stdout. `traces export` calls the customer trace -request-ID lookup for one explicit `trace_id`, then passes the returned IDs to -the existing request capture batch exporter. It never walks the project capture -catalog. Full per-request captures require the same explicit payload opt-in; -there is no unbounded all-traces operation. +prints raw payloads to stdout. `traces export` either resolves explicit trace +IDs through the customer trace request-ID endpoint or downloads one exact raw +workload day. Workload mode defaults to the rolling latest 24 hours and accepts +`--date YYYY-MM-DD` for a completed UTC calendar day. It requires the same +explicit payload opt-in, writes owner-private files, and emits only counts, +paths, and hashes; there is no unbounded all-traces operation. `evals build` +uses the same one-day primitive before handing semantic authoring to the coding +agent. `optimize-workload check` reads `.understudy/capture-evidence/` artifacts, fails closed on missing files, invalid JSON, stale baseline hashes, diff --git a/docs/privacy-and-data-boundaries.md b/docs/privacy-and-data-boundaries.md index 48b709d9..a5c6181d 100644 --- a/docs/privacy-and-data-boundaries.md +++ b/docs/privacy-and-data-boundaries.md @@ -74,16 +74,22 @@ only to local files (mode `600` on Unix), redacted `.summary.json` files cannot be mistaken for full-payload `.payload.json` files during resume, and stdout contains counts and paths rather than capture content. -`understudy traces export` applies the same rule to one explicit hosted -`trace_id` or a private file of explicit trace IDs. The CLI resolves membership -through the customer trace request-ID endpoint and passes the returned IDs to -the existing bounded capture batch exporter; it never scans project capture -history and exposes no unbounded `--all` operation. A private `trace.json` -records membership and counts without raw bodies. Full per-request files require -`--include-payload --yes`, and stdout still contains only counts and paths. -Owner-private files, disjoint summary/payload suffixes, bounded concurrency, -retries, resume, `failed-request-ids.txt`, and `failed-trace-ids.txt` preserve -the request-export boundary. +`understudy traces export` supports two bounded modes. Explicit mode accepts one +hosted `trace_id` or a private file of explicit trace IDs, resolves membership +through the customer trace request-ID endpoint, and passes those IDs to the +capture batch exporter. Workload-window mode requires an explicit workload and +exports exactly 24 hours: `--date YYYY-MM-DD` selects one completed UTC calendar +day, while omitting it selects the rolling 24 hours ending when the command +starts. It has no unbounded `--all` operation. Workload-window export always +contains full payloads and therefore requires `--include-payload --yes`. + +Both modes write payloads only to owner-private local files and never print +prompts, completions, or tool payloads to stdout. Explicit mode records a +private `trace.json`; workload-window mode records a final ordered +`source/index.jsonl` and `source/summary.json` only after the complete export. +Bounded concurrency, retries, atomic writes, and validation of the requested +organization, project, workload, and request identity preserve the export +boundary. Gateway probes are explicit live calls. BYOK provider keys are read only from an environment variable named by `--byok-env`; they are not requested in chat, not diff --git a/schemas/README.md b/schemas/README.md index 5fa78670..f13c2a09 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -5,27 +5,23 @@ app, skills, CLI, ladder). One spine, adopted everywhere. ## Local workload eval authoring -The `understudy.eval-project.v2`, export-proof, execution-index-row, metric, +The `understudy.eval-project.v2`, execution-index-row, metric, coverage, harness, environment, splits, check-fixtures, check-report, and approval schemas define the private coding-agent workspace checked by `understudy evals check`. The workload profile remains Markdown; its exact bytes are bound by both intent approval and the deterministic check-input hash. These contracts require a provider-free local environment replay, independent good/wrong evidence, explicit lineage -coverage, and a separate post-check owner approval. +coverage, and a separate post-check owner approval. The project source records +requested, materialized, and skipped capture counts and the private portable +`source/skipped.jsonl` index so agents can qualify coverage claims when a raw +object is no longer available. The `understudy.eval-publication.v1` and `understudy.eval-release.v1` JSON Schemas define the structural hosted boundary for this workflow. Publication -carries the checked hashes, a compact backend-verifiable source attestation, -final approval, executable layout, and bundle inventory. The server response -adds the immutable release seal. Neither contract contains raw source traces, -the expiring export receipt, the local export-proof file, or mutable authoring -state. - -Within the private project manifest, `source.export_proof_sha256` binds the -exact local export-proof file. Within the check report and hosted -publication/release source, that field instead binds the exact opaque -`source_attestation` token. The CLI verifies both links before upload, and the -backend verifies the attestation itself before sealing a release. +carries the exact 24-hour source window, local index/count/byte commitments, +checked hashes, final approval, executable layout, and bundle inventory. The +server response adds the immutable release seal. Neither contract contains raw +source traces or mutable authoring state. These Draft 2020-12 schemas do not express the release contract's cross-field path rules. Consumers must also parse publications with the package's exported diff --git a/schemas/understudy.eval-check.v1.schema.json b/schemas/understudy.eval-check.v1.schema.json index 22ec7728..03d54f2c 100644 --- a/schemas/understudy.eval-check.v1.schema.json +++ b/schemas/understudy.eval-check.v1.schema.json @@ -27,12 +27,11 @@ "source": { "type": "object", "additionalProperties": false, - "required": ["scope", "scope_sha256", "index_sha256", "export_proof_sha256", "capture_count", "size_bytes"], + "required": ["scope", "scope_sha256", "index_sha256", "capture_count", "size_bytes"], "properties": { "scope": { "$ref": "#/$defs/scope" }, "scope_sha256": { "$ref": "#/$defs/sha" }, "index_sha256": { "$ref": "#/$defs/sha" }, - "export_proof_sha256": { "$ref": "#/$defs/sha" }, "capture_count": { "type": "integer", "minimum": 0 }, "size_bytes": { "type": "integer", "minimum": 0 } } diff --git a/schemas/understudy.eval-draft-check.v1.schema.json b/schemas/understudy.eval-draft-check.v1.schema.json index 53be6b47..fc80f4e9 100644 --- a/schemas/understudy.eval-draft-check.v1.schema.json +++ b/schemas/understudy.eval-draft-check.v1.schema.json @@ -29,12 +29,11 @@ "source": { "type": "object", "additionalProperties": false, - "required": ["scope", "scope_sha256", "index_sha256", "export_proof_sha256", "capture_count", "size_bytes"], + "required": ["scope", "scope_sha256", "index_sha256", "capture_count", "size_bytes"], "properties": { "scope": { "$ref": "#/$defs/scope" }, "scope_sha256": { "$ref": "#/$defs/sha" }, "index_sha256": { "$ref": "#/$defs/sha" }, - "export_proof_sha256": { "$ref": "#/$defs/sha" }, "capture_count": { "type": "integer", "minimum": 0 }, "size_bytes": { "type": "integer", "minimum": 0 } } diff --git a/schemas/understudy.eval-export-proof.v1.schema.json b/schemas/understudy.eval-export-proof.v1.schema.json deleted file mode 100644 index f611fbd9..00000000 --- a/schemas/understudy.eval-export-proof.v1.schema.json +++ /dev/null @@ -1,57 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://understudylabs.com/schemas/understudy.eval-export-proof.v1.schema.json", - "title": "understudy.eval-export-proof.v1", - "type": "object", - "additionalProperties": false, - "required": ["schema_version", "canonical_scope", "segment_manifest_sha256", "terminal_receipt", "verified_receipt"], - "properties": { - "schema_version": { "const": "understudy.eval-export-proof.v1" }, - "canonical_scope": { "$ref": "#/$defs/scope" }, - "segment_manifest_sha256": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/sha" } }, - "terminal_receipt": { "$ref": "#/$defs/nonempty" }, - "verified_receipt": { - "type": "object", - "additionalProperties": false, - "required": ["verified", "scope_hash", "chain_id", "segment_id", "segment_index", "manifest_sha256", "previous_manifest_sha256", "cumulative_scanned", "cumulative_matched", "cumulative_exported", "total_bytes", "local_index_sha256", "expires_at", "canonical_scope", "source_attestation"], - "properties": { - "verified": { "const": true }, - "scope_hash": { "$ref": "#/$defs/sha" }, - "chain_id": { "$ref": "#/$defs/nonempty" }, - "segment_id": { "$ref": "#/$defs/sha" }, - "segment_index": { "$ref": "#/$defs/count" }, - "manifest_sha256": { "$ref": "#/$defs/sha" }, - "previous_manifest_sha256": { "oneOf": [{ "$ref": "#/$defs/sha" }, { "type": "null" }] }, - "cumulative_scanned": { "$ref": "#/$defs/count" }, - "cumulative_matched": { "$ref": "#/$defs/count" }, - "cumulative_exported": { "$ref": "#/$defs/count" }, - "total_bytes": { "$ref": "#/$defs/count" }, - "local_index_sha256": { "$ref": "#/$defs/sha" }, - "expires_at": { "$ref": "#/$defs/timestamp" }, - "canonical_scope": { "$ref": "#/$defs/scope" }, - "source_attestation": { "type": "string", "minLength": 1, "maxLength": 8192 } - } - } - }, - "$defs": { - "nonempty": { "type": "string", "minLength": 1 }, - "sha": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, - "count": { "type": "integer", "minimum": 0 }, - "timestamp": { "type": "string", "format": "date-time", "pattern": "Z$" }, - "scope": { - "type": "object", - "additionalProperties": false, - "required": ["schema_version", "selector", "org_id", "project_id", "workload_id", "from", "to", "ingestion_cutoff"], - "properties": { - "schema_version": { "const": "understudy.export-scope.v1" }, - "selector": { "const": "workload-window" }, - "org_id": { "$ref": "#/$defs/nonempty" }, - "project_id": { "$ref": "#/$defs/nonempty" }, - "workload_id": { "$ref": "#/$defs/nonempty" }, - "from": { "$ref": "#/$defs/timestamp" }, - "to": { "$ref": "#/$defs/timestamp" }, - "ingestion_cutoff": { "$ref": "#/$defs/timestamp" } - } - } - } -} diff --git a/schemas/understudy.eval-project.v2.schema.json b/schemas/understudy.eval-project.v2.schema.json index de22da0c..876a91ef 100644 --- a/schemas/understudy.eval-project.v2.schema.json +++ b/schemas/understudy.eval-project.v2.schema.json @@ -25,7 +25,7 @@ "source": { "type": "object", "additionalProperties": false, - "required": ["window", "capture_count", "size_bytes", "index", "index_sha256", "export_proof", "export_proof_sha256", "exported_capture_count", "exported_total_bytes", "terminal_receipt_verified"], + "required": ["window", "requested_count", "materialized_count", "skipped_count", "skipped_index", "capture_count", "size_bytes", "index", "index_sha256"], "properties": { "window": { "type": "object", @@ -42,15 +42,14 @@ "ingestion_cutoff": { "$ref": "#/$defs/timestamp" } } }, + "requested_count": { "$ref": "#/$defs/count" }, + "materialized_count": { "$ref": "#/$defs/count" }, + "skipped_count": { "$ref": "#/$defs/count" }, + "skipped_index": { "$ref": "#/$defs/path" }, "capture_count": { "$ref": "#/$defs/count" }, "size_bytes": { "$ref": "#/$defs/count" }, "index": { "$ref": "#/$defs/path" }, - "index_sha256": { "$ref": "#/$defs/sha" }, - "export_proof": { "$ref": "#/$defs/path" }, - "export_proof_sha256": { "$ref": "#/$defs/sha" }, - "exported_capture_count": { "$ref": "#/$defs/count" }, - "exported_total_bytes": { "$ref": "#/$defs/count" }, - "terminal_receipt_verified": { "const": true } + "index_sha256": { "$ref": "#/$defs/sha" } } }, "artifacts": { diff --git a/schemas/understudy.eval-publication.v1.schema.json b/schemas/understudy.eval-publication.v1.schema.json index ac346fab..aff41de3 100644 --- a/schemas/understudy.eval-publication.v1.schema.json +++ b/schemas/understudy.eval-publication.v1.schema.json @@ -30,16 +30,14 @@ "source": { "type": "object", "additionalProperties": false, - "required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256", "export_proof_sha256", "source_attestation"], + "required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256"], "properties": { "from": { "$ref": "#/$defs/timestamp" }, "to": { "$ref": "#/$defs/timestamp" }, "ingestion_cutoff": { "$ref": "#/$defs/timestamp" }, "capture_count": { "type": "integer", "minimum": 0 }, "total_bytes": { "type": "integer", "minimum": 0 }, - "local_index_sha256": { "$ref": "#/$defs/sha" }, - "export_proof_sha256": { "$ref": "#/$defs/sha" }, - "source_attestation": { "type": "string", "minLength": 1, "maxLength": 8192 } + "local_index_sha256": { "$ref": "#/$defs/sha" } } }, "artifacts": { diff --git a/schemas/understudy.eval-release.v1.schema.json b/schemas/understudy.eval-release.v1.schema.json index 4030f67e..ba27f9e6 100644 --- a/schemas/understudy.eval-release.v1.schema.json +++ b/schemas/understudy.eval-release.v1.schema.json @@ -33,11 +33,10 @@ "path": { "type": "string", "minLength": 1, "maxLength": 240, "pattern": "^(?!.*\\u0000)(?!/)(?![A-Za-z]:)(?!.*\\\\)(?!.*(?:^|/)(?:\\.|\\.\\.)(?:/|$))(?!.*//)(?!.*\/$).+$" }, "source": { "type": "object", "additionalProperties": false, - "required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256", "export_proof_sha256", "source_attestation"], + "required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256"], "properties": { "from": { "$ref": "#/$defs/timestamp" }, "to": { "$ref": "#/$defs/timestamp" }, "ingestion_cutoff": { "$ref": "#/$defs/timestamp" }, - "capture_count": { "type": "integer", "minimum": 0 }, "total_bytes": { "type": "integer", "minimum": 0 }, "local_index_sha256": { "$ref": "#/$defs/sha" }, - "export_proof_sha256": { "$ref": "#/$defs/sha" }, "source_attestation": { "type": "string", "minLength": 1, "maxLength": 8192 } + "capture_count": { "type": "integer", "minimum": 0 }, "total_bytes": { "type": "integer", "minimum": 0 }, "local_index_sha256": { "$ref": "#/$defs/sha" } } }, "artifacts": { diff --git a/skills/capture-evidence/SKILL.md b/skills/capture-evidence/SKILL.md index 3e136041..fe63d1b7 100644 --- a/skills/capture-evidence/SKILL.md +++ b/skills/capture-evidence/SKILL.md @@ -22,10 +22,12 @@ When the developer names a workload already captured by Understudy and the active credentials can read it, use the hosted-workload front door in [`references/hosted-workload-eval.md`](references/hosted-workload-eval.md). An active `understudy.eval-project.v2` is a separate, project-local branch: use -its exact seven-day source, author only inside that eval project, and make the +its exact one-day raw source, author only inside that eval project, and make the coding agent—not the CLI—the conversational frontend. Infer the goal, metric, and failure taxonomy from the traces and repository before asking targeted gap -questions. Even without owner confirmation, continue to an explicitly +questions. Reconcile any capture references recorded in +`source/skipped.jsonl` before making coverage claims. Even without owner +confirmation, continue to an explicitly provisional local draft and run `understudy evals check --draft`. Do not run the incumbent baseline, null floor, a provider model, or a hosted EvalWorkspace on that branch. Strict `understudy evals check`, final approval, @@ -205,10 +207,13 @@ understudy evals build \ --project \ --workload \ --name \ - --out .understudy/evals/ + --out .understudy/evals/ \ + --yes ``` -This is a local-authoring operation: the service selects and freezes the source +By default this downloads the rolling 24 hours ending when the command starts; +add `--date YYYY-MM-DD` for one completed UTC calendar day. This is a +local-authoring operation: the CLI mechanically transports and validates raw captures, while the coding agent owns lineage reconstruction, tool-call interpretation, task and contract proposals, verifier generation, and review. Do not silently substitute an older server-generated environment for the local diff --git a/skills/capture-evidence/references/hosted-workload-eval.md b/skills/capture-evidence/references/hosted-workload-eval.md index 1361bb34..c89f75c3 100644 --- a/skills/capture-evidence/references/hosted-workload-eval.md +++ b/skills/capture-evidence/references/hosted-workload-eval.md @@ -1,7 +1,7 @@ # Build a local eval from a hosted Understudy workload Use this branch when the developer names a workload already captured by -Understudy. The backend transports the exact frozen week; the coding agent owns +Understudy. The CLI transports and validates one exact raw workload day; the coding agent owns all workload understanding, case selection, environment design, verifier authoring, and the conversation with the developer. The CLI is a transport and validation primitive, not a questionnaire or eval author. No hosted eval @@ -12,7 +12,7 @@ when the person at the keyboard is not the workload owner. Owner confirmation determines whether the draft can become a release; it does not gate local exploration. -## 1. Materialize the exact week +## 1. Materialize one raw workload day If no active `understudy.eval-project.v2` exists, explain that the files contain prompts, completions, and tool payloads, obtain approval, then run: @@ -23,10 +23,14 @@ understudy evals build \ --workload \ --name \ --out .understudy/evals/ \ - --last 7d \ --yes ``` +Without another time option, the command selects the rolling 24 hours ending +when it starts. Add `--date YYYY-MM-DD` to select one completed UTC calendar +day. Do not invent a broader range: this primitive intentionally has one exact +24-hour shape. + After the build materializes the source, give the CLI-emitted coding-agent prompt to the active agent. That output is the canonical handoff and includes the exact eval directory and draft-check command; do not maintain a second @@ -36,18 +40,20 @@ Choose `` once as a filesystem-safe directory name and use that exact path below. The display name may contain spaces or punctuation; the directory path does not depend on the CLI's name-to-slug conversion. -Resume the same command after an interruption. Do not copy the week into a +Resume the same command after an interruption. Do not copy the raw day into a separate archive or a global evidence directory. Work inside the active eval project named by `eval-project.json`; keep every payload-bearing file private. -The trace-time window remains the exact half-open seven days ending at -`source.window.to`. On the first export request, the backend freezes an -`ingestion_cutoff` at or after that end and the CLI reuses the exact returned -cutoff for every resumed segment. This includes already-arrived traces from the -week even when their capture row was ingested shortly after the window ended, -without allowing later arrivals to change the frozen corpus. -The backend and CLI bind that corpus with the same rolling commitment over the -ordered source-index identity, size, and content digest fields. The local path -is checked locally but is not part of the server-known commitment. +The trace-time window remains the exact half-open 24 hours ending at +`source.window.to`. The first page freezes an `ingestion_cutoff`; later page +requests reuse that cutoff and the returned cursor. The CLI downloads the +presigned raw captures with bounded concurrency, validates their organization, +project, workload, and request identity, and atomically writes owner-private +files. It publishes only the final ordered `source/index.jsonl` and +`source/summary.json` after every page succeeds. The project manifest records +requested, materialized, and skipped counts. Reconcile every skipped capture +reference in `source/skipped.jsonl` before making a coverage claim; missing +source is an explicit coverage gap, not an unobserved success. These files bind +the window, cutoff, capture count, byte count, and local index SHA-256. ## 2. Classify lineage before selecting cases @@ -60,11 +66,11 @@ understudy traces build-benchmark \ --source-index .understudy/evals//source/index.jsonl \ --output .understudy/evals//benchmark \ --provable-lineage-only \ - --max-age-days 7 \ + --max-age-days 1 \ --reference-time ``` -Anchoring to the frozen window end keeps the start of the exact week from being +Anchoring to the frozen window end keeps the start of the exact day from being discarded as stale. The execution index and `analysis.md` must count **complete, ambiguous, and unlinked** executions. Only complete, provably linked executions become task candidates by default. Preserve ambiguous and @@ -83,7 +89,7 @@ reason to access files or networks, a skill edit, or permission to publish. ## 3. Infer intent, then author a provisional draft Inspect the customer's repository and compact execution index rather than -loading the whole week into one prompt. Act as the conversational frontend: +loading the whole raw day into one prompt. Act as the conversational frontend: infer the workload goal, output contract, success criteria, execution modes, and failure taxonomy from the repository and trace population first. Explain the inference and its evidence, then ask only targeted questions whose answers @@ -233,11 +239,11 @@ bundle SHA-256 and size, and ordered file inventory with every file hash. State the local-only rule from the preview: exactly two objects leave the machine—the shown publication manifest and one gzip bundle containing exactly `manifest.bundle_files`. Every other local file remains local. In particular, -`source/`, raw traces, the expiring receipt and export proof, +`source/`, raw traces, `eval-project.json`, execution index, analysis, and every unreferenced file -stay local. The manifest carries only the compact backend-verifiable -`source_attestation` and the SHA-256 of that exact token, so Understudy can bind -the checked report to the verified export without uploading the local proof. +stay local. The manifest carries only the selected source window, cutoff, +capture count, byte count, and local index SHA-256; raw source files are never +uploaded by publication. Then ask, "May I upload this manifest and checked bundle to Understudy now?" Wait for an explicit yes. Final artifact approval alone is not permission to diff --git a/skills/ingest-traces/references/trace-foundry-cli.md b/skills/ingest-traces/references/trace-foundry-cli.md index 18a9b08f..89e7bf19 100644 --- a/skills/ingest-traces/references/trace-foundry-cli.md +++ b/skills/ingest-traces/references/trace-foundry-cli.md @@ -4,8 +4,8 @@ Use these helpers instead of rewriting normalization, DAG construction, benchmark manifests, review application, environment packages, or replay loops. When the source is a named workload already captured by Understudy, the -one-command front door freezes and downloads an immutable cohort before invoking -this same compiler locally: +one-command front door downloads one exact raw workload day before handing the +source to the coding agent and this compiler locally: ```sh understudy evals build \ @@ -14,6 +14,9 @@ understudy evals build \ --name ``` +It defaults to the rolling 24 hours ending when the command starts. Add +`--date YYYY-MM-DD` to select one completed UTC calendar day. + See [`../../capture-evidence/references/hosted-workload-eval.md`](../../capture-evidence/references/hosted-workload-eval.md) for the service-versus-agent ownership boundary and privacy gates. diff --git a/skills/understand-workload/SKILL.md b/skills/understand-workload/SKILL.md index 37dda2f9..5e3cae6a 100644 --- a/skills/understand-workload/SKILL.md +++ b/skills/understand-workload/SKILL.md @@ -147,9 +147,11 @@ rather than blocking useful analysis. For a workload hosted by Understudy, follow the draft-first branch in [`../capture-evidence/references/hosted-workload-eval.md`](../capture-evidence/references/hosted-workload-eval.md). -Use the exact frozen seven-day corpus and repository to create the local draft, -then run `understudy evals check --draft`. Keep raw traces local. Strict checking, -final approval, and publication remain separate owner-confirmed release steps. +Use the exact one-day raw source and repository to create the local draft, then +run `understudy evals check --draft`. The CLI defaults to the rolling 24 hours +ending when export starts; `--date YYYY-MM-DD` selects a completed UTC day. +Keep raw traces local. Strict checking, final approval, and publication remain +separate owner-confirmed release steps. ## Output Standard diff --git a/skills/understudy/SKILL.md b/skills/understudy/SKILL.md index 80ebe1b8..d92ba300 100644 --- a/skills/understudy/SKILL.md +++ b/skills/understudy/SKILL.md @@ -151,8 +151,10 @@ Identify the developer's current stage and load exactly one: harness, traces, metric, splits, or incumbent baseline are missing, ambiguous, or stale → [`../capture-evidence/SKILL.md`](../capture-evidence/SKILL.md) (also owns repo inspection + eval-harness discovery/build). For a named - workload already captured by Understudy, this route uses `understudy evals - build` to freeze the source cohort and construct the verifier draft locally. + workload already captured by Understudy, the recommended route is + `understudy evals build`: it downloads one raw workload day and hands that + source to the coding agent for local verifier authoring. Do not use the + legacy sampled-cohort `evals create` path for this workflow. - **App is running but no traces exist yet** — the developer wants capture flowing in minutes with no app-code changes ("instrument my app", "start capturing my LLM calls") → [`../instrument/SKILL.md`](../instrument/SKILL.md) diff --git a/src/commands/evals.ts b/src/commands/evals.ts index 0e00293b..3c8e8618 100644 --- a/src/commands/evals.ts +++ b/src/commands/evals.ts @@ -9,40 +9,36 @@ import { runEvalCheck } from "../evals/check.js"; import { previewEvalPublication, publishEvalRelease } from "../evals/publish.js"; import { acquireEvalBuildLease, - assertWorkloadBuildStateMatches, - creatingWorkloadBuildState, + createPrivateDirectory, ensureUnderstudyGitExcluded, - initializeBuildCheckpoint, pathExists, - readEvalBuildState, - replacePrivateJson, - writePrivateJson, } from "../evals/build-state.js"; import { CatalogItemSchema, CatalogResponseSchema, CohortExportSchema, CohortSchema, - EvalWorkloadBuildStateSchema, - VerifyWorkloadCaptureExportReceiptResponseSchema, - WorkloadCaptureExportResponseSchema, type CatalogItem, - type EvalWorkloadBuildState, - type WorkloadCaptureExportResponse, - type WorkloadCaptureExportScope, } from "../evals/contracts.js"; import { assertEquivalentExport, assertExportLineage, downloadExport, EXPORT_EXPIRES_SECONDS, - materializeWorkloadExportSegment, } from "../evals/materialize.js"; -import { sourceIndexCommitmentSha256 } from "../evals/source-index.js"; import { request } from "../internal/http.js"; import { isJsonMode, runAction } from "../internal/output.js"; import { resolveProject, type ProjectResolutionOptions } from "../internal/projects.js"; import { resolveWorkload } from "../internal/workloads.js"; +import { + DEFAULT_WORKLOAD_TRACE_CONCURRENCY, + DEFAULT_WORKLOAD_TRACE_RETRIES, + exportWorkloadTraceWindow, + MAX_WORKLOAD_TRACE_CONCURRENCY, + MAX_WORKLOAD_TRACE_RETRIES, + resolveWorkloadTraceWindow, + type WorkloadTraceWindow, +} from "../workload-trace-export.js"; interface WorkloadOpts extends ProjectResolutionOptions { workload: string; @@ -85,11 +81,12 @@ interface GuidedCreateOpts extends WorkloadOpts { } interface BuildOpts extends WorkloadOpts { name: string; - last: string; + last?: string; + date?: string; out?: string; yes?: boolean; - maxAgeDays?: string; - batchSize: string; + concurrency: string; + retries: string; } interface CheckOpts { project: string; @@ -105,8 +102,21 @@ export function registerEvalsCommand(program: Command): void { const evals = program.command("evals") .description("Build, check, publish, and manage workload-scoped evaluations for coding agents."); + addWorkloadOptions(evals.command("build") + .description("Recommended: download one raw workload day for a coding agent to turn into an eval.") + .requiredOption("--name ", "Local eval project name.") + .option("--last ", "Rolling workload window (currently exactly 1d).") + .option("--date ", "Use one complete UTC calendar day instead of the latest 24 hours.") + .option("--out ", "Destination directory (default: .understudy/evals/).") + .option("--concurrency ", `Raw capture download concurrency, max ${MAX_WORKLOAD_TRACE_CONCURRENCY}.`, String(DEFAULT_WORKLOAD_TRACE_CONCURRENCY)) + .option("--retries ", `Retries for transient page or capture failures, max ${MAX_WORKLOAD_TRACE_RETRIES}.`, String(DEFAULT_WORKLOAD_TRACE_RETRIES)) + .option("--yes", "Approve downloading payload-bearing traces without prompting.")) + .action(async function (this: Command, opts: BuildOpts) { + await runAction(this, () => runBuild(this, opts)); + }); + addWorkloadOptions(addRecentSelectionOptions( - evals.command("create").description("Create a frozen eval set from a recent workload window."), + evals.command("create").description("Legacy: create a sampled hosted cohort; use `evals build` for coding-agent eval authoring."), "Cohort name.", ) .option("--out ", "Destination directory (default: .understudy/evals/).") @@ -116,18 +126,6 @@ export function registerEvalsCommand(program: Command): void { await runAction(this, () => runGuidedCreate(this, opts)); }); - addWorkloadOptions(evals.command("build") - .description("Download a complete seven-day workload source for a coding agent to turn into an eval.") - .requiredOption("--name ", "Local eval project name.") - .option("--last ", "Complete capture window (currently exactly 7d).", "7d") - .option("--out ", "Destination directory (default: .understudy/evals/).") - .option("--max-age-days ", "Record the source freshness horizon (default: 7).") - .option("--batch-size ", "Record the coding-agent processing batch size.", "10") - .option("--yes", "Approve downloading payload-bearing traces without prompting.")) - .action(async function (this: Command, opts: BuildOpts) { - await runAction(this, () => runBuild(this, opts)); - }); - evals.command("check") .description("Check a locally authored eval, its verifier fixtures, and artifact hashes without a model call.") .option("--project ", "Eval project directory containing eval-project.json.", ".") @@ -344,18 +342,7 @@ async function runGuidedCreate(cmd: Command, opts: GuidedCreateOpts): Promise { - const batchSize = parsePositiveInteger("--batch-size", opts.batchSize); - const windowMs = parseDuration(opts.last); - if (windowMs !== 7 * 86_400_000) { - throw new Error("understudy evals build currently requires the complete --last 7d window."); - } - const selectionDays = 7; - const maxAgeDays = opts.maxAgeDays === undefined - ? selectionDays - : parsePositiveInteger("--max-age-days", opts.maxAgeDays); - if (maxAgeDays < selectionDays) { - throw new Error(`--max-age-days must cover --last (${selectionDays} day(s)).`); - } + const window = resolveWorkloadTraceWindow({ date: opts.date, last: opts.last }); if (isJsonMode(cmd) && !opts.yes) { throw new Error("JSON mode cannot prompt. Re-run with --yes to approve the complete local trace download."); } @@ -369,7 +356,17 @@ async function runBuild(cmd: Command, opts: BuildOpts): Promise { ensureUnderstudyGitExcluded(output); const releaseLease = acquireEvalBuildLease(output); try { - await runBuildWithLease(cmd, opts, { batchSize, windowMs, maxAgeDays, output }); + await runBuildWithLease(cmd, opts, { + output, + window, + concurrency: parseBoundedInteger( + "--concurrency", + opts.concurrency, + 1, + MAX_WORKLOAD_TRACE_CONCURRENCY, + ), + retries: parseBoundedInteger("--retries", opts.retries, 0, MAX_WORKLOAD_TRACE_RETRIES), + }); } finally { releaseLease(); } @@ -378,265 +375,65 @@ async function runBuild(cmd: Command, opts: BuildOpts): Promise { async function runBuildWithLease( cmd: Command, opts: BuildOpts, - build: { batchSize: number; windowMs: number; maxAgeDays: number; output: string }, + build: { + output: string; + window: WorkloadTraceWindow; + concurrency: number; + retries: number; + }, ): Promise { - const { batchSize, windowMs, maxAgeDays, output } = build; + const { output, window, concurrency, retries } = build; if (pathExists(output)) { throw new Error(`Eval build destination already exists: ${output}. Choose a fresh --out directory.`); } const staging = join(dirname(output), `.${basename(output)}.eval-build`); - const pending = pathExists(staging) ? readEvalBuildState(staging) : null; + if (pathExists(staging)) createPrivateDirectory(staging); const context = await resolveContext(opts); const currentIdentity = identityFromContext(context); - let state: EvalWorkloadBuildState; - if (pending) { - assertWorkloadBuildStateMatches(pending, opts.name, currentIdentity, maxAgeDays, batchSize); - state = pending; - } else { - const to = new Date(); - state = creatingWorkloadBuildState({ - name: opts.name, - identity: currentIdentity, - source: { - from: new Date(to.getTime() - windowMs).toISOString(), - to: to.toISOString(), - ingestion_cutoff: null, - }, - maxAgeDays, - batchSize, - now: to, - }); - initializeBuildCheckpoint(staging, state); - } if (!opts.yes) { const approved = await confirm({ - message: `${pending ? "Resume" : "Download"} every retrievable capture in the frozen seven-day window for local eval “${opts.name}”? Files may contain prompts, completions, and tool payloads.`, + message: `${pathExists(staging) ? "Resume" : "Download"} every retrievable capture in this one-day window for local eval “${opts.name}”? Files may contain prompts, completions, and tool payloads.`, default: false, }); if (!approved) throw new Error("Eval build cancelled before payload download."); } - - if (state.status === "complete") { - const recovered = JSON.parse(readFileSync(join(staging, "eval-project.json"), "utf8")) as WorkloadEvalProjectBuildResult; - renameSync(staging, output); - recovered.project_file = join(output, "eval-project.json"); - emitWorkloadBuildResult(cmd, output, recovered); - return; - } - - while (state.status === "downloading") { - state = resetInterruptedInitialWorkloadExport(staging, state); - const segment = await fetchWorkloadExportSegment(context, state); - if (state.source.ingestion_cutoff === null) { - assertInitialWorkloadExportScopeMatchesState(segment, state); - state = persistWorkloadBuildState(staging, { - ...state, - source: { - ...state.source, - ingestion_cutoff: segment.canonical_scope.ingestion_cutoff, + const source = await exportWorkloadTraceWindow({ + orgId: currentIdentity.org_id, + projectId: currentIdentity.project_id, + workloadId: currentIdentity.workload_id, + outputDirectory: staging, + gatewayUrl: context.project.auth.gatewayUrl, + concurrency, + retries, + reuseStoredWindow: opts.date === undefined, + ...window, + onProgress: isJsonMode(cmd) + ? undefined + : (completed) => { + if (completed % 100 === 0) process.stderr.write(`Exported ${completed} raw captures...\n`); }, - }); - } - assertWorkloadExportSegmentMatchesState(segment, state); - await materializeWorkloadExportSegment({ - exportData: segment, - tracesDirectory: join(staging, "source", "traces"), - gatewayUrl: context.project.auth.gatewayUrl, - verifiedFiles: state.transport.verified_files, - onVerified(file) { - if (!state.transport.verified_files.some((existing) => existing.capture_key === file.capture_key)) { - state = persistWorkloadBuildState(staging, { - ...state, - transport: { - ...state.transport, - verified_files: [...state.transport.verified_files, file], - }, - }); - } - }, - }); - if (sourceIndexCommitmentSha256(state.transport.verified_files) !== segment.chain.local_index_sha256) { - throw new Error("Capture export source index commitment does not match its manifest items."); - } - state = persistWorkloadBuildState(staging, { - ...state, - status: segment.chain.terminal ? "receipt_pending" : "downloading", - transport: { - ...state.transport, - resume_cursor: segment.resume_cursor ?? null, - chain_id: segment.chain.chain_id, - next_segment_index: segment.chain.segment_index + 1, - previous_manifest_sha256: segment.chain.manifest_sha256, - segment_manifest_sha256: [...state.transport.segment_manifest_sha256, segment.chain.manifest_sha256], - cumulative_exported: segment.chain.cumulative_exported, - cumulative_total_bytes: segment.chain.cumulative_total_bytes, - terminal_receipt: segment.chain.terminal_receipt ?? null, - }, - }); - } - - if (!state.transport.terminal_receipt) throw new Error("Complete capture export is missing its terminal receipt."); - const receipt = await verifyWorkloadExportReceipt(context, state); - if ( - receipt.chain_id !== state.transport.chain_id || - receipt.cumulative_exported !== state.transport.cumulative_exported || - receipt.total_bytes !== state.transport.cumulative_total_bytes || - receipt.manifest_sha256 !== state.transport.previous_manifest_sha256 || - receipt.local_index_sha256 !== sourceIndexCommitmentSha256(state.transport.verified_files) - ) throw new Error("Verified capture export receipt does not match the downloaded source chain."); - + }); const project = buildWorkloadEvalProject({ output: staging, - name: state.name, - identity: state.identity, - canonicalScope: receipt.canonical_scope, - verifiedFiles: state.transport.verified_files, - segmentManifestSha256: state.transport.segment_manifest_sha256, - terminalReceipt: state.transport.terminal_receipt, - verifiedReceipt: receipt, - now: new Date(state.created_at), + name: opts.name, + identity: currentIdentity, + source, + now: new Date(), }); - state = persistWorkloadBuildState(staging, { ...state, status: "complete" }); renameSync(staging, output); project.project_file = join(output, "eval-project.json"); emitWorkloadBuildResult(cmd, output, project); } -async function fetchWorkloadExportSegment( - context: Awaited>, - state: EvalWorkloadBuildState, -): Promise { - if (state.transport.resume_cursor !== null && state.source.ingestion_cutoff === null) { - throw new Error("Resumed capture export is missing its frozen ingestion cutoff."); - } - const response = await request({ - url: `${context.base}/eval-capture-export`, - method: "POST", - orgId: context.project.auth.orgId, - signal: AbortSignal.timeout(60_000), - body: { - from: state.source.from, - to: state.source.to, - expires_seconds: EXPORT_EXPIRES_SECONDS, - ...(state.transport.resume_cursor === null - ? {} - : { - ingestion_cutoff: state.source.ingestion_cutoff, - resume_cursor: state.transport.resume_cursor, - }), - }, - }, WorkloadCaptureExportResponseSchema); - return response.data; -} - -function resetInterruptedInitialWorkloadExport( - staging: string, - state: EvalWorkloadBuildState, -): EvalWorkloadBuildState { - if (state.transport.resume_cursor !== null || state.source.ingestion_cutoff === null) return state; - if ( - state.transport.next_segment_index !== 0 || - state.transport.chain_id !== null || - state.transport.previous_manifest_sha256 !== null || - state.transport.segment_manifest_sha256.length !== 0 || - state.transport.cumulative_exported !== 0 || - state.transport.cumulative_total_bytes !== 0 || - state.transport.terminal_receipt !== null - ) { - throw new Error("Interrupted initial capture export has inconsistent chain state."); - } - rmSync(join(staging, "source", "traces"), { recursive: true, force: true }); - return persistWorkloadBuildState(staging, { - ...state, - source: { ...state.source, ingestion_cutoff: null }, - transport: { ...state.transport, verified_files: [] }, - }); -} - -async function verifyWorkloadExportReceipt( - context: Awaited>, - state: EvalWorkloadBuildState, -) { - const canonicalScope = workloadExportScope(state); - const response = await request({ - url: `${context.base}/eval-capture-export/verify`, - method: "POST", - orgId: context.project.auth.orgId, - signal: AbortSignal.timeout(60_000), - body: { terminal_receipt: state.transport.terminal_receipt, canonical_scope: canonicalScope }, - }, VerifyWorkloadCaptureExportReceiptResponseSchema); - if (JSON.stringify(response.data.canonical_scope) !== JSON.stringify(canonicalScope)) { - throw new Error("Verified capture export receipt returned a different canonical scope."); - } - return response.data; -} - -function workloadExportScope(state: EvalWorkloadBuildState): WorkloadCaptureExportScope { - const ingestionCutoff = state.source.ingestion_cutoff; - if (ingestionCutoff === null) { - throw new Error("Capture export has not returned its frozen ingestion cutoff."); - } - return { - schema_version: "understudy.export-scope.v1" as const, - selector: "workload-window" as const, - org_id: state.identity.org_id, - project_id: state.identity.project_id, - workload_id: state.identity.workload_id, - from: state.source.from, - to: state.source.to, - ingestion_cutoff: ingestionCutoff, - }; -} - -function assertInitialWorkloadExportScopeMatchesState( - segment: WorkloadCaptureExportResponse, - state: EvalWorkloadBuildState, -): void { - const scope = segment.canonical_scope; - const ingestionCutoffMs = Date.parse(scope.ingestion_cutoff); - if ( - scope.org_id !== state.identity.org_id || - scope.project_id !== state.identity.project_id || - scope.workload_id !== state.identity.workload_id || - scope.from !== state.source.from || - scope.to !== state.source.to || - ingestionCutoffMs < Date.parse(scope.to) || - ingestionCutoffMs > Date.now() + 60_000 - ) { - throw new Error("Capture export response does not match the requested workload window."); - } -} - -function assertWorkloadExportSegmentMatchesState( - segment: WorkloadCaptureExportResponse, - state: EvalWorkloadBuildState, -): void { - const expectedScope = workloadExportScope(state); - if (JSON.stringify(segment.canonical_scope) !== JSON.stringify(expectedScope)) { - throw new Error("Capture export response does not match the frozen workload window."); - } - if ( - segment.chain.segment_index !== state.transport.next_segment_index || - segment.chain.previous_manifest_sha256 !== state.transport.previous_manifest_sha256 || - (state.transport.chain_id !== null && segment.chain.chain_id !== state.transport.chain_id) || - segment.chain.cumulative_exported !== state.transport.cumulative_exported + segment.count || - segment.chain.cumulative_total_bytes !== state.transport.cumulative_total_bytes + segment.total_bytes - ) throw new Error("Capture export segment does not continue the persisted source chain."); -} - -function persistWorkloadBuildState(staging: string, candidate: EvalWorkloadBuildState): EvalWorkloadBuildState { - const state = EvalWorkloadBuildStateSchema.parse(candidate); - replacePrivateJson(join(staging, "build-state.json"), state); - return state; -} - function emitWorkloadBuildResult(cmd: Command, output: string, project: WorkloadEvalProjectBuildResult): void { - const checkArgs = ["evals", "check", "--draft", "--project", output]; + const checkArgs = ["--json", "evals", "check", "--draft", "--project", output]; + const skippedPath = join(output, project.source.skipped_index); const nextAction = { kind: "coding_agent_prompt" as const, command: { executable: "understudy", args: checkArgs }, prompt: [ - `Use the Understudy capture-evidence skill to build a provisional eval from the complete seven-day traces at ${output}.`, + `Use the Understudy capture-evidence skill to build a provisional eval from the one-day raw traces at ${output}.`, + `Reconcile ${project.source.skipped_count} skipped captures recorded at ${skippedPath} before making coverage claims.`, "Infer the workload goal, output contract, success criteria, execution modes, and failure taxonomy from these traces and the current repository.", "Explain your evidence and ask only targeted questions whose answers would materially change the metric, environment, or case selection.", "Author the project-local eval artifacts and mark unconfirmed semantics as provisional.", @@ -647,8 +444,9 @@ function emitWorkloadBuildResult(cmd: Command, output: string, project: Workload if (isJsonMode(cmd)) { process.stdout.write(`${JSON.stringify({ ...project, next_action: nextAction })}\n`); } else { - process.stdout.write(`${kleur.green("✓")} Materialized the complete seven-day source at ${output}\n`); + process.stdout.write(`${kleur.green("✓")} Materialized the one-day raw source at ${output}\n`); process.stdout.write(`Project manifest: ${project.project_file}\n`); + process.stdout.write(`Source: ${project.source.requested_count} requested, ${project.source.materialized_count} materialized, ${project.source.skipped_count} skipped (${project.source.skipped_index})\n`); process.stdout.write(`${kleur.yellow("warning")}: local files contain prompts, completions, or tool payloads; nothing was uploaded and no model provider was called\n`); process.stdout.write("Next, give this prompt to your coding agent:\n\n"); process.stdout.write(`${nextAction.prompt}\n`); @@ -857,10 +655,10 @@ function parseDuration(value: string): number { return durationMs; } -function parsePositiveInteger(name: string, value: string): number { +function parseBoundedInteger(name: string, value: string, minimum: number, maximum: number): number { const parsed = Number(value); - if (!Number.isInteger(parsed) || parsed <= 0) { - throw new Error(`${name} must be a positive integer.`); + if (!Number.isInteger(parsed) || parsed < minimum || parsed > maximum) { + throw new Error(`${name} must be between ${minimum} and ${maximum}.`); } return parsed; } diff --git a/src/commands/trace-exports.ts b/src/commands/trace-exports.ts index e11f068a..712b06c5 100644 --- a/src/commands/trace-exports.ts +++ b/src/commands/trace-exports.ts @@ -1,6 +1,6 @@ import { createHash } from "node:crypto"; import { existsSync, readFileSync, statSync } from "node:fs"; -import { basename, join } from "node:path"; +import { basename, join, resolve } from "node:path"; import { Command } from "commander"; import kleur from "kleur"; import { z } from "zod"; @@ -14,11 +14,20 @@ import { } from "./captures.js"; import { request, UnderstudyApiError } from "../internal/http.js"; import { isJsonMode, runAction } from "../internal/output.js"; +import { acquireEvalBuildLease } from "../evals/build-state.js"; import { resolveProject, type ProjectResolutionOptions, } from "../internal/projects.js"; import { resolveWorkload } from "../internal/workloads.js"; +import { + DEFAULT_WORKLOAD_TRACE_CONCURRENCY, + DEFAULT_WORKLOAD_TRACE_RETRIES, + exportWorkloadTraceWindow, + MAX_WORKLOAD_TRACE_CONCURRENCY, + MAX_WORKLOAD_TRACE_RETRIES, + resolveWorkloadTraceWindow, +} from "../workload-trace-export.js"; const TRACE_EXPORT_SCHEMA = "understudy.trace_export.v1"; const DEFAULT_EXPORT_CONCURRENCY = 4; @@ -37,6 +46,8 @@ interface TraceExportOpts extends ProjectResolutionOptions { workload?: string; out: string; traceIdsFile?: string; + date?: string; + last?: string; includePayload?: boolean; yes?: boolean; concurrency?: string; @@ -54,16 +65,18 @@ interface TraceExportResult { export function registerHostedTraceExportCommand(traces: Command): void { traces.command("export [trace-id]") - .description("Resolve hosted trace request ids and export their captures privately.") + .description("Export explicit hosted traces or one workload day privately.") .requiredOption("--out ", "Private output directory.") .option("--trace-ids-file ", "Batch export: one explicit trace id per line.") + .option("--date ", "Workload mode: one complete UTC calendar day.") + .option("--last ", "Workload mode: rolling window (currently exactly 1d).") .option("--project-id ", "Project id from `understudy projects list --json`.") .option("--project ", "Project slug to resolve to an id.") .option("--workload ", "Optional workload name or id.") .option("--org ", "Org id to use (default: local config or only org in credentials).") .option("--concurrency ", `Capture-fetch concurrency, max ${MAX_EXPORT_CONCURRENCY}.`, String(DEFAULT_EXPORT_CONCURRENCY)) .option("--retries ", `Retries for transient lookup or capture failures, max ${MAX_EXPORT_RETRIES}.`, String(DEFAULT_EXPORT_RETRIES)) - .option("--no-resume", "Re-download capture files that already exist.") + .option("--no-resume", "Explicit trace mode: re-download capture files that already exist.") .option("--include-payload", "Write full captures, including prompt/completion payloads.") .option("--yes", "Confirm full payload export without prompting.") .action(async function (this: Command, traceId: string | undefined, opts: TraceExportOpts) { @@ -78,11 +91,18 @@ async function runTraceExport( ): Promise { const traceId = traceIdInput?.trim(); const traceIdsFile = opts.traceIdsFile?.trim(); - if (Boolean(traceId) === Boolean(traceIdsFile)) { + if (traceId && traceIdsFile) { throw new Error( - "Provide exactly one of or --trace-ids-file .", + "Provide only one of or --trace-ids-file .", ); } + if (!traceId && !traceIdsFile) { + await runWorkloadTraceExport(cmd, opts); + return; + } + if (opts.date !== undefined || opts.last !== undefined) { + throw new Error("--date and --last are only valid for workload-day export without explicit trace ids."); + } if (traceId) validateTraceId(traceId); if (opts.includePayload && !opts.yes) { throw new Error( @@ -242,6 +262,91 @@ async function runTraceExport( if (retryTraceIds.length > 0) process.exitCode = 1; } +async function runWorkloadTraceExport(cmd: Command, opts: TraceExportOpts): Promise { + if (!opts.workload) { + throw new Error("Workload-day trace export requires --workload when no trace id is provided."); + } + if (!opts.includePayload || !opts.yes) { + throw new Error( + "Workload-day export contains raw prompts/completions. Re-run with --include-payload --yes to write it privately.", + ); + } + if (opts.resume === false) { + throw new Error("--no-resume is only valid for explicit trace ids. Choose a fresh --out directory for a new workload-day export."); + } + const window = resolveWorkloadTraceWindow({ date: opts.date, last: opts.last }); + const concurrency = parseBoundedInteger( + opts.concurrency, + "--concurrency", + 1, + MAX_WORKLOAD_TRACE_CONCURRENCY, + DEFAULT_WORKLOAD_TRACE_CONCURRENCY, + ); + const retries = parseBoundedInteger( + opts.retries, + "--retries", + 0, + MAX_WORKLOAD_TRACE_RETRIES, + DEFAULT_WORKLOAD_TRACE_RETRIES, + ); + const outputDirectory = resolve(opts.out); + const releaseOutputLease = acquireEvalBuildLease(outputDirectory); + try { + const project = await resolveProject(opts); + const workload = await resolveWorkload(project, opts.workload); + const result = await exportWorkloadTraceWindow({ + orgId: project.auth.orgId, + projectId: project.projectId, + workloadId: workload.id, + outputDirectory, + gatewayUrl: project.auth.gatewayUrl, + concurrency, + retries, + reuseStoredWindow: opts.date === undefined, + ...window, + onProgress: isJsonMode(cmd) + ? undefined + : (completed) => { + if (completed % 100 === 0) process.stderr.write(`Processed ${completed} raw capture references...\n`); + }, + }); + const source = { + window: result.canonicalScope, + requested_count: result.requestedCount, + materialized_count: result.captureCount, + skipped_count: result.skippedCount, + capture_count: result.captureCount, + size_bytes: result.sizeBytes, + index: "source/index.jsonl", + index_sha256: result.indexSha256, + }; + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify({ + ok: true, + mode: "workload_window", + org_id: project.auth.orgId, + project_id: project.projectId, + workload_id: workload.id, + output_directory: result.outputDirectory, + source, + written: result.writtenCount, + adopted: result.adoptedCount, + skipped: result.skippedCount, + warning: "files may contain prompts, completions, or tool payloads", + })}\n`); + return; + } + process.stdout.write( + `${kleur.green("✓")} Exported ${result.captureCount}/${result.requestedCount} raw workload captures` + + `${result.skippedCount > 0 ? ` (${result.skippedCount} no longer available)` : ""} -> ${result.outputDirectory}\n`, + ); + process.stdout.write(`Source index: ${result.indexPath}\n`); + process.stdout.write(`${kleur.yellow("warning")}: files may contain prompts, completions, or tool payloads\n`); + } finally { + releaseOutputLease(); + } +} + async function fetchTraceRequestIdsWithRetry( orgId: string, projectId: string, diff --git a/src/eval-project.ts b/src/eval-project.ts index 76e525f6..31004c2f 100644 --- a/src/eval-project.ts +++ b/src/eval-project.ts @@ -1,17 +1,12 @@ -import { createHash, randomUUID } from "node:crypto"; -import { chmodSync, mkdirSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { createHash } from "node:crypto"; +import { mkdirSync, writeFileSync } from "node:fs"; import { isAbsolute, join, relative, resolve, sep } from "node:path"; import { compileTraceFoundry, type FoundryResult } from "./trace-foundry.js"; -import { createPrivateDirectory } from "./evals/build-state.js"; -import { sourceIndexCommitmentSha256 } from "./evals/source-index.js"; import type { WorkloadEvalProject } from "./evals/authoring-contracts.js"; -import type { - EvalBuildIdentity, - VerifiedWorkloadCaptureFile, - VerifyWorkloadCaptureExportReceiptResponse, - WorkloadCaptureExportScope, -} from "./evals/contracts.js"; +import type { EvalBuildIdentity } from "./evals/contracts.js"; +import { replacePrivateText } from "./evals/build-state.js"; +import type { WorkloadTraceExportResult, WorkloadTraceExportScope } from "./workload-trace-export.js"; export interface EvalProjectIdentity { orgId: string; @@ -79,16 +74,15 @@ export interface WorkloadEvalProjectManifest { created_at: string; identity: EvalBuildIdentity; source: { - window: WorkloadCaptureExportScope; + window: WorkloadTraceExportScope; + requested_count: number; + materialized_count: number; + skipped_count: number; + skipped_index: string; capture_count: number; size_bytes: number; index: string; index_sha256: string; - export_proof: string; - export_proof_sha256: string; - exported_capture_count: number; - exported_total_bytes: number; - terminal_receipt_verified: true; }; artifacts: WorkloadEvalProject["artifacts"]; authoring: { @@ -102,11 +96,7 @@ export interface BuildWorkloadEvalProjectOptions { output: string; name: string; identity: EvalBuildIdentity; - canonicalScope: WorkloadCaptureExportScope; - verifiedFiles: VerifiedWorkloadCaptureFile[]; - segmentManifestSha256: string[]; - terminalReceipt: string; - verifiedReceipt: VerifyWorkloadCaptureExportReceiptResponse; + source: WorkloadTraceExportResult; now: Date; } @@ -117,7 +107,7 @@ export interface WorkloadEvalProjectBuildResult extends WorkloadEvalProjectManif export function deriveWorkloadEvalId(input: { name: string; identity: EvalBuildIdentity; - sourceWindow: WorkloadCaptureExportScope; + sourceWindow: WorkloadTraceExportScope; }): string { return `eval_${createHash("sha256").update(JSON.stringify({ schema_version: "understudy.eval-identity.v1", @@ -219,56 +209,29 @@ export function buildEvalProject(options: BuildEvalProjectOptions): EvalProjectB } export function buildWorkloadEvalProject(options: BuildWorkloadEvalProjectOptions): WorkloadEvalProjectBuildResult { - if (options.verifiedReceipt.cumulative_exported === 0) { + if (options.source.captureCount === 0) { throw new Error("No captures were exported for the frozen workload window; refusing to create an empty eval project."); } - const projectRoot = resolve(options.output); - const sourceRoot = join(projectRoot, "source"); - createPrivateDirectory(sourceRoot); - if ( - JSON.stringify(options.verifiedReceipt.canonical_scope) !== JSON.stringify(options.canonicalScope) || - options.verifiedReceipt.chain_id.length === 0 || - options.verifiedReceipt.manifest_sha256 !== options.segmentManifestSha256.at(-1) - ) throw new Error("Verified export receipt does not match the materialized source chain."); - - const unique = new Map(); - for (const file of options.verifiedFiles) { - const previous = unique.get(file.capture_key); - if (previous && JSON.stringify(previous) !== JSON.stringify(file)) { - throw new Error(`Capture source ledger conflicts for ${file.capture_key}.`); - } - unique.set(file.capture_key, file); + if (options.source.requestedCount !== options.source.captureCount + options.source.skippedCount) { + throw new Error("Raw trace source requested count must equal its materialized and skipped capture counts."); } - const files = [...unique.values()]; - const uniqueTotalBytes = files.reduce((sum, file) => sum + file.size_bytes, 0); + const projectRoot = resolve(options.output); + const scope = options.source.canonicalScope; if ( - files.length !== options.verifiedReceipt.cumulative_exported || - uniqueTotalBytes !== options.verifiedReceipt.total_bytes - ) { - throw new Error("Verified export receipt totals do not match unique materialized captures."); + scope.org_id !== options.identity.org_id || + scope.project_id !== options.identity.project_id || + scope.workload_id !== options.identity.workload_id + ) throw new Error("Raw trace source scope does not match the eval project identity."); + const indexPath = resolve(options.source.indexPath); + if (indexPath !== join(projectRoot, "source", "index.jsonl")) { + throw new Error("Raw trace source index must be source/index.jsonl inside the eval project."); } - const indexBody = files.map((file) => JSON.stringify(file)).join("\n") + (files.length > 0 ? "\n" : ""); - const indexSha256 = sourceIndexCommitmentSha256(files); - if (options.verifiedReceipt.local_index_sha256 !== indexSha256) { - throw new Error("Verified export receipt source index commitment does not match materialized captures."); - } - const indexPath = join(sourceRoot, "index.jsonl"); - replacePrivateText(indexPath, indexBody); - const proofPath = join(sourceRoot, "export-proof.json"); - const proofBody = `${JSON.stringify({ - schema_version: "understudy.eval-export-proof.v1", - canonical_scope: options.canonicalScope, - segment_manifest_sha256: options.segmentManifestSha256, - terminal_receipt: options.terminalReceipt, - verified_receipt: options.verifiedReceipt, - }, null, 2)}\n`; - replacePrivateText(proofPath, proofBody); const projectFile = join(projectRoot, "eval-project.json"); const evalId = deriveWorkloadEvalId({ name: options.name, identity: options.identity, - sourceWindow: options.canonicalScope, + sourceWindow: scope, }); const project: WorkloadEvalProjectManifest = { schema_version: "understudy.eval-project.v2", @@ -278,16 +241,15 @@ export function buildWorkloadEvalProject(options: BuildWorkloadEvalProjectOption created_at: options.now.toISOString(), identity: options.identity, source: { - window: options.canonicalScope, - capture_count: files.length, - size_bytes: uniqueTotalBytes, + window: scope, + requested_count: options.source.requestedCount, + materialized_count: options.source.captureCount, + skipped_count: options.source.skippedCount, + skipped_index: portableRelative(projectRoot, join(projectRoot, "source", "skipped.jsonl")), + capture_count: options.source.captureCount, + size_bytes: options.source.sizeBytes, index: portableRelative(projectRoot, indexPath), - index_sha256: indexSha256, - export_proof: portableRelative(projectRoot, proofPath), - export_proof_sha256: createHash("sha256").update(proofBody).digest("hex"), - exported_capture_count: options.verifiedReceipt.cumulative_exported, - exported_total_bytes: options.verifiedReceipt.total_bytes, - terminal_receipt_verified: true, + index_sha256: options.source.indexSha256, }, artifacts: { workload_profile: "workload-profile.md", @@ -314,14 +276,3 @@ export function buildWorkloadEvalProject(options: BuildWorkloadEvalProjectOption replacePrivateText(projectFile, `${JSON.stringify(project, null, 2)}\n`); return { ...project, project_file: projectFile }; } - -function replacePrivateText(path: string, body: string): void { - const temporary = `${path}.tmp-${randomUUID()}`; - try { - writeFileSync(temporary, body, { encoding: "utf8", mode: 0o600, flag: "wx" }); - renameSync(temporary, path); - chmodSync(path, 0o600); - } finally { - rmSync(temporary, { force: true }); - } -} diff --git a/src/evals/authoring-contracts.ts b/src/evals/authoring-contracts.ts index 8b4f00e1..a2ed8536 100644 --- a/src/evals/authoring-contracts.ts +++ b/src/evals/authoring-contracts.ts @@ -2,8 +2,7 @@ import { z } from "zod"; import { Sha256Schema, - VerifiedWorkloadCaptureFileSchema, - VerifyWorkloadCaptureExportReceiptResponseSchema, + WorkloadSourceRowSchema, WorkloadCaptureExportScopeSchema, } from "./contracts.js"; import { EvalReleaseArtifactPathSchema } from "./release-contracts.js"; @@ -40,16 +39,22 @@ export const WorkloadEvalProjectSchema = z.object({ }).strict(), source: z.object({ window: WorkloadCaptureExportScopeSchema.strict(), + requested_count: z.number().int().nonnegative(), + materialized_count: z.number().int().nonnegative(), + skipped_count: z.number().int().nonnegative(), + skipped_index: RelativeArtifactPathSchema, capture_count: z.number().int().nonnegative(), size_bytes: z.number().int().nonnegative(), index: RelativeArtifactPathSchema, index_sha256: Sha256Schema, - export_proof: RelativeArtifactPathSchema, - export_proof_sha256: Sha256Schema, - exported_capture_count: z.number().int().nonnegative(), - exported_total_bytes: z.number().int().nonnegative(), - terminal_receipt_verified: z.literal(true), - }).strict(), + }).strict().superRefine((source, context) => { + if (source.capture_count !== source.materialized_count) { + context.addIssue({ code: "custom", message: "capture_count must equal materialized_count" }); + } + if (source.requested_count !== source.materialized_count + source.skipped_count) { + context.addIssue({ code: "custom", message: "requested_count must equal materialized_count plus skipped_count" }); + } + }), artifacts: EvalProjectArtifactsSchema, authoring: z.object({ owner: z.literal("coding_agent"), @@ -63,16 +68,8 @@ export const WorkloadEvalProjectSchema = z.object({ }).strict(), }).strict(); -export const EvalSourceRowSchema = VerifiedWorkloadCaptureFileSchema.strict(); - -export const EvalExportProofSchema = z.object({ - schema_version: z.literal("understudy.eval-export-proof.v1"), - canonical_scope: WorkloadCaptureExportScopeSchema.strict(), - segment_manifest_sha256: z.array(Sha256Schema).min(1), - terminal_receipt: z.string().min(1), - verified_receipt: VerifyWorkloadCaptureExportReceiptResponseSchema.extend({ - canonical_scope: WorkloadCaptureExportScopeSchema.strict(), - }).strict(), +export const EvalSourceRowSchema = WorkloadSourceRowSchema.extend({ + local_path: RelativeArtifactPathSchema, }).strict(); const EvalExecutionSourceFileSchema = z.object({ @@ -325,7 +322,6 @@ export const EvalCheckReportSchema = z.object({ scope: WorkloadCaptureExportScopeSchema.strict(), scope_sha256: Sha256Schema, index_sha256: Sha256Schema, - export_proof_sha256: Sha256Schema, capture_count: z.number().int().nonnegative(), size_bytes: z.number().int().nonnegative(), }).strict(), @@ -373,5 +369,4 @@ export type EvalApproval = z.infer; export type EvalCheckReport = z.infer; export type EvalDraftSemanticAssumption = z.infer; export type EvalDraftCheckReport = z.infer; -export type EvalExportProof = z.infer; export type EvalExecutionIndexRow = z.infer; diff --git a/src/evals/build-state.ts b/src/evals/build-state.ts index 3fc7fcd1..68c22f16 100644 --- a/src/evals/build-state.ts +++ b/src/evals/build-state.ts @@ -3,21 +3,6 @@ import { spawnSync } from "node:child_process"; import { chmodSync, lstatSync, mkdirSync, readFileSync, realpathSync, renameSync, rmSync, writeFileSync } from "node:fs"; import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; -import { - EvalBuildCreatingStateSchema, - EvalBuildStateSchema, - EvalWorkloadBuildStateSchema, - type CatalogResponse, - type Cohort, - type EvalBuildCreatingState, - type EvalBuildIdentity, - type EvalLegacyBuildState, - type EvalBuildSelection, - type EvalBuildState, - type EvalWorkloadBuildState, - type FrozenCohort, -} from "./contracts.js"; - export function pathExists(path: string): boolean { try { lstatSync(path); @@ -47,9 +32,13 @@ export function writePrivateJson(path: string, value: unknown): void { } export function replacePrivateJson(path: string, value: unknown): void { + replacePrivateText(path, `${JSON.stringify(value, null, 2)}\n`); +} + +export function replacePrivateText(path: string, body: string): void { const temporary = `${path}.tmp-${randomUUID()}`; try { - writePrivateJson(temporary, value); + writeFileSync(temporary, body, { encoding: "utf8", mode: 0o600, flag: "wx" }); renameSync(temporary, path); chmodSync(path, 0o600); } finally { @@ -57,164 +46,6 @@ export function replacePrivateJson(path: string, value: unknown): void { } } -export function initializeBuildCheckpoint(staging: string, state: EvalBuildState): void { - if (pathExists(staging)) throw new Error(`Eval build checkpoint already exists: ${staging}`); - const temporary = join(dirname(staging), `.${basename(staging)}.init-${randomUUID()}`); - try { - createPrivateDirectory(temporary); - writePrivateJson(join(temporary, "build-state.json"), state); - renameSync(temporary, staging); - } finally { - rmSync(temporary, { recursive: true, force: true }); - } -} - -export function readEvalBuildState(staging: string): EvalBuildState { - const stagingStat = lstatSync(staging); - if (stagingStat.isSymbolicLink() || !stagingStat.isDirectory()) { - throw new Error(`Eval build staging path must be a real directory: ${staging}`); - } - chmodSync(staging, 0o700); - const statePath = join(staging, "build-state.json"); - const stateStat = lstatSync(statePath); - if (stateStat.isSymbolicLink() || !stateStat.isFile()) { - throw new Error(`Eval build state must be a real file: ${statePath}`); - } - chmodSync(statePath, 0o600); - return EvalBuildStateSchema.parse(JSON.parse(readFileSync(statePath, "utf8"))); -} - -export function creatingBuildState( - name: string, - description: string | undefined, - identity: EvalBuildIdentity, - catalog: CatalogResponse, - selection: EvalBuildSelection, - maxAgeDays: number, - batchSize: number, - now: Date, -): EvalBuildCreatingState { - const operationId = randomUUID(); - return EvalBuildCreatingStateSchema.parse({ - schema_version: "understudy.eval-build-state.v1", - status: "cohort_creating", - created_at: now.toISOString(), - name, - identity, - selection, - create_request: { - operation_id: operationId, - name, - selection: { - source: "explicit_capture_references", - ...(description === undefined ? {} : { description }), - sampling_seed: catalog.selection.sample_seed, - }, - captures: catalog.captures.map(({ capture_key, request_id, content_sha256 }) => ({ capture_key, request_id, content_sha256 })), - }, - compile: { max_age_days: maxAgeDays, batch_size: batchSize }, - }); -} - -export function buildState( - status: "cohort_frozen" | "complete", - name: string, - identity: EvalBuildIdentity, - cohort: FrozenCohort, - selection: EvalBuildSelection, - maxAgeDays: number, - batchSize: number, - now: Date, -) { - return { - schema_version: "understudy.eval-build-state.v1" as const, - status, - created_at: now.toISOString(), - name, - identity, - selection, - cohort, - compile: { max_age_days: maxAgeDays, batch_size: batchSize }, - }; -} - -export function assertBuildStateMatches( - state: EvalBuildState, - name: string, - identity: EvalBuildIdentity, - selection: EvalBuildSelection, - maxAgeDays: number, - batchSize: number, -): asserts state is EvalLegacyBuildState { - if (state.schema_version !== "understudy.eval-build-state.v1") { - throw new Error("Existing eval build state uses a different build workflow."); - } - if (state.status === "complete" || state.name !== name) { - throw new Error("Existing eval build state does not match this resumable build."); - } - for (const key of ["org_id", "project_id", "workload_id"] as const) { - if (state.identity[key] !== identity[key]) { - throw new Error(`Existing eval build state does not match ${key}.`); - } - } - if (state.compile.max_age_days !== maxAgeDays || state.compile.batch_size !== batchSize) { - throw new Error("Existing eval build state does not match the compile options."); - } - if (JSON.stringify(state.selection) !== JSON.stringify(selection)) { - throw new Error("Existing eval build state does not match the capture selection options."); - } -} - -export function creatingWorkloadBuildState(input: { - name: string; - identity: EvalBuildIdentity; - source: EvalWorkloadBuildState["source"]; - maxAgeDays: number; - batchSize: number; - now: Date; -}): EvalWorkloadBuildState { - return EvalWorkloadBuildStateSchema.parse({ - schema_version: "understudy.eval-build-state.v2", - status: "downloading", - created_at: input.now.toISOString(), - name: input.name, - identity: input.identity, - source: input.source, - compile: { max_age_days: input.maxAgeDays, batch_size: input.batchSize }, - transport: { - resume_cursor: null, - chain_id: null, - next_segment_index: 0, - previous_manifest_sha256: null, - segment_manifest_sha256: [], - cumulative_exported: 0, - cumulative_total_bytes: 0, - terminal_receipt: null, - verified_files: [], - }, - }); -} - -export function assertWorkloadBuildStateMatches( - state: EvalBuildState, - name: string, - identity: EvalBuildIdentity, - maxAgeDays: number, - batchSize: number, -): asserts state is EvalWorkloadBuildState { - if (state.schema_version !== "understudy.eval-build-state.v2" || state.name !== name) { - throw new Error("Existing eval build state does not match this resumable full-corpus build."); - } - for (const key of ["org_id", "project_id", "workload_id"] as const) { - if (state.identity[key] !== identity[key]) { - throw new Error(`Existing eval build state does not match ${key}.`); - } - } - if (state.compile.max_age_days !== maxAgeDays || state.compile.batch_size !== batchSize) { - throw new Error("Existing eval build state does not match the build options."); - } -} - export function ensureUnderstudyGitExcluded(output: string): void { const absoluteOutput = resolve(output); let existing = resolve(output); @@ -254,10 +85,6 @@ export function ensureUnderstudyGitExcluded(output: string): void { writeFileSync(excludePath, `${current}${separator}/.understudy/\n`, { encoding: "utf8", mode: 0o600 }); } -export function cohortFromResponse(cohort: Cohort): FrozenCohort { - return { id: cohort.id, cohort_sha256: cohort.cohort_sha256, capture_count: cohort.capture_count }; -} - interface LeaseOwner { token: string; pid: number; diff --git a/src/evals/check.ts b/src/evals/check.ts index 2cba7be4..9fc546d4 100644 --- a/src/evals/check.ts +++ b/src/evals/check.ts @@ -21,7 +21,6 @@ import { EvalDraftMetricSchema, EvalEnvironmentSchema, EvalExecutionIndexRowSchema, - EvalExportProofSchema, EvalHarnessSchema, EvalMetricSchema, EvalSourceRowSchema, @@ -36,6 +35,7 @@ import { import { deriveWorkloadEvalId } from "../eval-project.js"; import { replacePrivateJson } from "./build-state.js"; import { sourceIndexCommitmentSha256 } from "./source-index.js"; +import { WorkloadTraceExportSkippedCaptureSchema } from "./contracts.js"; import { canonicalJson, compareCodeUnits } from "./canonical.js"; import { runInProviderFreeSandbox, @@ -276,18 +276,10 @@ function sameReport(left: EvalCheckReportResult, right: EvalCheckReportResult): return JSON.stringify(leftStable) === JSON.stringify(rightStable); } -function sameJson(left: unknown, right: unknown): boolean { - return canonicalJson(left) === canonicalJson(right); -} - -function assertExactSourceProof( - project: ReturnType, - proof: ReturnType, - proofSha256: string, -): void { +function assertExactSource(project: ReturnType): void { const windowStart = new Date(project.source.window.from).valueOf(); const windowEnd = new Date(project.source.window.to).valueOf(); - if (windowEnd - windowStart !== 7 * 86_400_000) throw new Error("Eval source window must be exactly seven days."); + if (windowEnd - windowStart !== 86_400_000) throw new Error("Eval source window must be exactly 24 hours."); if (Date.parse(project.source.window.ingestion_cutoff) < windowEnd) { throw new Error("Eval source ingestion cutoff must be at or after the frozen window end."); } @@ -296,40 +288,72 @@ function assertExactSourceProof( throw new Error(`Eval source window ${key} does not match project identity.`); } } - if (proofSha256 !== project.source.export_proof_sha256) throw new Error("Export proof hash does not match eval-project.json."); - if (!sameJson(proof.canonical_scope, project.source.window)) throw new Error("Export proof canonical scope does not match eval-project.json."); - if (!sameJson(proof.verified_receipt.canonical_scope, proof.canonical_scope)) { - throw new Error("Verified export receipt canonical scope does not match its proof."); - } - const receipt = proof.verified_receipt; - const expectedScopeHash = sha256(JSON.stringify(proof.canonical_scope)); - if (receipt.scope_hash !== expectedScopeHash) { - throw new Error("Verified export receipt scope hash does not match the canonical scope."); - } - if (proof.segment_manifest_sha256.length !== receipt.segment_index + 1) { - throw new Error("Export proof manifest chain length does not match the verified terminal segment."); - } - if (new Set(proof.segment_manifest_sha256).size !== proof.segment_manifest_sha256.length) { - throw new Error("Export proof manifest chain contains duplicate segment hashes."); +} + +type SourceReference = { + request_id: string; + capture_key: string; + captured_at: string; +}; + +function compareSourceReferences(left: SourceReference, right: SourceReference): number { + for (const key of ["captured_at", "request_id", "capture_key"] as const) { + if (left[key] < right[key]) return -1; + if (left[key] > right[key]) return 1; } - if (proof.segment_manifest_sha256.at(-1) !== receipt.manifest_sha256) { - throw new Error("Export proof terminal manifest does not match the verified receipt."); + return 0; +} + +function assertSourceReferenceScope( + project: ReturnType, + row: SourceReference, + label: string, +): void { + if ( + !row.capture_key.startsWith(`${project.identity.org_id}/${project.identity.project_id}/`) || + !row.capture_key.endsWith(`/${row.request_id}.jsonl`) + ) { + throw new Error(`${label} capture reference does not match request ${row.request_id}.`); } - const expectedPrevious = receipt.segment_index === 0 ? null : proof.segment_manifest_sha256.at(-2) ?? null; - if (receipt.previous_manifest_sha256 !== expectedPrevious) { - throw new Error("Export proof previous manifest does not match the verified receipt chain."); + const capturedAt = Date.parse(row.captured_at); + if (capturedAt < Date.parse(project.source.window.from) || capturedAt >= Date.parse(project.source.window.to)) { + throw new Error(`${label} capture ${row.request_id} falls outside the eval source window.`); } - if ( - receipt.cumulative_exported !== project.source.exported_capture_count || - receipt.total_bytes !== project.source.exported_total_bytes - ) throw new Error("Verified export receipt totals do not match eval-project.json."); - if (receipt.local_index_sha256 !== project.source.index_sha256) { - throw new Error("Verified export receipt source index commitment does not match eval-project.json."); +} + +function assertSourceReferenceIndexes( + project: ReturnType, + sourceRows: SourceReference[], + skippedRows: SourceReference[], +): void { + const requestIds = new Set(); + const captureKeys = new Set(); + let sourceIndex = 0; + let skippedIndex = 0; + let previous: SourceReference | null = null; + while (sourceIndex < sourceRows.length || skippedIndex < skippedRows.length) { + const source = sourceRows[sourceIndex]; + const skipped = skippedRows[skippedIndex]; + const useSource = source !== undefined && + (skipped === undefined || compareSourceReferences(source, skipped) <= 0); + const row = useSource ? source! : skipped!; + const label = useSource ? "Source" : "Skipped source"; + if (useSource) sourceIndex += 1; + else skippedIndex += 1; + assertSourceReferenceScope(project, row, label); + if (requestIds.has(row.request_id)) { + throw new Error(`Source and skipped indexes contain duplicate request id ${row.request_id}.`); + } + if (captureKeys.has(row.capture_key)) { + throw new Error(`Source and skipped indexes contain duplicate capture key ${row.capture_key}.`); + } + if (previous !== null && compareSourceReferences(previous, row) >= 0) { + throw new Error(`Source and skipped indexes are not strictly ordered at request ${row.request_id}.`); + } + requestIds.add(row.request_id); + captureKeys.add(row.capture_key); + previous = row; } - if ( - project.source.capture_count !== project.source.exported_capture_count || - project.source.size_bytes !== project.source.exported_total_bytes - ) throw new Error("Local eval source totals do not match the verified export totals."); } export async function runEvalCheck(projectInput: string, options: RunEvalCheckOptions = {}): Promise { @@ -342,7 +366,8 @@ export async function runEvalCheck(projectInput: string, options: RunEvalCheckOp } const expectedEvalId = deriveWorkloadEvalId({ name: project.name, identity: project.identity, sourceWindow: project.source.window }); if (project.eval_id !== expectedEvalId) throw new Error("Eval id does not match the project name, identity, and frozen source window."); - const declaredPaths = [project.source.index, project.source.export_proof, ...Object.values(project.artifacts)]; + assertExactSource(project); + const declaredPaths = [project.source.index, project.source.skipped_index, ...Object.values(project.artifacts)]; if (new Set(declaredPaths).size !== declaredPaths.length) throw new Error("Eval project artifact paths must be unique; duplicate aliases are not allowed."); if (declaredPaths.includes(DRAFT_CHECK_REPORT_PATH)) { throw new Error(`${DRAFT_CHECK_REPORT_PATH} is reserved for the distinct local draft check report.`); @@ -350,6 +375,19 @@ export async function runEvalCheck(projectInput: string, options: RunEvalCheckOp const indexPath = existingProjectPath(projectRoot, project.source.index, "source index"); const indexBytes = regularFile(indexPath, "source index"); + const skippedIndexPath = existingProjectPath(projectRoot, project.source.skipped_index, "skipped source index"); + const skippedRows = regularFile(skippedIndexPath, "skipped source index") + .toString("utf8").split(/\r?\n/).filter(Boolean).map((line, index) => { + let value: unknown; + try { value = JSON.parse(line); } + catch (error) { throw new Error(`Invalid skipped source index line ${index + 1}: ${error instanceof Error ? error.message : String(error)}`); } + const parsed = WorkloadTraceExportSkippedCaptureSchema.safeParse(value); + if (!parsed.success) throw new Error(`Invalid skipped source index line ${index + 1}: ${z.prettifyError(parsed.error)}`); + return parsed.data; + }); + if (skippedRows.length !== project.source.skipped_count) { + throw new Error("Skipped source index count does not match eval-project.json."); + } const sourceRows = indexBytes.toString("utf8").split(/\r?\n/).filter(Boolean).map((line, index) => { let value: unknown; try { value = JSON.parse(line); } @@ -363,15 +401,13 @@ export async function runEvalCheck(projectInput: string, options: RunEvalCheckOp } if (sourceRows.length !== project.source.capture_count) throw new Error("Source index capture count does not match eval-project.json."); if (sourceRows.reduce((sum, row) => sum + row.size_bytes, 0) !== project.source.size_bytes) throw new Error("Source index byte count does not match eval-project.json."); + assertSourceReferenceIndexes(project, sourceRows, skippedRows); const sourcePaths = new Set(); - const sourceCaptureKeys = new Set(); const sourceRowsByPath = new Map(); const sourceCapturePaths: string[] = []; for (const row of sourceRows) { if (sourcePaths.has(row.local_path)) throw new Error(`Source index contains duplicate local path ${row.local_path}.`); - if (sourceCaptureKeys.has(row.capture_key)) throw new Error(`Source index contains duplicate capture key ${row.capture_key}.`); sourcePaths.add(row.local_path); - sourceCaptureKeys.add(row.capture_key); sourceRowsByPath.set(row.local_path, row); const capturePath = existingProjectPath(projectRoot, row.local_path, "source capture"); sourceCapturePaths.push(capturePath); @@ -380,15 +416,6 @@ export async function runEvalCheck(projectInput: string, options: RunEvalCheckOp throw new Error(`Source capture integrity check failed for request ${row.request_id}.`); } } - const proofPath = existingProjectPath(projectRoot, project.source.export_proof, "export proof"); - const proofBytes = regularFile(proofPath, "export proof"); - const proof = parseJson(proofPath, EvalExportProofSchema, "export-proof.json"); - const localExportProofSha256 = sha256(proofBytes); - assertExactSourceProof(project, proof, localExportProofSha256); - // The project hash protects the private proof file. The report field with - // the same legacy name is the durable backend-verifiable attestation hash. - const sourceAttestationSha256 = sha256(proof.verified_receipt.source_attestation); - const profilePath = existingProjectPath(projectRoot, project.artifacts.workload_profile, "workload profile"); const profileBytes = regularFile(profilePath, "workload profile"); if (profileBytes.toString("utf8").trim().length < 20) throw new Error("Workload profile is missing or too short to record inferred or confirmed intent."); @@ -539,12 +566,11 @@ export async function runEvalCheck(projectInput: string, options: RunEvalCheckOp resolve(projectRoot, project.artifacts.check_report), resolve(projectRoot, DRAFT_CHECK_REPORT_PATH), ]); - if ([indexPath, proofPath, ...sourceCapturePaths, ...fixtureDataPaths].some((path) => generatedReportPaths.has(path))) { + if ([indexPath, ...sourceCapturePaths, ...fixtureDataPaths].some((path) => generatedReportPaths.has(path))) { throw new Error("Source and fixture data cannot alias generated check report paths."); } const protectedPaths = [ indexPath, - proofPath, ...sourceCapturePaths, ...fixtureDataPaths, ...generatedReportPaths, @@ -587,9 +613,8 @@ export async function runEvalCheck(projectInput: string, options: RunEvalCheckOp const verifierSha256 = execution.verifier.sha256; const sourceBinding = { scope: project.source.window, - scope_sha256: proof.verified_receipt.scope_hash, + scope_sha256: sha256(JSON.stringify(project.source.window)), index_sha256: project.source.index_sha256, - export_proof_sha256: sourceAttestationSha256, capture_count: project.source.capture_count, size_bytes: project.source.size_bytes, }; diff --git a/src/evals/contracts.ts b/src/evals/contracts.ts index 62173eef..a433a5da 100644 --- a/src/evals/contracts.ts +++ b/src/evals/contracts.ts @@ -75,201 +75,51 @@ export const WorkloadCaptureExportScopeSchema = z.object({ from: z.string().datetime(), to: z.string().datetime(), ingestion_cutoff: z.string().datetime(), -}); - -export const WorkloadCaptureExportManifestItemSchema = z.object({ - request_id: z.string().min(1), - key: z.string().min(1), - size: z.number().int().nonnegative(), - content_sha256: Sha256Schema, - url: z.string().url(), -}); - -export const WorkloadCaptureExportManifestHeaderSchema = z.object({ - record_type: z.literal("understudy_capture_export_chain_v1"), - chain_id: z.string().min(1), - segment_id: Sha256Schema, - segment_index: z.number().int().nonnegative(), - previous_manifest_sha256: Sha256Schema.nullable(), - cumulative_scanned: z.number().int().nonnegative(), - cumulative_matched: z.number().int().nonnegative(), - cumulative_exported: z.number().int().nonnegative(), - cumulative_total_bytes: z.number().int().nonnegative(), - terminal: z.boolean(), -}); - -export const WorkloadCaptureExportResponseSchema = z.object({ - export_id: z.string().min(1), - count: z.number().int().nonnegative(), - total_bytes: z.number().int().nonnegative(), - manifest_url: z.string().url(), - expires_at: z.string().datetime(), - truncated: z.boolean(), - resume_cursor: z.string().min(1).optional(), - canonical_scope: WorkloadCaptureExportScopeSchema, - chain: z.object({ - chain_id: z.string().min(1), - segment_id: Sha256Schema, - segment_index: z.number().int().nonnegative(), - previous_manifest_sha256: Sha256Schema.nullable(), - manifest_sha256: Sha256Schema, - cumulative_scanned: z.number().int().nonnegative(), - cumulative_matched: z.number().int().nonnegative(), - cumulative_exported: z.number().int().nonnegative(), - cumulative_total_bytes: z.number().int().nonnegative(), - local_index_sha256: Sha256Schema, - terminal: z.boolean(), - terminal_receipt: z.string().min(1).optional(), - }), -}); +}).strict(); -export const VerifyWorkloadCaptureExportReceiptResponseSchema = z.object({ - verified: z.literal(true), - scope_hash: Sha256Schema, - chain_id: z.string().min(1), - segment_id: Sha256Schema, - segment_index: z.number().int().nonnegative(), - manifest_sha256: Sha256Schema, - previous_manifest_sha256: Sha256Schema.nullable(), - cumulative_scanned: z.number().int().nonnegative(), - cumulative_matched: z.number().int().nonnegative(), - cumulative_exported: z.number().int().nonnegative(), - total_bytes: z.number().int().nonnegative(), - local_index_sha256: Sha256Schema, - expires_at: z.string().datetime(), - canonical_scope: WorkloadCaptureExportScopeSchema, - source_attestation: z.string().min(1).max(8_192), -}); +export const WorkloadTraceWindowBindingSchema = z.object({ + schema_version: z.literal("understudy.trace-export-window.v1"), + selection_mode: z.enum(["rolling", "date"]), + org_id: z.string().min(1), + project_id: z.string().min(1), + workload_id: z.string().min(1), + from: z.string().datetime(), + to: z.string().datetime(), +}).strict(); -export const VerifiedWorkloadCaptureFileSchema = z.object({ +export const WorkloadSourceRowSchema = z.object({ schema_version: z.literal(EVAL_SOURCE_ROW_SCHEMA_VERSION), request_id: z.string().min(1), capture_key: z.string().min(1), + captured_at: z.string().datetime(), size_bytes: z.number().int().nonnegative(), content_sha256: Sha256Schema, local_path: z.string().min(1), -}); - -export const EvalBuildStateBaseSchema = z.object({ - schema_version: z.literal("understudy.eval-build-state.v1"), - created_at: z.string().datetime(), - name: z.string().min(1).max(120), - identity: z.object({ - org_id: z.string(), - project_id: z.string(), - workload_id: z.string(), - workload_name: z.string(), - }), - compile: z.object({ - max_age_days: z.number().int().positive(), - batch_size: z.number().int().positive(), - }), - selection: z.object({ - last: z.string(), - limit: z.number().int().min(1).max(100), - seed: z.string(), - description: z.string().min(1).max(1000).nullable(), - requested_model: z.string().nullable(), - served_model: z.string().nullable(), - status_code: z.number().int().min(100).max(599).nullable(), - requires_tools: z.boolean(), - requires_structured_output: z.boolean(), - }), -}); - -export const FrozenCohortSchema = z.object({ - id: z.string(), - cohort_sha256: Sha256Schema, - capture_count: z.number().int().positive(), -}); +}).strict(); -export const EvalBuildCreatingStateSchema = EvalBuildStateBaseSchema.extend({ - status: z.literal("cohort_creating"), - create_request: z.object({ - operation_id: z.string().uuid(), - name: z.string().min(1).max(120), - selection: CohortSelectionSchema, - captures: z.array(z.object({ - capture_key: z.string(), - request_id: z.string(), - content_sha256: Sha256Schema, - })).min(1).max(500), - }), -}); - -export const EvalBuildFrozenStateSchema = EvalBuildStateBaseSchema.extend({ - status: z.enum(["cohort_frozen", "complete"]), - cohort: FrozenCohortSchema, -}); - -export const EvalLegacyBuildStateSchema = z.discriminatedUnion("status", [ - EvalBuildCreatingStateSchema, - EvalBuildFrozenStateSchema, -]); - -export const EvalWorkloadBuildStateSchema = z.object({ - schema_version: z.literal("understudy.eval-build-state.v2"), - status: z.enum(["downloading", "receipt_pending", "complete"]), - created_at: z.string().datetime(), - name: z.string().min(1).max(120), - identity: z.object({ - org_id: z.string().min(1), - project_id: z.string().min(1), - workload_id: z.string().min(1), - workload_name: z.string().min(1), - }), - source: z.object({ - from: z.string().datetime(), - to: z.string().datetime(), - ingestion_cutoff: z.string().datetime().nullable(), - }).superRefine((source, context) => { - if ( - source.ingestion_cutoff !== null && - Date.parse(source.ingestion_cutoff) < Date.parse(source.to) - ) { - context.addIssue({ - code: "custom", - path: ["ingestion_cutoff"], - message: "the frozen ingestion cutoff must be at or after the source window end", - }); - } - }), - compile: z.object({ - max_age_days: z.number().int().positive(), - batch_size: z.number().int().positive(), - }), - transport: z.object({ - resume_cursor: z.string().min(1).nullable(), - chain_id: z.string().min(1).nullable(), - next_segment_index: z.number().int().nonnegative(), - previous_manifest_sha256: Sha256Schema.nullable(), - segment_manifest_sha256: z.array(Sha256Schema), - cumulative_exported: z.number().int().nonnegative(), - cumulative_total_bytes: z.number().int().nonnegative(), - terminal_receipt: z.string().min(1).nullable(), - verified_files: z.array(VerifiedWorkloadCaptureFileSchema), - }), +/** A capture requested from the frozen source window but unavailable from R2. */ +export const WorkloadTraceExportSkippedCaptureSchema = z.object({ + request_id: z.string().min(1), + capture_key: z.string().min(1), + captured_at: z.string().datetime(), + reason: z.literal("not_found"), +}).strict(); +export type WorkloadTraceExportSkippedCapture = z.infer< + typeof WorkloadTraceExportSkippedCaptureSchema +>; + +export const EvalBuildIdentitySchema = z.object({ + org_id: z.string().min(1), + project_id: z.string().min(1), + workload_id: z.string().min(1), + workload_name: z.string().min(1), }); -export const EvalBuildStateSchema = z.union([ - EvalLegacyBuildStateSchema, - EvalWorkloadBuildStateSchema, -]); - export type CatalogItem = z.infer; export type CatalogResponse = z.infer; export type Cohort = z.infer; export type CohortExport = z.infer; -export type EvalBuildState = z.infer; -export type EvalLegacyBuildState = z.infer; -export type EvalBuildCreatingState = z.infer; -export type EvalBuildIdentity = z.infer["identity"]; -export type EvalBuildSelection = z.infer["selection"]; -export type FrozenCohort = z.infer; +export type EvalBuildIdentity = z.infer; export type WorkloadCaptureExportScope = z.infer; -export type WorkloadCaptureExportManifestItem = z.infer; -export type WorkloadCaptureExportManifestHeader = z.infer; -export type WorkloadCaptureExportResponse = z.infer; -export type VerifyWorkloadCaptureExportReceiptResponse = z.infer; -export type VerifiedWorkloadCaptureFile = z.infer; -export type EvalWorkloadBuildState = z.infer; +export type WorkloadSourceRow = z.infer; +export type WorkloadTraceWindowBinding = z.infer; diff --git a/src/evals/materialize.ts b/src/evals/materialize.ts index 7d7c9998..5075b775 100644 --- a/src/evals/materialize.ts +++ b/src/evals/materialize.ts @@ -1,17 +1,9 @@ import { createHash } from "node:crypto"; -import { closeSync, createReadStream, lstatSync, openSync, renameSync, rmSync, writeSync } from "node:fs"; -import { dirname, join, relative, resolve, sep } from "node:path"; +import { closeSync, openSync, renameSync, rmSync, writeSync } from "node:fs"; +import { join, resolve } from "node:path"; import { createPrivateDirectory, pathExists, writePrivateJson } from "./build-state.js"; -import { - WorkloadCaptureExportManifestHeaderSchema, - WorkloadCaptureExportManifestItemSchema, - type CohortExport, - type VerifiedWorkloadCaptureFile, - type WorkloadCaptureExportManifestHeader, - type WorkloadCaptureExportManifestItem, - type WorkloadCaptureExportResponse, -} from "./contracts.js"; +import { type CohortExport } from "./contracts.js"; export const EXPORT_EXPIRES_SECONDS = 3600; const CAPTURE_DOWNLOAD_TIMEOUT_MS = 60_000; @@ -191,235 +183,7 @@ export function reserveDownloadedChunk( return captureBytes + chunkBytes; } -export function reserveReceiptDrivenChunk( - requestId: string, - captureBytes: number, - chunkBytes: number, - expectedBytes: number, -): number { - const next = captureBytes + chunkBytes; - if ( - !Number.isSafeInteger(captureBytes) || captureBytes < 0 || - !Number.isSafeInteger(chunkBytes) || chunkBytes < 0 || - !Number.isSafeInteger(expectedBytes) || expectedBytes < 0 || - !Number.isSafeInteger(next) || next > expectedBytes - ) { - throw new Error(`Capture ${requestId} exceeds its authenticated ${expectedBytes}-byte manifest size.`); - } - return next; -} - -export async function materializeWorkloadExportSegment(input: { - exportData: WorkloadCaptureExportResponse; - tracesDirectory: string; - gatewayUrl: string; - verifiedFiles: VerifiedWorkloadCaptureFile[]; - onVerified: (file: VerifiedWorkloadCaptureFile) => void | Promise; -}): Promise<{ - header: WorkloadCaptureExportManifestHeader; - items: WorkloadCaptureExportManifestItem[]; - manifest_sha256: string; -}> { - const manifestUrl = allowedCaptureUrl(input.exportData.manifest_url, input.gatewayUrl); - const manifestResponse = await fetch(manifestUrl, { - headers: { Accept: "application/x-ndjson" }, - redirect: "error", - signal: AbortSignal.timeout(CAPTURE_DOWNLOAD_TIMEOUT_MS), - }); - if (!manifestResponse.ok) { - throw new Error(`Capture export manifest download failed with status ${manifestResponse.status}.`); - } - const manifestBody = await manifestResponse.text(); - const manifestSha256 = createHash("sha256").update(manifestBody).digest("hex"); - if (manifestSha256 !== input.exportData.chain.manifest_sha256) { - throw new Error("Capture export manifest failed SHA-256 verification."); - } - const lines = manifestBody.split("\n").filter((line) => line.length > 0); - if (lines.length === 0) throw new Error("Capture export manifest is empty."); - const header = WorkloadCaptureExportManifestHeaderSchema.parse(JSON.parse(lines[0]!)); - const items = lines.slice(1).map((line) => WorkloadCaptureExportManifestItemSchema.parse(JSON.parse(line))); - assertWorkloadManifestLineage(input.exportData, header, items, manifestSha256); - - const tracesDirectory = resolve(input.tracesDirectory); - const projectRoot = dirname(dirname(tracesDirectory)); - createPrivateDirectory(tracesDirectory); - const verifiedByKey = new Map(); - for (const file of input.verifiedFiles) { - const previous = verifiedByKey.get(file.capture_key); - if (previous && JSON.stringify(previous) !== JSON.stringify(file)) { - throw new Error(`Verified capture ledger contains conflicting entries for ${file.capture_key}.`); - } - verifiedByKey.set(file.capture_key, file); - } - - for (const item of items) { - const fileName = portableCaptureFileName( - item.request_id, - `-${createHash("sha256").update(item.key).digest("hex").slice(0, 12)}`, - ); - const expectedLocalPath = relative(projectRoot, join(tracesDirectory, fileName)).split(sep).join("/"); - const existing = verifiedByKey.get(item.key); - if (existing) { - if ( - existing.request_id !== item.request_id || existing.size_bytes !== item.size || - existing.content_sha256 !== item.content_sha256 || existing.local_path !== expectedLocalPath - ) throw new Error(`Verified capture ledger does not match export item ${item.request_id}.`); - const existingPath = resolveLedgerPath(projectRoot, existing.local_path); - const hashed = await hashLocalCapture(existingPath); - if (hashed.sizeBytes !== existing.size_bytes || hashed.digest !== existing.content_sha256) { - throw new Error(`Verified local capture ${item.request_id} no longer matches its ledger.`); - } - continue; - } - - const finalPath = join(tracesDirectory, fileName); - if (pathExists(finalPath)) { - const recovered = await hashLocalCapture(finalPath); - if (recovered.sizeBytes !== item.size || recovered.digest !== item.content_sha256) { - throw new Error(`Untracked capture file does not match export item ${item.request_id}.`); - } - const verified: VerifiedWorkloadCaptureFile = { - schema_version: "understudy.eval-source-capture.v1", - request_id: item.request_id, - capture_key: item.key, - size_bytes: recovered.sizeBytes, - content_sha256: recovered.digest, - local_path: expectedLocalPath, - }; - verifiedByKey.set(item.key, verified); - await input.onVerified(verified); - continue; - } - const downloaded = await downloadReceiptDrivenCapture(item, finalPath, input.gatewayUrl); - const verified: VerifiedWorkloadCaptureFile = { - schema_version: "understudy.eval-source-capture.v1", - request_id: item.request_id, - capture_key: item.key, - size_bytes: downloaded.sizeBytes, - content_sha256: downloaded.digest, - local_path: expectedLocalPath, - }; - verifiedByKey.set(item.key, verified); - await input.onVerified(verified); - } - return { header, items, manifest_sha256: manifestSha256 }; -} - -function assertWorkloadManifestLineage( - exportData: WorkloadCaptureExportResponse, - header: WorkloadCaptureExportManifestHeader, - items: WorkloadCaptureExportManifestItem[], - manifestSha256: string, -): void { - const chain = exportData.chain; - for (const key of ["chain_id", "segment_id", "segment_index", "previous_manifest_sha256", "terminal"] as const) { - if (header[key] !== chain[key]) throw new Error(`Capture export manifest ${key} does not match its response.`); - } - if ( - manifestSha256 !== chain.manifest_sha256 || - header.cumulative_scanned !== chain.cumulative_scanned || - header.cumulative_matched !== chain.cumulative_matched || - header.cumulative_exported !== chain.cumulative_exported || - header.cumulative_total_bytes !== chain.cumulative_total_bytes - ) throw new Error("Capture export manifest cumulative lineage does not match its response."); - const totalBytes = items.reduce((sum, item) => sum + item.size, 0); - if (items.length !== exportData.count || totalBytes !== exportData.total_bytes) { - throw new Error("Capture export manifest totals do not match its response."); - } - if (chain.terminal === exportData.truncated) { - throw new Error("Capture export terminal state is inconsistent."); - } - if (chain.terminal) { - if (!chain.terminal_receipt || exportData.resume_cursor) { - throw new Error("Terminal capture export segment is missing its receipt."); - } - } else if (!exportData.resume_cursor || chain.terminal_receipt) { - throw new Error("Non-terminal capture export segment is missing its resume cursor."); - } -} - -async function downloadReceiptDrivenCapture( - item: WorkloadCaptureExportManifestItem, - finalPath: string, - gatewayUrl: string, -): Promise<{ digest: string; sizeBytes: number }> { - const url = allowedCaptureUrl(item.url, gatewayUrl); - const partialPath = `${finalPath}.partial`; - let descriptor: number | null = null; - let complete = false; - try { - const download = await fetch(url, { - headers: { Accept: "application/x-ndjson" }, - redirect: "error", - signal: AbortSignal.timeout(CAPTURE_DOWNLOAD_TIMEOUT_MS), - }); - if (!download.ok) throw new Error(`Capture ${item.request_id} download failed with status ${download.status}.`); - const declaredLength = download.headers.get("content-length"); - if (declaredLength !== null && Number(declaredLength) !== item.size) { - throw new Error(`Capture ${item.request_id} content length does not match its authenticated manifest size.`); - } - if (!download.body) throw new Error(`Capture ${item.request_id} download returned no body.`); - descriptor = openSync(partialPath, "wx", 0o600); - const hash = createHash("sha256"); - const reader = download.body.getReader(); - let sizeBytes = 0; - while (true) { - const chunk = await reader.read(); - if (chunk.done) break; - try { - sizeBytes = reserveReceiptDrivenChunk(item.request_id, sizeBytes, chunk.value.byteLength, item.size); - } catch (error) { - await reader.cancel(); - throw error; - } - hash.update(chunk.value); - let written = 0; - while (written < chunk.value.byteLength) { - const count = writeSync(descriptor, chunk.value, written, chunk.value.byteLength - written); - if (count <= 0) throw new Error(`Capture ${item.request_id} could not be written completely.`); - written += count; - } - } - if (sizeBytes !== item.size) { - throw new Error(`Capture ${item.request_id} ended before its authenticated manifest size.`); - } - closeSync(descriptor); - descriptor = null; - const digest = hash.digest("hex"); - if (digest !== item.content_sha256) { - throw new Error(`Capture ${item.request_id} failed authenticated SHA-256 verification.`); - } - renameSync(partialPath, finalPath); - complete = true; - return { digest, sizeBytes }; - } finally { - if (descriptor !== null) closeSync(descriptor); - if (!complete) rmSync(partialPath, { force: true }); - } -} - -function resolveLedgerPath(projectRoot: string, localPath: string): string { - const absolute = resolve(projectRoot, localPath); - const relativePath = relative(projectRoot, absolute); - if (!relativePath || relativePath === ".." || relativePath.startsWith(`..${sep}`)) { - throw new Error("Verified capture ledger path leaves the eval project."); - } - return absolute; -} - -async function hashLocalCapture(path: string): Promise<{ digest: string; sizeBytes: number }> { - const stat = lstatSync(path); - if (stat.isSymbolicLink() || !stat.isFile()) throw new Error(`Verified capture must be a real file: ${path}.`); - const hash = createHash("sha256"); - let sizeBytes = 0; - for await (const chunk of createReadStream(path)) { - sizeBytes += chunk.length; - hash.update(chunk); - } - return { digest: hash.digest("hex"), sizeBytes }; -} - -function allowedCaptureUrl(raw: string, gatewayUrl: string): string { +export function allowedCaptureUrl(raw: string, gatewayUrl: string): string { const url = new URL(raw); if (url.username || url.password) throw new Error("Capture download URL must not contain credentials."); const trustedR2 = url.protocol === "https:" && (url.port === "" || url.port === "443") && /^[a-z0-9-]+\.r2\.cloudflarestorage\.com$/i.test(url.hostname); @@ -430,7 +194,7 @@ function allowedCaptureUrl(raw: string, gatewayUrl: string): string { throw new Error(`Refusing capture download from untrusted origin ${url.origin}.`); } -function portableCaptureFileName(requestId: string, suffix = ""): string { +export function portableCaptureFileName(requestId: string, suffix = ""): string { const tail = `${suffix}${CAPTURE_FILE_EXTENSION}`; const stemBytes = MAX_PORTABLE_FILE_NAME_BYTES - Buffer.byteLength(tail); if (stemBytes < 1) throw new Error(`Capture filename suffix is too long for request ${requestId}.`); diff --git a/src/evals/publish.ts b/src/evals/publish.ts index 6a3542b6..2ec0e215 100644 --- a/src/evals/publish.ts +++ b/src/evals/publish.ts @@ -20,7 +20,6 @@ import { EvalCheckFixturesSchema, EvalCheckReportSchema, EvalEnvironmentSchema, - EvalExportProofSchema, EvalHarnessSchema, EvalSourceRowSchema, WorkloadEvalProjectSchema, @@ -371,17 +370,10 @@ export async function prepareEvalPublication( const sourceIndexEntry = readStableFile(projectRoot, project.source.index, "source index", null); const sourceIndex = parseSourceIndex(sourceIndexEntry); assertHash("Source index commitment", sourceIndex.commitmentSha256, project.source.index_sha256); - const exportProofEntry = readStableFile(projectRoot, project.source.export_proof, "export proof", null); - assertHash("Export proof", exportProofEntry.sha256, project.source.export_proof_sha256); - const exportProof = parseJson(exportProofEntry.bytes, EvalExportProofSchema, "source/export-proof.json"); - assertHash("Verified source index commitment", exportProof.verified_receipt.local_index_sha256, project.source.index_sha256); - const sourceAttestation = exportProof.verified_receipt.source_attestation; - const sourceAttestationSha256 = sha256(sourceAttestation); const sourcePaths = sourceIndex.paths; const forbiddenPaths = new Set([ "eval-project.json", project.source.index, - project.source.export_proof, project.artifacts.execution_index, project.artifacts.analysis, ...sourcePaths, @@ -473,7 +465,7 @@ export async function prepareEvalPublication( if (canonicalJson(checkReport) !== canonicalJson(checked.report)) { throw new Error("Snapshotted check report does not match the passing eval check."); } - assertHash("Checked source attestation", checkReport.source.export_proof_sha256, sourceAttestationSha256); + assertHash("Checked source scope", checkReport.source.scope_sha256, sha256(JSON.stringify(project.source.window))); assertHash("Checked source index commitment", checkReport.source.index_sha256, project.source.index_sha256); const assertFixtureBinding = ( @@ -522,8 +514,6 @@ export async function prepareEvalPublication( capture_count: project.source.capture_count, total_bytes: project.source.size_bytes, local_index_sha256: project.source.index_sha256, - export_proof_sha256: sourceAttestationSha256, - source_attestation: sourceAttestation, }, artifacts: { eval_set_sha256: checked.hashes.eval_set_sha256, @@ -573,7 +563,6 @@ export async function prepareEvalPublication( "eval-project.json", project.artifacts.analysis, project.artifacts.execution_index, - project.source.export_proof, project.source.index, "source/", "traces/", diff --git a/src/evals/release-contracts.ts b/src/evals/release-contracts.ts index a2cedc58..f3df7391 100644 --- a/src/evals/release-contracts.ts +++ b/src/evals/release-contracts.ts @@ -32,11 +32,9 @@ export const EvalReleaseSourceSchema = z.object({ capture_count: z.number().int().nonnegative(), total_bytes: z.number().int().nonnegative(), local_index_sha256: EvalReleaseSha256Schema, - export_proof_sha256: EvalReleaseSha256Schema, - source_attestation: z.string().min(1).max(8_192), }).strict().superRefine((source, context) => { - if (Date.parse(source.to) - Date.parse(source.from) !== 7 * 24 * 60 * 60 * 1_000) { - context.addIssue({ code: "custom", path: ["to"], message: "the source window must be exactly seven days" }); + if (Date.parse(source.to) - Date.parse(source.from) !== 24 * 60 * 60 * 1_000) { + context.addIssue({ code: "custom", path: ["to"], message: "the source window must be exactly 24 hours" }); } if (Date.parse(source.ingestion_cutoff) < Date.parse(source.to)) { context.addIssue({ code: "custom", path: ["ingestion_cutoff"], message: "the frozen ingestion cutoff must be at or after the source window end" }); diff --git a/src/evals/source-index.ts b/src/evals/source-index.ts index c149f318..7825d08f 100644 --- a/src/evals/source-index.ts +++ b/src/evals/source-index.ts @@ -7,21 +7,35 @@ export interface EvalSourceCommitmentRow { schema_version: typeof EVAL_SOURCE_ROW_SCHEMA_VERSION; request_id: string; capture_key: string; + captured_at: string; size_bytes: number; content_sha256: string; } -export function sourceIndexCommitmentSha256(rows: Iterable): string { - let commitment = createHash("sha256").update(SOURCE_INDEX_COMMITMENT_DOMAIN, "utf8").digest(); - for (const row of rows) { +export class SourceIndexCommitment { + #commitment = createHash("sha256").update(SOURCE_INDEX_COMMITMENT_DOMAIN, "utf8").digest(); + + update(row: EvalSourceCommitmentRow): void { const canonicalLine = `${JSON.stringify({ schema_version: row.schema_version, request_id: row.request_id, capture_key: row.capture_key, + captured_at: row.captured_at, size_bytes: row.size_bytes, content_sha256: row.content_sha256, })}\n`; - commitment = createHash("sha256").update(commitment).update(canonicalLine, "utf8").digest(); + this.#commitment = createHash("sha256").update(this.#commitment).update(canonicalLine, "utf8").digest(); + } + + digest(): string { + return this.#commitment.toString("hex"); + } +} + +export function sourceIndexCommitmentSha256(rows: Iterable): string { + const commitment = new SourceIndexCommitment(); + for (const row of rows) { + commitment.update(row); } - return commitment.toString("hex"); + return commitment.digest(); } diff --git a/src/trace-foundry.ts b/src/trace-foundry.ts index a8db68e2..d01ae2c5 100644 --- a/src/trace-foundry.ts +++ b/src/trace-foundry.ts @@ -6,7 +6,7 @@ import { traceFoundryViewer } from "./trace-foundry-viewer.js"; import { bumpVersion, classifyTaskChange, computeTaskContentHashes, validateBenchmarkManifest } from "./benchmark.js"; import { FOUNDRY_SELF_CHECK_SCHEMA, REVIEW_DECISIONS, TRACE_FOUNDRY_SCHEMA, captureFileId, readJsonlFile, readReviews, toPortablePath } from "./benchmark-artifacts.js"; import { buildRejectionGuidance, loadGuidanceFile } from "./rejection-guidance.js"; -import { VerifiedWorkloadCaptureFileSchema } from "./evals/contracts.js"; +import { WorkloadSourceRowSchema } from "./evals/contracts.js"; type J = null | boolean | number | string | J[] | { [key: string]: J }; type Obj = Record; @@ -132,7 +132,7 @@ function hostedSourceFiles(source: string, files: string[], sourceIndexInput: st let value: unknown; try { value = JSON.parse(line); } catch (error) { throw new Error(`Invalid hosted source index line ${index + 1}: ${error instanceof Error ? error.message : String(error)}`); } - const parsed = VerifiedWorkloadCaptureFileSchema.strict().safeParse(value); + const parsed = WorkloadSourceRowSchema.strict().safeParse(value); if (!parsed.success) throw new Error(`Invalid hosted source index line ${index + 1}.`); return parsed.data; }); diff --git a/src/workload-trace-export.ts b/src/workload-trace-export.ts new file mode 100644 index 00000000..ead47a30 --- /dev/null +++ b/src/workload-trace-export.ts @@ -0,0 +1,838 @@ +import { createHash, randomUUID } from "node:crypto"; +import { + chmodSync, + closeSync, + createReadStream, + existsSync, + lstatSync, + openSync, + readdirSync, + readFileSync, + renameSync, + rmSync, + writeSync, +} from "node:fs"; +import { join, relative, resolve, sep } from "node:path"; +import { z } from "zod"; + +import { createPrivateDirectory, pathExists, replacePrivateText } from "./evals/build-state.js"; +import { + WorkloadCaptureExportScopeSchema, + WorkloadSourceRowSchema, + WorkloadTraceExportSkippedCaptureSchema, + WorkloadTraceWindowBindingSchema, + type WorkloadCaptureExportScope, + type WorkloadSourceRow, + type WorkloadTraceWindowBinding, +} from "./evals/contracts.js"; +import { allowedCaptureUrl, portableCaptureFileName } from "./evals/materialize.js"; +import { EVAL_SOURCE_ROW_SCHEMA_VERSION, SourceIndexCommitment } from "./evals/source-index.js"; +import { request, UnderstudyApiError } from "./internal/http.js"; + +export const DEFAULT_WORKLOAD_TRACE_CONCURRENCY = 4; +export const MAX_WORKLOAD_TRACE_CONCURRENCY = 16; +export const DEFAULT_WORKLOAD_TRACE_RETRIES = 2; +export const MAX_WORKLOAD_TRACE_RETRIES = 5; +const DOWNLOAD_TIMEOUT_MS = 60_000; +const MAX_CAPTURE_BYTES = 16 * 1024 * 1024; +const MAX_PAGE_CAPTURES = 1_000; + +export const WORKLOAD_CAPTURE_EXPORT_ROUTE_PATTERN = + "/orgs/:org_id/projects/:project_id/workloads/:workload_id/captures/export" as const; + +export const WorkloadTraceExportCaptureSchema = z.object({ + request_id: z.string().min(1), + capture_key: z.string().min(1), + captured_at: z.string().datetime(), + url: z.string().url(), +}).strict(); + +export const WorkloadTraceExportPageSchema = z.object({ + canonical_scope: WorkloadCaptureExportScopeSchema, + captures: z.array(WorkloadTraceExportCaptureSchema).max(MAX_PAGE_CAPTURES), + next_cursor: z.string().min(1).max(8_192).nullable(), +}).strict(); + +export const WorkloadTraceExportPageRequestSchema = z.object({ + from: z.string().datetime(), + to: z.string().datetime(), + ingestion_cutoff: z.string().datetime().optional(), + cursor: z.string().min(1).max(8_192).optional(), +}).strict().superRefine((request, context) => { + if (Boolean(request.cursor) !== Boolean(request.ingestion_cutoff)) { + context.addIssue({ + code: "custom", + path: request.cursor ? ["ingestion_cutoff"] : ["cursor"], + message: "ingestion_cutoff and cursor must be supplied together", + }); + } +}); + +const WorkloadTraceExportSummarySchema = z.object({ + schema_version: z.literal("understudy.trace-source.v1"), + window: WorkloadCaptureExportScopeSchema, + requested_count: z.number().int().nonnegative(), + materialized_count: z.number().int().nonnegative(), + skipped_count: z.number().int().nonnegative(), + skipped_index: z.literal("source/skipped.jsonl"), + capture_count: z.number().int().nonnegative(), + size_bytes: z.number().int().nonnegative(), + index: z.literal("source/index.jsonl"), + index_sha256: z.string().regex(/^[a-f0-9]{64}$/), +}).strict(); + +export interface WorkloadTraceWindow { + from: string; + to: string; +} + +export type WorkloadTraceExportScope = WorkloadCaptureExportScope; +export type WorkloadTraceExportCapture = z.infer; +export type WorkloadTraceExportPage = z.infer; +type WorkloadTraceExportSkippedCapture = z.infer; + +export interface WorkloadTraceExportResult { + outputDirectory: string; + indexPath: string; + canonicalScope: WorkloadTraceExportScope; + captureCount: number; + sizeBytes: number; + indexSha256: string; + requestedCount: number; + skippedCount: number; + writtenCount: number; + adoptedCount: number; +} + +export interface WorkloadTraceExportInput extends WorkloadTraceWindow { + orgId: string; + projectId: string; + workloadId: string; + outputDirectory: string; + gatewayUrl: string; + concurrency?: number; + retries?: number; + reuseStoredWindow?: boolean; + onProgress?: (completed: number, written: number, adopted: number) => void; + requestPage?: (body: WorkloadTraceExportPageRequest) => Promise; + fetchCapture?: typeof fetch; +} + +export type WorkloadTraceExportPageRequest = z.infer; + +export function resolveWorkloadTraceWindow(input: { + date?: string; + last?: string; + now?: Date; +}): WorkloadTraceWindow { + if (input.date !== undefined && input.last !== undefined) { + throw new Error("Choose either --date or --last, not both."); + } + const now = input.now ?? new Date(); + if (!Number.isFinite(now.valueOf())) throw new Error("Current time is invalid."); + if (input.last !== undefined && input.last !== "1d") { + throw new Error("Workload trace export currently supports exactly 1d."); + } + if (input.date === undefined) { + return { + from: new Date(now.valueOf() - 86_400_000).toISOString(), + to: now.toISOString(), + }; + } + if (!/^\d{4}-\d{2}-\d{2}$/.test(input.date)) { + throw new Error("--date must use YYYY-MM-DD."); + } + const from = new Date(`${input.date}T00:00:00.000Z`); + if (Number.isNaN(from.valueOf()) || from.toISOString().slice(0, 10) !== input.date) { + throw new Error("--date must be a valid UTC calendar date."); + } + const to = new Date(from.valueOf() + 86_400_000); + if (to.valueOf() > now.valueOf()) { + throw new Error("--date must select a complete UTC calendar day whose end is not in the future."); + } + return { from: from.toISOString(), to: to.toISOString() }; +} + +export async function exportWorkloadTraceWindow( + input: WorkloadTraceExportInput, +): Promise { + validateInput(input); + const outputDirectory = resolve(input.outputDirectory); + createPrivateDirectory(outputDirectory); + const sourceDirectory = join(outputDirectory, "source"); + createPrivateDirectory(sourceDirectory); + const tracesDirectory = join(sourceDirectory, "traces"); + const effectiveWindow = bindExportWindow(sourceDirectory, tracesDirectory, input); + const effectiveInput = { ...input, ...effectiveWindow }; + const indexPath = join(sourceDirectory, "index.jsonl"); + const summaryPath = join(sourceDirectory, "summary.json"); + const completedExport = await readCompletedExport( + outputDirectory, + indexPath, + summaryPath, + effectiveInput, + ); + if (completedExport !== null) return completedExport; + rmSync(indexPath, { force: true }); + rmSync(summaryPath, { force: true }); + const skippedPath = join(sourceDirectory, "skipped.jsonl"); + rmSync(skippedPath, { force: true }); + createPrivateDirectory(tracesDirectory); + const requestPage = input.requestPage ?? ((body) => requestHostedPage(effectiveInput, body)); + const fetchCapture = input.fetchCapture ?? fetch; + let indexTemporary = `${indexPath}.tmp-${randomUUID()}`; + let skippedTemporary = `${skippedPath}.tmp-${randomUUID()}`; + let indexDescriptor: number | null = null; + let skippedDescriptor: number | null = null; + let completedExportWritten = false; + let canonicalScope: WorkloadTraceExportScope | null = null; + let cursor: string | null = null; + let previousOrder: CaptureOrder | null = null; + let requestedCount = 0; + let materializedCount = 0; + let skippedCount = 0; + let sizeBytes = 0; + const commitment = new SourceIndexCommitment(); + let completed = 0; + let writtenCount = 0; + let adoptedCount = 0; + + try { + indexDescriptor = openSync(indexTemporary, "wx", 0o600); + skippedDescriptor = openSync(skippedTemporary, "wx", 0o600); + do { + const body: WorkloadTraceExportPageRequest = cursor === null + ? { from: effectiveInput.from, to: effectiveInput.to } + : { + from: effectiveInput.from, + to: effectiveInput.to, + ingestion_cutoff: canonicalScope!.ingestion_cutoff, + cursor, + }; + const rawPage = await retryOperation( + () => requestPage(body), + input.retries ?? DEFAULT_WORKLOAD_TRACE_RETRIES, + ); + const page = WorkloadTraceExportPageSchema.parse(rawPage); + assertPageScope(effectiveInput, page.canonical_scope, canonicalScope); + canonicalScope ??= page.canonical_scope; + if (page.captures.length === 0 && page.next_cursor !== null) { + throw new Error("Workload trace export returned an empty non-terminal page."); + } + for (const capture of page.captures) { + assertCaptureReference(effectiveInput, capture); + previousOrder = assertStrictlyIncreasingCaptureOrder(previousOrder, capture); + } + requestedCount += page.captures.length; + + const pageRows = new Array(page.captures.length).fill(null); + const pageSkipped = new Array(page.captures.length).fill(null); + await runWithConcurrency( + page.captures, + input.concurrency ?? DEFAULT_WORKLOAD_TRACE_CONCURRENCY, + async (capture, index) => { + const fileName = portableCaptureFileName( + capture.request_id, + `-${createHash("sha256").update(capture.capture_key).digest("hex")}`, + ); + const finalPath = join(tracesDirectory, fileName); + const localPath = relative(outputDirectory, finalPath).split(sep).join("/"); + let materialized: Awaited> | + Awaited>; + try { + materialized = existsSync(finalPath) + ? await adoptExistingCapture(finalPath, effectiveInput, capture) + : await downloadCaptureWithRetry( + finalPath, + effectiveInput, + capture, + fetchCapture, + input.retries ?? DEFAULT_WORKLOAD_TRACE_RETRIES, + ); + } catch (error) { + if (!isMissingCapture(error)) throw error; + pageSkipped[index] = { + request_id: capture.request_id, + capture_key: capture.capture_key, + captured_at: capture.captured_at, + reason: "not_found", + }; + completed += 1; + input.onProgress?.(completed, writtenCount, adoptedCount); + return; + } + if (materialized.adopted) adoptedCount += 1; + else writtenCount += 1; + pageRows[index] = { + schema_version: EVAL_SOURCE_ROW_SCHEMA_VERSION, + request_id: capture.request_id, + capture_key: capture.capture_key, + captured_at: capture.captured_at, + size_bytes: materialized.sizeBytes, + content_sha256: materialized.digest, + local_path: localPath, + }; + completed += 1; + input.onProgress?.(completed, writtenCount, adoptedCount); + }, + ); + for (const row of pageRows) { + if (row === null) continue; + appendJsonLine(indexDescriptor, row); + commitment.update(row); + materializedCount += 1; + sizeBytes += row.size_bytes; + } + for (const skipped of pageSkipped) { + if (skipped === null) continue; + appendJsonLine(skippedDescriptor, skipped); + skippedCount += 1; + } + cursor = page.next_cursor; + } while (cursor !== null); + + if (canonicalScope === null || materializedCount === 0) { + throw new Error("No raw captures could be materialized in the selected workload day."); + } + closeSync(indexDescriptor); + indexDescriptor = null; + closeSync(skippedDescriptor); + skippedDescriptor = null; + const indexSha256 = commitment.digest(); + const summaryBody = `${JSON.stringify({ + schema_version: "understudy.trace-source.v1", + window: canonicalScope, + requested_count: requestedCount, + materialized_count: materializedCount, + skipped_count: skippedCount, + skipped_index: "source/skipped.jsonl", + capture_count: materializedCount, + size_bytes: sizeBytes, + index: "source/index.jsonl", + index_sha256: indexSha256, + }, null, 2)}\n`; + renameSync(indexTemporary, indexPath); + indexTemporary = ""; + chmodSync(indexPath, 0o600); + renameSync(skippedTemporary, skippedPath); + skippedTemporary = ""; + chmodSync(skippedPath, 0o600); + replacePrivateText(summaryPath, summaryBody); + completedExportWritten = true; + return { + outputDirectory, + indexPath, + canonicalScope, + captureCount: materializedCount, + sizeBytes, + indexSha256, + requestedCount, + skippedCount, + writtenCount, + adoptedCount, + }; + } finally { + if (indexDescriptor !== null) closeSync(indexDescriptor); + if (skippedDescriptor !== null) closeSync(skippedDescriptor); + if (indexTemporary) rmSync(indexTemporary, { force: true }); + if (skippedTemporary) rmSync(skippedTemporary, { force: true }); + if (!completedExportWritten) { + rmSync(indexPath, { force: true }); + rmSync(skippedPath, { force: true }); + rmSync(summaryPath, { force: true }); + } + } +} + +type CaptureOrder = Pick; + +function assertStrictlyIncreasingCaptureOrder( + previous: CaptureOrder | null, + capture: CaptureOrder, +): CaptureOrder { + if (previous === null) return capture; + const comparison = compareCaptureOrder(previous, capture); + if (comparison >= 0) { + throw new Error(`Workload trace export returned repeated or out-of-order capture ${capture.request_id}.`); + } + return capture; +} + +function compareCaptureOrder(left: CaptureOrder, right: CaptureOrder): number { + for (const key of ["captured_at", "request_id", "capture_key"] as const) { + if (left[key] < right[key]) return -1; + if (left[key] > right[key]) return 1; + } + return 0; +} + +function appendJsonLine(descriptor: number, value: unknown): void { + const bytes = Buffer.from(`${JSON.stringify(value)}\n`, "utf8"); + let written = 0; + while (written < bytes.byteLength) { + const count = writeSync(descriptor, bytes, written, bytes.byteLength - written); + if (count <= 0) throw new Error("Could not write workload trace export index."); + written += count; + } +} + +async function readCompletedExport( + outputDirectory: string, + indexPath: string, + summaryPath: string, + input: WorkloadTraceExportInput, +): Promise { + if (!pathExists(indexPath) || !pathExists(summaryPath)) return null; + const summary = WorkloadTraceExportSummarySchema.parse( + JSON.parse(readFileSync(summaryPath, "utf8")), + ); + assertPageScope(input, summary.window, null); + const skippedPath = resolve(outputDirectory, summary.skipped_index); + for (const path of [indexPath, skippedPath, summaryPath]) { + const stat = lstatSync(path); + if (stat.isSymbolicLink() || !stat.isFile()) { + throw new Error(`Completed workload trace export marker must be a real file: ${path}.`); + } + } + const tracesDirectory = resolve(outputDirectory, "source", "traces"); + const tracesStat = lstatSync(tracesDirectory); + if (tracesStat.isSymbolicLink() || !tracesStat.isDirectory()) { + throw new Error(`Completed workload trace export trace path must be a real directory: ${tracesDirectory}.`); + } + const rowIterator = readJsonl(indexPath, "source index"); + const skippedIterator = readJsonl(skippedPath, "skipped index"); + let nextRow = await rowIterator.next(); + let nextSkipped = await skippedIterator.next(); + let previousOrder: CaptureOrder | null = null; + let materializedCount = 0; + let skippedCount = 0; + let sizeBytes = 0; + const commitment = new SourceIndexCommitment(); + while (!nextRow.done || !nextSkipped.done) { + const row = nextRow.done ? null : WorkloadSourceRowSchema.parse(nextRow.value); + const skipped = nextSkipped.done ? null : WorkloadTraceExportSkippedCaptureSchema.parse(nextSkipped.value); + if (row !== null && (skipped === null || compareCaptureOrder(row, skipped) <= 0)) { + assertCaptureReference(input, row); + previousOrder = assertStrictlyIncreasingCaptureOrder(previousOrder, row); + const expectedLocalPath = relative( + outputDirectory, + join( + tracesDirectory, + portableCaptureFileName( + row.request_id, + `-${createHash("sha256").update(row.capture_key).digest("hex")}`, + ), + ), + ).split(sep).join("/"); + if (row.local_path !== expectedLocalPath) { + throw new Error(`Completed workload trace export has an unexpected local path ${row.local_path}.`); + } + const capturePath = resolve(outputDirectory, row.local_path); + const stat = lstatSync(capturePath); + if ( + stat.isSymbolicLink() || + !stat.isFile() || + stat.size !== row.size_bytes || + stat.size > MAX_CAPTURE_BYTES + ) { + throw new Error(`Completed workload trace export capture is missing or changed: ${row.local_path}.`); + } + await assertRawCaptureIdentity(capturePath, input, row); + const hashed = await hashFile(capturePath); + if (hashed.sizeBytes !== row.size_bytes || hashed.digest !== row.content_sha256) { + throw new Error(`Completed workload trace export capture is missing or changed: ${row.local_path}.`); + } + commitment.update(row); + materializedCount += 1; + sizeBytes += row.size_bytes; + nextRow = await rowIterator.next(); + } else if (skipped !== null) { + assertCaptureReference(input, skipped); + previousOrder = assertStrictlyIncreasingCaptureOrder(previousOrder, skipped); + skippedCount += 1; + nextSkipped = await skippedIterator.next(); + } + } + if (materializedCount === 0) { + throw new Error("Completed workload trace export has no materialized raw captures."); + } + const indexSha256 = commitment.digest(); + if ( + summary.capture_count !== materializedCount || + summary.materialized_count !== materializedCount || + summary.skipped_count !== skippedCount || + summary.requested_count !== materializedCount + skippedCount || + summary.size_bytes !== sizeBytes || + summary.index_sha256 !== indexSha256 + ) { + throw new Error("Completed workload trace export summary does not match its source index."); + } + return { + outputDirectory, + indexPath, + canonicalScope: summary.window, + captureCount: materializedCount, + sizeBytes, + indexSha256, + requestedCount: summary.requested_count, + skippedCount: summary.skipped_count, + writtenCount: 0, + adoptedCount: materializedCount, + }; +} + +async function* readJsonl(path: string, description: string): AsyncGenerator { + let remainder = ""; + let lineNumber = 0; + for await (const chunk of createReadStream(path, { encoding: "utf8" })) { + remainder += chunk; + while (true) { + const newline = remainder.indexOf("\n"); + if (newline === -1) break; + const line = remainder.slice(0, newline).replace(/\r$/, ""); + remainder = remainder.slice(newline + 1); + lineNumber += 1; + if (line.length === 0) continue; + yield parseJsonlValue(line, description, lineNumber); + } + } + if (remainder.length > 0) { + lineNumber += 1; + yield parseJsonlValue(remainder.replace(/\r$/, ""), description, lineNumber); + } +} + +function parseJsonlValue(line: string, description: string, lineNumber: number): unknown { + try { + return JSON.parse(line); + } catch (error) { + throw new Error(`Invalid ${description} line ${lineNumber}: ${error instanceof Error ? error.message : String(error)}`); + } +} + +function bindExportWindow( + sourceDirectory: string, + tracesDirectory: string, + input: WorkloadTraceExportInput, +): WorkloadTraceWindow { + const bindingPath = join(sourceDirectory, "window.json"); + const requested: WorkloadTraceWindowBinding = { + schema_version: "understudy.trace-export-window.v1", + selection_mode: input.reuseStoredWindow ? "rolling" : "date", + org_id: input.orgId, + project_id: input.projectId, + workload_id: input.workloadId, + from: input.from, + to: input.to, + }; + if (pathExists(bindingPath)) { + const stored = WorkloadTraceWindowBindingSchema.parse(JSON.parse(readFileSync(bindingPath, "utf8"))); + if ( + stored.org_id !== requested.org_id || + stored.project_id !== requested.project_id || + stored.workload_id !== requested.workload_id + ) { + throw new Error("Existing raw captures belong to a different organization, project, or workload. Choose a fresh --out directory."); + } + if (stored.selection_mode !== requested.selection_mode) { + throw new Error("Existing raw captures use a different window selection mode. Choose a fresh --out directory."); + } + if (requested.selection_mode === "date" && (stored.from !== requested.from || stored.to !== requested.to)) { + throw new Error("Existing raw captures belong to a different day. Choose a fresh --out directory."); + } + return { from: stored.from, to: stored.to }; + } + if (pathExists(tracesDirectory) && readdirSync(tracesDirectory).length > 0) { + throw new Error("Existing raw captures have no saved workload-day binding. Choose a fresh --out directory."); + } + replacePrivateText(bindingPath, `${JSON.stringify(requested, null, 2)}\n`); + return { from: requested.from, to: requested.to }; +} + +async function requestHostedPage( + input: WorkloadTraceExportInput, + body: WorkloadTraceExportPageRequest, +): Promise { + const route = WORKLOAD_CAPTURE_EXPORT_ROUTE_PATTERN + .replace(":org_id", encodeURIComponent(input.orgId)) + .replace(":project_id", encodeURIComponent(input.projectId)) + .replace(":workload_id", encodeURIComponent(input.workloadId)); + const response = await request({ + url: `/admin/v1${route}`, + method: "POST", + orgId: input.orgId, + signal: AbortSignal.timeout(60_000), + body: WorkloadTraceExportPageRequestSchema.parse(body), + }, WorkloadTraceExportPageSchema); + return response.data; +} + +function validateInput(input: WorkloadTraceExportInput): void { + for (const [name, value] of [ + ["org id", input.orgId], + ["project id", input.projectId], + ["workload id", input.workloadId], + ] as const) { + if (!value) throw new Error(`Workload trace export ${name} is required.`); + } + const from = Date.parse(input.from); + const to = Date.parse(input.to); + if (!Number.isFinite(from) || !Number.isFinite(to) || to - from !== 86_400_000) { + throw new Error("Workload trace export window must be exactly 24 hours."); + } + const concurrency = input.concurrency ?? DEFAULT_WORKLOAD_TRACE_CONCURRENCY; + if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > MAX_WORKLOAD_TRACE_CONCURRENCY) { + throw new Error(`Workload trace export concurrency must be between 1 and ${MAX_WORKLOAD_TRACE_CONCURRENCY}.`); + } + const retries = input.retries ?? DEFAULT_WORKLOAD_TRACE_RETRIES; + if (!Number.isInteger(retries) || retries < 0 || retries > MAX_WORKLOAD_TRACE_RETRIES) { + throw new Error(`Workload trace export retries must be between 0 and ${MAX_WORKLOAD_TRACE_RETRIES}.`); + } +} + +function assertPageScope( + input: WorkloadTraceExportInput, + scope: WorkloadTraceExportScope, + frozen: WorkloadTraceExportScope | null, +): void { + if ( + scope.org_id !== input.orgId || + scope.project_id !== input.projectId || + scope.workload_id !== input.workloadId || + scope.from !== input.from || + scope.to !== input.to || + Date.parse(scope.ingestion_cutoff) < Date.parse(scope.to) || + Date.parse(scope.ingestion_cutoff) > Date.now() + 60_000 + ) { + throw new Error("Workload trace export response does not match the requested organization, project, workload, or window."); + } + if (frozen !== null && JSON.stringify(scope) !== JSON.stringify(frozen)) { + throw new Error("Workload trace export response changed its frozen scope between pages."); + } +} + +function assertCaptureReference( + input: WorkloadTraceExportInput, + capture: Pick, +): void { + if ( + !capture.capture_key.startsWith(`${input.orgId}/${input.projectId}/`) || + !capture.capture_key.endsWith(`/${capture.request_id}.jsonl`) + ) { + throw new Error(`Workload trace capture reference does not match request ${capture.request_id}.`); + } + const capturedAt = Date.parse(capture.captured_at); + if (capturedAt < Date.parse(input.from) || capturedAt >= Date.parse(input.to)) { + throw new Error(`Workload trace capture ${capture.request_id} falls outside the requested window.`); + } +} + +async function adoptExistingCapture( + path: string, + input: WorkloadTraceExportInput, + capture: WorkloadTraceExportCapture, +): Promise<{ adopted: true; digest: string; sizeBytes: number }> { + const stat = lstatSync(path); + if (stat.isSymbolicLink() || !stat.isFile()) { + throw new Error(`Existing raw capture must be a real file: ${path}.`); + } + if (stat.size > MAX_CAPTURE_BYTES) { + throw new Error(`Existing raw capture ${capture.request_id} exceeds the local size limit.`); + } + await assertRawCaptureIdentity(path, input, capture); + const hashed = await hashFile(path); + chmodSync(path, 0o600); + return { adopted: true, ...hashed }; +} + +async function downloadCaptureWithRetry( + finalPath: string, + input: WorkloadTraceExportInput, + capture: WorkloadTraceExportCapture, + fetchCapture: typeof fetch, + retries: number, +): Promise<{ adopted: false; digest: string; sizeBytes: number }> { + for (let attempt = 0; ; attempt += 1) { + try { + const downloaded = await downloadCaptureOnce(finalPath, input, capture, fetchCapture); + return { adopted: false, ...downloaded }; + } catch (error) { + if (attempt >= retries || !isRetryable(error)) throw error; + await delay(250 * 2 ** attempt); + } + } +} + +async function downloadCaptureOnce( + finalPath: string, + input: WorkloadTraceExportInput, + capture: WorkloadTraceExportCapture, + fetchCapture: typeof fetch, +): Promise<{ digest: string; sizeBytes: number }> { + const url = allowedCaptureUrl(capture.url, input.gatewayUrl); + const partialPath = `${finalPath}.${process.pid}.${randomUUID()}.partial`; + let descriptor: number | null = null; + try { + const response = await fetchCapture(url, { + headers: { Accept: "application/x-ndjson" }, + redirect: "error", + signal: AbortSignal.timeout(DOWNLOAD_TIMEOUT_MS), + }); + if (!response.ok) { + await response.body?.cancel(); + throw new CaptureDownloadStatusError(capture.request_id, response.status); + } + const declaredLength = response.headers.get("content-length"); + if (declaredLength !== null && Number(declaredLength) > MAX_CAPTURE_BYTES) { + await response.body?.cancel(); + throw new Error(`Raw capture ${capture.request_id} exceeds the ${MAX_CAPTURE_BYTES}-byte local limit.`); + } + if (!response.body) throw new Error(`Raw capture ${capture.request_id} returned no body.`); + descriptor = openSync(partialPath, "wx", 0o600); + const hash = createHash("sha256"); + const reader = response.body.getReader(); + let sizeBytes = 0; + while (true) { + const chunk = await reader.read(); + if (chunk.done) break; + sizeBytes += chunk.value.byteLength; + if (sizeBytes > MAX_CAPTURE_BYTES) { + await reader.cancel(); + throw new Error(`Raw capture ${capture.request_id} exceeds the ${MAX_CAPTURE_BYTES}-byte local limit.`); + } + hash.update(chunk.value); + let written = 0; + while (written < chunk.value.byteLength) { + const count = writeSync(descriptor, chunk.value, written, chunk.value.byteLength - written); + if (count <= 0) throw new Error(`Raw capture ${capture.request_id} could not be written completely.`); + written += count; + } + } + closeSync(descriptor); + descriptor = null; + await assertRawCaptureIdentity(partialPath, input, capture); + renameSync(partialPath, finalPath); + chmodSync(finalPath, 0o600); + return { digest: hash.digest("hex"), sizeBytes }; + } finally { + if (descriptor !== null) closeSync(descriptor); + rmSync(partialPath, { force: true }); + } +} + +async function assertRawCaptureIdentity( + path: string, + input: WorkloadTraceExportInput, + capture: Pick, +): Promise { + const firstLine = await readFirstNonEmptyLine(path, capture.request_id); + let raw: unknown; + try { + raw = firstLine === undefined ? null : JSON.parse(firstLine); + } catch { + raw = null; + } + const value = typeof raw === "object" && raw !== null && !Array.isArray(raw) + ? raw as Record + : {}; + const workloadId = typeof value.workload_id === "string" + ? value.workload_id + : typeof value.placement_id === "string" ? value.placement_id : null; + if ( + value.request_id !== capture.request_id || + value.workos_org_id !== input.orgId || + value.project_id !== input.projectId || + workloadId !== input.workloadId + ) { + throw new Error(`Raw capture identity does not match export reference ${capture.request_id}.`); + } +} + +async function readFirstNonEmptyLine(path: string, requestId: string): Promise { + let remainder = ""; + for await (const chunk of createReadStream(path, { encoding: "utf8" })) { + remainder += chunk; + if (Buffer.byteLength(remainder, "utf8") > MAX_CAPTURE_BYTES) { + throw new Error(`Raw capture ${requestId} exceeds the local size limit.`); + } + while (true) { + const newline = remainder.indexOf("\n"); + if (newline === -1) break; + const line = remainder.slice(0, newline).replace(/\r$/, ""); + remainder = remainder.slice(newline + 1); + if (line.trim().length > 0) return line; + } + } + const finalLine = remainder.replace(/\r$/, ""); + return finalLine.trim().length > 0 ? finalLine : undefined; +} + +async function hashFile(path: string): Promise<{ digest: string; sizeBytes: number }> { + const hash = createHash("sha256"); + let sizeBytes = 0; + for await (const chunk of createReadStream(path)) { + sizeBytes += chunk.length; + hash.update(chunk); + } + return { digest: hash.digest("hex"), sizeBytes }; +} + +async function runWithConcurrency( + values: T[], + concurrency: number, + worker: (value: T, index: number) => Promise, +): Promise { + let nextIndex = 0; + let failed = false; + let failure: unknown; + const workers = Array.from({ length: Math.min(concurrency, values.length) }, async () => { + while (!failed && nextIndex < values.length) { + const index = nextIndex; + nextIndex += 1; + try { + await worker(values[index]!, index); + } catch (error) { + failed = true; + failure = error; + } + } + }); + await Promise.all(workers); + if (failed) throw failure; +} + +async function retryOperation(operation: () => Promise, retries: number): Promise { + for (let attempt = 0; ; attempt += 1) { + try { + return await operation(); + } catch (error) { + if (attempt >= retries || !isRetryable(error)) throw error; + await delay(250 * 2 ** attempt); + } + } +} + +function isRetryable(error: unknown): boolean { + if (error instanceof UnderstudyApiError || error instanceof CaptureDownloadStatusError) { + return error.status === 408 || error.status === 425 || error.status === 429 || error.status >= 500; + } + return error instanceof TypeError || + (error instanceof DOMException && error.name === "TimeoutError"); +} + +function isMissingCapture(error: unknown): boolean { + return error instanceof CaptureDownloadStatusError && error.status === 404; +} + +class CaptureDownloadStatusError extends Error { + readonly status: number; + + constructor(requestId: string, status: number, cause?: unknown) { + super(`Raw capture ${requestId} download failed with status ${status}.`, { cause }); + this.status = status; + } +} + +function delay(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} diff --git a/tests/cli.test.mjs b/tests/cli.test.mjs index b9fa17f7..1221df4d 100644 --- a/tests/cli.test.mjs +++ b/tests/cli.test.mjs @@ -7,8 +7,6 @@ import { tmpdir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { describe, it } from "node:test"; -import { sourceIndexCommitmentSha256 } from "../dist/evals/source-index.js"; - const cli = ["node", resolve("dist/bin.js")]; const uvAvailable = spawnSync("uv", ["--version"], { encoding: "utf8" }).status === 0; const pythonAvailable = spawnSync("python3", ["--version"], { encoding: "utf8" }).status === 0; @@ -181,11 +179,12 @@ async function withHostedFixture(fn) { evalCaptureDeclaredLength: null, evalExportCohortSha: "a".repeat(64), evalExportExpiries: [], - evalWorkloadManifests: new Map(), - evalWorkloadIngestionCutoffs: new Map(), - evalWorkloadIngestionCutoffOffsetMs: 1_000, - evalWorkloadIndexInvalid: false, - evalWorkloadReceiptInvalid: false, + rawExportPages: new Map(), + rawCaptureFailures: 0, + rawCaptureUnavailableIndexes: new Set(), + rawCaptureDelayMs: 0, + rawCaptureActive: 0, + rawCaptureMaxActive: 0, }; const server = createServer(async (req, res) => { @@ -527,129 +526,65 @@ async function withHostedFixture(fn) { const evalBase = "/admin/v1/orgs/org_1/projects/proj_1/workloads/usp_classify"; const rawCapture = `${JSON.stringify(state.captures[0])}\n`; const rawCaptureSha = createHash("sha256").update(rawCapture).digest("hex"); - const workloadBodies = state.captures.slice(0, 2).map((capture) => `${JSON.stringify(capture)}\n`); - const workloadSourceRows = state.captures.slice(0, 2).map((capture, index) => ({ - schema_version: "understudy.eval-source-capture.v1", - request_id: capture.request_id, - capture_key: `org_1/proj_1/key_${index === 0 ? "z" : "a"}/2026/08/30/${capture.request_id}.jsonl`, - size_bytes: Buffer.byteLength(workloadBodies[index]), - content_sha256: createHash("sha256").update(workloadBodies[index]).digest("hex"), - })); - if (req.method === "POST" && url.pathname === `${evalBase}/eval-capture-export`) { - const sourceKey = `${body.from}|${body.to}`; - const frozenIngestionCutoff = state.evalWorkloadIngestionCutoffs.get(sourceKey) ?? - new Date(Date.parse(body.to) + state.evalWorkloadIngestionCutoffOffsetMs).toISOString(); - state.evalWorkloadIngestionCutoffs.set(sourceKey, frozenIngestionCutoff); - const resume = body.resume_cursor !== undefined; + if (req.method === "POST" && url.pathname === `${evalBase}/captures/export`) { + const page = body.cursor === undefined ? 0 : 1; + const cutoff = state.rawExportPages.get(`${body.from}|${body.to}`)?.cutoff ?? + new Date(Math.max(Date.parse(body.to), Date.now() - 1_000)).toISOString(); if ( - (!resume && body.ingestion_cutoff !== undefined) || - (resume && body.ingestion_cutoff !== frozenIngestionCutoff) - ) { - return send(400, { message: "synthetic ingestion cutoff mismatch" }); - } - const canonicalScope = { - schema_version: "understudy.export-scope.v1", - selector: "workload-window", - org_id: "org_1", - project_id: "proj_1", - workload_id: "usp_classify", - from: body.from, - to: body.to, - ingestion_cutoff: frozenIngestionCutoff, - }; - const makeManifest = (segmentIndex, previousManifestSha256) => { - const capture = state.captures[segmentIndex]; - const sourceRow = workloadSourceRows[segmentIndex]; - const item = { - request_id: capture.request_id, - key: sourceRow.capture_key, - size: sourceRow.size_bytes, - content_sha256: sourceRow.content_sha256, - url: `${gatewayUrl}/eval-workload-capture-${segmentIndex}`, - }; - const terminal = segmentIndex === 1; - const header = { - record_type: "understudy_capture_export_chain_v1", - chain_id: "chain_fixture", - segment_id: (segmentIndex === 0 ? "b" : "c").repeat(64), - segment_index: segmentIndex, - previous_manifest_sha256: previousManifestSha256, - cumulative_scanned: segmentIndex + 1, - cumulative_matched: segmentIndex + 1, - cumulative_exported: segmentIndex + 1, - cumulative_total_bytes: workloadBodies.slice(0, segmentIndex + 1).reduce((sum, value) => sum + Buffer.byteLength(value), 0), - terminal, - }; - const manifest = `${JSON.stringify(header)}\n${JSON.stringify(item)}\n`; - return { header, item, manifest, sha256: createHash("sha256").update(manifest).digest("hex") }; - }; - const first = makeManifest(0, null); - const second = makeManifest(1, first.sha256); - state.evalWorkloadManifests.set("/eval-workload-manifest-0", first.manifest); - state.evalWorkloadManifests.set("/eval-workload-manifest-1", second.manifest); - const segmentIndex = resume ? 1 : 0; - const segment = segmentIndex === 0 ? first : second; + (page === 0 && (body.cursor !== undefined || body.ingestion_cutoff !== undefined)) || + (page === 1 && (body.cursor !== "raw_cursor_1" || body.ingestion_cutoff !== cutoff)) + ) return send(400, { message: "synthetic raw export cursor mismatch" }); + state.rawExportPages.set(`${body.from}|${body.to}`, { cutoff }); + const capture = state.captures[page]; + const capturedAt = new Date(Date.parse(body.from) + (page + 1) * 1_000).toISOString(); return send(200, { - export_id: `exp_fixture_${segmentIndex}`, - count: 1, - total_bytes: segment.item.size, - manifest_url: `${gatewayUrl}/eval-workload-manifest-${segmentIndex}`, - expires_at: new Date(Date.now() + 60 * 60 * 1000).toISOString(), - truncated: !segment.header.terminal, - ...(segment.header.terminal ? {} : { resume_cursor: "cursor_fixture_1" }), - canonical_scope: canonicalScope, - chain: { - chain_id: segment.header.chain_id, - segment_id: segment.header.segment_id, - segment_index: segment.header.segment_index, - previous_manifest_sha256: segment.header.previous_manifest_sha256, - manifest_sha256: segment.sha256, - cumulative_scanned: segment.header.cumulative_scanned, - cumulative_matched: segment.header.cumulative_matched, - cumulative_exported: segment.header.cumulative_exported, - cumulative_total_bytes: segment.header.cumulative_total_bytes, - local_index_sha256: state.evalWorkloadIndexInvalid - ? "f".repeat(64) - : sourceIndexCommitmentSha256(workloadSourceRows.slice(0, segmentIndex + 1)), - terminal: segment.header.terminal, - ...(segment.header.terminal ? { terminal_receipt: "terminal_receipt_fixture" } : {}), + canonical_scope: { + schema_version: "understudy.export-scope.v1", + selector: "workload-window", + org_id: "org_1", + project_id: "proj_1", + workload_id: "usp_classify", + from: body.from, + to: body.to, + ingestion_cutoff: cutoff, }, + captures: [{ + request_id: capture.request_id, + capture_key: `org_1/proj_1/key_raw/2026/08/29/${capture.request_id}.jsonl`, + captured_at: capturedAt, + url: `${gatewayUrl}/raw-workload-capture-${page}`, + }], + next_cursor: page === 0 ? "raw_cursor_1" : null, }); } - if (req.method === "POST" && url.pathname === `${evalBase}/eval-capture-export/verify`) { - const manifests = [0, 1].map((index) => state.evalWorkloadManifests.get(`/eval-workload-manifest-${index}`)); - const manifestSha256 = createHash("sha256").update(manifests[1]).digest("hex"); - const previousManifestSha256 = createHash("sha256").update(manifests[0]).digest("hex"); - return send(200, { - verified: true, - scope_hash: "d".repeat(64), - chain_id: state.evalWorkloadReceiptInvalid ? "wrong_chain" : "chain_fixture", - segment_id: "c".repeat(64), - segment_index: 1, - manifest_sha256: manifestSha256, - previous_manifest_sha256: previousManifestSha256, - cumulative_scanned: 2, - cumulative_matched: 2, - cumulative_exported: 2, - total_bytes: state.captures.slice(0, 2).reduce((sum, capture) => sum + Buffer.byteLength(`${JSON.stringify(capture)}\n`), 0), - local_index_sha256: sourceIndexCommitmentSha256(workloadSourceRows), - expires_at: new Date(Date.now() + 60 * 60 * 1000).toISOString(), - canonical_scope: body.canonical_scope, - source_attestation: "signed-cli-source-attestation", - }); - } - if (req.method === "GET" && state.evalWorkloadManifests.has(url.pathname)) { - return sendBytes(200, state.evalWorkloadManifests.get(url.pathname)); - } - const evalWorkloadCapture = url.pathname.match(/^\/eval-workload-capture-(\d)$/); - if (req.method === "GET" && evalWorkloadCapture) { - if (state.evalCaptureDelayMs > 0) await new Promise((resolve) => setTimeout(resolve, state.evalCaptureDelayMs)); - if (state.evalCaptureFailures > 0) { - state.evalCaptureFailures -= 1; - return send(503, { message: "synthetic eval capture failure" }); + const rawWorkloadCapture = url.pathname.match(/^\/raw-workload-capture-(\d)$/); + if (req.method === "GET" && rawWorkloadCapture) { + state.rawCaptureActive += 1; + state.rawCaptureMaxActive = Math.max(state.rawCaptureMaxActive, state.rawCaptureActive); + try { + if (state.rawCaptureDelayMs > 0) await new Promise((resolve) => setTimeout(resolve, state.rawCaptureDelayMs)); + if (state.rawCaptureFailures > 0) { + state.rawCaptureFailures -= 1; + return send(503, { message: "synthetic raw capture failure" }); + } + const index = Number(rawWorkloadCapture[1]); + if (state.rawCaptureUnavailableIndexes.has(index)) { + return send(404, { message: "synthetic raw capture is no longer available" }); + } + const capture = state.captures[index]; + return sendBytes(200, `${JSON.stringify({ + schema_version: 4, + request_id: capture.request_id, + ts: capture.ts, + workos_org_id: "org_1", + project_id: "proj_1", + workload_id: "usp_classify", + customer_request_body: capture.customer_request_body, + response_body: capture.response_body, + })}\n`); + } finally { + state.rawCaptureActive -= 1; } - const index = Number(evalWorkloadCapture[1]); - return sendBytes(200, `${JSON.stringify(state.captures[index])}\n`); } if (req.method === "GET" && url.pathname === `${evalBase}/eval-capture-catalog`) { return send(200, { @@ -4211,6 +4146,87 @@ class ScoreWithFeedback: }); }); + it("exports one workload day through the generic raw trace endpoint", async () => { + await withHostedFixture(async ({ home, repo, requests }) => { + const env = { HOME: home, USERPROFILE: home }; + const outputDir = join(repo, ".understudy", "traces", "synthetic-day"); + + const competingWindow = await runWithEnvAsync([ + "--json", "traces", "export", "--project", "rehearsal", "--workload", "classify", + "--date", "2026-08-29", "--last", "1d", "--out", outputDir, + "--include-payload", "--yes", + ], env, repo); + assert.notEqual(competingWindow.status, 0); + assert.match(competingWindow.stderr, /either --date or --last/i); + assert.equal(requests.length, 0, "invalid window options fail before hosted reads"); + + const noResume = await runWithEnvAsync([ + "--json", "traces", "export", "--project", "rehearsal", "--workload", "classify", + "--date", "2026-08-29", "--out", outputDir, "--no-resume", + "--include-payload", "--yes", + ], env, repo); + assert.notEqual(noResume.status, 0); + assert.match(noResume.stderr, /only valid for explicit trace ids.*fresh --out/i); + assert.equal(requests.length, 0, "unsupported workload no-resume fails before hosted reads"); + + const exported = await runWithEnvAsync([ + "--json", "traces", "export", "--project", "rehearsal", "--workload", "classify", + "--date", "2026-08-29", "--out", outputDir, + "--include-payload", "--yes", "--concurrency", "2", + ], env, repo); + assert.equal(exported.status, 0, exported.stderr); + assert.doesNotMatch(exported.stdout + exported.stderr, /SECRET_PROMPT|SECRET_COMPLETION/); + const payload = JSON.parse(exported.stdout); + assert.equal(payload.mode, "workload_window"); + assert.equal(payload.source.requested_count, 2); + assert.equal(payload.source.materialized_count, 2); + assert.equal(payload.source.skipped_count, 0); + assert.equal(payload.source.capture_count, 2); + assert.equal(payload.skipped, 0); + const exportRequests = requests.filter((entry) => entry.path.endsWith("/captures/export")); + assert.equal(exportRequests.length, 2); + assert.deepEqual(exportRequests[0].body, { + from: "2026-08-29T00:00:00.000Z", + to: "2026-08-30T00:00:00.000Z", + }); + assert.equal(exportRequests[1].body.cursor, "raw_cursor_1"); + assert.equal(exportRequests[1].body.ingestion_cutoff, payload.source.window.ingestion_cutoff); + const rows = readFileSync(join(outputDir, "source/index.jsonl"), "utf8") + .trim().split("\n").map((line) => JSON.parse(line)); + assert.deepEqual(rows.map((row) => row.request_id), ["req_123", "req_456"]); + assert.match(readFileSync(join(outputDir, rows[0].local_path), "utf8"), /SECRET_PROMPT/); + assert.equal(existsSync(join(outputDir, "source/summary.json")), true); + assert.equal(existsSync(join(outputDir, "build-state.json")), false); + }); + }); + + it("excludes concurrent standalone workload exports from the same output", async () => { + await withHostedFixture(async ({ home, repo, requests, state }) => { + const env = { HOME: home, USERPROFILE: home }; + const outputDir = join(repo, ".understudy", "traces", "concurrent-day"); + const args = [ + "--json", "traces", "export", "--project", "rehearsal", "--workload", "classify", + "--date", "2026-08-29", "--out", outputDir, "--include-payload", "--yes", + ]; + state.rawCaptureDelayMs = 400; + const readsBefore = requests.filter((entry) => entry.path === "/raw-workload-capture-0").length; + const first = runWithEnvAsync(args, env, repo); + const deadline = Date.now() + 5_000; + while (requests.filter((entry) => entry.path === "/raw-workload-capture-0").length === readsBefore) { + if (Date.now() > deadline) throw new Error("first standalone export did not reach capture download"); + await new Promise((resolve) => setTimeout(resolve, 10)); + } + + const second = await runWithEnvAsync(args, env, repo); + assert.notEqual(second.status, 0); + assert.match(second.stderr, /already owns/); + const completed = await first; + state.rawCaptureDelayMs = 0; + assert.equal(completed.status, 0, completed.stderr); + assert.equal(existsSync(join(outputDir, "source", "summary.json")), true); + }); + }); + it("selects, freezes, and materializes a workload-scoped eval cohort", async () => { await withHostedFixture(async ({ home, repo, requests }) => { const env = { HOME: home, USERPROFILE: home }; @@ -4285,11 +4301,18 @@ class ScoreWithFeedback: }); }); - it("builds a receipt-verified v2 project from every segment in a frozen seven-day workload window", async () => { - await withHostedFixture(async ({ gatewayUrl, home, repo, requests, state }) => { + it("recommends raw-day eval building and labels cohort creation legacy", () => { + const help = run(["evals", "--help"]); + assert.equal(help.status, 0, help.stderr); + assert.match(help.stdout, /build \[options\]\s+Recommended:[\s\S]*raw workload day/i); + assert.match(help.stdout, /create \[options\]\s+Legacy:[\s\S]*cohort/i); + }); + + it("builds a v2 eval project from the generic one-day raw trace export", async () => { + await withHostedFixture(async ({ gatewayUrl, home, repo, requests }) => { const env = { HOME: home, USERPROFILE: home }; assert.equal(spawnSync("git", ["init", "-q", repo]).status, 0); - const outputDir = join(repo, ".understudy", "evals", "complete week's draft"); + const outputDir = join(repo, ".understudy", "evals", "one day's draft"); const reservedOutput = join(repo, ".understudy"); rmSync(reservedOutput, { recursive: true, force: true }); @@ -4303,31 +4326,19 @@ class ScoreWithFeedback: assert.equal(requests.length, 0, "the reserved private root fails before hosted reads"); writeHostedConfig({ home, repo, gatewayUrl }); - const unsafeOutput = join(repo, "evals", "unsafe-week"); + const unsafeOutput = join(repo, "evals", "unsafe-day"); const blockedOutput = await runWithEnvAsync([ "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", - "--name", "unsafe-week", "--out", unsafeOutput, "--yes", + "--name", "unsafe-day", "--out", unsafeOutput, "--yes", ], env, repo); assert.notEqual(blockedOutput.status, 0); assert.match(blockedOutput.stderr, /must use a destination under .*\.understudy/); assert.equal(existsSync(unsafeOutput), false); assert.equal(requests.length, 0, "unsafe repository paths fail before hosted reads"); - state.evalCaptureFailures = 1; - const interrupted = await runWithEnvAsync([ - "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", - "--name", "complete-week", "--out", outputDir, "--yes", - ], env, repo); - assert.notEqual(interrupted.status, 0); - assert.equal(existsSync(outputDir), false); - const checkpointPath = join(repo, ".understudy", "evals", ".complete week's draft.eval-build", "build-state.json"); - assert.equal(existsSync(checkpointPath), true); - const interruptedState = JSON.parse(readFileSync(checkpointPath, "utf8")); - assert.ok(Date.parse(interruptedState.source.ingestion_cutoff) > Date.parse(interruptedState.source.to)); - const built = await runWithEnvAsync([ "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", - "--name", "complete-week", "--out", outputDir, "--yes", + "--name", "one-day", "--date", "2026-08-29", "--out", outputDir, "--yes", ], env, repo); assert.equal(built.status, 0, built.stderr); assert.doesNotMatch(built.stdout + built.stderr, /SECRET_PROMPT|SECRET_COMPLETION/); @@ -4335,86 +4346,82 @@ class ScoreWithFeedback: assert.equal(builtPayload.next_action.kind, "coding_agent_prompt"); assert.deepEqual(builtPayload.next_action.command, { executable: "understudy", - args: ["evals", "check", "--draft", "--project", outputDir], + args: ["--json", "evals", "check", "--draft", "--project", outputDir], }); assert.match(builtPayload.next_action.prompt, /infer the workload goal/i); + assert.match(builtPayload.next_action.prompt, /reconcile 0 skipped captures.*source\/skipped\.jsonl.*before.*coverage claims/i); assert.match(builtPayload.next_action.prompt, /exact argument array/i); assert.match(builtPayload.next_action.prompt, new RegExp(outputDir.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"))); const project = JSON.parse(readFileSync(join(outputDir, "eval-project.json"), "utf8")); assert.equal(project.schema_version, "understudy.eval-project.v2"); assert.match(project.eval_id, /^eval_[a-f0-9]{24}$/); - assert.equal(project.name, "complete-week"); + assert.equal(project.name, "one-day"); assert.equal(project.status, "source_materialized"); + assert.equal(project.source.requested_count, 2); + assert.equal(project.source.materialized_count, 2); + assert.equal(project.source.skipped_count, 0); + assert.equal(project.source.skipped_index, "source/skipped.jsonl"); assert.equal(project.source.capture_count, 2); - assert.equal(project.source.terminal_receipt_verified, true); assert.equal(project.authoring.owner, "coding_agent"); assert.equal(project.authoring.semantic_preparation_performed, false); const indexRows = readFileSync(join(outputDir, "source", "index.jsonl"), "utf8") .trim().split("\n").map((line) => JSON.parse(line)); assert.deepEqual(indexRows.map((row) => row.request_id), ["req_123", "req_456"]); assert.ok(indexRows.every((row) => /^[a-f0-9]{64}$/.test(row.content_sha256))); + assert.ok(indexRows.every((row) => typeof row.captured_at === "string")); assert.match(readFileSync(join(outputDir, indexRows[0].local_path), "utf8"), /SECRET_PROMPT/); + assert.equal(existsSync(join(outputDir, "source", "summary.json")), true); + assert.equal(existsSync(join(outputDir, "build-state.json")), false); assert.equal( readFileSync(join(repo, ".git", "info", "exclude"), "utf8") .split(/\r?\n/).filter((line) => line === "/.understudy/").length, 1, ); - const exportRequests = requests.filter((entry) => entry.path.endsWith("/eval-capture-export") && entry.method === "POST"); - assert.equal(exportRequests.length, 3, "the failed first segment is retried, then its resume cursor fetches segment two"); - assert.equal(exportRequests.at(-1).body.resume_cursor, "cursor_fixture_1"); + const exportRequests = requests.filter((entry) => entry.path.endsWith("/captures/export") && entry.method === "POST"); + assert.equal(exportRequests.length, 2); + assert.equal(exportRequests.at(-1).body.cursor, "raw_cursor_1"); assert.equal( Date.parse(exportRequests[0].body.to) - Date.parse(exportRequests[0].body.from), - 7 * 24 * 60 * 60 * 1000, + 24 * 60 * 60 * 1000, ); - const initialRequests = exportRequests.filter((request) => request.body.resume_cursor === undefined); - assert.equal(initialRequests.length, 2, "an interrupted first page restarts as a server-frozen initial export"); - assert.ok(initialRequests.every((request) => request.body.ingestion_cutoff === undefined)); - assert.ok(Date.parse(project.source.window.ingestion_cutoff) > Date.parse(project.source.window.to)); - for (const request of exportRequests.filter((entry) => entry.body.resume_cursor !== undefined)) { - assert.equal(request.body.ingestion_cutoff, project.source.window.ingestion_cutoff, "resumed segments reuse the backend cutoff"); - } - assert.ok(requests.some((entry) => entry.path.endsWith("/eval-capture-export/verify"))); + assert.deepEqual( + { from: exportRequests[0].body.from, to: exportRequests[0].body.to }, + { from: "2026-08-29T00:00:00.000Z", to: "2026-08-30T00:00:00.000Z" }, + ); + assert.equal(exportRequests[0].body.ingestion_cutoff, undefined); + assert.equal(exportRequests[1].body.ingestion_cutoff, project.source.window.ingestion_cutoff); assert.equal(requests.some((entry) => entry.path.endsWith("/eval-cohorts") && entry.method === "POST"), false); }); }); - it("rejects an implausibly future backend ingestion cutoff", async () => { - await withHostedFixture(async ({ home, repo, state }) => { - const env = { HOME: home, USERPROFILE: home }; - assert.equal(spawnSync("git", ["init", "-q", repo]).status, 0); - state.evalWorkloadIngestionCutoffOffsetMs = 5 * 60 * 1000; - const outputDir = join(repo, ".understudy", "evals", "future-cutoff"); - - const result = await runWithEnvAsync([ - "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", - "--name", "future-cutoff", "--out", outputDir, "--yes", - ], env, repo); - - assert.notEqual(result.status, 0); - assert.match(result.stderr, /does not match the requested workload window/); - assert.equal(existsSync(outputDir), false); - }); - }); - - it("rejects an export chain that does not bind the downloaded corpus", async () => { + it("surfaces skipped raw captures in the eval build JSON handoff", async () => { await withHostedFixture(async ({ home, repo, state }) => { const env = { HOME: home, USERPROFILE: home }; assert.equal(spawnSync("git", ["init", "-q", repo]).status, 0); - state.evalWorkloadIndexInvalid = true; - const outputDir = join(repo, ".understudy", "evals", "wrong-corpus"); + state.rawCaptureUnavailableIndexes.add(1); + const outputDir = join(repo, ".understudy", "evals", "incomplete-day"); - const result = await runWithEnvAsync([ + const built = await runWithEnvAsync([ "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", - "--name", "wrong-corpus", "--out", outputDir, "--yes", + "--name", "incomplete-day", "--date", "2026-08-29", "--out", outputDir, "--yes", ], env, repo); + assert.equal(built.status, 0, built.stderr); + const payload = JSON.parse(built.stdout); + assert.equal(payload.source.requested_count, 2); + assert.equal(payload.source.materialized_count, 1); + assert.equal(payload.source.skipped_count, 1); + assert.equal(payload.source.skipped_index, "source/skipped.jsonl"); + assert.match(payload.next_action.prompt, /reconcile 1 skipped capture.*source\/skipped\.jsonl.*before.*coverage claims/i); - assert.notEqual(result.status, 0); - assert.match(result.stderr, /source index commitment does not match its manifest items/i); - assert.equal(existsSync(outputDir), false); + const project = JSON.parse(readFileSync(join(outputDir, "eval-project.json"), "utf8")); + assert.deepEqual(project.source, payload.source); + const skipped = readFileSync(join(outputDir, project.source.skipped_index), "utf8") + .trim().split("\n").map(JSON.parse); + assert.deepEqual(skipped.map((row) => row.request_id), ["req_456"]); }); }); - it("publishes a validated checkpoint and excludes a concurrent builder from the same output", async () => { + it("publishes a complete raw dump and excludes a concurrent builder from the same output", async () => { await withHostedFixture(async ({ home, repo, requests, state }) => { const env = { HOME: home, USERPROFILE: home }; const invalidOutput = join(repo, ".understudy", "evals", "invalid-checkpoint"); @@ -4422,10 +4429,10 @@ class ScoreWithFeedback: const invalid = await runWithEnvAsync([ "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", "--name", "invalid-checkpoint", "--out", invalidOutput, - "--batch-size", "0", "--yes", + "--concurrency", "0", "--yes", ], env, repo); assert.notEqual(invalid.status, 0); - assert.match(invalid.stderr, /--batch-size must be a positive integer/); + assert.match(invalid.stderr, /--concurrency must be between 1 and 16/); assert.equal(existsSync(invalidStaging), false, "invalid state is never published as a resumable checkpoint"); const corrected = await runWithEnvAsync([ @@ -4453,14 +4460,14 @@ class ScoreWithFeedback: rmSync(staleLock, { recursive: true, force: true }); const concurrentOutput = join(repo, ".understudy", "evals", "concurrent-builder"); - state.evalCaptureDelayMs = 400; - const captureReadsBefore = requests.filter((entry) => entry.path === "/eval-workload-capture-0").length; + state.rawCaptureDelayMs = 400; + const captureReadsBefore = requests.filter((entry) => entry.path === "/raw-workload-capture-0").length; const first = runWithEnvAsync([ "--json", "evals", "build", "--project", "rehearsal", "--workload", "classify", "--name", "concurrent-builder", "--out", concurrentOutput, "--yes", ], env, repo); const deadline = Date.now() + 5_000; - while (requests.filter((entry) => entry.path === "/eval-workload-capture-0").length === captureReadsBefore) { + while (requests.filter((entry) => entry.path === "/raw-workload-capture-0").length === captureReadsBefore) { if (Date.now() > deadline) throw new Error("first eval builder did not reach capture download"); await new Promise((resolve) => setTimeout(resolve, 10)); } @@ -4471,7 +4478,7 @@ class ScoreWithFeedback: assert.notEqual(second.status, 0); assert.match(second.stderr, /already owns/); const completed = await first; - state.evalCaptureDelayMs = 0; + state.rawCaptureDelayMs = 0; assert.equal(completed.status, 0, completed.stderr); assert.equal(existsSync(join(concurrentOutput, "eval-project.json")), true); }); diff --git a/tests/eval-authoring-schema-drift.test.mjs b/tests/eval-authoring-schema-drift.test.mjs index c4c3ddee..4c0f29c6 100644 --- a/tests/eval-authoring-schema-drift.test.mjs +++ b/tests/eval-authoring-schema-drift.test.mjs @@ -15,7 +15,6 @@ import { EvalDraftMetricSchema, EvalEnvironmentSchema, EvalExecutionIndexRowSchema, - EvalExportProofSchema, EvalHarnessSchema, EvalMetricSchema, EvalSplitsSchema, @@ -28,12 +27,10 @@ import { } from "../dist/evals/release-contracts.js"; const sha = "a".repeat(64); -const sourceAttestation = "signed-source-attestation"; -const sourceAttestationSha = createHash("sha256").update(sourceAttestation).digest("hex"); const timestamp = "2026-08-30T12:00:00.000Z"; // Keep these digests in sync with the server-side release contract test. -const GOLDEN_PUBLICATION_SHA256 = "e5f1300027c2ec46607243b47b205efd48d0bc0aacf3ec14bfb8db9d736dd24a"; -const GOLDEN_RELEASE_SHA256 = "f3f4efaea8d188ab02a00200e88d426577686a1e7e00dfb2b502750f9868e35e"; +const GOLDEN_PUBLICATION_SHA256 = "3582d4a8c497191edb1ae68cf5949bed1371315a39b1e6dbb7685a19fd4e84e2"; +const GOLDEN_RELEASE_SHA256 = "ae1899288cd0869bf81453e8bc817e1b2e6e15aea4c5b2d2276348b0dc2ca23c"; const pathPatternValue = "environment/replay.mjs"; const scope = { schema_version: "understudy.export-scope.v1", selector: "workload-window", org_id: "org", project_id: "project", workload_id: "workload", from: timestamp, to: timestamp, ingestion_cutoff: timestamp }; @@ -139,14 +136,12 @@ const publicationValue = { eval_id: "eval_0123456789abcdef01234567", name: "weekly eval", source: { - from: "2026-08-23T12:00:00.000Z", + from: "2026-08-29T12:00:00.000Z", to: timestamp, ingestion_cutoff: timestamp, capture_count: 1, total_bytes: 12, local_index_sha256: sha, - export_proof_sha256: sourceAttestationSha, - source_attestation: sourceAttestation, }, artifacts: { eval_set_sha256: sha, @@ -196,9 +191,11 @@ const samples = { identity: { org_id: "org", project_id: "project", workload_id: "workload", workload_name: "support" }, source: { window: scope, + requested_count: 1, + materialized_count: 1, + skipped_count: 0, + skipped_index: "source/skipped.jsonl", capture_count: 1, size_bytes: 12, index: "source/index.jsonl", index_sha256: sha, - export_proof: "source/export-proof.json", export_proof_sha256: sha, exported_capture_count: 1, exported_total_bytes: 12, - terminal_receipt_verified: true, }, artifacts: { workload_profile: "workload-profile.md", coverage: "coverage.json", harness: "harness.json", environment: "environment.json", metric: "metric.json", splits: "splits.json", tasks: "benchmark/tasks.jsonl", execution_index: "benchmark/execution-index.jsonl", analysis: "benchmark/analysis.md", verifier: "verifier", approval: "approval.json", check_report: "checks/report.json" }, authoring: { owner: "coding_agent", semantic_preparation_performed: true }, @@ -216,33 +213,6 @@ const samples = { value: { schema_version: "understudy.eval-draft-coverage.v1", lineage: { execution_index_sha256: sha, counts: { complete: 1, ambiguous: 0, unlinked: 0 } }, execution_modes: [{ name: "write", observed_count: 1, task_ids: ["task-1"], disposition: "covered" }], failure_classes: [{ name: "wrong", observed_count: 0, task_ids: [], disposition: "agent_proposed_uncovered", agent_note: "No independently confirmed negative exists yet." }] }, reject: (value) => { value.failure_classes[0].agent_note = ""; }, }, - "export-proof.v1": { - runtime: EvalExportProofSchema, - value: { - schema_version: "understudy.eval-export-proof.v1", - canonical_scope: scope, - segment_manifest_sha256: [sha], - terminal_receipt: "signed receipt", - verified_receipt: { - verified: true, - scope_hash: sha, - chain_id: "chain", - segment_id: sha, - segment_index: 0, - manifest_sha256: sha, - previous_manifest_sha256: null, - cumulative_scanned: 1, - cumulative_matched: 1, - cumulative_exported: 1, - total_bytes: 12, - local_index_sha256: sha, - expires_at: timestamp, - canonical_scope: scope, - source_attestation: sourceAttestation, - }, - }, - reject: (value) => { value.verified_receipt.verified = false; }, - }, "execution-index-row.v1": { runtime: EvalExecutionIndexRowSchema, value: { schema_version: "understudy.eval-execution-index-row.v1", source_status: "included", execution_group: "execution-1", lineage_status: "complete", capture_count: 1, source_files: [{ local_path: "source/traces/one.jsonl", content_sha256: sha }], task_id: "task-1", exclusion_reasons: [] }, @@ -290,12 +260,12 @@ const samples = { }, "check.v1": { runtime: EvalCheckReportSchema, - value: { schema_version: "understudy.eval-check.v1", checked_at: timestamp, status: "passed", task_count: 1, representative_replay: { ...outcome, provider_called: false }, oracle_fixture: outcome, wrong_fixture: { ...outcome, result: "rejected", feedback: "wrong" }, source: { scope, scope_sha256: sha, index_sha256: sha, export_proof_sha256: sha, capture_count: 1, size_bytes: 12 }, check_input_sha256: sha, eval_set_sha256: sha, coverage_sha256: sha, environment_sha256: sha, verifier_sha256: sha }, + value: { schema_version: "understudy.eval-check.v1", checked_at: timestamp, status: "passed", task_count: 1, representative_replay: { ...outcome, provider_called: false }, oracle_fixture: outcome, wrong_fixture: { ...outcome, result: "rejected", feedback: "wrong" }, source: { scope, scope_sha256: sha, index_sha256: sha, capture_count: 1, size_bytes: 12 }, check_input_sha256: sha, eval_set_sha256: sha, coverage_sha256: sha, environment_sha256: sha, verifier_sha256: sha }, reject: (value) => { value.wrong_fixture.result = "passed"; }, }, "draft-check.v1": { runtime: EvalDraftCheckReportSchema, - value: { schema_version: "understudy.eval-draft-check.v1", checked_at: timestamp, status: "passed", publishable: false, task_count: 1, representative_replay: { ...draftOutcome, provider_called: false }, oracle_fixture: draftOutcome, wrong_fixture: { ...draftOutcome, result: "rejected", feedback: "wrong" }, source: { scope, scope_sha256: sha, index_sha256: sha, export_proof_sha256: sha, capture_count: 1, size_bytes: 12 }, check_input_sha256: sha, eval_set_sha256: sha, coverage_sha256: sha, environment_sha256: sha, verifier_sha256: sha, semantic_assumptions: [{ kind: "workload_goal", reference: "workload-profile.md", statement: "The inferred workload goal still needs owner confirmation." }] }, + value: { schema_version: "understudy.eval-draft-check.v1", checked_at: timestamp, status: "passed", publishable: false, task_count: 1, representative_replay: { ...draftOutcome, provider_called: false }, oracle_fixture: draftOutcome, wrong_fixture: { ...draftOutcome, result: "rejected", feedback: "wrong" }, source: { scope, scope_sha256: sha, index_sha256: sha, capture_count: 1, size_bytes: 12 }, check_input_sha256: sha, eval_set_sha256: sha, coverage_sha256: sha, environment_sha256: sha, verifier_sha256: sha, semantic_assumptions: [{ kind: "workload_goal", reference: "workload-profile.md", statement: "The inferred workload goal still needs owner confirmation." }] }, reject: (value) => { value.publishable = true; }, }, "publication.v1": { @@ -316,7 +286,7 @@ test("publication and release golden bytes match the cross-repository digests", assert.equal(digest(releaseValue), GOLDEN_RELEASE_SHA256); }); -test("release sources preserve a backend cutoff at or after the exact seven-day window", () => { +test("release sources preserve a backend cutoff at or after the exact 24-hour window", () => { const delayed = structuredClone(publicationValue); delayed.source.ingestion_cutoff = "2026-08-30T12:00:01.000Z"; assert.equal(EvalPublicationSchema.safeParse(delayed).success, true); diff --git a/tests/eval-build-state.test.mjs b/tests/eval-build-state.test.mjs index 890021a8..8bce2e94 100644 --- a/tests/eval-build-state.test.mjs +++ b/tests/eval-build-state.test.mjs @@ -59,15 +59,12 @@ test("a recycled live pid is stale only when its process instance can be disting } }); -test("a full-corpus checkpoint freezes the absolute window and locally ignores private eval data", async () => { +test("private eval data is idempotently ignored by the local repository", async () => { const root = mkdtempSync(join(tmpdir(), "understudy-eval-build-state-")); try { - const { - creatingWorkloadBuildState, - ensureUnderstudyGitExcluded, - initializeBuildCheckpoint, - readEvalBuildState, - } = await import(`../dist/evals/build-state.js?full-corpus=${Date.now()}`); + const { ensureUnderstudyGitExcluded } = await import( + `../dist/evals/build-state.js?git-exclude=${Date.now()}` + ); const repo = join(root, "synthetic-repo"); mkdirSync(repo, { mode: 0o700 }); const initialized = childProcess.spawnSync("git", ["init", "-q", repo]); @@ -80,42 +77,6 @@ test("a full-corpus checkpoint freezes the absolute window and locally ignores p .split(/\r?\n/).filter((line) => line === "/.understudy/").length, 1, ); - - const staging = join(repo, ".understudy", "evals", ".weekly.eval-build"); - const state = creatingWorkloadBuildState({ - name: "weekly", - identity: { - org_id: "org_synthetic", - project_id: "proj_synthetic", - workload_id: "workload_synthetic", - workload_name: "synthetic", - }, - source: { - from: "2026-08-23T12:00:00.000Z", - to: "2026-08-30T12:00:00.000Z", - ingestion_cutoff: null, - }, - maxAgeDays: 7, - batchSize: 10, - now: new Date("2026-08-30T12:00:00.000Z"), - }); - initializeBuildCheckpoint(staging, state); - const stored = readEvalBuildState(staging); - assert.equal(stored.schema_version, "understudy.eval-build-state.v2"); - assert.equal(stored.status, "downloading"); - assert.deepEqual(stored.source, state.source); - assert.equal(stored.source.ingestion_cutoff, null, "the backend freezes the cutoff on the first export response"); - assert.deepEqual(stored.transport, { - resume_cursor: null, - chain_id: null, - next_segment_index: 0, - previous_manifest_sha256: null, - segment_manifest_sha256: [], - cumulative_exported: 0, - cumulative_total_bytes: 0, - terminal_receipt: null, - verified_files: [], - }); } finally { rmSync(root, { recursive: true, force: true }); } diff --git a/tests/eval-materialize.test.mjs b/tests/eval-materialize.test.mjs index 0c61ce91..f4adf681 100644 --- a/tests/eval-materialize.test.mjs +++ b/tests/eval-materialize.test.mjs @@ -1,17 +1,15 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; -import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, it } from "node:test"; import { downloadExport, - materializeWorkloadExportSegment, MAX_CAPTURE_BYTES, MAX_COHORT_BYTES, reserveDownloadedChunk, - reserveReceiptDrivenChunk, } from "../dist/evals/materialize.js"; describe("eval materialization byte budgets", () => { @@ -91,249 +89,3 @@ describe("eval materialization filenames", () => { } }); }); - -describe("complete workload export materialization", () => { - it("rejects same-size capture bytes not bound by the authenticated manifest", async () => { - const root = mkdtempSync(join(tmpdir(), "understudy-workload-export-digest-")); - const traces = join(root, "source", "traces"); - const requestId = "req-hash"; - const expectedBody = '{"capture":"a"}\n'; - const corruptedBody = '{"capture":"b"}\n'; - const key = `org/proj/apk/2026/08/30/${requestId}.jsonl`; - const item = { - request_id: requestId, - key, - size: Buffer.byteLength(expectedBody), - content_sha256: createHash("sha256").update(expectedBody).digest("hex"), - url: `http://localhost:8787/captures/${requestId}`, - }; - const header = { - record_type: "understudy_capture_export_chain_v1", - chain_id: "chain-digest", - segment_id: "a".repeat(64), - segment_index: 0, - previous_manifest_sha256: null, - cumulative_scanned: 1, - cumulative_matched: 1, - cumulative_exported: 1, - cumulative_total_bytes: item.size, - terminal: true, - }; - const manifest = `${JSON.stringify(header)}\n${JSON.stringify(item)}\n`; - const manifestSha256 = createHash("sha256").update(manifest).digest("hex"); - const response = { - export_id: "exp-digest", - count: 1, - total_bytes: item.size, - manifest_url: "http://localhost:8787/manifests/digest", - expires_at: new Date(Date.now() + 60 * 60_000).toISOString(), - truncated: false, - canonical_scope: { - schema_version: "understudy.export-scope.v1", - selector: "workload-window", - org_id: "org", - project_id: "proj", - workload_id: "workload", - from: "2026-08-23T00:00:00.000Z", - to: "2026-08-30T00:00:00.000Z", - ingestion_cutoff: "2026-08-30T00:00:01.000Z", - }, - chain: { - chain_id: header.chain_id, - segment_id: header.segment_id, - segment_index: header.segment_index, - previous_manifest_sha256: header.previous_manifest_sha256, - manifest_sha256: manifestSha256, - cumulative_scanned: header.cumulative_scanned, - cumulative_matched: header.cumulative_matched, - cumulative_exported: header.cumulative_exported, - cumulative_total_bytes: header.cumulative_total_bytes, - terminal: header.terminal, - terminal_receipt: "signed-terminal-receipt", - }, - }; - const originalFetch = globalThis.fetch; - let captureRequests = 0; - globalThis.fetch = async (rawUrl) => { - const url = new URL(rawUrl); - if (url.pathname === "/manifests/digest") return new Response(manifest); - captureRequests += 1; - return new Response(corruptedBody, { - headers: { "content-length": String(Buffer.byteLength(corruptedBody)) }, - }); - }; - - try { - await assert.rejects( - materializeWorkloadExportSegment({ - exportData: response, - tracesDirectory: traces, - gatewayUrl: "http://localhost:8787", - verifiedFiles: [], - onVerified() {}, - }), - /authenticated SHA-256 verification/, - ); - const localName = `${requestId}-${createHash("sha256").update(key).digest("hex").slice(0, 12)}.jsonl`; - const localPath = join(traces, localName); - assert.equal(existsSync(localPath), false, "a mismatched download must not be published"); - - writeFileSync(localPath, corruptedBody); - await assert.rejects( - materializeWorkloadExportSegment({ - exportData: response, - tracesDirectory: traces, - gatewayUrl: "http://localhost:8787", - verifiedFiles: [], - onVerified() {}, - }), - /Untracked capture file does not match/, - ); - assert.equal(captureRequests, 1, "same-size recovery is rejected before another capture download"); - } finally { - globalThis.fetch = originalFetch; - rmSync(root, { recursive: true, force: true }); - } - }); - - it("resumes without redownloading verified files and uses manifest sizes instead of sample-era limits", async () => { - assert.equal( - reserveReceiptDrivenChunk("req_large", 256 * 1024 * 1024, 1, 300 * 1024 * 1024), - 256 * 1024 * 1024 + 1, - "the full-corpus path must not retain the old 16 MiB or 256 MiB ceilings", - ); - - const root = mkdtempSync(join(tmpdir(), "understudy-workload-export-")); - const traces = join(root, "source", "traces"); - const bodies = new Map([ - ["req-a", '{"capture":"a"}\n'], - ["req-b", '{"capture":"b"}\n'], - ]); - const items = [...bodies].map(([request_id, body]) => ({ - request_id, - key: `org/proj/apk/2026/08/30/${request_id}.jsonl`, - size: Buffer.byteLength(body), - content_sha256: createHash("sha256").update(body).digest("hex"), - url: `http://localhost:8787/captures/${request_id}`, - })); - const header = { - record_type: "understudy_capture_export_chain_v1", - chain_id: "chain-1", - segment_id: "a".repeat(64), - segment_index: 0, - previous_manifest_sha256: null, - cumulative_scanned: 2, - cumulative_matched: 2, - cumulative_exported: 2, - cumulative_total_bytes: items.reduce((sum, item) => sum + item.size, 0), - terminal: true, - }; - const manifest = `${JSON.stringify(header)}\n${items.map((item) => JSON.stringify(item)).join("\n")}\n`; - const manifestSha256 = createHash("sha256").update(manifest).digest("hex"); - const response = { - export_id: "exp-1", - count: 2, - total_bytes: items.reduce((sum, item) => sum + item.size, 0), - manifest_url: "http://localhost:8787/manifests/segment-0", - expires_at: new Date(Date.now() + 60 * 60_000).toISOString(), - truncated: false, - canonical_scope: { - schema_version: "understudy.export-scope.v1", - selector: "workload-window", - org_id: "org", - project_id: "proj", - workload_id: "workload", - from: "2026-08-23T00:00:00.000Z", - to: "2026-08-30T00:00:00.000Z", - ingestion_cutoff: "2026-08-30T00:00:01.000Z", - }, - chain: { - chain_id: "chain-1", - segment_id: header.segment_id, - segment_index: 0, - previous_manifest_sha256: null, - manifest_sha256: manifestSha256, - cumulative_scanned: 2, - cumulative_matched: 2, - cumulative_exported: 2, - cumulative_total_bytes: responseTotal(items), - terminal: true, - terminal_receipt: "signed-terminal-receipt", - }, - }; - const requests = []; - const originalFetch = globalThis.fetch; - globalThis.fetch = async (rawUrl) => { - const url = new URL(rawUrl); - requests.push(url.pathname); - if (url.pathname === "/manifests/segment-0") return new Response(manifest); - const requestId = url.pathname.split("/").at(-1); - const body = bodies.get(requestId); - return body === undefined - ? new Response("not found", { status: 404 }) - : new Response(body, { headers: { "content-length": String(Buffer.byteLength(body)) } }); - }; - - const verified = []; - let interruptedFile; - try { - await assert.rejects( - materializeWorkloadExportSegment({ - exportData: response, - tracesDirectory: traces, - gatewayUrl: "http://localhost:8787", - verifiedFiles: verified, - onVerified(file) { - interruptedFile = file; - throw new Error("synthetic interruption"); - }, - }), - /synthetic interruption/, - ); - assert.equal(verified.length, 0, "the simulated crash happens before checkpoint persistence"); - assert.equal(existsSync(join(root, interruptedFile.local_path)), true); - const firstDownloadedPath = requests.find((path) => path.startsWith("/captures/")); - - const resumed = await materializeWorkloadExportSegment({ - exportData: response, - tracesDirectory: traces, - gatewayUrl: "http://localhost:8787", - verifiedFiles: verified, - onVerified(file) { - verified.push(file); - }, - }); - assert.equal(resumed.manifest_sha256, manifestSha256); - assert.equal(verified.length, 2); - assert.equal( - requests.filter((path) => path === firstDownloadedPath).length, - 1, - "an atomically published capture from the crash window must be adopted, not downloaded again", - ); - assert.deepEqual(verified.map((file) => file.request_id).sort(), ["req-a", "req-b"]); - - const captureRequests = requests.filter((path) => path.startsWith("/captures/")).length; - await materializeWorkloadExportSegment({ - exportData: response, - tracesDirectory: traces, - gatewayUrl: "http://localhost:8787", - verifiedFiles: verified, - onVerified() { - throw new Error("verified files must not be checkpointed twice"); - }, - }); - assert.equal( - requests.filter((path) => path.startsWith("/captures/")).length, - captureRequests, - "checkpointed captures must be rehashed locally, not downloaded again", - ); - } finally { - globalThis.fetch = originalFetch; - rmSync(root, { recursive: true, force: true }); - } - }); -}); - -function responseTotal(items) { - return items.reduce((sum, item) => sum + item.size, 0); -} diff --git a/tests/eval-project.test.mjs b/tests/eval-project.test.mjs index 0a5459cc..c5c8d559 100644 --- a/tests/eval-project.test.mjs +++ b/tests/eval-project.test.mjs @@ -1,59 +1,48 @@ import assert from "node:assert/strict"; -import { existsSync, mkdtempSync, rmSync } from "node:fs"; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { test } from "node:test"; import { buildWorkloadEvalProject } from "../dist/eval-project.js"; -test("a verified empty export cannot create an unusable workload eval project", () => { - const root = mkdtempSync(join(tmpdir(), "understudy-empty-eval-project-")); - const output = join(root, "empty-week"); - const scope = { - schema_version: "understudy.export-scope.v1", - selector: "workload-window", - org_id: "org_synthetic", - project_id: "proj_synthetic", - workload_id: "workload_synthetic", - from: "2026-08-23T12:00:00.000Z", - to: "2026-08-30T12:00:00.000Z", - ingestion_cutoff: "2026-08-30T12:00:00.000Z", +const scope = { + schema_version: "understudy.export-scope.v1", + selector: "workload-window", + org_id: "org_synthetic", + project_id: "proj_synthetic", + workload_id: "workload_synthetic", + from: "2026-08-29T00:00:00.000Z", + to: "2026-08-30T00:00:00.000Z", + ingestion_cutoff: "2026-08-30T00:00:01.000Z", +}; + +function source(output, captureCount = 1, skippedCount = 0) { + return { + outputDirectory: output, + indexPath: join(output, "source", "index.jsonl"), + canonicalScope: scope, + captureCount, + sizeBytes: captureCount === 0 ? 0 : 17, + indexSha256: "a".repeat(64), + requestedCount: captureCount + skippedCount, + skippedCount, + writtenCount: captureCount, + adoptedCount: 0, }; - const terminalManifestSha256 = "c".repeat(64); +} +test("an empty raw dump cannot create an unusable workload eval project", () => { + const root = mkdtempSync(join(tmpdir(), "understudy-empty-eval-project-")); + const output = join(root, "empty-day"); try { assert.throws( () => buildWorkloadEvalProject({ output, - name: "empty-week", - identity: { - org_id: scope.org_id, - project_id: scope.project_id, - workload_id: scope.workload_id, - workload_name: "synthetic", - }, - canonicalScope: scope, - verifiedFiles: [], - segmentManifestSha256: [terminalManifestSha256], - terminalReceipt: "signed-empty-terminal-receipt", - verifiedReceipt: { - verified: true, - scope_hash: "d".repeat(64), - chain_id: "chain_empty_week", - segment_id: "e".repeat(64), - segment_index: 0, - manifest_sha256: terminalManifestSha256, - previous_manifest_sha256: null, - cumulative_scanned: 0, - cumulative_matched: 0, - cumulative_exported: 0, - total_bytes: 0, - local_index_sha256: "4da6e1855a6868d3caa47455d7b802a3e9d737e9d157434cff60e26d9a8345b0", - expires_at: "2026-08-30T13:00:00.000Z", - canonical_scope: scope, - source_attestation: "signed-empty-source-attestation", - }, - now: new Date("2026-08-30T12:00:00.000Z"), + name: "empty-day", + identity: { org_id: scope.org_id, project_id: scope.project_id, workload_id: scope.workload_id, workload_name: "synthetic" }, + source: source(output, 0), + now: new Date("2026-08-30T00:00:01.000Z"), }), /no captures were exported.*refusing to create an empty eval project/i, ); @@ -63,69 +52,29 @@ test("a verified empty export cannot create an unusable workload eval project", } }); -test("a repeated capture key across export segments cannot complete a workload eval build", () => { +test("a one-day raw dump creates only local index provenance in the project manifest", () => { const root = mkdtempSync(join(tmpdir(), "understudy-eval-project-")); - const output = join(root, "weekly"); - const scope = { - schema_version: "understudy.export-scope.v1", - selector: "workload-window", - org_id: "org_synthetic", - project_id: "proj_synthetic", - workload_id: "workload_synthetic", - from: "2026-08-23T12:00:00.000Z", - to: "2026-08-30T12:00:00.000Z", - ingestion_cutoff: "2026-08-30T12:00:00.000Z", - }; - const repeatedCapture = { - schema_version: "understudy.eval-source-capture.v1", - request_id: "req_repeated", - capture_key: "org_synthetic/proj_synthetic/workload_synthetic/req_repeated.jsonl", - size_bytes: 17, - content_sha256: "a".repeat(64), - local_path: "source/traces/req_repeated.jsonl", - }; - const terminalManifestSha256 = "c".repeat(64); - + const output = join(root, "one-day"); try { - assert.throws( - () => buildWorkloadEvalProject({ - output, - name: "duplicate-segment-week", - identity: { - org_id: scope.org_id, - project_id: scope.project_id, - workload_id: scope.workload_id, - workload_name: "synthetic", - }, - canonicalScope: scope, - // Each entry represents one segment. The project ledger deduplicates - // their shared key, while the terminal receipt counts both exports. - verifiedFiles: [repeatedCapture, repeatedCapture], - segmentManifestSha256: ["b".repeat(64), terminalManifestSha256], - terminalReceipt: "signed-terminal-receipt", - verifiedReceipt: { - verified: true, - scope_hash: "d".repeat(64), - chain_id: "chain_duplicate_segment", - segment_id: "e".repeat(64), - segment_index: 1, - manifest_sha256: terminalManifestSha256, - previous_manifest_sha256: "b".repeat(64), - cumulative_scanned: 2, - cumulative_matched: 2, - cumulative_exported: 2, - total_bytes: repeatedCapture.size_bytes * 2, - local_index_sha256: "f".repeat(64), - expires_at: "2026-08-30T13:00:00.000Z", - canonical_scope: scope, - source_attestation: "signed-duplicate-segment-source-attestation", - }, - now: new Date("2026-08-30T12:00:00.000Z"), - }), - /receipt totals do not match unique materialized captures/i, - ); - assert.equal(existsSync(join(output, "eval-project.json")), false); - assert.equal(existsSync(join(output, "source", "index.jsonl")), false); + mkdirSync(join(output, "source"), { recursive: true }); + writeFileSync(join(output, "source", "index.jsonl"), "{}\n"); + writeFileSync(join(output, "source", "skipped.jsonl"), "{\"request_id\":\"missing\"}\n"); + const project = buildWorkloadEvalProject({ + output, + name: "one-day", + identity: { org_id: scope.org_id, project_id: scope.project_id, workload_id: scope.workload_id, workload_name: "synthetic" }, + source: source(output, 1, 1), + now: new Date("2026-08-30T00:00:01.000Z"), + }); + assert.deepEqual(Object.keys(project.source).sort(), [ + "capture_count", "index", "index_sha256", "materialized_count", "requested_count", + "size_bytes", "skipped_count", "skipped_index", "window", + ]); + assert.equal(project.source.capture_count, 1); + assert.equal(project.source.requested_count, 2); + assert.equal(project.source.materialized_count, 1); + assert.equal(project.source.skipped_count, 1); + assert.equal(project.source.skipped_index, "source/skipped.jsonl"); } finally { rmSync(root, { recursive: true, force: true }); } diff --git a/tests/eval-source-index.test.mjs b/tests/eval-source-index.test.mjs index 133d1fc7..99622e3c 100644 --- a/tests/eval-source-index.test.mjs +++ b/tests/eval-source-index.test.mjs @@ -1,12 +1,13 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { sourceIndexCommitmentSha256 } from "../dist/evals/source-index.js"; +import { SourceIndexCommitment, sourceIndexCommitmentSha256 } from "../dist/evals/source-index.js"; const first = { schema_version: "understudy.eval-source-capture.v1", request_id: "req-1", capture_key: "org/proj/apk/2026/08/23/req-1.jsonl", + captured_at: "2026-08-23T00:00:00.000Z", size_bytes: 12, content_sha256: "a".repeat(64), }; @@ -14,6 +15,7 @@ const second = { schema_version: "understudy.eval-source-capture.v1", request_id: "req-2", capture_key: "org/proj/apk/2026/08/24/req-2.jsonl", + captured_at: "2026-08-24T00:00:00.000Z", size_bytes: 34, content_sha256: "b".repeat(64), }; @@ -25,14 +27,25 @@ test("source index commitment matches the cross-repository rolling digest", () = ); assert.equal( sourceIndexCommitmentSha256([first]), - "fc24fc7e56f59b721914d84b9e6b333ad8b19fa1dd7789b83c72a68269d1a832", + "0358aac87d6193bd56af2df07aaa02fbc0214d7640993d0e596a89cdd4879913", ); assert.equal( sourceIndexCommitmentSha256([first, second]), - "fcf0ab494f9878bf18ce9a2ebfe762c0c5621237351d9e079a25cfc7c0741998", + "acf3bd481cc77dfa3fd39cef65f613929350f72df8b9f56f154114ec5ec84075", ); assert.notEqual( sourceIndexCommitmentSha256([second, first]), sourceIndexCommitmentSha256([first, second]), ); }); + +test("source index commitment binds captured_at and matches incremental updates", () => { + const incremental = new SourceIndexCommitment(); + incremental.update(first); + incremental.update(second); + assert.equal(incremental.digest(), sourceIndexCommitmentSha256([first, second])); + assert.notEqual( + sourceIndexCommitmentSha256([{ ...first, captured_at: "2026-08-23T00:00:01.000Z" }, second]), + sourceIndexCommitmentSha256([first, second]), + ); +}); diff --git a/tests/evals-check.test.mjs b/tests/evals-check.test.mjs index 8d64eb5d..6c064151 100644 --- a/tests/evals-check.test.mjs +++ b/tests/evals-check.test.mjs @@ -14,18 +14,6 @@ import { buildEvalProject as buildProject } from "./helpers/eval-project.mjs"; const sha = (value) => createHash("sha256").update(value).digest("hex"); const writeJson = (path, value) => writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 }); -function rewriteProof(project, mutate) { - const manifestPath = join(project, "eval-project.json"); - const proofPath = join(project, "source/export-proof.json"); - const manifest = JSON.parse(readFileSync(manifestPath, "utf8")); - const proof = JSON.parse(readFileSync(proofPath, "utf8")); - mutate({ manifest, proof }); - const body = `${JSON.stringify(proof, null, 2)}\n`; - writeFileSync(proofPath, body, { mode: 0o600 }); - manifest.source.export_proof_sha256 = sha(body); - writeJson(manifestPath, manifest); -} - function rewriteExecutionIndex(project, mutate) { const indexPath = join(project, "benchmark/execution-index.jsonl"); const rows = readFileSync(indexPath, "utf8").trim().split("\n").map(JSON.parse); @@ -38,6 +26,15 @@ function rewriteExecutionIndex(project, mutate) { writeJson(coveragePath, coverage); } +function writeSkippedIndex(project, body, count) { + writeFileSync(join(project, "source/skipped.jsonl"), body, { mode: 0o600 }); + const manifestPath = join(project, "eval-project.json"); + const manifest = JSON.parse(readFileSync(manifestPath, "utf8")); + manifest.source.skipped_count = count; + manifest.source.requested_count = manifest.source.materialized_count + count; + writeJson(manifestPath, manifest); +} + test("evals check hashes module trees in global code-unit path order", async () => { const root = mkdtempSync(join(tmpdir(), "understudy-evals-module-order-")); try { @@ -202,14 +199,9 @@ test("evals check replays representative/good/wrong fixtures without a provider assert.equal(first.report.representative_replay.provider_called, false); assert.equal(first.report.oracle_fixture.result, "passed"); assert.equal(first.report.wrong_fixture.result, "rejected"); - const proof = JSON.parse(readFileSync(join(project, "source/export-proof.json"), "utf8")); const projectManifest = JSON.parse(readFileSync(join(project, "eval-project.json"), "utf8")); - assert.equal(first.report.source.export_proof_sha256, sha(proof.verified_receipt.source_attestation)); - assert.notEqual( - first.report.source.export_proof_sha256, - projectManifest.source.export_proof_sha256, - "the hosted source proof binds the durable attestation, while the private project hash binds the local proof file", - ); + assert.equal(first.report.source.scope_sha256, sha(JSON.stringify(projectManifest.source.window))); + assert.equal(first.report.source.index_sha256, projectManifest.source.index_sha256); assert.equal(existsSync(marker), false, "trace text is inert evidence, never an instruction"); const firstReport = readFileSync(join(project, "checks/report.json"), "utf8"); @@ -442,7 +434,7 @@ test("evals check rejects unknown task references, mismatched lineage, and dupli } }); -test("evals check binds deterministic identity and the exact verified seven-day export proof", async () => { +test("evals check binds deterministic identity and exact one-day local source provenance", async () => { const root = mkdtempSync(join(tmpdir(), "understudy-evals-check-proof-")); try { const stable = buildProject(join(root, "stable")); @@ -458,31 +450,31 @@ test("evals check binds deterministic identity and the exact verified seven-day ); const delayedIngestion = buildProject(join(root, "delayed-ingestion")); - rewriteProof(delayedIngestion.project, ({ manifest, proof }) => { + { + const path = join(delayedIngestion.project, "eval-project.json"); + const manifest = JSON.parse(readFileSync(path, "utf8")); manifest.source.window.ingestion_cutoff = "2026-08-30T12:00:01.000Z"; - proof.canonical_scope = manifest.source.window; - proof.verified_receipt.canonical_scope = proof.canonical_scope; - proof.verified_receipt.scope_hash = sha(JSON.stringify(proof.canonical_scope)); manifest.eval_id = deriveWorkloadEvalId({ name: manifest.name, identity: manifest.identity, sourceWindow: manifest.source.window, }); - }); + writeJson(path, manifest); + } assert.equal((await runEvalCheck(delayedIngestion.project)).status, "passed"); const prematureCutoff = buildProject(join(root, "premature-cutoff")); - rewriteProof(prematureCutoff.project, ({ manifest, proof }) => { + { + const path = join(prematureCutoff.project, "eval-project.json"); + const manifest = JSON.parse(readFileSync(path, "utf8")); manifest.source.window.ingestion_cutoff = "2026-08-30T11:59:59.999Z"; - proof.canonical_scope = manifest.source.window; - proof.verified_receipt.canonical_scope = proof.canonical_scope; - proof.verified_receipt.scope_hash = sha(JSON.stringify(proof.canonical_scope)); manifest.eval_id = deriveWorkloadEvalId({ name: manifest.name, identity: manifest.identity, sourceWindow: manifest.source.window, }); - }); + writeJson(path, manifest); + } await assert.rejects( () => runEvalCheck(prematureCutoff.project), /ingestion cutoff must be at or after the frozen window end/i, @@ -503,53 +495,31 @@ test("evals check binds deterministic identity and the exact verified seven-day await assert.rejects(() => runEvalCheck(renamed.project), /Eval id does not match/); const shortWindow = buildProject(join(root, "window")); - rewriteProof(shortWindow.project, ({ manifest, proof }) => { - manifest.source.window.from = "2026-08-24T12:00:00.000Z"; - proof.canonical_scope = manifest.source.window; - proof.verified_receipt.canonical_scope = manifest.source.window; + { + const path = join(shortWindow.project, "eval-project.json"); + const manifest = JSON.parse(readFileSync(path, "utf8")); + manifest.source.window.from = "2026-08-29T13:00:00.000Z"; manifest.eval_id = deriveWorkloadEvalId({ name: manifest.name, identity: manifest.identity, sourceWindow: manifest.source.window }); - }); - await assert.rejects(() => runEvalCheck(shortWindow.project), /exactly seven days/); + writeJson(path, manifest); + } + await assert.rejects(() => runEvalCheck(shortWindow.project), /exactly 24 hours/); const scopeMismatch = buildProject(join(root, "scope")); - rewriteProof(scopeMismatch.project, ({ proof }) => { - proof.canonical_scope = { ...proof.canonical_scope, workload_id: "different-workload" }; - proof.verified_receipt.canonical_scope = proof.canonical_scope; - }); - await assert.rejects(() => runEvalCheck(scopeMismatch.project), /proof canonical scope does not match/i); - - const unverified = buildProject(join(root, "unverified")); - rewriteProof(unverified.project, ({ proof }) => { proof.verified_receipt.verified = false; }); - await assert.rejects(() => runEvalCheck(unverified.project), /Invalid export-proof|verified/i); - - const chain = buildProject(join(root, "chain")); - rewriteProof(chain.project, ({ proof }) => { proof.verified_receipt.manifest_sha256 = "d".repeat(64); }); - await assert.rejects(() => runEvalCheck(chain.project), /terminal manifest does not match/i); - - const scopeHash = buildProject(join(root, "scope-hash")); - rewriteProof(scopeHash.project, ({ proof }) => { proof.verified_receipt.scope_hash = "b".repeat(64); }); - await assert.rejects(() => runEvalCheck(scopeHash.project), /receipt scope hash does not match/i); - - const substitutedCorpus = buildProject(join(root, "substituted-corpus")); - const substitutedIndexPath = join(substitutedCorpus.project, "source/index.jsonl"); - const substitutedRows = readFileSync(substitutedIndexPath, "utf8").trim().split("\n").map(JSON.parse); - substitutedRows[0].capture_key = "captures/substituted/capture.json"; - writeFileSync(substitutedIndexPath, `${substitutedRows.map(JSON.stringify).join("\n")}\n`, { mode: 0o600 }); - const substitutedManifestPath = join(substitutedCorpus.project, "eval-project.json"); - const substitutedManifest = JSON.parse(readFileSync(substitutedManifestPath, "utf8")); - substitutedManifest.source.index_sha256 = sourceIndexCommitmentSha256(substitutedRows); - writeJson(substitutedManifestPath, substitutedManifest); - await assert.rejects( - () => runEvalCheck(substitutedCorpus.project), - /receipt source index commitment does not match/i, - ); + const scopeManifestPath = join(scopeMismatch.project, "eval-project.json"); + const scopeManifest = JSON.parse(readFileSync(scopeManifestPath, "utf8")); + scopeManifest.source.window.workload_id = "different-workload"; + scopeManifest.eval_id = deriveWorkloadEvalId({ name: scopeManifest.name, identity: scopeManifest.identity, sourceWindow: scopeManifest.source.window }); + writeJson(scopeManifestPath, scopeManifest); + await assert.rejects(() => runEvalCheck(scopeMismatch.project), /window workload_id does not match project identity/i); const totals = buildProject(join(root, "totals")); - rewriteProof(totals.project, ({ manifest, proof }) => { - manifest.source.exported_capture_count = 2; - proof.verified_receipt.cumulative_exported = 2; - }); - await assert.rejects(() => runEvalCheck(totals.project), /Local eval source totals do not match/i); + const totalsPath = join(totals.project, "eval-project.json"); + const totalsManifest = JSON.parse(readFileSync(totalsPath, "utf8")); + totalsManifest.source.capture_count = 2; + totalsManifest.source.materialized_count = 2; + totalsManifest.source.requested_count = 2; + writeJson(totalsPath, totalsManifest); + await assert.rejects(() => runEvalCheck(totals.project), /source index capture count/i); } finally { rmSync(root, { recursive: true, force: true }); } @@ -590,29 +560,68 @@ test("evals check reconciles every execution row to every frozen source file exa sourceRows.push({ schema_version: "understudy.eval-source-capture.v1", request_id: "req-synthetic-2", - capture_key: "captures/synthetic/capture-2.json", + capture_key: "org_synthetic/proj_synthetic/key_synthetic/2026/08/29/req-synthetic-2.jsonl", + captured_at: "2026-08-29T13:00:00.000Z", size_bytes: Buffer.byteLength(secondBody), content_sha256: sha(secondBody), local_path: "source/traces/capture-2.json", }); const sourceIndexBody = `${sourceRows.map(JSON.stringify).join("\n")}\n`; writeFileSync(sourceIndexPath, sourceIndexBody, { mode: 0o600 }); - rewriteProof(omitted.project, ({ manifest, proof }) => { - manifest.source.capture_count = 2; - manifest.source.size_bytes += Buffer.byteLength(secondBody); - manifest.source.index_sha256 = sourceIndexCommitmentSha256(sourceRows); - manifest.source.exported_capture_count = 2; - manifest.source.exported_total_bytes = manifest.source.size_bytes; - proof.verified_receipt.cumulative_exported = 2; - proof.verified_receipt.total_bytes = manifest.source.size_bytes; - proof.verified_receipt.local_index_sha256 = manifest.source.index_sha256; - }); + const omittedManifestPath = join(omitted.project, "eval-project.json"); + const omittedManifest = JSON.parse(readFileSync(omittedManifestPath, "utf8")); + omittedManifest.source.capture_count = 2; + omittedManifest.source.materialized_count = 2; + omittedManifest.source.requested_count = 2; + omittedManifest.source.size_bytes += Buffer.byteLength(secondBody); + omittedManifest.source.index_sha256 = sourceIndexCommitmentSha256(sourceRows); + writeJson(omittedManifestPath, omittedManifest); await assert.rejects(() => runEvalCheck(omitted.project), /capture total does not match|does not account for every frozen source file/i); } finally { rmSync(root, { recursive: true, force: true }); } }); +test("evals check validates skipped rows, scope, strict ordering, and overlap", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-evals-check-skipped-index-")); + const skipped = (requestId, capturedAt, captureKey = `org_synthetic/proj_synthetic/key_synthetic/2026/08/29/${requestId}.jsonl`) => ({ + request_id: requestId, + capture_key: captureKey, + captured_at: capturedAt, + reason: "not_found", + }); + try { + const malformed = buildProject(join(root, "malformed")); + writeSkippedIndex(malformed.project, "{not-json}\n", 1); + await assert.rejects(() => runEvalCheck(malformed.project), /Invalid skipped source index line 1/i); + + const outOfScope = buildProject(join(root, "out-of-scope")); + writeSkippedIndex(outOfScope.project, `${JSON.stringify(skipped("req-synthetic-2", "2026-08-30T12:00:00.000Z"))}\n`, 1); + await assert.rejects(() => runEvalCheck(outOfScope.project), /falls outside the eval source window/i); + + const wrongKey = buildProject(join(root, "wrong-key")); + writeSkippedIndex(wrongKey.project, `${JSON.stringify(skipped("req-synthetic-2", "2026-08-29T13:00:00.000Z", "org_other/proj_other/req-synthetic-2.jsonl"))}\n`, 1); + await assert.rejects(() => runEvalCheck(wrongKey.project), /capture reference does not match request/i); + + const outOfOrder = buildProject(join(root, "out-of-order")); + writeSkippedIndex(outOfOrder.project, [ + skipped("req-synthetic-3", "2026-08-29T14:00:00.000Z"), + skipped("req-synthetic-2", "2026-08-29T13:00:00.000Z"), + ].map(JSON.stringify).join("\n") + "\n", 2); + await assert.rejects(() => runEvalCheck(outOfOrder.project), /not strictly ordered/i); + + const overlap = buildProject(join(root, "overlap")); + writeSkippedIndex(overlap.project, `${JSON.stringify(skipped( + "req-synthetic-1", + "2026-08-29T12:00:00.000Z", + "org_synthetic/proj_synthetic/key_synthetic/2026/08/29/req-synthetic-1.jsonl", + ))}\n`, 1); + await assert.rejects(() => runEvalCheck(overlap.project), /duplicate request id/i); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + test("evals check enforces created, metric, intent, check, and final approval chronology", async () => { const root = mkdtempSync(join(tmpdir(), "understudy-evals-check-time-")); try { diff --git a/tests/evals-publish.test.mjs b/tests/evals-publish.test.mjs index 8d5813d2..82eafbeb 100644 --- a/tests/evals-publish.test.mjs +++ b/tests/evals-publish.test.mjs @@ -114,11 +114,10 @@ test("evals publish preview exposes the exact non-uploaded release and binds the assert.equal(preview.bundle.size_bytes > 0, true); assert.equal(preview.bundle.r2_key, preview.manifest.artifacts.bundle_r2_key); assert.deepEqual(preview.bundle.files, preview.manifest.bundle_files); - assert.equal(preview.manifest.source.source_attestation, "signed-synthetic-source-attestation"); - assert.equal( - preview.manifest.source.export_proof_sha256, - createHash("sha256").update(preview.manifest.source.source_attestation).digest("hex"), - ); + assert.deepEqual(Object.keys(preview.manifest.source).sort(), [ + "capture_count", "from", "ingestion_cutoff", "local_index_sha256", "to", "total_bytes", + ]); + assert.equal(Date.parse(preview.manifest.source.to) - Date.parse(preview.manifest.source.from), 86_400_000); assert.match(preview.local_only.policy, /exactly two objects.*publication manifest.*gzip bundle.*every other file.*stays local/i); assert.deepEqual(preview.local_only.explicitly_excluded, [ ".understudy/", @@ -223,7 +222,6 @@ test("evals publish deterministically packages exactly the checked release allow for (const forbidden of [ "eval-project.json", "source/index.jsonl", - "source/export-proof.json", "source/traces/capture.json", "benchmark/execution-index.jsonl", "benchmark/analysis.md", @@ -306,15 +304,6 @@ test("evals publish reruns the check and refuses stale approval or symlinked rel /source index commitment changed after the passing eval check/i, ); - const exportProofMutation = buildEvalProject(join(root, "export-proof-mutation")); - await finalizeApproval(exportProofMutation.project); - await assert.rejects( - () => prepareEvalPublication(exportProofMutation.project, { - afterCheck: () => writeFileSync(join(exportProofMutation.project, "source/export-proof.json"), "\n", { flag: "a" }), - }), - /export proof changed after the passing eval check/i, - ); - const moduleMutation = buildEvalProject(join(root, "module-mutation")); await finalizeApproval(moduleMutation.project); await assert.rejects( diff --git a/tests/evaluation-evidence-gates.test.mjs b/tests/evaluation-evidence-gates.test.mjs index 5de69bf3..4f14d21a 100644 --- a/tests/evaluation-evidence-gates.test.mjs +++ b/tests/evaluation-evidence-gates.test.mjs @@ -54,15 +54,17 @@ test("hosted workload eval authoring stays project-local, provider-free, and tre assert.match(hosted, /--expect-release-id /i); assert.match(hosted, /does not match.*preview.*before.*upload/is); assert.match(hosted, /exactly two objects.*publication manifest.*gzip bundle/is); - assert.match(hosted, /source_attestation.*SHA-256.*exact token/is); - assert.match(hosted, /backend freezes.*ingestion_cutoff.*at or after.*reuses the exact returned\s+cutoff/is); - assert.match(hosted, /rolling commitment.*ordered source-index.*local path.*not part/is); + assert.match(hosted, /one exact raw workload day/i); + assert.match(hosted, /first page freezes an `ingestion_cutoff`.*later page.*reuse/is); + assert.match(hosted, /final ordered `source\/index\.jsonl` and\s+`source\/summary\.json` after every page succeeds/is); + assert.match(hosted, /reconcile.*`source\/skipped\.jsonl`.*before.*coverage claim/is); + assert.match(hosted, /window,\s+cutoff, capture count, byte count, and local index SHA-256/is); assert.match(hosted, /--source-index .*source\/index\.jsonl/i); assert.match(hosted, /--out \.understudy\/evals\//i); }); test("local workload eval contracts are packaged as versioned JSON schemas", () => { - for (const name of ["project.v2", "export-proof.v1", "execution-index-row.v1", "metric.v1", "coverage.v1", "harness.v1", "environment.v1", "splits.v1", "check-fixtures.v1", "check.v1", "approval.v1"]) { + for (const name of ["project.v2", "execution-index-row.v1", "metric.v1", "coverage.v1", "harness.v1", "environment.v1", "splits.v1", "check-fixtures.v1", "check.v1", "approval.v1"]) { const schema = JSON.parse(read(`schemas/understudy.eval-${name}.schema.json`)); assert.equal(schema.$schema, "https://json-schema.org/draft/2020-12/schema"); assert.equal(schema.title, `understudy.eval-${name}`); diff --git a/tests/helpers/eval-project.mjs b/tests/helpers/eval-project.mjs index 3504e5ab..79a66d71 100644 --- a/tests/helpers/eval-project.mjs +++ b/tests/helpers/eval-project.mjs @@ -24,7 +24,8 @@ export function buildEvalProject(root, overrides = {}) { const sourceRow = { schema_version: "understudy.eval-source-capture.v1", request_id: "req-synthetic-1", - capture_key: "captures/synthetic/capture.json", + capture_key: "org_synthetic/proj_synthetic/key_synthetic/2026/08/29/req-synthetic-1.jsonl", + captured_at: "2026-08-29T12:00:00.000Z", size_bytes: Buffer.byteLength(traceBody), content_sha256: sha(traceBody), local_path: "source/traces/capture.json", @@ -32,6 +33,7 @@ export function buildEvalProject(root, overrides = {}) { const sourceIndex = `${JSON.stringify(sourceRow)}\n`; const sourceIndexSha256 = sourceIndexCommitmentSha256([sourceRow]); writeFileSync(join(project, "source/index.jsonl"), sourceIndex, { mode: 0o600 }); + writeFileSync(join(project, "source/skipped.jsonl"), "", { mode: 0o600 }); const task = { schema_version: "understudy.benchmark_task.v1", @@ -151,33 +153,8 @@ export function verify({ replay }) { }); const identity = { org_id: "org_synthetic", project_id: "proj_synthetic", workload_id: "workload_synthetic", workload_name: "synthetic" }; - const sourceWindow = { schema_version: "understudy.export-scope.v1", selector: "workload-window", org_id: "org_synthetic", project_id: "proj_synthetic", workload_id: "workload_synthetic", from: "2026-08-23T12:00:00.000Z", to: "2026-08-30T12:00:00.000Z", ingestion_cutoff: "2026-08-30T12:00:00.000Z" }; - const proof = { - schema_version: "understudy.eval-export-proof.v1", - canonical_scope: sourceWindow, - segment_manifest_sha256: ["a".repeat(64)], - terminal_receipt: "signed-synthetic-terminal-receipt", - verified_receipt: { - verified: true, - scope_hash: sha(JSON.stringify(sourceWindow)), - chain_id: "synthetic-chain", - segment_id: "c".repeat(64), - segment_index: 0, - manifest_sha256: "a".repeat(64), - previous_manifest_sha256: null, - cumulative_scanned: 1, - cumulative_matched: 1, - cumulative_exported: 1, - total_bytes: Buffer.byteLength(traceBody), - local_index_sha256: sourceIndexSha256, - expires_at: "2026-08-30T13:00:00.000Z", - canonical_scope: sourceWindow, - source_attestation: "signed-synthetic-source-attestation", - }, - }; - const proofBody = `${JSON.stringify(proof, null, 2)}\n`; - writeFileSync(join(project, "source/export-proof.json"), proofBody, { mode: 0o600 }); - const projectName = "weekly synthetic eval"; + const sourceWindow = { schema_version: "understudy.export-scope.v1", selector: "workload-window", org_id: "org_synthetic", project_id: "proj_synthetic", workload_id: "workload_synthetic", from: "2026-08-29T12:00:00.000Z", to: "2026-08-30T12:00:00.000Z", ingestion_cutoff: "2026-08-30T12:00:00.000Z" }; + const projectName = "one-day synthetic eval"; const projectManifest = { schema_version: "understudy.eval-project.v2", eval_id: deriveWorkloadEvalId({ name: projectName, identity, sourceWindow }), @@ -187,15 +164,14 @@ export function verify({ replay }) { identity, source: { window: sourceWindow, + requested_count: 1, + materialized_count: 1, + skipped_count: 0, + skipped_index: "source/skipped.jsonl", capture_count: 1, size_bytes: Buffer.byteLength(traceBody), index: "source/index.jsonl", index_sha256: sourceIndexSha256, - export_proof: "source/export-proof.json", - export_proof_sha256: sha(proofBody), - exported_capture_count: 1, - exported_total_bytes: Buffer.byteLength(traceBody), - terminal_receipt_verified: true, }, artifacts: { workload_profile: "workload-profile.md", coverage: "coverage.json", harness: "harness.json", diff --git a/tests/trace-foundry.test.mjs b/tests/trace-foundry.test.mjs index 7270f637..bff7c554 100644 --- a/tests/trace-foundry.test.mjs +++ b/tests/trace-foundry.test.mjs @@ -88,6 +88,7 @@ test("hosted eval lineage reports complete, ambiguous, and unlinked executions a schema_version: "understudy.eval-source-capture.v1", request_id: row.request_id, capture_key: `captures/test/${index}`, + captured_at: row.ts ?? "2026-07-20T12:00:00.000Z", size_bytes: Buffer.byteLength(body), content_sha256: createHash("sha256").update(body).digest("hex"), local_path: localPath, diff --git a/tests/workload-trace-export.test.mjs b/tests/workload-trace-export.test.mjs new file mode 100644 index 00000000..832edc3d --- /dev/null +++ b/tests/workload-trace-export.test.mjs @@ -0,0 +1,765 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; + +import { + WORKLOAD_CAPTURE_EXPORT_ROUTE_PATTERN, + WorkloadTraceExportPageRequestSchema, + WorkloadTraceExportPageSchema, + exportWorkloadTraceWindow, + resolveWorkloadTraceWindow, +} from "../dist/workload-trace-export.js"; + +// Keep this digest in sync with the platform raw export contract test. +const GOLDEN_WORKLOAD_CAPTURE_EXPORT_SHA256 = + "9033a98092e7d28879ee92e0734eae20b51e7983b77516b55d3d8cf1f54717b3"; + +const identity = { + orgId: "org_synthetic", + projectId: "proj_synthetic", + workloadId: "usp_synthetic", +}; +const window = { + from: "2026-08-29T00:00:00.000Z", + to: "2026-08-30T00:00:00.000Z", +}; +const cutoff = "2026-08-30T00:00:01.000Z"; + +function goldenWorkloadCaptureExportContract() { + const scope = { + schema_version: "understudy.export-scope.v1", + selector: "workload-window", + org_id: "org_1", + project_id: "proj_1", + workload_id: "workload_1", + from: "2026-08-29T12:00:00.000Z", + to: "2026-08-30T12:00:00.000Z", + ingestion_cutoff: "2026-08-30T12:00:01.000Z", + }; + return { + route_pattern: "/orgs/:org_id/projects/:project_id/workloads/:workload_id/captures/export", + first_request: { from: scope.from, to: scope.to }, + continued_request: { + from: scope.from, + to: scope.to, + ingestion_cutoff: scope.ingestion_cutoff, + cursor: "opaque-cursor", + }, + response: { + canonical_scope: scope, + captures: [{ + request_id: "request_1", + capture_key: "org_1/proj_1/key/2026/08/29/request_1.jsonl", + captured_at: "2026-08-29T12:00:00.000Z", + url: "https://example.r2.cloudflarestorage.com/request_1.jsonl", + }], + next_cursor: "opaque-cursor", + }, + }; +} + +function captureReference(index) { + const requestId = `req_${index}`; + return { + request_id: requestId, + capture_key: `${identity.orgId}/${identity.projectId}/key_synthetic/2026/08/29/${requestId}.jsonl`, + captured_at: `2026-08-29T00:00:0${index}.000Z`, + url: `http://127.0.0.1:8789/raw/${requestId}`, + }; +} + +function captureBody(reference, overrides = {}) { + return `${JSON.stringify({ + schema_version: 4, + request_id: reference.request_id, + ts: reference.captured_at, + workos_org_id: identity.orgId, + project_id: identity.projectId, + workload_id: identity.workloadId, + ...overrides, + })}\n`; +} + +function responsePage(captures, nextCursor = null, scope = {}) { + return { + canonical_scope: { + schema_version: "understudy.export-scope.v1", + selector: "workload-window", + org_id: identity.orgId, + project_id: identity.projectId, + workload_id: identity.workloadId, + ...window, + ingestion_cutoff: cutoff, + ...scope, + }, + captures, + next_cursor: nextCursor, + }; +} + +test("workload trace export pins the cross-repository API contract", () => { + const golden = goldenWorkloadCaptureExportContract(); + const digest = createHash("sha256").update(JSON.stringify(golden)).digest("hex"); + assert.equal(digest, GOLDEN_WORKLOAD_CAPTURE_EXPORT_SHA256); + assert.equal(WORKLOAD_CAPTURE_EXPORT_ROUTE_PATTERN, golden.route_pattern); + assert.deepEqual( + WorkloadTraceExportPageRequestSchema.parse(golden.first_request), + golden.first_request, + ); + assert.deepEqual( + WorkloadTraceExportPageRequestSchema.parse(golden.continued_request), + golden.continued_request, + ); + assert.deepEqual(WorkloadTraceExportPageSchema.parse(golden.response), golden.response); +}); + +test("workload trace export defaults to the rolling 24 hours ending now", () => { + const now = new Date("2026-08-31T17:42:19.123Z"); + assert.deepEqual(resolveWorkloadTraceWindow({ now }), { + from: "2026-08-30T17:42:19.123Z", + to: "2026-08-31T17:42:19.123Z", + }); +}); + +test("workload trace export accepts one complete UTC calendar day", () => { + assert.deepEqual( + resolveWorkloadTraceWindow({ + date: "2026-08-29", + now: new Date("2026-08-31T17:42:19.123Z"), + }), + { + from: "2026-08-29T00:00:00.000Z", + to: "2026-08-30T00:00:00.000Z", + }, + ); +}); + +test("workload trace export rejects competing, invalid, or incomplete future windows", () => { + const now = new Date("2026-08-31T17:42:19.123Z"); + assert.throws( + () => resolveWorkloadTraceWindow({ date: "2026-08-29", last: "1d", now }), + /either --date or --last/i, + ); + assert.throws( + () => resolveWorkloadTraceWindow({ date: "08\/29\/2026", now }), + /YYYY-MM-DD/, + ); + assert.throws( + () => resolveWorkloadTraceWindow({ date: "2026-02-30", now }), + /valid UTC calendar date/i, + ); + assert.throws( + () => resolveWorkloadTraceWindow({ date: "2026-08-31", now }), + /complete UTC calendar day/i, + ); + assert.throws( + () => resolveWorkloadTraceWindow({ last: "3d", now }), + /exactly 1d/i, + ); +}); + +test("workload trace export paginates a frozen scope and downloads raw files concurrently in response order", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-")); + const references = [0, 1, 2, 3, 4].map(captureReference); + const requestBodies = []; + let active = 0; + let maxActive = 0; + try { + const result = await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 3, + retries: 0, + async requestPage(body) { + requestBodies.push(body); + return requestBodies.length === 1 + ? responsePage(references.slice(0, 3), "cursor_1") + : responsePage(references.slice(3)); + }, + async fetchCapture(url) { + active += 1; + maxActive = Math.max(maxActive, active); + await new Promise((resolve) => setTimeout(resolve, 20)); + active -= 1; + const reference = references.find((entry) => entry.url === String(url)); + return new Response(captureBody(reference), { status: 200 }); + }, + }); + + assert.deepEqual(requestBodies, [ + { ...window }, + { ...window, ingestion_cutoff: cutoff, cursor: "cursor_1" }, + ]); + assert.ok(maxActive > 1, `expected concurrent downloads, saw ${maxActive}`); + assert.ok(maxActive <= 3, `concurrency exceeded the configured bound: ${maxActive}`); + assert.equal(result.captureCount, 5); + const rows = readFileSync(join(root, "source/index.jsonl"), "utf8") + .trim().split("\n").map((line) => JSON.parse(line)); + assert.deepEqual(rows.map((row) => row.request_id), references.map((entry) => entry.request_id)); + assert.ok(rows.every((row) => row.captured_at.startsWith("2026-08-29T"))); + const summary = JSON.parse(readFileSync(join(root, "source/summary.json"), "utf8")); + assert.equal(summary.requested_count, 5); + assert.equal(summary.materialized_count, 5); + assert.equal(summary.skipped_count, 0); + assert.equal(summary.skipped_index, "source/skipped.jsonl"); + assert.equal(readFileSync(join(root, summary.skipped_index), "utf8"), ""); + assert.equal(summary.capture_count, 5); + assert.equal(summary.index_sha256, result.indexSha256); + if (process.platform !== "win32") { + assert.equal(statSync(join(root, "source/index.jsonl")).mode & 0o777, 0o600); + assert.equal(statSync(join(root, "source/traces")).mode & 0o777, 0o700); + assert.ok(rows.every((row) => (statSync(join(root, row.local_path)).mode & 0o777) === 0o600)); + } + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("completed workload trace exports are returned without network calls or marker replacement", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-complete-rerun-")); + const references = [0, 1].map(captureReference); + try { + const first = await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage(references), + async fetchCapture(url) { + const reference = references.find((entry) => entry.url === String(url)); + return new Response(captureBody(reference), { status: 200 }); + }, + }); + const indexPath = join(root, "source/index.jsonl"); + const skippedPath = join(root, "source/skipped.jsonl"); + const summaryPath = join(root, "source/summary.json"); + const originalIndex = readFileSync(indexPath, "utf8"); + const originalSkipped = readFileSync(skippedPath, "utf8"); + const originalSummary = readFileSync(summaryPath, "utf8"); + let pageRequests = 0; + let captureRequests = 0; + + const rerun = await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => { + pageRequests += 1; + throw new Error("completed exports must not refresh"); + }, + fetchCapture: async () => { + captureRequests += 1; + throw new Error("completed exports must not redownload"); + }, + }); + + assert.equal(pageRequests, 0); + assert.equal(captureRequests, 0); + assert.equal(rerun.captureCount, first.captureCount); + assert.equal(rerun.requestedCount, first.requestedCount); + assert.equal(rerun.adoptedCount, first.captureCount); + assert.equal(rerun.writtenCount, 0); + assert.equal(readFileSync(indexPath, "utf8"), originalIndex); + assert.equal(readFileSync(skippedPath, "utf8"), originalSkipped); + assert.equal(readFileSync(summaryPath, "utf8"), originalSummary); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("completed workload trace exports reject equal-size capture tampering without replacing markers", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-complete-tamper-")); + const reference = captureReference(0); + try { + await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage([reference]), + fetchCapture: async () => new Response(captureBody(reference), { status: 200 }), + }); + const indexPath = join(root, "source/index.jsonl"); + const skippedPath = join(root, "source/skipped.jsonl"); + const summaryPath = join(root, "source/summary.json"); + const originalIndex = readFileSync(indexPath, "utf8"); + const originalSkipped = readFileSync(skippedPath, "utf8"); + const originalSummary = readFileSync(summaryPath, "utf8"); + const row = JSON.parse(originalIndex); + const capturePath = join(root, row.local_path); + const originalCapture = readFileSync(capturePath, "utf8"); + const tamperedCapture = originalCapture.replace('"schema_version":4', '"schema_version":5'); + assert.notEqual(tamperedCapture, originalCapture); + assert.equal(Buffer.byteLength(tamperedCapture), Buffer.byteLength(originalCapture)); + writeFileSync(capturePath, tamperedCapture, { mode: 0o600 }); + let pageRequests = 0; + let captureRequests = 0; + + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => { + pageRequests += 1; + throw new Error("completed exports must not refresh"); + }, + fetchCapture: async () => { + captureRequests += 1; + throw new Error("completed exports must not redownload"); + }, + }), + /missing or changed/i, + ); + + assert.equal(pageRequests, 0); + assert.equal(captureRequests, 0); + assert.equal(readFileSync(indexPath, "utf8"), originalIndex); + assert.equal(readFileSync(skippedPath, "utf8"), originalSkipped); + assert.equal(readFileSync(summaryPath, "utf8"), originalSummary); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export records missing raw objects and continues with materialized captures", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-missing-object-")); + const references = [0, 1, 2].map(captureReference); + let missingFetchCount = 0; + try { + const result = await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 2, + retries: 2, + requestPage: async () => responsePage(references), + async fetchCapture(url) { + const reference = references.find((entry) => entry.url === String(url)); + if (reference === references[1]) { + missingFetchCount += 1; + return new Response("missing", { status: 404 }); + } + return new Response(captureBody(reference), { status: 200 }); + }, + }); + + assert.equal(result.requestedCount, 3); + assert.equal(result.captureCount, 2); + assert.equal(result.skippedCount, 1); + assert.equal(result.writtenCount, 2); + assert.equal(missingFetchCount, 1, "a missing object should not consume transient retries"); + const rows = readFileSync(join(root, "source/index.jsonl"), "utf8") + .trim().split("\n").map((line) => JSON.parse(line)); + assert.deepEqual(rows.map((row) => row.request_id), ["req_0", "req_2"]); + const summary = JSON.parse(readFileSync(join(root, "source/summary.json"), "utf8")); + assert.equal(summary.requested_count, 3); + assert.equal(summary.materialized_count, 2); + assert.equal(summary.capture_count, 2); + assert.equal(summary.skipped_count, 1); + const skipped = readFileSync(join(root, summary.skipped_index), "utf8").trim().split("\n").map(JSON.parse); + assert.deepEqual(skipped, [{ + request_id: references[1].request_id, + capture_key: references[1].capture_key, + captured_at: references[1].captured_at, + reason: "not_found", + }]); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export rejects repeated or out-of-order capture pages before download", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-order-")); + const first = captureReference(1); + const earlier = captureReference(0); + try { + let requests = 0; + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + retries: 0, + requestPage: async () => { + requests += 1; + return requests === 1 ? responsePage([first], "cursor_1") : responsePage([earlier]); + }, + fetchCapture: async () => new Response(captureBody(first), { status: 200 }), + }), + /repeated or out-of-order/i, + ); + assert.equal(existsSync(join(root, "source/index.jsonl")), false); + assert.equal(existsSync(join(root, "source/summary.json")), false); + + const repeatedRoot = mkdtempSync(join(tmpdir(), "understudy-workload-traces-repeat-")); + try { + requests = 0; + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: repeatedRoot, + gatewayUrl: "http://127.0.0.1:8789", + retries: 0, + requestPage: async () => { + requests += 1; + return requests === 1 ? responsePage([first], "cursor_1") : responsePage([first]); + }, + fetchCapture: async () => new Response(captureBody(first), { status: 200 }), + }), + /repeated or out-of-order/i, + ); + assert.equal(existsSync(join(repeatedRoot, "source/index.jsonl")), false); + assert.equal(existsSync(join(repeatedRoot, "source/summary.json")), false); + } finally { + rmSync(repeatedRoot, { recursive: true, force: true }); + } + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export leaves no completion markers when every raw object is missing", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-all-missing-")); + const references = [0, 1].map(captureReference); + try { + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + retries: 0, + requestPage: async () => responsePage(references), + fetchCapture: async () => new Response("missing", { status: 404 }), + }), + /no raw captures could be materialized/i, + ); + const source = join(root, "source"); + assert.equal(existsSync(join(source, "index.jsonl")), false); + assert.equal(existsSync(join(source, "skipped.jsonl")), false); + assert.equal(existsSync(join(source, "summary.json")), false); + assert.ok(readdirSync(source).every((name) => !name.includes(".tmp-"))); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export validates response and raw capture identity locally", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-scope-")); + const reference = captureReference(0); + try { + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage([reference], null, { workload_id: "usp_other" }), + fetchCapture: async () => new Response(captureBody(reference), { status: 200 }), + }), + /does not match the requested organization, project, workload, or window/i, + ); + + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage([reference], null, { + ingestion_cutoff: new Date(Date.now() + 120_000).toISOString(), + }), + fetchCapture: async () => new Response(captureBody(reference), { status: 200 }), + }), + /does not match the requested organization, project, workload, or window/i, + ); + + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage([reference]), + fetchCapture: async () => new Response(captureBody(reference, { request_id: "req_other" }), { status: 200 }), + }), + /raw capture identity does not match/i, + ); + assert.equal(existsSync(join(root, "source/index.jsonl")), false); + assert.equal(existsSync(join(root, "source/summary.json")), false); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("failed export leaves resumable raw files but no final index, then adopts valid files on rerun", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-rerun-")); + const references = [0, 1].map(captureReference); + const fetched = []; + try { + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + reuseStoredWindow: true, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage(references), + async fetchCapture(url) { + fetched.push(String(url)); + const reference = references.find((entry) => entry.url === String(url)); + return reference === references[1] + ? new Response("temporary failure", { status: 503 }) + : new Response(captureBody(reference), { status: 200 }); + }, + }), + /status 503/i, + ); + assert.equal(existsSync(join(root, "source/index.jsonl")), false); + assert.equal(existsSync(join(root, "source/summary.json")), false); + assert.ok(readdirSync(join(root, "source/traces")).every((name) => !name.includes(".partial"))); + + fetched.length = 0; + const requestBodies = []; + const result = await exportWorkloadTraceWindow({ + ...identity, + from: "2026-08-30T00:00:00.000Z", + to: "2026-08-31T00:00:00.000Z", + reuseStoredWindow: true, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async (body) => { + requestBodies.push(body); + return responsePage(references); + }, + async fetchCapture(url) { + fetched.push(String(url)); + const reference = references.find((entry) => entry.url === String(url)); + return new Response(captureBody(reference), { status: 200 }); + }, + }); + assert.equal(result.captureCount, 2); + assert.equal(result.adoptedCount, 1); + assert.deepEqual(fetched, [references[1].url]); + assert.deepEqual(requestBodies, [{ ...window }], "a rolling rerun must resume the originally bound day"); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export retries transient failures and resumes after a later page fails", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-paginated-resume-")); + const references = [0, 1].map(captureReference); + const firstPage = responsePage([references[0]], "cursor_1"); + const secondPage = responsePage([references[1]]); + let initialPageAttempts = 0; + let laterPageAttempts = 0; + let firstCaptureAttempts = 0; + try { + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 1, + async requestPage(body) { + if (body.cursor === undefined) { + initialPageAttempts += 1; + if (initialPageAttempts === 1) throw new TypeError("transient initial page failure"); + return firstPage; + } + laterPageAttempts += 1; + throw new TypeError("page two remains unavailable"); + }, + async fetchCapture(url) { + assert.equal(String(url), references[0].url); + firstCaptureAttempts += 1; + return firstCaptureAttempts === 1 + ? new Response("temporary failure", { status: 503 }) + : new Response(captureBody(references[0]), { status: 200 }); + }, + }), + /page two remains unavailable/i, + ); + assert.equal(initialPageAttempts, 2, "the initial page should succeed on its configured retry"); + assert.equal(firstCaptureAttempts, 2, "the first capture should succeed on its configured retry"); + assert.equal(laterPageAttempts, 2, "the later page should exhaust its configured retry"); + const sourceDirectory = join(root, "source"); + const tracesDirectory = join(sourceDirectory, "traces"); + assert.equal(readdirSync(tracesDirectory).length, 1, "the completed first-page download remains resumable"); + assert.equal(existsSync(join(sourceDirectory, "index.jsonl")), false); + assert.equal(existsSync(join(sourceDirectory, "skipped.jsonl")), false); + assert.equal(existsSync(join(sourceDirectory, "summary.json")), false); + assert.ok(readdirSync(sourceDirectory).every((name) => !name.includes(".tmp-"))); + + const rerunPageBodies = []; + const rerunCaptureUrls = []; + const resumed = await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + async requestPage(body) { + rerunPageBodies.push(body); + return body.cursor === undefined ? firstPage : secondPage; + }, + async fetchCapture(url) { + rerunCaptureUrls.push(String(url)); + assert.equal(String(url), references[1].url); + return new Response(captureBody(references[1]), { status: 200 }); + }, + }); + + assert.deepEqual(rerunPageBodies, [ + { ...window }, + { ...window, ingestion_cutoff: cutoff, cursor: "cursor_1" }, + ]); + assert.deepEqual(rerunCaptureUrls, [references[1].url]); + assert.equal(resumed.adoptedCount, 1); + assert.equal(resumed.writtenCount, 1); + assert.deepEqual( + readFileSync(join(sourceDirectory, "index.jsonl"), "utf8") + .trim().split("\n").map((line) => JSON.parse(line).request_id), + references.map((reference) => reference.request_id), + ); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export retries timeout errors", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-timeout-retry-")); + const reference = captureReference(0); + let attempts = 0; + try { + const result = await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 1, + requestPage: async () => { + attempts += 1; + if (attempts === 1) throw new DOMException("timed out", "TimeoutError"); + return responsePage([reference]); + }, + fetchCapture: async () => new Response(captureBody(reference), { status: 200 }), + }); + assert.equal(attempts, 2); + assert.equal(result.captureCount, 1); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export refuses to mix a different selected day into an existing output", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-bound-day-")); + const reference = captureReference(0); + try { + await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage([reference]), + fetchCapture: async () => new Response(captureBody(reference), { status: 200 }), + }); + const indexPath = join(root, "source/index.jsonl"); + const originalIndex = readFileSync(indexPath, "utf8"); + let requested = false; + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + from: "2026-08-28T00:00:00.000Z", + to: "2026-08-29T00:00:00.000Z", + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => { + requested = true; + return responsePage([reference]); + }, + }), + /different day.*fresh --out/i, + ); + assert.equal(requested, false); + assert.equal(readFileSync(indexPath, "utf8"), originalIndex, "scope rejection must preserve the completed export"); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("workload trace export refuses to reinterpret an explicit date as a rolling resume", async () => { + const root = mkdtempSync(join(tmpdir(), "understudy-workload-traces-window-mode-")); + const reference = captureReference(0); + try { + await exportWorkloadTraceWindow({ + ...identity, + ...window, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => responsePage([reference]), + fetchCapture: async () => new Response(captureBody(reference), { status: 200 }), + }); + let requested = false; + await assert.rejects( + () => exportWorkloadTraceWindow({ + ...identity, + from: "2026-08-30T00:00:00.000Z", + to: "2026-08-31T00:00:00.000Z", + reuseStoredWindow: true, + outputDirectory: root, + gatewayUrl: "http://127.0.0.1:8789", + concurrency: 1, + retries: 0, + requestPage: async () => { + requested = true; + return responsePage([reference]); + }, + }), + /different window selection mode.*fresh --out/i, + ); + assert.equal(requested, false); + } finally { + rmSync(root, { recursive: true, force: true }); + } +});