Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,8 @@ understudy captures list --project rehearsal --workload classify
understudy captures export --request-ids-file request-ids.txt --project rehearsal --out .understudy/capture-batch --include-payload --yes
understudy traces export <trace-id> --project rehearsal --out .understudy/trace-exports --include-payload --yes
understudy traces export --trace-ids-file trace-ids.txt --project rehearsal --out .understudy/trace-exports --include-payload --yes
understudy traces export --project rehearsal --workload classify --date 2026-08-29 --out .understudy/trace-exports/day --include-payload --yes
understudy evals build --project rehearsal --workload classify --name classification-day --out .understudy/evals/classification-day --yes
understudy routes set classify --project rehearsal --model-id glm-5.1 --traffic-pct 10
understudy routes show classify --project rehearsal
understudy routes clear classify --project rehearsal
Expand Down Expand Up @@ -324,13 +326,15 @@ file-only, and requires `--include-payload --yes`. For a customer-owned batch,
put one request id per line in a file and pass `--request-ids-file`; the CLI
retries transient failures, resumes from completed files, and writes
`failed-request-ids.txt`. Redacted batch files use `.summary.json`, keeping
them distinct from full-payload `.payload.json` files. `traces export` resolves
one explicit `trace_id` through the customer trace request-ID endpoint, then
reuses that same bounded request exporter for every returned ID. Use a
positional trace ID or an explicit `--trace-ids-file`; there is no unbounded
`--all` trace scan. Each trace writes a private `trace.json` membership manifest,
per-request summary or payload files, `failed-request-ids.txt`, and a batch-level
`failed-trace-ids.txt`. `models list` shows public Understudy model IDs and
them distinct from full-payload `.payload.json` files. `traces export` can
resolve explicit trace IDs through the customer trace request-ID endpoint, or
download exactly one raw workload day through `--workload`. Workload mode uses
the rolling latest 24 hours by default; `--date YYYY-MM-DD` selects a completed
UTC calendar day. There is no unbounded `--all` trace scan. Explicit traces
write a private `trace.json` membership manifest, per-request files, and batch
failure manifests. Workload days write only a final ordered
`source/index.jsonl` and `source/summary.json` after success. `models list`
shows public Understudy model IDs and
display names only.

If the coding agent has an approved native email connector, it may complete the
Expand Down
15 changes: 10 additions & 5 deletions docs/current-functionality.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ understudy captures export <request-id> --out .understudy/captures/<request-id>.
understudy captures export --request-ids-file request-ids.txt --project rehearsal --out .understudy/capture-batch --include-payload --yes
understudy traces export <trace-id> --project rehearsal --out .understudy/trace-exports --include-payload --yes
understudy traces export --trace-ids-file trace-ids.txt --project rehearsal --out .understudy/trace-exports --include-payload --yes
understudy traces export --project rehearsal --workload classify --date 2026-08-29 --out .understudy/trace-exports/day --include-payload --yes
understudy evals build --project rehearsal --workload classify --name classification-day --out .understudy/evals/classification-day --yes
understudy routes show classify --project rehearsal
understudy routes set classify --project rehearsal --model-id glm-5.1 --traffic-pct 10
understudy routes clear classify --project rehearsal
Expand Down Expand Up @@ -150,11 +152,14 @@ model and what remains passthrough/frontier.
Hosted capture commands are metadata-first. `captures list` and `captures get`
redact prompt/completion-bearing fields into presence booleans. Full capture
export is opt-in with `--include-payload --yes`, writes only to a file, and never
prints raw payloads to stdout. `traces export` calls the customer trace
request-ID lookup for one explicit `trace_id`, then passes the returned IDs to
the existing request capture batch exporter. It never walks the project capture
catalog. Full per-request captures require the same explicit payload opt-in;
there is no unbounded all-traces operation.
prints raw payloads to stdout. `traces export` either resolves explicit trace
IDs through the customer trace request-ID endpoint or downloads one exact raw
workload day. Workload mode defaults to the rolling latest 24 hours and accepts
`--date YYYY-MM-DD` for a completed UTC calendar day. It requires the same
explicit payload opt-in, writes owner-private files, and emits only counts,
paths, and hashes; there is no unbounded all-traces operation. `evals build`
uses the same one-day primitive before handing semantic authoring to the coding
agent.

`optimize-workload check` reads `.understudy/capture-evidence/`
artifacts, fails closed on missing files, invalid JSON, stale baseline hashes,
Expand Down
26 changes: 16 additions & 10 deletions docs/privacy-and-data-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,16 +74,22 @@ only to local files (mode `600` on Unix), redacted `.summary.json` files cannot
be mistaken for full-payload `.payload.json` files during resume, and stdout
contains counts and paths rather than capture content.

`understudy traces export` applies the same rule to one explicit hosted
`trace_id` or a private file of explicit trace IDs. The CLI resolves membership
through the customer trace request-ID endpoint and passes the returned IDs to
the existing bounded capture batch exporter; it never scans project capture
history and exposes no unbounded `--all` operation. A private `trace.json`
records membership and counts without raw bodies. Full per-request files require
`--include-payload --yes`, and stdout still contains only counts and paths.
Owner-private files, disjoint summary/payload suffixes, bounded concurrency,
retries, resume, `failed-request-ids.txt`, and `failed-trace-ids.txt` preserve
the request-export boundary.
`understudy traces export` supports two bounded modes. Explicit mode accepts one
hosted `trace_id` or a private file of explicit trace IDs, resolves membership
through the customer trace request-ID endpoint, and passes those IDs to the
capture batch exporter. Workload-window mode requires an explicit workload and
exports exactly 24 hours: `--date YYYY-MM-DD` selects one completed UTC calendar
day, while omitting it selects the rolling 24 hours ending when the command
starts. It has no unbounded `--all` operation. Workload-window export always
contains full payloads and therefore requires `--include-payload --yes`.

Both modes write payloads only to owner-private local files and never print
prompts, completions, or tool payloads to stdout. Explicit mode records a
private `trace.json`; workload-window mode records a final ordered
`source/index.jsonl` and `source/summary.json` only after the complete export.
Bounded concurrency, retries, atomic writes, and validation of the requested
organization, project, workload, and request identity preserve the export
boundary.

Gateway probes are explicit live calls. BYOK provider keys are read only from an
environment variable named by `--byok-env`; they are not requested in chat, not
Expand Down
22 changes: 9 additions & 13 deletions schemas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,23 @@ app, skills, CLI, ladder). One spine, adopted everywhere.

## Local workload eval authoring

The `understudy.eval-project.v2`, export-proof, execution-index-row, metric,
The `understudy.eval-project.v2`, execution-index-row, metric,
coverage, harness, environment, splits, check-fixtures, check-report, and approval schemas define the private
coding-agent workspace checked by `understudy evals check`. The workload
profile remains Markdown; its exact bytes are bound by both intent approval and
the deterministic check-input hash. These contracts require a provider-free
local environment replay, independent good/wrong evidence, explicit lineage
coverage, and a separate post-check owner approval.
coverage, and a separate post-check owner approval. The project source records
requested, materialized, and skipped capture counts and the private portable
`source/skipped.jsonl` index so agents can qualify coverage claims when a raw
object is no longer available.

The `understudy.eval-publication.v1` and `understudy.eval-release.v1` JSON
Schemas define the structural hosted boundary for this workflow. Publication
carries the checked hashes, a compact backend-verifiable source attestation,
final approval, executable layout, and bundle inventory. The server response
adds the immutable release seal. Neither contract contains raw source traces,
the expiring export receipt, the local export-proof file, or mutable authoring
state.

Within the private project manifest, `source.export_proof_sha256` binds the
exact local export-proof file. Within the check report and hosted
publication/release source, that field instead binds the exact opaque
`source_attestation` token. The CLI verifies both links before upload, and the
backend verifies the attestation itself before sealing a release.
carries the exact 24-hour source window, local index/count/byte commitments,
checked hashes, final approval, executable layout, and bundle inventory. The
server response adds the immutable release seal. Neither contract contains raw
source traces or mutable authoring state.

These Draft 2020-12 schemas do not express the release contract's cross-field
path rules. Consumers must also parse publications with the package's exported
Expand Down
3 changes: 1 addition & 2 deletions schemas/understudy.eval-check.v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,11 @@
"source": {
"type": "object",
"additionalProperties": false,
"required": ["scope", "scope_sha256", "index_sha256", "export_proof_sha256", "capture_count", "size_bytes"],
"required": ["scope", "scope_sha256", "index_sha256", "capture_count", "size_bytes"],
"properties": {
"scope": { "$ref": "#/$defs/scope" },
"scope_sha256": { "$ref": "#/$defs/sha" },
"index_sha256": { "$ref": "#/$defs/sha" },
"export_proof_sha256": { "$ref": "#/$defs/sha" },
"capture_count": { "type": "integer", "minimum": 0 },
"size_bytes": { "type": "integer", "minimum": 0 }
}
Expand Down
3 changes: 1 addition & 2 deletions schemas/understudy.eval-draft-check.v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,11 @@
"source": {
"type": "object",
"additionalProperties": false,
"required": ["scope", "scope_sha256", "index_sha256", "export_proof_sha256", "capture_count", "size_bytes"],
"required": ["scope", "scope_sha256", "index_sha256", "capture_count", "size_bytes"],
"properties": {
"scope": { "$ref": "#/$defs/scope" },
"scope_sha256": { "$ref": "#/$defs/sha" },
"index_sha256": { "$ref": "#/$defs/sha" },
"export_proof_sha256": { "$ref": "#/$defs/sha" },
"capture_count": { "type": "integer", "minimum": 0 },
"size_bytes": { "type": "integer", "minimum": 0 }
}
Expand Down
57 changes: 0 additions & 57 deletions schemas/understudy.eval-export-proof.v1.schema.json

This file was deleted.

13 changes: 6 additions & 7 deletions schemas/understudy.eval-project.v2.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"source": {
"type": "object",
"additionalProperties": false,
"required": ["window", "capture_count", "size_bytes", "index", "index_sha256", "export_proof", "export_proof_sha256", "exported_capture_count", "exported_total_bytes", "terminal_receipt_verified"],
"required": ["window", "requested_count", "materialized_count", "skipped_count", "skipped_index", "capture_count", "size_bytes", "index", "index_sha256"],
"properties": {
"window": {
"type": "object",
Expand All @@ -42,15 +42,14 @@
"ingestion_cutoff": { "$ref": "#/$defs/timestamp" }
}
},
"requested_count": { "$ref": "#/$defs/count" },
"materialized_count": { "$ref": "#/$defs/count" },
"skipped_count": { "$ref": "#/$defs/count" },
"skipped_index": { "$ref": "#/$defs/path" },
"capture_count": { "$ref": "#/$defs/count" },
"size_bytes": { "$ref": "#/$defs/count" },
"index": { "$ref": "#/$defs/path" },
"index_sha256": { "$ref": "#/$defs/sha" },
"export_proof": { "$ref": "#/$defs/path" },
"export_proof_sha256": { "$ref": "#/$defs/sha" },
"exported_capture_count": { "$ref": "#/$defs/count" },
"exported_total_bytes": { "$ref": "#/$defs/count" },
"terminal_receipt_verified": { "const": true }
"index_sha256": { "$ref": "#/$defs/sha" }
}
},
"artifacts": {
Expand Down
6 changes: 2 additions & 4 deletions schemas/understudy.eval-publication.v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,14 @@
"source": {
"type": "object",
"additionalProperties": false,
"required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256", "export_proof_sha256", "source_attestation"],
"required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256"],
"properties": {
"from": { "$ref": "#/$defs/timestamp" },
"to": { "$ref": "#/$defs/timestamp" },
"ingestion_cutoff": { "$ref": "#/$defs/timestamp" },
"capture_count": { "type": "integer", "minimum": 0 },
"total_bytes": { "type": "integer", "minimum": 0 },
"local_index_sha256": { "$ref": "#/$defs/sha" },
"export_proof_sha256": { "$ref": "#/$defs/sha" },
"source_attestation": { "type": "string", "minLength": 1, "maxLength": 8192 }
"local_index_sha256": { "$ref": "#/$defs/sha" }
}
},
"artifacts": {
Expand Down
5 changes: 2 additions & 3 deletions schemas/understudy.eval-release.v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,10 @@
"path": { "type": "string", "minLength": 1, "maxLength": 240, "pattern": "^(?!.*\\u0000)(?!/)(?![A-Za-z]:)(?!.*\\\\)(?!.*(?:^|/)(?:\\.|\\.\\.)(?:/|$))(?!.*//)(?!.*\/$).+$" },
"source": {
"type": "object", "additionalProperties": false,
"required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256", "export_proof_sha256", "source_attestation"],
"required": ["from", "to", "ingestion_cutoff", "capture_count", "total_bytes", "local_index_sha256"],
"properties": {
"from": { "$ref": "#/$defs/timestamp" }, "to": { "$ref": "#/$defs/timestamp" }, "ingestion_cutoff": { "$ref": "#/$defs/timestamp" },
"capture_count": { "type": "integer", "minimum": 0 }, "total_bytes": { "type": "integer", "minimum": 0 }, "local_index_sha256": { "$ref": "#/$defs/sha" },
"export_proof_sha256": { "$ref": "#/$defs/sha" }, "source_attestation": { "type": "string", "minLength": 1, "maxLength": 8192 }
"capture_count": { "type": "integer", "minimum": 0 }, "total_bytes": { "type": "integer", "minimum": 0 }, "local_index_sha256": { "$ref": "#/$defs/sha" }
}
},
"artifacts": {
Expand Down
13 changes: 9 additions & 4 deletions skills/capture-evidence/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@ When the developer names a workload already captured by Understudy and the
active credentials can read it, use the hosted-workload front door in
[`references/hosted-workload-eval.md`](references/hosted-workload-eval.md).
An active `understudy.eval-project.v2` is a separate, project-local branch: use
its exact seven-day source, author only inside that eval project, and make the
its exact one-day raw source, author only inside that eval project, and make the
coding agent—not the CLI—the conversational frontend. Infer the goal, metric,
and failure taxonomy from the traces and repository before asking targeted gap
questions. Even without owner confirmation, continue to an explicitly
questions. Reconcile any capture references recorded in
`source/skipped.jsonl` before making coverage claims. Even without owner
confirmation, continue to an explicitly
provisional local draft and run `understudy evals check --draft`.
Do not run the incumbent baseline, null floor, a provider model, or a hosted
EvalWorkspace on that branch. Strict `understudy evals check`, final approval,
Expand Down Expand Up @@ -205,10 +207,13 @@ understudy evals build \
--project <project> \
--workload <workload> \
--name <eval-name> \
--out .understudy/evals/<eval-dir>
--out .understudy/evals/<eval-dir> \
--yes
```

This is a local-authoring operation: the service selects and freezes the source
By default this downloads the rolling 24 hours ending when the command starts;
add `--date YYYY-MM-DD` for one completed UTC calendar day. This is a
local-authoring operation: the CLI mechanically transports and validates raw
captures, while the coding agent owns lineage reconstruction, tool-call
interpretation, task and contract proposals, verifier generation, and review.
Do not silently substitute an older server-generated environment for the local
Expand Down
Loading