Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
schema: ratchet
created: 2026-07-09
standards: [delegated-lifecycle, documentation, testing]
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
Feature: Agent-cmd override prints a one-line notice
As a ratchet operator
I want an active RATCHET_BATCH_AGENT_CMD / RATCHET_EVAL_AGENT_CMD to be loudly surfaced
So that a leftover test override can never silently replace the configured coding agent

Scenario: batch apply text output carries the override notice
Given RATCHET_BATCH_AGENT_CMD is set to a stand-in command
When `ratchet batch apply` runs a step that spawns the agent
Then the rendered step result includes the one-line notice "⚠ agent overridden by RATCHET_BATCH_AGENT_CMD"

Scenario: batch apply --json output carries agentOverride
Given RATCHET_BATCH_AGENT_CMD is set to a stand-in command
When `ratchet batch apply --json` runs a step that spawns the agent
Then the emitted step-result JSON has an "agentOverride" field set to true

Scenario: eval run text output carries the override notice
Given RATCHET_EVAL_AGENT_CMD is set to a stand-in command
When `ratchet eval run` executes
Then the scorecard output includes the one-line notice "⚠ agent overridden by RATCHET_EVAL_AGENT_CMD"

Scenario: eval run --json output carries agentOverride
Given RATCHET_EVAL_AGENT_CMD is set to a stand-in command
When `ratchet eval run --json` executes
Then the emitted run JSON has an "agentOverride" field set to true

Scenario: no override means no notice and no flag
Given neither RATCHET_BATCH_AGENT_CMD nor RATCHET_EVAL_AGENT_CMD is set
When `ratchet batch apply` or `ratchet eval run` executes
Then no override notice line is printed
And the --json output carries no "agentOverride" field
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
Feature: Override provenance is stamped on journal entries and run records
As a ratchet operator auditing a batch or eval run
I want every journal entry and run record produced under an agent-cmd override marked "via: env-override"
So that synthetic runs are distinguishable from real agent work after the fact

Scenario: engine transition-outcome journal entry is stamped
Given RATCHET_BATCH_AGENT_CMD is set to a stand-in command
When the engine spawns a step and records its transition-outcome journal entry
Then the appended journal entry carries "via": "env-override"

Scenario: agent-reported journal entries are stamped
Given a `ratchet batch report` invocation whose process environment carries an active RATCHET_BATCH_AGENT_CMD
When it appends a progress, blocker, needs-input, or completion entry
Then the appended journal entry carries "via": "env-override"

Scenario: eval run record is stamped
Given RATCHET_EVAL_AGENT_CMD is set to a stand-in command
When `ratchet eval run` persists the run under .ratchet/evals/runs/
Then the persisted run record carries "via": "env-override"

Scenario: work produced without an override stays unstamped
Given neither RATCHET_BATCH_AGENT_CMD nor RATCHET_EVAL_AGENT_CMD is set
When journal entries and eval run records are produced
Then none of them carry a "via" field
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Feature: One shared helper owns override-aware spawn-request construction
As a ratchet maintainer
I want the env-override gate and spawn-request construction to exist in exactly one helper
So that the batch engine, the eval judge, and the mutation harness cannot drift apart (the #67 triplication)

Scenario: the three spawn seams share one override gate
Given the batch engine, the eval judge, and the mutation harness each build an agent spawn request
When their override env var is set to a stand-in command
Then each produces the same `bash -c <override>` request shape through the shared helper
And each reports that the agent was overridden

Scenario: a whitespace-only override is inactive
Given RATCHET_BATCH_AGENT_CMD is set to only whitespace
When a spawn request is built
Then the configured adapter path is used
And the agent is not reported as overridden

Scenario: an override-built request threads env like any other request
Given a spawn request built under an active override with a per-step env var
When a rex runtime builds the launch command for it
Then the launch command exports the per-step env var before invoking the override command
65 changes: 65 additions & 0 deletions .ratchet/changes/gate-and-mark-agent-cmd-override/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# gate-and-mark-agent-cmd-override

## Why

`RATCHET_BATCH_AGENT_CMD` / `RATCHET_EVAL_AGENT_CMD` are read unconditionally at spawn time in production code, in three drifting copies (`src/core/batch/engine/engine.ts:835`, `src/core/eval/judge.ts:274`, `src/core/eval/mutation-harness.ts:154`): any leftover value — from an eval session, CI, a `.envrc` — silently replaces the configured coding agent with `bash -c <value>`, with no console signal and journal entries/run records indistinguishable from real agent work (issue #80; the audit hole behind #78's self-attested completions). This change makes an active override loud (one-line notice + `agentOverride: true` in `--json`), auditable (`via: "env-override"` provenance on journal entries and eval run records), and single-sourced (one shared override-aware spawn-request helper, coordinating with the #67 triplication).

## What Changes

- A shared override-aware spawn-request helper in `src/core/batch/engine/agent.ts` (exported through `src/core/batch/engine/index.ts`, which `judge.ts`/`mutation-harness.ts` already import from): `activeAgentCmdOverride(envVar, env)` (trimmed non-empty value or `undefined`), `buildAgentSpawnRequest({ overrideEnvVar, instructions, cwd, env, buildAdapterRequest })` returning `{ request, agentOverride }`, `agentOverrideNotice(envVar)` (the `⚠ agent overridden by <VAR>` line), and an `ENV_OVERRIDE_PROVENANCE = 'env-override'` constant. All three spawn seams — `engine.ts` `buildSpawnRequest`, `judge.ts` `buildVoteRequest`, `mutation-harness.ts` `buildSeedRequest` — delegate their override branch to it. Implements `features/agent-cmd-override/shared-spawn-helper.feature`.
- `JournalEntry` (`src/core/batch/journal.ts:55`) gains an optional `via?: 'env-override'` field; the engine stamps it on the transition-outcome entry it appends (`engine.ts:807`) when the step's spawn used the override, and `ratchet batch report` (`src/commands/batch/report.ts:84-106`) stamps every entry it appends when its own process environment carries an active `RATCHET_BATCH_AGENT_CMD` (the spawned stand-in inherits the var, so stub-reported completions become auditable — the #78 hole). Implements `features/agent-cmd-override/override-provenance.feature`.
- `StepResult` (`src/core/batch/engine/contract.ts:230`) and `EngineStepOutcome` gain an optional `agentOverride?: true`; `batch apply --json` therefore carries the field verbatim (`renderResult`, `src/commands/batch/apply.ts:1085`), and the text path prints the one-line notice when set. Implements `features/agent-cmd-override/override-notice.feature`.
- `EvalRun` (`src/core/eval/run.ts:63`) gains an optional `via?: 'env-override'` stamped at run persistence when `RATCHET_EVAL_AGENT_CMD` is active; `eval run --json` adds a top-level `agentOverride: true` and the text scorecard prints the notice line (`src/commands/eval/run.ts:76-97`).
- Both rex runtime test suites assert an override-built request (`bash -c <override>` + per-step env) threads env through the launch command like any adapter-built request, so the phase proof-of-work run (`npm test -- test/batch-engine/rex-sidecar-runtime.test.ts test/batch-engine/rex-remote-runtime.test.ts`) genuinely covers the override path's composition with #89's env threading.
- Reference docs updated in the same change: `docs/engine/agent-runtime.md` (override seam: notice, `agentOverride` JSON field, `via: env-override` journal provenance) and `docs/eval-mutation-harness.md` (its two existing `RATCHET_EVAL_AGENT_CMD` mentions gain the notice/provenance contract); `README.md` checked for affected surfaces.
- Field additions are optional and absent when no override is active — no journal/run-record migration, byte-identical output for override-free runs. Not breaking.
- Out of scope (kept thin per the vertical-slice strategy): the issue's "consider an explicit opt-in flag / NODE_ENV gate" (proposal item 3) — the definition of done requires notice + provenance + shared helper, and an opt-in gate would break every existing e2e/eval harness invocation; revisit if #78's corroboration work (phase 2) still needs it. Narrowing what env reaches the agent stays #86.

## Design

**One helper, three seams (`delegated-lifecycle`, #67 coordination).** The override gate moves into `agent.ts` — already the home of `AgentSpawnRequest`, the adapter registry, and `resolveAdapter` — as `buildAgentSpawnRequest`, which checks `activeAgentCmdOverride(overrideEnvVar, process.env)` and returns either the `bash -c <override>` request (`agentOverride: true`) or `buildAdapterRequest()`'s result (`agentOverride: false`). Site-specific adapter resolution (the engine's stage-map/spec/`emitsStreamJson` logic, the eval side's bare `resolveAdapter(agentName)`) stays at each call site inside the `buildAdapterRequest` closure — the helper owns only the shared override semantics (trim check, request shape, flag), so the gate exists in exactly one place without flattening genuinely different adapter paths. This satisfies the phase criterion "spawn-request construction lives in one shared helper" at the engine layer; `runtime/spawn-command.ts` (the sibling change's shell-serialization helper) stays a separate module because it operates at a different layer (rex launch-command text, consumed by both runtimes), and merging the two would couple engine-level request construction to rex-only shell details. Per the `delegated-lifecycle` standard the helper is purely mechanical — it carries no instruction text, no standards loading, and no done-semantics; lifecycle authorship stays in the shared skill/workflow layer untouched.

**Notice rides the result, not a side-channel print.** The engine sets `agentOverride: true` on the step outcome; `toStepResult` carries it to `StepResult`. `renderResult` in `--json` mode then emits the field with zero extra plumbing (it already `JSON.stringify`s the whole result), and in text mode prints `⚠ agent overridden by RATCHET_BATCH_AGENT_CMD` as its first line for the step. This keeps `--json` output a single well-formed document (no notice line interleaved into JSON stdout) and the notice appears exactly when a spawn actually ran under the override. Pre-spawn parks (`notAdvanced`) print no notice — nothing was spawned. `eval run` follows the same shape: `executeRun` records the override state once for the run (`EvalRun.via`), and the command layer adds `agentOverride: true` to the JSON payload / a notice line atop the text scorecard. The eval stamp/notice keys on the var being ACTIVE for the run (deterministic, documented), not on whether a given case happened to spawn — a run executed with the seam armed is synthetic evidence regardless of which contributors fired.

**Provenance at both producers.** Journal entries have two producers: the engine host loop (transition-outcome entry, `engine.ts:807`) and the `ratchet batch report` verb invoked *by the spawned agent*. The engine stamps from its in-hand `agentOverride` flag; `report.ts` stamps from its own process env via the same `activeAgentCmdOverride` helper — the spawned stand-in inherits `RATCHET_BATCH_AGENT_CMD` (the engine builds `env` from `process.env`, engine.ts:332, and #89's threading now delivers it), so a stub's `--complete` entry is stamped even though the engine never sees that append. `via` is optional and only ever `'env-override'` today (a string-literal union left open to widen later); readers ignore it, so `readJournal`/status/fold paths need no change. `ProofOfWorkRecord` is not stamped: the boundary proof is host-run (never through the agent seam), and stamping it is #82/phase-2 territory.

**Ecosystem/agent neutrality.** Nothing ships into consuming repos (`generalizable-defaults`: the notice text and provenance constant are CLI/runtime output, not generated artifacts) and no skill/template changes (`instruction-fed-config` untouched). The helper treats every adapter uniformly and names no specific agent (`multi-agent-support`): there are no per-agent outputs to enumerate.

**Testing (per the `testing` standard — right layer, pyramid-weighted).** Unit tests (no fs, no spawn) cover the helper: active/whitespace-only/unset var, `bash -c` request shape carrying instructions/cwd/env, fallback closure invoked exactly once when inactive, notice text per var name. Integration tests cover each producer/surface: `test/batch-engine/engine-agent-override.test.ts` extends to assert `StepResult.agentOverride` and the stamped outcome entry; `test/commands/batch/report.test.ts` asserts entries appended under an active var carry `via: 'env-override'` (and stay unstamped without it); `test/commands/batch/apply.test.ts` asserts the text notice line and the `--json` field; `test/commands/eval/run.test.ts` asserts the persisted run record stamp, the JSON `agentOverride`, and the text notice; existing judge/mutation-harness tests keep passing with the delegated override branch. The rex runtime suites gain one assertion each: an override-shaped request threads a per-step env var into the launch command (sidecar run-op / remote `/execute` body), tying this change into the phase proof-of-work run. Test headers name the `.feature` files they implement. Full suite and the coverage gate stay green at or above the enforced `COVERAGE_THRESHOLD`.

**Documentation (per the `documentation` standard).** `docs/engine/agent-runtime.md` (existing Reference doc for this core flow, already touched by the sibling change) documents the override seam contract — the notice line, the `agentOverride` JSON field, and `via: env-override` journal provenance — and `docs/eval-mutation-harness.md`'s existing `RATCHET_EVAL_AGENT_CMD` entries (lines 126, 230) gain the same contract for eval runs and run records. Existing diagrams in both docs are re-verified against the changed flow (the override seam adds fields, not new components — no new diagram, per the standard's "use diagrams deliberately"). `README.md` is checked for any description of the override/journal surfaces and updated to match (expected no-op).

## Tasks

**1. Shared override-aware spawn-request helper**

- [x] 1.1 Add `activeAgentCmdOverride`, `buildAgentSpawnRequest`, `agentOverrideNotice`, and `ENV_OVERRIDE_PROVENANCE` to `src/core/batch/engine/agent.ts`; export them via `src/core/batch/engine/index.ts`
- [x] 1.2 Unit tests (no fs, no spawn) in `test/batch-engine/agent.test.ts`: active/whitespace-only/unset override, `bash -c` request shape (instructions/cwd/env carried), fallback closure used exactly once when inactive, notice text per env-var name; header names `features/agent-cmd-override/shared-spawn-helper.feature`

**2. Batch engine + apply surface**

- [x] 2.1 `engine.ts` `buildSpawnRequest` delegates its override branch to `buildAgentSpawnRequest` and returns the `agentOverride` flag; the step path threads it onto `EngineStepOutcome`/`StepResult` (`agentOverride?: true`) and stamps `via: 'env-override'` on the transition-outcome journal entry (`engine.ts:807`) when set
- [x] 2.2 Add `via?: 'env-override'` to `JournalEntry` (`src/core/batch/journal.ts`); `src/commands/batch/report.ts` stamps every entry it appends when `activeAgentCmdOverride('RATCHET_BATCH_AGENT_CMD', process.env)` is active
- [x] 2.3 `renderResult` (`src/commands/batch/apply.ts`) prints the one-line `⚠ agent overridden by RATCHET_BATCH_AGENT_CMD` notice in text mode when `result.agentOverride` is set (`--json` carries the field via the existing stringify)
- [x] 2.4 Integration tests: `engine-agent-override.test.ts` asserts `StepResult.agentOverride` + stamped outcome entry; `test/commands/batch/report.test.ts` asserts stamped/unstamped entries by env; `test/commands/batch/apply.test.ts` asserts the text notice and `--json` field; headers name `features/agent-cmd-override/override-notice.feature` / `override-provenance.feature`

**3. Eval surface**

- [x] 3.1 `judge.ts` `buildVoteRequest` and `mutation-harness.ts` `buildSeedRequest` delegate their override branch to `buildAgentSpawnRequest` (adapter resolution stays in each `buildAdapterRequest` closure)
- [x] 3.2 `executeRun` stamps `via: 'env-override'` on the persisted `EvalRun` when `RATCHET_EVAL_AGENT_CMD` is active; `src/commands/eval/run.ts` adds top-level `agentOverride: true` to the `--json` payload and a notice line atop the text scorecard
- [x] 3.3 Integration tests in `test/commands/eval/run.test.ts`: persisted run record stamped under an active var (unstamped without), `--json` `agentOverride`, text notice; judge/mutation-harness suites stay green on the delegated branch; headers name the implemented `.feature` files

**4. Rex runtime composition (phase proof-of-work)**

- [x] 4.1 Add one assertion to each of `test/batch-engine/rex-sidecar-runtime.test.ts` and `test/batch-engine/rex-remote-runtime.test.ts`: an override-shaped request (`bash -c <cmd>` with a per-step env var) has that var exported in the built launch command — the override path composes with #89's env threading; headers reference `features/agent-cmd-override/shared-spawn-helper.feature`

**5. Documentation (mandatory — `documentation` standard, Reference docs)**

- [x] 5.1 Update `docs/engine/agent-runtime.md`: document the override seam contract — the one-line notice, `agentOverride: true` in `--json`, `via: env-override` journal provenance, and the single shared helper; re-verify the doc's existing diagram/tables still depict the code accurately
- [x] 5.2 Update `docs/eval-mutation-harness.md`: the `RATCHET_EVAL_AGENT_CMD` entries state the notice and the `via: env-override` run-record stamp
- [x] 5.3 Check `README.md` for any description of the override/journal/run-record surfaces this change alters and update it to match (expected no-op)

**6. Verification**

- [x] 6.1 Run `npm test -- test/batch-engine/rex-sidecar-runtime.test.ts test/batch-engine/rex-remote-runtime.test.ts` — exit code 0 with the override-composition assertions in place (phase proof-of-work)
- [x] 6.2 Run the full test suite and the coverage gate — green, coverage at or above the enforced `COVERAGE_THRESHOLD` (`testing` standard)
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
schema: ratchet
created: 2026-07-09
standards: [documentation, testing]
Loading
Loading