Repository navigation
feat(harness): close ten harness gaps found against OpenCode/Hermes, and measure them - #632
Merged
0x-copilot-dev merged 190 commits intoAug 14, 2026
Merged
Conversation
…y PR `grep -rln 'desktop-journeys' .github/workflows/` returned zero files. The harness that drives the real packaged app — supervised Electron + embedded PostgreSQL + all three Python services — and that repeatedly caught "unit-tests-pass-but-the-real-app-is-broken" regressions ran on no PR at all. The nearest thing, desktop-supervised-boot-drill.yml, is schedule + dispatch only and boots the stack as a STAGING process rather than through the shipped app. Adds job `desktop-journey-first-run` to ci-desktop.yml: stage fresh from the PR's tree, build the desktop bundle, drive first_run.py's FR-0 against it. Three things had to exist first. * JOURNEY_PHASES (_lib.py). Since e8622a1 the phases in a journey file share one boot and consume each other's state on purpose, so a whole-file per-PR run inherits stale route/rail/env/run-history and fails in a LATER phase with a symptom-shaped message that reads like a product bug. The selector runs the named phases and DROPS the rest (dropped, not skipped — a skipped phase is deliberately non-zero). A group with no selected phase is never booted. An id matching no declared phase exits 1: a CI gate running zero phases and reporting success is the failure this harness exists to catch. * FR-0 (first_run.py). Every phase in the file hung off a setup that loaded a BYOK key, so the file was unrunnable without a secret and nothing ever asserted the keyless state a new user actually meets. Sign-in becomes the setup; the key add becomes FR-1 and SKIPS without a key instead of failing the group; FR-2..FR-6 declare that dependency. FR-0 asserts the BYOK gate and that the catalog, read through the app, reports nothing configured — no key, no model call, no third-party request. The boot also clears the forwarded provider-key env vars, so FR-0 and FR-3 stop depending on the developer's shell. * tools/test_desktop_journey_ci.py, wired into ci-gates (which runs on every PR, unlike the macOS job it guards). It checks the selector's behaviour and the seam both ways: the workflow pins exactly one phase, that phase exists in first_run.PHASES, it needs no key, and the stage is built fresh rather than cached. RUNTIME_FAKE_MODEL is deliberately NOT set. buildServiceEnv starts from {} and copies only ENV_PASSTHROUGH_ALLOWLIST (service-env.ts), which omits it — the fail-closed guarantee in fake_model.py. Setting it here would be a no-op that reads like coverage, so the hermetic run->stream tail stays in the boot drill. Not a required status check: a new e2e gate earns that after a few runs of observed stability. Verified by running the job's exact command locally against the real supervised stack (embedded Postgres + three services under production posture): FR-0 passed, 301 models offered, none configured, exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…a a ~ glob
The journey job uploaded evidence from two paths, one of them
`~/Library/Application Support/0xCopilot/journey-first-run-*/logs/**`.
A leading `~` in an upload `path:` is expanded by @actions/glob rather
than by the shell, and `if-no-files-found: warn` downgrades a miss to a
warning — so if that expansion does not behave as assumed, the four
supervised services' logs vanish silently from precisely the red run
that needed them, and the artifact still looks populated because the
screenshots came from the other path.
Copy them with `$HOME` in a shell step instead, into the run dir the
screenshots already live in, and upload one in-workspace path. Shell
expansion is verifiable: the step body was extracted from the parsed
YAML and executed under `bash --noprofile --norc -e -o pipefail`, the
same shell Actions uses, against the userData tree a real local FR-0 run
produced — it collected ai-backend / backend / backend-facade / postgres
logs and exited 0.
No `${{ }}` inside the script body (ci-gates header: a substituted value
is a script-injection vector). Every command is guarded, so the step
cannot turn an already-red job into a confusing one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…desktop `ToolResultAdmissionAdapter` bounds an oversized tool result by *moving* it: the bytes go to an object store and the model gets a 4 KiB stub plus a read-back reference. Only the desktop file backend has that object store, so `FileStoreWorkerWiring.tool_result_admission()` returns `None` on web, postgres and in-memory — and `ToolBudgetGuard.admit_tool_result` answered a `None` adapter by returning the raw result. A 5 MB MCP read entered model context whole on every image except the desktop. There were two holes, not one. The second is upstream: when the org has no configured tool budgets `_build_tool_budget_guard` returns `None`, and `RuntimeControlMiddleware._execute`/`_aexecute` returned `handler(request)` directly, skipping `_admit_result` entirely. Measured on the live seam, an in-memory run handed the model all 90,032 characters on both paths. Adds `ToolResultCap`: the same bound expressed for a backend that can retain nothing. Both of its numbers are read off the adapter rather than restated — oversize at `DEFAULT_INLINE_TOKEN_BUDGET` tokens' worth of characters, output within `DEFAULT_OFFLOADED_MODEL_CONTENT_LIMIT_CHARS` — so the two lanes make the same decision and there is no third constant to keep in sync. It keeps a bounded head and states in the message how much was dropped and what to do instead, and it is honest that this backend stored nothing: repeating the offload lane's "read the reference" would cost the model a turn fetching something that does not exist. The cut is by characters, not lines, so a single-line 70 KB JSON blob still previews. Both holes route through the one `_admit_result` seam. A result that fits is returned as the identical object, so a run on a previously unbounded backend behaves reference-for-reference as before until a result is genuinely oversized. The file store's offload path is untouched; a configured adapter's failure still propagates rather than falling back to the cap, because an offload failure on a backend that promised read-back is a broken promise. Also renames `test_non_file_runtime_preserves_no_admission_behavior` — that name asserted the defect — and corrects the `file_store_wiring` module docstring, which claimed the non-file path was "inline behavior, byte-identical". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A provider 429 twenty tool calls into a turn used to cost the whole turn.
The only model-path retry we owned was the run-claim retry in
runtime_worker/loop.py, which restarts the turn and re-pays for every tool
call that already succeeded; underneath it, each provider SDK ran its own
silent retry schedule (openai/anthropic 2, google-genai 6) that we neither
configured, observed, nor bounded.
Adds ModelCallRetryPolicy: classify, honour provider retry headers, else
bounded jittered exponential backoff, give up into the existing typed error
taxonomy. Tuned through hyperparameters.json's new `model_retry` section.
Wired at both branches of the one model-call choke point:
* ModelInvocationMiddleware._awrap_occupancy_only — the branch every shipped
deployment takes, since FeatureModeSet.f10 ships OFF and no release config
exists in the repo. It called the handler exactly once; it now dispatches
under the policy.
* the F10 journaled retry loop, which decided *whether* to retry but
re-dispatched instantly — answering a 429 with an identical request
microseconds later. Admission still owns the attempt budget; the policy
contributes pacing only, so there are not two ceilings.
Three details that carry the correctness:
* No second classifier. Retryability reads the ModelFailureClass that
ProviderFailureClassifier already produced from adapter-attested SDK facts.
The reference implementation's regexes over provider prose are deliberately
not ported — model_failure_adapters states that str(exception) is never
consulted, and a message-shaped rule would retry a 400 whose body happens
to say "timeout".
* Classification outside F10 goes by SDK exception identity, not by the
model's _llm_type. The latter is LangChain's name ("openai-chat"), matches
no adapter key, and would have classified every provider failure as UNKNOWN
— i.e. never retry — making the whole policy dead on the live path.
* Each attempt is still bounded by the model client's own `timeout`; the
backoff is spent between attempts, and SDK-level retries are switched off
so `timeout` bounds one attempt rather than being multiplied by a hidden
attempt count.
A provider-stated wait is clamped to 30s: execution.worker_lock_seconds is
60, so honouring a literal `retry-after: 3600` would let a second worker
claim a run this one is still executing. A call that already streamed
visible text is never re-dispatched, and cancellation is never retried.
_mark_failure documents why the claim retry does not double-count: an
exhausted model call re-raises the same typed error it would have raised
anyway, and the two scopes recover different things.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-in gate
The per-PR journey job would have been red on its first run, for a reason
that has nothing to do with the product.
`DriverSession.start` returns when the driver's control server answers, and
that happens as soon as the FIRST WINDOW exists — but the first window is the
boot screen (`BootProgress.tsx`, `data-testid=boot-gate`), not the app. The
three supervised services behind it are still starting. `sign_in_local` then
went straight to `wait_for("[data-testid=sign-in-button]")`, whose default
budget is 60s.
Measured on a warm M-series laptop against an already-staged runtime: the
window appears at ~9s and the sign-in gate at ~110s. So the wait expired
mid-boot and the journey reported
AssertionError: sign-in gate never appeared
which is a box timeout wearing a product failure's clothes — the same trap
`driver.mjs` already documents for `electron.launch`, one layer up.
`sign_in_local` now calls `wait_for_app_ready()` first, which spends the
existing `BOOT_TIMEOUT_S` budget (default 260s) rather than introducing a
second knob — and which `ci-desktop.yml` already raises to 420 for the hosted
runner, so that setting is now load-bearing for both waits instead of only the
control-server probe. A fatal boot error fails immediately with the app's own
message instead of burning the budget, and a timeout names the stage it was
stuck on. A longer wait can only turn a timeout into a real verdict, so no
currently-passing journey changes behaviour; all 19 `sign_in_local` callers
inherit the fix.
Verified end to end: `JOURNEY_PHASES=FR-0 ... first_run.py` now exits 0
("app ready after 94s of boot", 360 models offered, none configured) where
before it exited 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-clock deadline `recursion_limit` appeared nowhere in the service, so every run inherited LangGraph's own default and failed with an opaque `GraphRecursionError`. There was also no run-level wall clock: `ModelConfig.timeout_seconds` bounds a single invocation and re-arms on every graph entry, so a run that streams, parks on an approval and resumes had no ceiling of its own. Three changes, each wired to a real entry point: - `recursion_limit` becomes a bounded hyperparameter (default 500) that reaches Pregel as a TOP-LEVEL `RunnableConfig` key via `AgentRuntimeContext`. Under `configurable` it would be inert, which is the silent-no-op version of this fix, so a test asserts the key's position and that the step counter honours it. - `GraphRecursionError` is translated into `RuntimeErrorCode.RECURSION_LIMIT_EXCEEDED` and marked non-retryable. It subclasses `RuntimeError`, so the generic handler was classifying it as a retryable external-service error and would have re-run the same loop at the same cost. - `RunDeadline` adds the run-level wall clock, anchored at `started_at` so a re-claim inherits spent time. It terminates through the existing `RunTerminationCoordinator`, preserving the causal-prefix seal, and reports `RUN_DEADLINE_EXCEEDED` — distinct from `RUN_TIMEOUT`, since both surface as the same `TimeoutError` and only the scope's own flag can tell them apart. The default is derived from a measurement against the real Deep Agents graph: a tool round costs 6 + 4*rounds super-steps in the production middleware shape (langgraph 1.2.9 / deepagents 0.7.1), so 500 is ~125 rounds — an order of magnitude clear of the per-tool-name budget of 10, and a bounded worst case against a BYOK key. Two tests pin that arithmetic against the real graph. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-clock deadline `recursion_limit` appeared nowhere in the service, so every run inherited the library's own default and failed with an opaque `GraphRecursionError`. There was also no run-level wall clock: `ModelConfig.timeout_seconds` bounds a single invocation and re-arms on every graph entry, so a run that streams, parks on an approval and resumes had no ceiling of its own. The preceding snapshot commit left this half-landed in the repo's signature shape: `execution/run_deadline.py` and its 297-line test were on disk with ZERO callers in `src/`, while the seven files that wire them were reverted. This re-lands the wiring and closes the seam that made it invisible. Three changes, each wired to a real entry point: - `recursion_limit` becomes a bounded hyperparameter (default 500) that reaches Pregel as a TOP-LEVEL `RunnableConfig` key via `AgentRuntimeContext`. Under `configurable` it would be inert, which is the silent-no-op version of this fix, so a test asserts the key's position and that the step counter honours it. - `GraphRecursionError` is translated into `RuntimeErrorCode.RECURSION_LIMIT_EXCEEDED` and marked non-retryable. Verified against the installed library: it subclasses `RecursionError` -> `RuntimeError`, so the generic handler was classifying it as a retryable external-service error and would have re-run the same loop at the same cost. - `RunDeadline` adds the run-level wall clock, anchored at `started_at` so a re-claim inherits spent time. It terminates through the existing `RunTerminationCoordinator`, preserving the causal-prefix seal, and reports `RUN_DEADLINE_EXCEEDED` — distinct from `RUN_TIMEOUT`, since both surface as the same `TimeoutError` and only the scope's own flag can tell them apart. Two additions beyond restoring the wiring, both aimed at the same failure mode: - A test for the settings -> persisted-context hop, which nothing covered. Every other test here builds a context by hand, and the field's default is the SAME number as the settings default, so deleting the one line in `RunCoordinator` that joins configuration to the run left the whole suite green. The new test configures a non-default 137; removing that line fails it with `assert 500 == 137`. - An SSOT row + ceiling gate for `recursion_limit`. `AgentRuntimeContext` must hold the default and the bound as literals — importing the document's values cycles through `delegation.subagents` and breaks every import of `agent_runtime` (confirmed empirically; it is the same cycle `deep_agent_builder._DEFAULT_TOOL_CALL_BUDGET` documents) — so the copies are pinned by the gate this repo already uses for exactly that situation. Note on the default: two library constants are in play and neither is ours. `langchain_core` defines `DEFAULT_RECURSION_LIMIT = 25`, but that is not what this graph gets — `langgraph._internal._config` supplies 10007, i.e. ~2500 tool rounds and no meaningful bound at all, which on a BYOK key is 2500 completions of a loop the user is paying for. 500 is ~125 rounds at the measured cost, an order of magnitude clear of the per-tool-name budget of 10 and a bounded worst case. Two tests pin that arithmetic against the real Deep Agents graph. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The orphan ratchet catches a module nothing imports and
check_dark_capabilities catches an env flag no test turns on. Both are
blind below the module line, and a nine-dimension audit found that is
where the population actually lives: six of the nine dimensions carried a
correct mechanism whose last seam was missing, every instance inside a
module that is imported, exercised and green.
check_dark_wiring adds three detectors for the shapes those gates miss:
* test-only - a callable no src module calls, a None-default contract
field threaded through constructors only as a copy of
itself, or a payload key the client mentions solely as a
bare string.
* backend-only - a store capability on one runtime_adapters backend, on
no sibling and no Protocol port, named by nothing
outside its own package.
* prompt-only - a validated field whose only cross-package read is an
f-string in a prompt: advice to the model, enforced by
nothing.
Wired unconditionally into ci-gates.yml, whose repo-gates context is a
required check on dev, with its own tests running before it - a gate that
runs nowhere is a comment. 306 already-dark symbols are recorded with a
reason each in tools/dark_wiring_baseline.txt; the 307th fails, and so
does any baseline line that has gone stale, so the file only shrinks.
There is deliberately no --update-baseline writer: a gate must not be
able to bless its own findings.
The detectors are biased toward silence throughout - ambiguous evidence
makes a symbol look live, not dark - because a gate that reddens a build
over a false positive does not survive its first month. Tuning that bias
took the population from 1,607 findings to 306; each rule that did the
cutting is pinned by a regression test named for the false positive it
removed, and every one was confirmed to fail when the rule is reverted.
Verified against the five defects the audit found by hand: FTS5
conversation search, the export and import archives, grant_options,
approvals.expires_at, and the skills manifest's allowed_tools.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine dimensions plus two adversarial critics. 6 of 18 claimed wins refuted; the dominant pattern is reachability, not correctness.
…-hermes-compare-19f93a
…mes-compare-19f93a
…rness-opencode-hermes-compare-19f93a
…ode-hermes-compare-19f93a
…e-hermes-compare-19f93a
Recovered from snapshot 582a5223 (parent e3faebc, same base as this branch). Unreviewed and untested at this commit -- committed verbatim so the audit diff that follows is readable and the prior agent's work cannot be lost again. Nothing here has a caller yet.
The stalled agent's route read `app.state.host_write_undo_service` and nothing ever assigned it, so every call answered 503 and the capture side filled a ledger no one could read back. Compose it where every other satellite store is composed: one `host_write_journal_store` port on `RuntimePorts`, populated on the file backend from the SAME layout and object store the worker's capture side opens, and `None` elsewhere so the routes 503 honestly.
…l when no scope was sent The WIP added `decision_scope` unconditionally, which rewrote the persisted LangGraph resume value for every plain ask-a-question into one carrying a key the caller never sent. `_interpret_resume` reads it with `.get`, so absent and None are the same answer to the only reader — emit the key only when a scope was actually named. Restores the two `test_approval_ask_a_question` payload assertions without touching them.
…s tools 20 tests over the PRODUCTION composition test_host_floor.py uses -- HostFilesystemRules + FilesystemBackend(virtual_mode=False) + the floor -- driving the real write_file/edit_file tools against real disk. Covers the six properties the design has to hold: byte-identical restore, one tool call reverted while a later write stands, a revertible delete, a binary round-trip, retention pruning records AND their orphaned blobs, and four ways a revert must refuse to write outside the granted floor (outside root, sibling prefix, planted symlink, traversal).
An aggregating connector (mcp-proxy, metamcp) re-advertises its upstreams' tools already carrying an mcp__<upstream>__ prefix. The recovered draft absorbed ANY prefix, so mcp__github__search and mcp__gitlab__search both composed to mcp__proxy__search — one registered tool and one dropped with the exact DUPLICATE_DESCRIPTOR_NAME failure the namespace exists to remove. Absorb the prefix only when it names this same server. Adds the module's own property tests: strip is the exact inverse of compose, compose is idempotent (including on the truncated/digested path) and injective over the sanitized pair, and every composed name fits OpenAI's tool-name charset and 64-character limit.
…ists
The recovered WIP was written against a summarization middleware this
runtime does not have. Verified absent: no SummarizationMiddleware is
installed anywhere in src/ (capabilities/middleware/ holds only
display_metadata + runtime_tool_control), and both 'compact_conversation'
and '_get_effective_messages' occur ONLY inside the WIP's own files.
Deleted, because each was justified by that fiction:
* ContextCompactionRecorder -- a ContextVar-bound, idempotent recorder
guarding against a folded request replayed on every later model call.
Nothing replays; compaction is decided once per tool result.
* messages_before/messages_after/messages_folded on the producer --
nothing in this runtime can honestly supply conversation message
counts, so the fields would be dark code carrying invented numbers.
* CompactionTrigger.TOOL_REQUEST -- named a tool that does not exist.
Kept and reshaped: CompactionNotice as the typed carrier, now adapted
from the ToolResultAdmission the runtime really produces, plus the
producer's 'derive the difference here so it cannot disagree' rule,
applied to tokens_saved.
No caller yet -- wiring lands in the next commits.
The stalled agent swapped Messages.Validation.schema_type_required's docstring for a 'TEMPORARY probe shim' note while running an unpatched baseline and never put it back.
…wn fold, and drop dead surface Three things the stalled WIP left inconsistent: - `RunDecisionLedger._is_allowed` required EVERY subject of a pending ask to match an ALLOW rule, while `PermissionRuleset.verdict` — the fold the PDP actually applies — takes the most-restrictive MATCHED rule and lets unmatched subjects contribute nothing. A sibling call to the same tool with different arguments therefore dispatched on replay while the ledger still reported it as pending. Retroactive resolution is a report about what the PDP will do, so it now asks the PDP's question. - `reply` had two guards where the second subsumed the first. - `PermissionRuleset.is_empty` and `RunDecisionLedgers.discard` had no caller.
…cted ToolResultAdmissionAdapter has always built a ContextCompressionEvent for every decision it makes; ToolResultOffloader.apply then returned only the rewritten payload and dropped it. The user saw a bounded preview with no account of where the rest of the content went. apply_with_notice returns both. stream_tools awaits the compression_note immediately BEFORE the TOOL_RESULT event it explains -- same pre-terminal async pass, so the note is provably inside the run's causal prefix and survives replay. A fire-and-forget task would have risked landing after the terminal event, where append_api_event fails closed with LedgerSealViolation. apply() is retained as a thin wrapper so no existing caller changes. runtime_worker suite: 905 passed.
`run_steered` has been a durable event inside the run's causal prefix since the steer endpoint landed, and nothing drew it: the user's own words, sent into their own run, absent from their own transcript. - `TcChat.steerNotes` joins the seq interleave AFTER the compaction pass, so a steer sharing a seq with a card reads as a reaction to it. - Steers count toward `nothingToShow`, unlike compaction notices: a run steered before it produced anything would otherwise draw "No messages yet." over a sentence the reader had just typed. - `absorbThoughtActivity` will not fold one into a thought — the user's words are not the agent's working, and an interjection ENDS the absorbed run. - `.tc-steer*` in the package's own `review-surfaces.css`: a LEADING-edge rule (the row is attributable) with wrapping, never-ellipsised text. - `TcChat.steering` swaps the placeholder to "Steer this run — ⏎ to send". Wiring of the projection and the POST lands in the next commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s blind spot tool_rounds could not see a call in flight when the run died, and that nearly lost the ceiling finding. Adding metrics without saying what each one cannot see would repeat the mistake, so every column now carries its blind spot in the module header. New per-run columns, all offline from the run store: tool_invocations (rows are TRANSITIONS, not calls, so they are grouped by invocation_id first), orphaned_rounds, reconciled_rounds, peak_parallel, delegated_rounds, namespaced_tools, tool_name_collisions, model_calls, peak_result_tokens, budget_notes, super_steps_estimate. Re-scoring the existing arms re-derives FINDINGS.md section 2 independently: reconciled_rounds names limit=25's write_todos as closed-by-reconciler rather than thrown, and super_steps_estimate puts that run at 22 of its 25-step ceiling. tool_rounds keeps its old 'succeeded' meaning so the published numbers do not move.
…un-blocked The design test caught this in heavy_tasks_ab's own Arm.prerequisite: it returned None for a CONNECTED_MCP task whenever setup had recorded no blocked reason, so a setup that returned early — an exception, a reordering — silently promoted H7 to runnable and it would have measured MCP namespacing on a profile with no connectors, and reported it. Availability is now a positive set that setup adds to only after checking. Adds tools/test_harness_bench_heavy_tasks.py: 17 offline checks over the task set (no host-absolute path in a grant-free prompt, no grep/glob on the memory route where they answer EMPTY rather than failing, no plan at the per-tool-name budget, every task carries a checkable finish state, derived answers are unique) and over the scorer (rows are transitions, orphans counted, reconciler-closed calls distinguished, overlap vs adjacency, delegation, namespacing, never-ran tasks excluded).
Two facade routes have been reachable by nothing:
GET /v1/agent/runs/{run_id}/host-writes
POST /v1/agent/runs/{run_id}/host-writes/revert
Three pieces, none wired yet:
- hostWrites.ts groups the flat listing by tool_call_id, which is the ONLY key
the revert route accepts. A group counts DISTINCT PATHS because the server
collapses per path (HostWriteReverter.select keeps the oldest record per
path), so counting records would print a number the undo cannot deliver.
- useHostWrites binds both routes through the Transport port. 503 is surfaced
as `unavailable`, not `error`: every non-desktop image composes no object
store, so that answer is the capability speaking, not a fault.
- HostWritesTab renders the groups. An undo ARMS a confirmation that prints the
paths in full, because the journal captures the pre-image and nothing
captures what is there now — the undo of an undo does not exist.
check_dark_wiring tokenises file TEXT, so naming WorkerAuditEmitter's tool-call-outcome method in a docstring made the gate report the baseline line as stale - i.e. a comment would have silenced a real 'no production caller' finding. Reference it by file:line and baseline line number instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e the gate Rescoring both existing arms with the extended scorer puts numbers on the gap: peak 3 completed tool rounds, 22 estimated super-steps of a 25 ceiling, 122-token peak tool result against an 8,192 cap, zero delegated rounds, zero namespaced MCP names. Those four claims are unmeasured because nothing in the set touches them, not because they held. Also records that reconciled_rounds re-derives finding 2 unprompted, and that orphaned_rounds cannot — two metrics, two blind spots, neither a substitute for the other. Three papercuts fixed while here: both report writers emit a trailing newline; the heavy arm writes its report after EVERY task so a crash in task six does not discard five paid runs; and runs/*.json is prettier-ignored, because json.dumps always expands arrays and prettier collapses short ones, so every rescore was landing a whitespace-only diff over real measurements. tools/test_harness_bench_heavy_tasks.py joins the repo-gates test list. repo-gates is unconditional, so no paths filter is added.
…g a list Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…costs one run
Validating the plumbing previously meant paying for a whole arm. HEAVY_TASKS
pins tasks by id with the same fail-loud contract as JOURNEY_PHASES: an id
matching no declared task raises rather than running nothing, because a
caller that pins a renamed task and gets a clean empty run has proven
nothing while reporting success.
Also covers the fixture-substitution path offline. h6-bigread's {ledger} /
{emea} placeholders are substituted mid-arm — after the boot, the key and
the grant — so a KeyError there fires at the most expensive possible
moment. It is now exercised for free.
…e that would The only staged runtime on this machine is four days behind the tree, so a live arm would run old backend code — and specifically the error taxonomy this file's finding 2 already corrected. Recording that rather than running it, since a stale-stage verdict looks impeccable and is wrong. Adds the re-stage-then-one-task-then-arms recipe and states the expected cost, so a number far from ~45-60 model calls per arm is itself readable as a finding rather than as noise.
…er-call undo RunDestination binds useHostWrites off the bound run and hands the result to RunWorkspaceRail, which grows a conditional fifth tab: Changes — what the run WROTE, as the counterpart to Sources, which is what it read. Conditional because the tab has two states with nothing to say, and a rail tab that is permanently empty is this package's own cautionary tale (the Sources tab): a deployment that captures no writes (every non-desktop image answers 503 by design) and a run that touched no files both produce `undefined`, and the rail is byte-identical to the four-tab v3 rail in each. An error DOES draw it — silence there is indistinguishable from "this run changed nothing". The tab's badge counts distinct FILES, not journal records, because the server collapses per path when it reverts.
wip checkpoint - composer submit-while-running seam + cockpit dispatch.
A mistyped HEAVY_TASKS id was raising inside the phase — after initdb, migrations and three uvicorns, about 110s of boot, to then report a failure a string comparison could have caught. JourneyPlan pins its phases before its factory for exactly this reason; this now matches. Verified against the real staged runtime: preflight passes, a bad pin exits before any boot, and a good pin resolves to the single task.
The failure this program keeps hitting is a feature that is landed and unreachable, which no unit test of a projection can detect. So the load-bearing test mounts the real RunDestination against a real Transport, finds the affordance the way a person would — rail tab, row, control — and asserts the POST that leaves the client. Without the wiring it fails at the first step: there is no Changes tab to find. Three safety properties get their own coverage: - no posting control is one click from rest, and the confirm testid is deliberately outside the arm's prefix, so a journey pressing every `host-writes-undo-*` cannot reach the control the arming step withholds; - every deciding name is scoped by its group, because two tool calls on screen make an unscoped name ambiguous and an ambiguous selector is a decision that never happened; - the revert POST always carries a tool_call_id. Omitting it is the whole-run revert, which no control here performs. Layout is read through getComputedStyle: jsdom runs no layout, so the row's give-way order is the only thing that decides whether a narrow rail clips the Undo button off the end of its own row.
…d run
The three existing steer suites are all handed the projection directly, so
none of them can see whether anything ever produces one. This starts a layer
out, at the two facts that make the feature real: an in-chat ⏎ mid-run reaches
POST /v1/agent/runs/{id}/steer and starts no second run, and the run_steered
frame off the OPEN session stream draws with no local echo.
Mounts the real AssistantComposer through renderComposer the way
destinationBinders.tsx:1546 does — the base <Composer> fallback inside TcChat
is never handed `running`, so asserting there would assert a branch no host
takes.
Removing either half of the wiring reds 2 of the 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eaches nothing Running the test found the assertion was wrong about its own subject: arming REMOVES the arm, so `[data-testid^=host-writes-undo-]` matches zero afterwards, not one. The corrected form is the stronger property — at rest the prefix matches exactly the arm; armed, it matches nothing at all — so a journey pressing every `host-writes-undo-*` cannot post under either state.
…e-hermes-compare-19f93a
…e-hermes-compare-19f93a
…e-hermes-compare-19f93a
…e-hermes-compare-19f93a
…plementation Both UI agents independently built the once/always grant control — HEAD's scopeApprove/allowsRunScopedGrant and the branch's grantAlways/renderGrantAlways. Keeping both would put two competing scope controls on one approval card, so every conflicted hunk resolves to HEAD, whose version is already consistent across TcChat, approvalProjection, the barrels and the stylesheet, and which has its own coverage in approvalProjection.grantScope + TcChat.grantScope. The branch's unique contribution — the host-write revert lane — is in separate files (hostWrites.ts, useHostWrites.ts and their tests) and merges untouched.
…implementation Also records a worktree toolchain trap: node_modules/@0x-copilot/* symlinks resolve to the MAIN checkout's packages, so a worktree typecheck silently checks cross-package types against dev. Ten of eleven errors here were that artifact; only this one was real.
Beats its own -1,326 prediction at -1,757. Cumulative across the wave: -2,634 cold tokens (-11.4%), -12.5% total, completion 3/4 -> 4/4.
Overlapping work reconciled rather than picked, because dev and this branch independently solved two of the same problems: - context origin/attribution: dev's ToolOriginResolver wins over our ToolSchemaOriginFallback — dev is the trunk — and our now-dead context_installed_tools.py is DELETED rather than baselined. - atlas_task_tool: taking dev's wholesale would have silently dropped our delegation-depth cap, so dev's declare_context_origin wrapper is spliced onto our version and both survive. - factory: dev deleted ToolBudgetGuardedTool, so our import and the _canonical_graph_tool unwrap loop (no callers either side) go with it; our run_tool_program builder and skills-index imports stay. - one dev test asserted whole-request equality on the occupancy-off path, which our always-on retry callback now breaks; narrowed to the fields that test is actually about.
- lint-and-secrets: ruff-format on the bench files, which never ran because the merge commits used --no-verify. - repo-gates: the dark-wiring gate's own test asserted SkillManifest.allowed_tools is STILL dark. dev enforced it at the tool surface, so the gate correctly stops reporting it and the case is retired — a case asserting a symbol stays dark has to go the moment the symbol is wired, or it fails the build for the fix. - test-and-audit: the model-delete test pinned a MECHANISM (deepagents' catch-all DENY) that holds on 0.7.1 and not on the 0.7.4 the service pins, so it passed on a dev venv and failed in CI. It now pins the property the feature exists for and which holds on both: a model-driven delete is either refused, or captured so it can be reverted — never silently unrecoverable.
Pinned ruff 0.15.12, matching .pre-commit-config.yaml, so local and CI agree.
0x-copilot-dev
added a commit
that referenced
this pull request
Aug 26, 2026
…s-comparison-a91a25 Merged on local verification, not on CI. GitHub Actions never started a job for e62dbf6 — 5 runs sat queued for ~50 minutes with zero check-runs registered, and cancel had no effect. Not red; never run. Local coverage substituted for each CI job: - ai-backend suite: 10,103 passed. The 2 failures are the borrowed venv's version pins (deepagents 0.7.1 vs 0.7.4, langgraph 1.2.9 vs 1.2.10), which CI pins correctly. - repo-gates: dark-wiring 306 baselined / 0 new, its own 32 tests pass; harness gates 90 passed. - ci-host-typecheck: desktop + frontend tsc clean. - chat-surface: 4,093 passed, 2 failed — BOTH proven pre-existing on dev, not introduced here. canvasLifecycle's Python differential imports agent_runtime.presentation.lifecycle, a module that exists nowhere (documented as pre-existing in #632). ChatsArchive FR-G.3 was demonstrated by reverting this branch's only TypeScript change (one union member added to ModelFailureClass) to dev's version and re-running: it fails identically. - An earlier SHA of this PR did run CI and passed 10 of 11 including test-and-audit, typecheck-and-test, build-and-audit and the packaged-app journey; the one failure was repo-gates, fixed in 8cac45c and re-verified locally above.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the ten harness gaps found by a three-way comparison against
sst/opencode and NousResearch/hermes-agent, then measures
whether they were worth building. Comparison:
docs/audit/harness/OPENCODE-HERMES-COMPARISON.md.Measurements:
tools/harness-bench/FINDINGS.md.Measured outcomes, from live runs of the packaged app against a real model
run_tool_programgated, attributionCumulative: −2,634 cold tokens (−11.4%), −12.5% total, completion 3/4 → 4/4.
The completion win is the step ceiling:
recursion_limitwas inheriting LangGraph's default of25 super-steps. A/B with only that knob moved, same stage and tasks: 3/4 → 4/4 for +0.1% tokens.
What landed
Loop control (
recursion_limit,RunDeadline, a real model-call retry policy) · MCP tool-namenamespacing + vendor schema repair · tool-result cap on every store backend · policy rules with
DESTRUCTIVEaboveBYPASS· delegation depth caps · a typed hook seam · agent-write snapshotswith per-tool-call revert · mid-run steering · batched tool execution · skills progressive
disclosure · first-party tool-schema disclosure · the four UI halves (steer composer, compaction
divider, revert affordance, once/always grants) · a dark-wiring ratchet + a per-PR journey job ·
and
tools/harness-bench/, the instrument all the numbers above come from.Verification
Python 9,984 passed · chat-surface 4,011 passed · typecheck clean · all six repo gates green.
Three known failures, all pre-existing on
devand unrelated: two version-pin tests that compareinstalled dist versions against
requirements.txt(the borrowed venv has deepagents 0.7.1 /langgraph 1.2.9 vs pins 0.7.4 / 1.2.10), and
canvasLifecycle's Python differential, which importsagent_runtime.presentation.lifecycle— a module that exists nowhere, including at the branch point.Reviewer notes — things I'd want a second opinion on
origin/devconflicts were resolved by splicing, not picking.atlas_task_tool.pywouldhave silently lost our delegation-depth cap had dev's version been taken wholesale; dev's
declare_context_originwrapper is spliced onto ours instead. On context attribution the reverse:dev is the trunk, so its
ToolOriginResolverwins and our now-deadcontext_installed_tools.pyis deleted, not baselined.
always-on retry callback now binds to the model, so the assertion is scoped to the fields the test
is actually about.
ArtifactToolFamilyExposureshipsALWAYSon purpose. Its 805 + 900 tokens stay residentbecause the only other state withholds
publish_artifactfrom the model entirely — a livecapability change that wants an owner's decision, not a default flip.
prompt-trimming change could in principle degrade answers and still show 4/4.
Honest record
FINDINGS.mddocuments two instrument failures rather than editing them away. The first scorerreported 0 tokens for every task (wrong event matcher — indistinguishable from a cheap run). The
second inferred "did the ceiling bind?" from completed tool rounds, which cannot observe a run that
died with a call in flight; that reading nearly buried the only real win in this branch.
🤖 Generated with Claude Code