diff --git a/.github/workflows/fork-ci.yml b/.github/workflows/fork-ci.yml index 70aeeba157c..bde59e76cff 100644 --- a/.github/workflows/fork-ci.yml +++ b/.github/workflows/fork-ci.yml @@ -246,6 +246,28 @@ jobs: "$(rustup which cargo)" test --locked --target x86_64-unknown-linux-gnu \ -p jcode-app-core --lib retention_readiness -- --nocapture + - name: Run shared-control cancellation and publication regressions + shell: bash + run: | + # ACP wire scenarios do not execute app-core's deterministic dispatch + # interleavings. Keep the original Phase 1 race evidence on this head. + cargo_bin="$(rustup which cargo)" + for filter in \ + server::client_lifecycle::turn_event_channel_tests \ + cancel_after_turn_claim_is_preserved \ + user_response_socket_registration_follows_explicit_capability \ + agent::turn_ownership::tests; do + python3 .github/scripts/run_with_timeout.py 600 \ + "$cargo_bin" test --locked --target x86_64-unknown-linux-gnu \ + -p jcode-app-core --lib "$filter" -- --nocapture + done + python3 .github/scripts/run_with_timeout.py 600 \ + "$cargo_bin" test --locked --target x86_64-unknown-linux-gnu \ + -p jcode-protocol --lib turn_identity -- --nocapture + python3 .github/scripts/run_with_timeout.py 600 \ + "$cargo_bin" test --locked --target x86_64-unknown-linux-gnu \ + -p jcode-harness-api-server --lib session_catalog -- --nocapture + - name: Run disconnect telemetry responsiveness regressions shell: bash run: | diff --git a/crates/jcode-app-core/src/agent/elicitation.rs b/crates/jcode-app-core/src/agent/elicitation.rs index 150c21c56dd..8c82c11ea73 100644 --- a/crates/jcode-app-core/src/agent/elicitation.rs +++ b/crates/jcode-app-core/src/agent/elicitation.rs @@ -16,8 +16,13 @@ impl Agent { .await; } let mut allowed = self.allowed_tools.clone(); - if applet && let Some(allowed) = allowed.as_mut() { - allowed.insert("applet".into()); + if let Some(allowed) = allowed.as_mut() { + if applet { + allowed.insert("applet".into()); + } + if desktop { + allowed.insert("user_response".into()); + } } self._tool_policy_registration = crate::tool::register_session_tool_policy( &self.session.id, diff --git a/crates/jcode-app-core/src/agent/turn_execution.rs b/crates/jcode-app-core/src/agent/turn_execution.rs index 448554e8975..a20a7e5e148 100644 --- a/crates/jcode-app-core/src/agent/turn_execution.rs +++ b/crates/jcode-app-core/src/agent/turn_execution.rs @@ -677,6 +677,12 @@ impl Agent { { allowed.insert("applet".into()); } + if self.ui_capabilities.0 + && enabled.is_none() + && let Some(allowed) = allowed.as_mut() + { + allowed.insert("user_response".into()); + } let mut tools = self.registry.definitions(allowed.as_ref()).await; if enabled.is_none() && !self.disabled_tools.is_empty() { tools.retain(|tool| { @@ -1156,6 +1162,7 @@ impl Agent { if let Some(allowed) = self.allowed_tools.as_ref() && !self.registry.tool_is_allowed(allowed, name) && !(name == "applet" && self.ui_capabilities.1) + && !(name == "user_response" && self.ui_capabilities.0) { return Err(anyhow::anyhow!("Tool '{}' is not allowed", name)); } diff --git a/docs/verification/acp-v1-acceptance.md b/docs/verification/acp-v1-acceptance.md new file mode 100644 index 00000000000..bbbf131c2f6 --- /dev/null +++ b/docs/verification/acp-v1-acceptance.md @@ -0,0 +1,134 @@ +# ACP v1 acceptance reconciliation + +## Scope and provenance + +This record reconciles the original Phase 0 through Phase 3 target against +integrated code and executable evidence. Historical planning prose is not a +list of missing implementation. The v2 track and optional features are not +part of v1 acceptance. + +Starting source: `1bdb614590763855b85919fda839903457149e51`. +The `custom` merge, `6766a2ed419721594b62c12bb92dbc2c09b3df6a`, has an +identical tree (`git diff 1bdb61459 6766a2ed4` is empty). + +Verified starting-head Fork CI: + +- `42738049168`, pull request event, source `1bdb614590763855b85919fda839903457149e51`. +- `42740276146`, push event, source `6766a2ed419721594b62c12bb92dbc2c09b3df6a`. +- Both Quality Guardrails and Build & Test passed on each run. +- Run `42738049168` executed 143 ACP adapter tests, one separately selected + private-daemon reexec test, and 100 e2e tests. One adapter test and seven + e2e tests were ignored. These counts are not a claim that the ignored live + provider and scale tests passed. + +The acceptance repair/fixture head +`5fa1db619a52084128be56a15a024b3e831325a1` also passed both jobs in +Fork CI `42907124856` and `42907113965`. These runs include the newly selected +deterministic regressions, the Desktop capability repair and the skill-clear +fixture. Later changes, including the command workflow tests below, require +their own exact-head CI results. None of these agent-side runs certifies an +installed consumer release. + +## Original criteria and existing evidence + +The named tests below are existing evidence anchors. Consumer results are +reported separately, because an agent-side wire test cannot establish the +consumer's resulting state. + +| Original criterion | Code-owned evidence anchor | Reconciliation | +| --- | --- | --- | +| Phase 0: fork quality and e2e workflow | `.github/workflows/fork-ci.yml`, runs above | Existing implementation, green starting-head CI | +| Phase 0: fork-owned budget baseline | `scripts/fork/`, Quality Guardrails | Existing implementation, green starting-head CI | +| Phase 0: contracts and pinned references | `docs/ACP_FORK_INTERFACE.md`, `docs/FORK_BOUNDARY_STRATEGY.md` | Existing documents, planning maintained in its companion | +| Phase 0: record a green `custom` baseline | Run `42740276146` | Verified on the exact starting `custom` head | +| Phase 1: attach, replay and peer observation | `acp_conformance_create_list_load`, `acp_conformance_interleavings`, `acp_client_mirrors_peer_turns_and_keeps_its_own_prompt_separate` | Passed in starting-head e2e CI | +| Phase 1: steer, adopt and recover stranded input | `acp_conformance_shared_observe_steer`, `acp_conformance_seeded_three_client_invariants`, retirement tests | Passed in starting-head e2e CI | +| Phase 1: stop then immediately prompt | `acp_conformance_interleavings`, `acp_conformance_stop_observed_bash_wait` | Passed in starting-head e2e CI | +| Phase 1: close an observing view without stopping work | `acp_conformance_close_owner_aware_v1`, `acp_conformance_observer_close_bracket_contract` | Passed in starting-head e2e CI | +| Phase 1: model, effort, skills and command discovery | `acp_conformance_config_commands_v1`, `acp_conformance_config_peer_update_contract`, `acp_skill_clear_round_trip` | Passed in starting-head e2e CI, consumer controls need separate evidence | +| Phase 1: compact history and launch a live subagent | `acp_conformance_compact_command_compacts_real_history`, `acp_conformance_subagent_command_spawns_live_worker` | Added actual daemon-workflow tests, named local run passed; final-head CI required | +| Phase 1 D1: atomic cancellation relative to replacement | `cancellation_dispatch_cannot_signal_a_successor`, `completed_turn_is_not_reported_interrupted_by_a_late_stop`, `late_stop_preserves_task_for_completion_reconciliation` | Existing deterministic regressions, added to Fork CI in this acceptance slice | +| Phase 1 D2: capture cancellation epoch at claim | `cancel_after_turn_claim_is_preserved` | Existing deterministic regression, added to Fork CI in this acceptance slice | +| Phase 1: completion, failure, panic and producer abort | `turn_event_channel_tests`, `acp_conformance_panicking_producer_is_failed`, `acp_conformance_aborted_producer_after_stop_is_cancelled` | Wire tests passed, deterministic publication regressions added to Fork CI | +| Phase 1: two steering producers and disconnect around insertion | `two_steering_producers_withdraw_on_either_side_of_insertion`, `acp_conformance_transport_loss_withdraws_waiting_steer`, `acp_conformance_transport_loss_preserves_inserted_work` | Socket regressions passed, deterministic core regression added to Fork CI | +| Phase 1: truthful initialize and attachment capability metadata | `initialize_advertises_jcode_capability_meta_in_every_profile`, `acp_capability_metadata_identifies_each_attachment`, `acp_conformance_negotiation_legacy_v1` | Passed in starting-head CI | +| Phase 1: mixed-version lifecycle delivery | `undecodable_lines_are_skipped_and_never_end_the_session`, `obsolete_turn_ended_is_not_a_daemon_wire_event` | R1 removed the competing event, adapter compatibility passed, protocol regressions added to Fork CI | +| Phase 2 R1: one authoritative lifecycle, compatible classifications | `docs/verification/r1-single-lifecycle.md`, classified scoped terminals in `wire/events.rs` and the ACP router | Integrated already, no `TurnEnded` daemon variant or duplicate retirement flag remains | +| Phase 2 R2: stored-session catalog facade | `translate/session_catalog.rs`, `acp_conformance_list_metadata_and_pagination` | Integrated facade, wire behavior passed, catalog regressions added to Fork CI | +| Phase 2 R3 through R6 | Fork boundary strategy | Opportunistic, not mandatory phase deliverables without measured recurring conflict burden | +| Phase 3: structured reasoning, plans and tool details | `acp_conformance_maps_reasoning_tool_calls_and_plan`, `event_mapper_reports_no_diff_for_a_failed_edit` | Passed in starting-head CI | +| Phase 3: prompt turn ID and role correlation | `acp_conformance_shared_observe_steer`, `acp_conformance_interleavings`, config-command non-turn result | Passed in starting-head e2e CI, consumer deduplication requires separate evidence | +| Phase 3: attachment-local skill selection and `skill: null` clear | `acp_skill_clear_round_trip`, ACP skill metadata unit tests | Passed in starting-head CI | +| Phase 3: shared transcript corpus with drift detection | `tests/fixtures/acp/`, `tests/e2e/test_support/acp_fixture.rs` | Agent-side replay passed, consumer-side replay requires separate evidence | +| Phase 3: tolerate session MCP servers, advertise limitation and notice | `acp_ignored_mcp_servers_notice_follows_the_response` | Passed in starting-head e2e CI | +| Phase 3: detach preserves work and withdraws only departing pending input | `acp_conformance_detach_originator`, `acp_conformance_detach_observer`, `acp_conformance_detach_adopted_steer`, `acp_conformance_detach_withdraws_pending_steer` | Passed in starting-head e2e CI | +| Phase 3: legacy setters and legacy clients | `acp_conformance_config_commands_v1`, `acp_conformance_negotiation_legacy_v1` | Passed in starting-head e2e CI | +| Phase 3 completion: Paseo fixture replay and real build on dedicated socket | Consumer acceptance results below | Not established by the starting-head agent CI alone | + +## Gaps closed by this acceptance slice + +The starting workflow executed adapter and wire tests but did not execute the +original deterministic app-core cancellation and publication regressions. +The existing tests are now selected explicitly in Fork CI. This closes a +verification omission, not a missing production implementation. The filters +cover the turn event channel, cancellation at claim, turn ownership, protocol +turn identity and the stored-session catalog facade. + +The acceptance run also exposed a real Desktop response-tool allow-list defect. +The explicit UI capability registered `user_response`, but the turn tool filter +then hid it. Commit `c92bdffad54e2a360db2591b6ce6cc70df97e308` preserves that +capability through both definition filtering and dispatch validation, without +overriding an explicit SDK enabled/disabled tool list. The exact named +`user_response_socket_registration_follows_explicit_capability` regression +passed after repair and is selected in Fork CI. + +Commit `5fa1db619a52084128be56a15a024b3e831325a1` adds the real-wire +`12-skill-selection-clear-v1.jsonl` transcript. The skill round-trip test +passed both recording and normal replay. The named corpus-integrity test +passed across all 12 transcript files, checking direction, contiguous sequence, +JSON-RPC shape, paired requests/responses and an executable replay reference +for every file. This is agent-side evidence, not consumer replay evidence. + +The new `commands_v1.rs` tests close a separate evidence gap: command +advertisement and parsing did not establish actual daemon scheduling. The +compaction test now witnesses a real summarization call and proves that the +next provider request replaces older history with that summary while retaining +recent turns. The subagent test separates the launch tool receipt from the +spawned child's mid-turn attach, gated live output and completed terminal. +The named `commands_v1` filter passed all three selected tests (the two new +workflows plus the existing config-command regression). This is mock-provider +workflow evidence, not a claim about provider-native compaction, visible-window +spawning or a paid external model. + +The historical D3-pending notes in `p1-cancel-race.md` are superseded by the +integrated R1 result in `r1-single-lifecycle.md`. Keeping the shared +`TurnEndStatus` enum as classified scoped-terminal metadata is intentional, +not a second lifecycle or unfinished removal of `TurnEnded`. + +## Consumer and final-head acceptance + +The real Paseo connector completed all 12 live scenarios with zero skips: +import/replay, bidirectional steering, one correlated execution with concurrent +steering and exact nonzero usage, immediate prompt after Stop, Stop of a peer +turn, bidirectional model/effort propagation, textual and explicit skills with +sticky selection and null clear, owner-aware close, and both originator and +observer detach. The five direct ACP companion probes also passed. These are +separate evidence layers: direct probes do not substitute for consumer tests. + +The reproducible runner is `scripts/live-acceptance/run-live-acceptance.sh`. +It uses an isolated real daemon and the consumer's source modules, records +binary and consumer provenance, and uses a deterministic local model backend. +Runtime production source is `c92bdffad54e2a360db2591b6ce6cc70df97e308`; +subsequent changes in this acceptance series are tests, documentation and CI. +The consumer replays all 12 canonical tapes through its actual ACP connector: +15 tests pass with both its vendored snapshot and the canonical source corpus. + +Consumer fixes include nested prompt-correlation parsing, balanced shared-turn +settlement, exact skill/null propagation, and explicit detach serialized with +close/reload in one lifecycle lane. Deferred regressions verify that detach +never closes a successor runtime or closes after an unsupported/failed detach. + +Final integration still requires green CI on the exact submitted and merged +heads. The canonical companion strategy state records those immutable heads +and run IDs once available. This source acceptance is not a consumer release +or shared-daemon deployment, and does not open the v2 gate. diff --git a/scripts/live-acceptance/README.md b/scripts/live-acceptance/README.md new file mode 100644 index 00000000000..9dbf254af5b --- /dev/null +++ b/scripts/live-acceptance/README.md @@ -0,0 +1,86 @@ +# Live v1 acceptance harness + +Durable, reproducible live evidence for the ACP v1 contract +(`docs/ACP_FORK_INTERFACE.md`) and strategy Phase 3. It does **not** replay +scripted ACP fixtures: a real `jcode` binary serves ACP, and the real Paseo ACP +connector drives it. + +## What runs + +| Artifact | Path | Proves | +| --- | --- | --- | +| Connector harness | `jcode-live-acceptance.ts` | Real Paseo daemon + `plugins/jcode-provider` + real `jcode acp` child: import/replay, forward steer, reverse steer, single-settlement turn correlation (concurrent steer into one held execution, one terminal, one usage total), Stop then immediate prompt, Stop of an observed peer turn, model/effort propagation both directions, textual and explicit skill selection with `null` clear, owner-aware close, observer detach (peer work survives, updates stop, session reloads), and explicit detach that preserves inserted work. | +| ACP companion probe | `jcode-acp-probe.ts` | Real `jcode acp` process: `interfaceVersion`/extensions/`sharedControl` (all four booleans), prompt correlation roles (`origin`/`steer`/`none`), explicit `_meta.jcode.skill` select and `null` clear, `_jcode/detach` semantics and `-32602`, config options and legacy aliases. | +| Runner | `run-live-acceptance.sh` | Provenance gate, isolated environment, evidence directory. | + +Both run their own `jcode serve` on a private socket with private +`JCODE_HOME`, `JCODE_RUNTIME_DIR` and (for the connector) `PASEO_HOME`. They +never talk to the shared daemon and never change user config. The model backend +is a local OpenAI-compatible HTTP server with deliberately released responses, +so turns are deterministic and no provider tokens are spent. + +The connector harness imports the consumer modules from +`PASEO_CONSUMER_ROOT` by absolute path. The runner defaults it to the sibling +checkout `../paseo-v1-acceptance` next to this repository; set it explicitly +when the consumer lives elsewhere. It never copies, stubs, or edits consumer +source. + +The runner starts Node with `--conditions=source`, so `@getpaseo/plugin` +resolves to the checkout's TypeScript source rather than a possibly stale +`dist`. The evidence records the resolved `pluginEntry` and the consumer +`head`, `tree` and worktree digest, so the exact code under test is auditable. +When the consumer has not built `dist`, this is also the only way to exercise +the edits under review. + +## Run + +Run from a checkout with a current binary and a sibling consumer checkout: + +```bash +JCODE_E2E_BINARY="$PWD/target/debug/jcode" \ +JCODE_EXPECTED_PROVENANCE= \ + scripts/live-acceptance/run-live-acceptance.sh +``` + +Or point at the consumer explicitly: + +```bash +PASEO_CONSUMER_ROOT=/path/to/paseo \ +JCODE_E2E_BINARY=/abs/path/to/jcode \ + scripts/live-acceptance/run-live-acceptance.sh +``` + +Outputs go to `JCODE_ACCEPTANCE_OUT_DIR` when set, otherwise to +`JCODE_SCRATCH_DIR/live-acceptance/live-run-` when `JCODE_SCRATCH_DIR` is +set, otherwise to `/target/live-acceptance/live-run-`: + +- `provenance.txt` - binary version, sha256, consumer head/branch/dirt, node +- `connector.evidence.json`, `connector.log` +- `probe.evidence.json`, `probe.log` +- `summary.txt` + +Each evidence file has `provenance`, `environment.isolated`, and a `scenarios` +object with `pass`/`fail` and the observed wire/state detail. The scripted +model can attach an OpenAI streaming `usage` chunk to a reply; `s04` uses +distinct per-call counters to prove the connector reports one execution total +instead of dropping or double-counting a call. + +Useful debug knobs for the connector harness: + +- `JCODE_ACCEPTANCE_SCENARIOS=s04,s09` runs only matching scenario names or + prefixes; the rest are recorded as `skip`. +- `JCODE_ACCEPTANCE_KEEP=1` keeps the isolated workspace and prints its path + instead of deleting it. + +## Honest scope limits + +- The probe is live binary evidence but **not** Paseo-connector proof. The + report labels it as a companion. +- `sharedControl.observe` has no separate consumer switch; it is exercised by a + Paseo attachment observing a peer-origin turn, which is what the boolean + gates. +- A fresh `fetchAgentTimeline` for a closed record can load provider history + again. `s12` therefore proves "updates stop" on the push channel + (`agent_stream` messages after detach), not by re-reading the timeline. +- A stale `JCODE_E2E_BINARY` (for example the 2026-11-09 `900265ac7` build) + produces smoke evidence only. The runner warns when it sees that version. diff --git a/scripts/live-acceptance/jcode-acp-probe.ts b/scripts/live-acceptance/jcode-acp-probe.ts new file mode 100644 index 00000000000..a91abf8afcb --- /dev/null +++ b/scripts/live-acceptance/jcode-acp-probe.ts @@ -0,0 +1,594 @@ +// Live ACP companion probe, independently checking capability advertisement, +// prompt-response correlation, +// explicit skill selection and `null` clear, `_jcode/detach`, and config +// aliases. +// +// This talks to the real `jcode acp` process built from the same binary the +// Paseo connector run uses, on its own isolated daemon socket/home. It is live +// binary evidence, not a scripted ACP fixture, but it is NOT Paseo-connector +// proof; the final report labels it that way. +import path from "node:path"; +import { spawn, type ChildProcess } from "node:child_process"; +import { createRequire } from "node:module"; +import { Readable, Writable } from "node:stream"; +import { pathToFileURL } from "node:url"; +import { + ALT_MODEL, + DEFAULT_MODEL, + SKILL_MARKER, + binaryProvenance, + bodyText, + connectPeer, + createWorkspace, + daemonEnv, + findEffort, + lastUserText, + newEvidence, + requireBinary, + runScenario, + safeJson, + sleep, + startJcodeDaemon, + startMockModel, + summaryLine, + turnIdentityKey, + waitFor, + writeEvidence, + type JcodeDaemon, + type LivePeer, + type MockModel, +} from "./jcode-live-common.ts"; + +const CONSUMER_ROOT = + process.env.PASEO_CONSUMER_ROOT ?? process.cwd(); + +async function loadSdk(): Promise { + // Resolve the ACP SDK from the consumer checkout so the probe and the + // connector use the same protocol implementation. + const requireFromConsumer = createRequire(path.join(CONSUMER_ROOT, "package.json")); + return import(pathToFileURL(requireFromConsumer.resolve("@agentclientprotocol/sdk")).href); +} + +function withTimeout(promise: Promise, ms: number, label: string): Promise { + return Promise.race([ + promise, + sleep(ms).then(() => { + throw new Error(`timed out after ${ms}ms waiting for ${label}`); + }), + ]); +} + +async function main() { + const evidence = newEvidence(); + const binary = requireBinary(); + evidence.provenance = { jcode: binaryProvenance(binary) }; + evidence.environment = { + node: process.version, + platform: `${process.platform}/${process.arch}`, + consumerRoot: CONSUMER_ROOT, + isolation: "dedicated daemon socket, JCODE_HOME, JCODE_RUNTIME_DIR; no shared daemon", + }; + + const workspace = await createWorkspace("jcode-acp-probe-"); + const model = await startMockModel(); + let daemon: JcodeDaemon | undefined; + let peer: LivePeer | undefined; + let acp: ChildProcess | undefined; + let connection: any; + const stderr: string[] = []; + + try { + daemon = await startJcodeDaemon({ binary, workspace, modelBaseUrl: model.baseUrl }); + peer = await connectPeer(daemon.socketPath); + evidence.environment.isolated = { + root: workspace.root, + socket: workspace.socketPath, + home: workspace.home, + runtime: workspace.runtime, + }; + + const sdk = await loadSdk(); + acp = spawn(binary, ["acp"], { + cwd: workspace.cwd, + env: daemonEnv(workspace, model.baseUrl), + stdio: ["pipe", "pipe", "pipe"], + }); + acp.stderr?.on("data", (data) => stderr.push(String(data))); + const stream = sdk.ndJsonStream( + Writable.toWeb(acp.stdin) as any, + Readable.toWeb(acp.stdout) as any, + ); + + const updates: any[] = []; + const notifications: Array<{ method: string; params: any }> = []; + connection = new sdk.ClientSideConnection( + () => ({ + async sessionUpdate(params: any) { + updates.push(params); + }, + async requestPermission(params: any) { + const allow = (params.options ?? []).find((option: any) => + String(option.kind ?? "").startsWith("allow"), + ); + return { outcome: { outcome: "selected", optionId: allow?.optionId } }; + }, + async extNotification(method: string, params: any) { + notifications.push({ method, params }); + }, + }), + stream, + ); + + const initialize = await withTimeout( + connection.initialize({ protocolVersion: 1 }), + 20_000, + "ACP initialize", + ); + const connectionMeta = initialize.agentCapabilities?._meta?.jcode; + const created = await withTimeout( + connection.newSession({ cwd: workspace.cwd, mcpServers: [] }), + 20_000, + "ACP session/new", + ); + const attachmentMeta = created._meta?.jcode; + const protocolSessionId: string = created.sessionId; + evidence.environment.protocolSessionId = protocolSessionId; + // The peer is the TUI stand-in and must share the same daemon session, so + // attach it only after the ACP attachment has created that session. + const session = await peer.subscribe(workspace.cwd, "probe-peer", protocolSessionId); + evidence.environment.attachedSessionId = session.session_id; + if (session.session_id !== protocolSessionId) { + throw new Error( + `peer attached to ${safeJson(session.session_id)} instead of ${protocolSessionId}`, + ); + } + + const prompt = async (text: string, meta?: Record) => { + const params: any = { + sessionId: protocolSessionId, + prompt: [{ type: "text", text }], + }; + if (meta) params._meta = meta; + return withTimeout(connection.prompt(params), 30_000, `prompt ${JSON.stringify(text)}`); + }; + + // --- p01: capability and attachment advertisement --------------------- + await runScenario(evidence, "p01-capability-advertisement", async () => { + if (!connectionMeta) throw new Error("initialize has no agentCapabilities._meta.jcode"); + if (connectionMeta.interfaceVersion !== 1) { + throw new Error(`interfaceVersion ${safeJson(connectionMeta.interfaceVersion)}`); + } + const required = ["shared_control", "peer_turns", "turn_correlation", "skill_selection"]; + const missing = required.filter((name) => !(connectionMeta.extensions ?? []).includes(name)); + if (missing.length > 0) throw new Error(`missing extensions: ${missing.join(", ")}`); + if (connectionMeta.sessionMcpServers !== false) { + throw new Error(`sessionMcpServers ${safeJson(connectionMeta.sessionMcpServers)}`); + } + if (!attachmentMeta) throw new Error("session/new has no _meta.jcode"); + if (typeof attachmentMeta.clientInstanceId !== "string") { + throw new Error("attachment has no clientInstanceId"); + } + const expected = { + observe: true, + stopObservedTurn: true, + detach: true, + implicitSteering: true, + }; + for (const [field, value] of Object.entries(expected)) { + if (attachmentMeta.sharedControl?.[field] !== value) { + throw new Error( + `sharedControl.${field} is ${safeJson(attachmentMeta.sharedControl?.[field])}, expected ${value}`, + ); + } + } + if ("interfaceVersion" in attachmentMeta) { + throw new Error("attachment metadata repeats interfaceVersion"); + } + return { + connectionMeta, + attachmentMeta, + configOptions: created.configOptions, + }; + }); + + const markerMessages = (body: Record) => + (Array.isArray(body.messages) ? (body.messages as Array>) : []).filter( + (message) => JSON.stringify(message).includes(SKILL_MARKER), + ).length; + + // --- p02: explicit skill select, sticky, and null clear --------------- + await runScenario(evidence, "p02-skill-select-and-clear", async () => { + const before = model.requests.length; + const selectedPending = prompt("SKILL-SELECT", { jcode: { skill: "acceptance-skill" } }); + const selectedRequest = await model.takeContaining("SKILL-SELECT"); + const selectedReminders = markerMessages(selectedRequest.body); + if (selectedReminders !== 1) { + throw new Error(`expected one skill reminder on selection, got ${selectedReminders}`); + } + model.replyText(selectedRequest, "SKILL-SELECT-ANSWER"); + const selected = await selectedPending; + const selectedCorrelation = selected._meta?.jcode; + if (selectedCorrelation?.role !== "origin" || !selectedCorrelation?.turnId) { + throw new Error(`unexpected correlation ${safeJson(selectedCorrelation)}`); + } + + const stickyPending = prompt("STICKY-SELECTED"); + const stickyRequest = await model.takeContaining("STICKY-SELECTED"); + const stickyReminders = markerMessages(stickyRequest.body); + model.replyText(stickyRequest, "STICKY-ANSWER"); + await stickyPending; + if (stickyReminders !== 1) { + throw new Error(`sticky prompt appended ${stickyReminders - 1} extra skill reminders`); + } + + const clearedPromise = prompt("CLEARED-TEXT", { jcode: { skill: null } }); + const clearedRequest = await model.takeContaining("CLEARED-TEXT"); + const clearedReminders = markerMessages(clearedRequest.body); + const clearedLastUser = lastUserText(clearedRequest.body); + model.replyText(clearedRequest, "CLEARED-ANSWER"); + const clearedText = await clearedPromise; + if (clearedReminders !== 1) { + throw new Error(`clearing prompt appended a skill reminder (${clearedReminders})`); + } + if (!clearedLastUser.endsWith("CLEARED-TEXT")) { + throw new Error(`cleared prompt text was modified: ${safeJson(clearedLastUser)}`); + } + if (clearedText._meta?.jcode?.role !== "origin") { + throw new Error(`cleared text correlation ${safeJson(clearedText._meta?.jcode)}`); + } + + // Re-selecting the same skill after the clear must append a new hidden + // reminder. If the clear had not changed the attachment state, this + // selection would be a no-op and the count would stay at one. + const reselectPending = prompt("RESELECT-A", { jcode: { skill: "acceptance-skill" } }); + const reselectRequest = await model.takeContaining("RESELECT-A"); + const reselectReminders = markerMessages(reselectRequest.body); + model.replyText(reselectRequest, "RESELECT-ANSWER"); + await reselectPending; + if (reselectReminders !== 2) { + throw new Error( + `re-selection did not append a reminder (${reselectReminders}); the clear did not clear the attachment skill`, + ); + } + + const requestsBeforeClear = model.requests.length; + const cleared = await prompt("", { jcode: { skill: null } }); + await sleep(300); + if (model.requests.length !== requestsBeforeClear) { + throw new Error("selection-only clear started a model turn"); + } + const clearCorrelation = cleared._meta?.jcode; + if (clearCorrelation?.role !== "none" || clearCorrelation?.turnId !== null) { + throw new Error(`selection-only clear correlation ${safeJson(clearCorrelation)}`); + } + + const afterClearPending = prompt("AFTER-CLEAR"); + const afterClearRequest = await model.takeContaining("AFTER-CLEAR"); + const afterClearReminders = markerMessages(afterClearRequest.body); + model.replyText(afterClearRequest, "AFTER-CLEAR-ANSWER"); + await afterClearPending; + if (afterClearReminders !== 2) { + throw new Error(`prompt after clear appended a skill reminder (${afterClearReminders})`); + } + return { + selectionTurnId: selectedCorrelation.turnId, + reminderMessages: { + afterSelect: selectedReminders, + afterSticky: stickyReminders, + afterClearWithText: clearedReminders, + afterReselect: reselectReminders, + afterSelectionOnlyClear: afterClearReminders, + }, + clearedLastUser, + selectionOnlyClearRole: clearCorrelation.role, + requestCountDelta: model.requests.length - before, + }; + }); + + // --- p03: peer turn observation and steer correlation ----------------- + await runScenario(evidence, "p03-peer-observe-and-steer-correlation", async () => { + peer!.message(30, "PEER-BASE"); + const baseRequest = await model.takeContaining("PEER-BASE"); + const started = await waitFor( + "peer _jcode/turn started", + () => notifications.find((entry) => entry.method === "_jcode/turn" && entry.params?.state === "started"), + 15_000, + ); + const turnId = started.params.turnId; + const joinPromise = prompt("ACP-JOIN"); + model.replyText(baseRequest, "PEER-BASE-ANSWER"); + const joinRequest = await waitFor( + "provider request containing steer text", + () => model.pending.find((entry) => bodyText(entry.body).includes("ACP-JOIN")), + 20_000, + ); + model.replyText(joinRequest, "JOIN-ANSWER"); + const joined = await joinPromise; + const correlation = joined._meta?.jcode; + if (correlation?.role !== "steer") { + throw new Error(`steer response role ${safeJson(correlation)}`); + } + if (correlation?.turnId !== turnId) { + throw new Error( + `steer turnId ${safeJson(correlation?.turnId)} != notification turnId ${safeJson(turnId)}`, + ); + } + await waitFor( + "peer turn completed notification", + () => + notifications.find( + (entry) => + entry.method === "_jcode/turn" && + entry.params?.turnId === turnId && + entry.params?.state === "completed", + ), + 15_000, + ); + const starts = notifications.filter( + (entry) => entry.method === "_jcode/turn" && entry.params?.turnId === turnId && entry.params?.state === "started", + ); + if (starts.length !== 1) throw new Error(`expected one started notification, got ${starts.length}`); + await waitFor( + "peer completed terminal", + () => + peer! + .turnEnds("completed") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(turnId)), + 10_000, + ); + const peerStarts = peer! + .turnStarts() + .filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(turnId)); + const peerCompleted = peer! + .turnEnds("completed") + .filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(turnId)); + if (peerStarts.length !== 1 || peerCompleted.length !== 1) { + throw new Error( + `daemon terminal mismatch: starts=${peerStarts.length} completed=${peerCompleted.length} turnId=${safeJson(turnId)} observedStarts=${safeJson(peer!.turnStarts().map((event) => ({ identity: event.identity, content: event.user_content })))}`, + ); + } + return { + observedTurnId: turnId, + steerRole: correlation.role, + steerTurnId: correlation.turnId, + peerTerminals: { + completed: peer!.turnEnds("completed").map((event) => event.identity), + cancelled: peer!.turnEnds("cancelled").map((event) => event.identity), + }, + }; + }); + + // --- p04: detach leaves inserted work running ------------------------- + await runScenario(evidence, "p04-detach", async () => { + const promptPromise = prompt("DETACH-BASE"); + const request = await model.takeContaining("DETACH-BASE"); + const turnId = await waitFor( + "detach turn identity", + () => peer!.identityFor("DETACH-BASE"), + 15_000, + ); + const detach = await withTimeout( + connection.extMethod("_jcode/detach", { sessionId: protocolSessionId }), + 10_000, + "detach", + ); + if (safeJson(detach) !== "{}") throw new Error(`detach result ${safeJson(detach)}`); + const settled = await promptPromise; + const correlation = settled._meta?.jcode; + if (settled.stopReason !== "cancelled") { + throw new Error(`detached prompt stopReason ${safeJson(settled.stopReason)}`); + } + if ( + turnIdentityKey(correlation?.turnId) !== turnIdentityKey(turnId) || + correlation?.role !== "origin" + ) { + throw new Error( + `detached prompt correlation ${safeJson({ correlation, peerIdentity: turnId, peerStarts: peer!.turnStarts().map((event) => ({ identity: event.identity, content: event.user_content })) })}`, + ); + } + if ( + peer! + .turnEnds("cancelled") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(turnId)) + ) { + throw new Error("detach cancelled inserted work"); + } + const updatesAfterDetach = updates.length; + await sleep(300); + if (updates.length !== updatesAfterDetach) { + throw new Error("session updates continued after the detach response"); + } + model.replyText(request, "DETACH-ANSWER"); + await waitFor( + "inserted work completed after detach", + () => + peer! + .turnEnds("completed") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(turnId)), + 20_000, + ); + const repeated = await withTimeout( + connection.extMethod("_jcode/detach", { sessionId: protocolSessionId }), + 10_000, + "repeated detach", + ); + if (safeJson(repeated) !== "{}") throw new Error(`repeated detach ${safeJson(repeated)}`); + let malformedCode: number | null = null; + try { + await connection.extMethod("_jcode/detach", { sessionId: 123 }); + } catch (error) { + malformedCode = (error as { code?: number }).code ?? null; + } + if (malformedCode !== -32602) { + throw new Error(`malformed detach error code ${safeJson(malformedCode)}`); + } + // Resume by replaying history, then run a normal turn on the new attachment. + const reloaded = await withTimeout( + connection.loadSession({ sessionId: protocolSessionId, cwd: workspace.cwd, mcpServers: [] }), + 20_000, + "session/load after detach", + ); + const resumedPending = prompt("AFTER-DETACH"); + const resumedRequest = await model.takeContaining("AFTER-DETACH"); + model.replyText(resumedRequest, "AFTER-DETACH-ANSWER"); + const resumed = await resumedPending; + if (resumed._meta?.jcode?.role !== "origin") { + throw new Error(`resumed prompt correlation ${safeJson(resumed._meta?.jcode)}`); + } + return { + detachedTurnId: turnId, + detachedStopReason: settled.stopReason, + insertedWorkContinued: true, + repeatedDetachResult: repeated, + malformedDetachCode: malformedCode, + updatesAfterDetach: 0, + reloadAttachment: reloaded._meta?.jcode ?? null, + resumedRole: resumed._meta?.jcode?.role, + }; + }); + + // --- p05: model/effort config and legacy aliases ---------------------- + await runScenario(evidence, "p05-model-effort-config", async () => { + const modelSet = await withTimeout( + connection.setSessionConfigOption({ + sessionId: protocolSessionId, + configId: "model", + value: ALT_MODEL, + }), + 15_000, + "set model", + ); + const modelOption = (modelSet.configOptions ?? []).find((option: any) => option.id === "model"); + if (modelOption?.currentValue !== ALT_MODEL) { + throw new Error(`model option ${safeJson(modelOption?.currentValue)}`); + } + const effortSet = await withTimeout( + connection.setSessionConfigOption({ + sessionId: protocolSessionId, + configId: "reasoning_effort", + value: "high", + }), + 15_000, + "set reasoning_effort", + ); + const effortOption = (effortSet.configOptions ?? []).find( + (option: any) => option.id === "reasoning_effort", + ); + if (effortOption?.currentValue !== "high") { + throw new Error(`effort option ${safeJson(effortOption)}`); + } + const configuredPending = prompt("CONFIG-PROBE"); + const configuredRequest = await model.takeContaining("CONFIG-PROBE"); + const summary = { + model: configuredRequest.body.model, + effort: findEffort(configuredRequest.body), + }; + model.replyText(configuredRequest, "CONFIG-ANSWER"); + const configured = await configuredPending; + if (summary.model !== ALT_MODEL) throw new Error(`provider model ${safeJson(summary.model)}`); + if (!summary.effort || summary.effort[1] !== "high") { + throw new Error(`provider effort ${safeJson(summary.effort)}`); + } + if (configured._meta?.jcode?.role !== "origin") { + throw new Error(`config prompt correlation ${safeJson(configured._meta?.jcode)}`); + } + + const legacyModel = await withTimeout( + connection.extMethod("session/set_model", { + sessionId: protocolSessionId, + modelId: DEFAULT_MODEL, + }), + 15_000, + "legacy set_model", + ); + const legacyOption = (legacyModel.configOptions ?? legacyModel.result?.configOptions ?? []).find( + (option: any) => option.id === "model", + ); + const legacyEffort = await withTimeout( + connection.extMethod("session/set_reasoning_effort", { + sessionId: protocolSessionId, + effort: "low", + }), + 15_000, + "legacy set_reasoning_effort", + ); + const legacyEffortOption = ( + legacyEffort.configOptions ?? + legacyEffort.result?.configOptions ?? + [] + ).find((option: any) => option.id === "reasoning_effort"); + const legacyPromptPending = prompt("LEGACY-ALIAS-PROBE"); + const legacyRequest = await model.takeContaining("LEGACY-ALIAS-PROBE"); + const legacySummary = { + model: legacyRequest.body.model, + effort: findEffort(legacyRequest.body), + lastUser: lastUserText(legacyRequest.body), + }; + model.replyText(legacyRequest, "LEGACY-ANSWER"); + await legacyPromptPending; + if (legacySummary.model !== DEFAULT_MODEL) { + throw new Error(`legacy model alias did not apply: ${safeJson(legacySummary.model)}`); + } + return { + configOptions: { model: modelOption, effort: effortOption }, + providerRequest: summary, + legacyModelOption: legacyOption ?? null, + legacyEffortOption: legacyEffortOption ?? null, + legacyProviderRequest: legacySummary, + }; + }); + + evidence.notes.push( + "Probe covers capability advertisement, all four sharedControl booleans, correlation roles, explicit skill select plus null clear, _jcode/detach, and config aliases against the real binary.", + ); + evidence.notes.push( + "This is not Paseo-connector proof. Connector-level evidence lives in jcode-live-acceptance.ts.", + ); + } catch (error) { + evidence.notes.push( + `setup failure: ${error instanceof Error ? error.stack ?? error.message : String(error)}`, + ); + throw error; + } finally { + try { + await connection?.close?.(); + } catch { + // ignore + } + if (acp && acp.exitCode === null && acp.signalCode === null) { + acp.kill("SIGTERM"); + await Promise.race([ + new Promise((resolve) => acp!.once("close", () => resolve())), + sleep(2_000), + ]); + if (acp.exitCode === null && acp.signalCode === null) acp.kill("SIGKILL"); + } + peer?.destroy(); + try { + await daemon?.stop(); + } catch { + // ignore + } + try { + await model.close(); + } catch { + // ignore + } + const evidencePath = + process.env.JCODE_ACCEPTANCE_OUT ?? + path.join(process.env.JCODE_SCRATCH_DIR ?? "/tmp", "jcode-acp-probe.evidence.json"); + evidence.notes.push(`acp stderr tail: ${stderr.join("").slice(-2000).replace(/\s+/g, " ")}`); + writeEvidence(evidencePath, evidence); + console.log(`evidence: ${evidencePath}`); + console.log(`scenarios: ${summaryLine(evidence)}`); + await workspace.cleanup().catch(() => undefined); + if (Object.values(evidence.scenarios).some((scenario) => scenario.status === "fail")) { + process.exitCode = 1; + } + } +} + +main().catch((error) => { + console.error(error instanceof Error ? error.stack ?? error.message : error); + process.exitCode = 1; +}); diff --git a/scripts/live-acceptance/jcode-live-acceptance.ts b/scripts/live-acceptance/jcode-live-acceptance.ts new file mode 100644 index 00000000000..c4be74555b4 --- /dev/null +++ b/scripts/live-acceptance/jcode-live-acceptance.ts @@ -0,0 +1,1153 @@ +// Live v1 acceptance through the real Paseo ACP connector. +// +// The consumer modules are imported from the consumer-owned checkout +// (PASEO_CONSUMER_ROOT, supplied by run-live-acceptance.sh). No +// consumer source is copied or stubbed: the real Paseo daemon creates the real +// `jcode acp` child and this harness drives it exactly as the app does. +// +// The jcode side is a real binary on an isolated socket/home. Only the model is +// scripted (local OpenAI-compatible HTTP server) so turn order is deterministic. +import path from "node:path"; +import { createRequire } from "node:module"; +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; +import { spawnSync } from "node:child_process"; +import { + ALT_MODEL, + DEFAULT_MODEL, + SKILL_MARKER, + binaryProvenance, + bodyText, + connectPeer, + createWorkspace, + findEffort, + lastUserText, + linkBinary, + newEvidence, + requireBinary, + runScenario, + safeJson, + startJcodeDaemon, + startMockModel, + summaryLine, + turnIdentityKey, + waitFor, + writeEvidence, + type JcodeDaemon, + type LivePeer, + type MockModel, +} from "./jcode-live-common.ts"; + +const CONSUMER_ROOT = + process.env.PASEO_CONSUMER_ROOT ?? process.cwd(); + +async function loadConsumer(relativePath: string): Promise { + return import(pathToFileURL(path.join(CONSUMER_ROOT, relativePath)).href); +} + +function gitValue(root: string, args: string[]): string { + const result = spawnSync("git", ["-C", root, ...args], { encoding: "utf8" }); + return (result.stdout ?? "").trim(); +} + +function consumerWorktreeDigest(): string { + const digest = createHash("sha256"); + digest.update(gitValue(CONSUMER_ROOT, ["diff", "HEAD"])); + digest.update(gitValue(CONSUMER_ROOT, ["status", "--porcelain"])); + const untracked = gitValue(CONSUMER_ROOT, ["ls-files", "--others", "--exclude-standard"]) + .split("\n") + .filter(Boolean) + .sort(); + for (const file of untracked) { + digest.update(file); + try { + digest.update(readFileSync(path.join(CONSUMER_ROOT, file))); + } catch { + digest.update(""); + } + } + return digest.digest("hex"); +} + +const CONSUMER_SOURCE_FILES = [ + "packages/plugin/src/server/acp-internal/connection.ts", + "packages/plugin/src/server/acp.ts", + "packages/plugin/src/server/provider.ts", + "packages/server/src/server/session.ts", + "packages/server/src/server/agent/agent-manager.ts", + "packages/server/src/server/agent/plugin-provider.ts", + "packages/server/src/server/agent/provider-registry.ts", + "packages/protocol/src/messages.ts", + "plugins/jcode-provider/index.server.ts", +]; + +function consumerSourceDigest(): string { + const digest = createHash("sha256"); + for (const file of CONSUMER_SOURCE_FILES) { + digest.update(file); + try { + digest.update(readFileSync(path.join(CONSUMER_ROOT, file))); + } catch { + digest.update(""); + } + } + return digest.digest("hex"); +} + +function consumerProvenance() { + const require = createRequire(path.join(CONSUMER_ROOT, "package.json")); + let pluginEntry: string | null = null; + try { + pluginEntry = require.resolve("@getpaseo/plugin/server"); + } catch { + pluginEntry = null; + } + return { + root: CONSUMER_ROOT, + head: gitValue(CONSUMER_ROOT, ["rev-parse", "HEAD"]), + tree: gitValue(CONSUMER_ROOT, ["rev-parse", "HEAD^{tree}"]), + worktreeSha256: consumerWorktreeDigest(), + sourceSha256: consumerSourceDigest(), + sourceFiles: CONSUMER_SOURCE_FILES, + branch: gitValue(CONSUMER_ROOT, ["branch", "--show-current"]), + dirty: gitValue(CONSUMER_ROOT, ["status", "--porcelain"]).split("\n").filter(Boolean).length, + // Record the module the daemon actually imported. Under the default + // conditions this is compiled dist, which can lag the checkout; the + // runner passes --conditions=source so the true source tree is exercised. + pluginEntry, + nodeConditions: process.execArgv.filter((arg) => arg.includes("conditions")), + pluginDist: path.join(CONSUMER_ROOT, "packages/plugin/dist/server/acp-internal/connection.js"), + pluginSource: path.join(CONSUMER_ROOT, "packages/plugin/src/server/acp-internal/connection.ts"), + }; +} + +function userTexts(entries: Array<{ item: any }>): string[] { + return entries + .map((entry) => entry.item) + .filter((item) => item?.type === "user_message") + .map((item) => String(item.text ?? "")); +} + +function peerTurnIdentities(peer: LivePeer) { + return { + started: peer.turnStarts().map((event) => ({ identity: event.identity, content: event.user_content, origin: event.origin })), + completed: peer.turnEnds("completed").map((event) => event.identity), + cancelled: peer.turnEnds("cancelled").map((event) => event.identity), + }; +} + +function requestSummary(request: { body: Record } | undefined) { + if (!request) return null; + return { + model: request.body.model ?? null, + effort: findEffort(request.body), + stream: request.body.stream ?? null, + hasTools: Array.isArray(request.body.tools) && request.body.tools.length > 0, + }; +} + +async function main() { + const evidence = newEvidence(); + evidence.provenance = { + jcode: binaryProvenance(requireBinary()), + consumer: consumerProvenance(), + }; + evidence.environment = { + node: process.version, + platform: `${process.platform}/${process.arch}`, + consumerRoot: CONSUMER_ROOT, + isolatedWorkspacePrefix: "jcode-live-", + }; + + const binary = requireBinary(); + const workspace = await createWorkspace(); + const model = await startMockModel(); + linkBinary(binary, workspace.binDir); + + let daemon: JcodeDaemon | undefined; + let peer: LivePeer | undefined; + let paseo: any; + let client: any; + const restoreEnv: Array<() => void> = []; + const setEnv = (key: string, value: string | undefined) => { + const previous = process.env[key]; + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + restoreEnv.push(() => { + if (previous === undefined) delete process.env[key]; + else process.env[key] = previous; + }); + }; + + try { + daemon = await startJcodeDaemon({ binary, workspace, modelBaseUrl: model.baseUrl }); + peer = await connectPeer(daemon.socketPath); + const session = await peer.subscribe(workspace.cwd, "acceptance-peer"); + const sessionId: string = session.session_id; + evidence.environment.sessionId = sessionId; + evidence.environment.isolated = { + root: workspace.root, + home: workspace.home, + runtime: workspace.runtime, + socket: workspace.socketPath, + }; + + // Seed the shared session from the peer (TUI stand-in), then reply. + peer.message(2, "BASELINE"); + const baselineRequest = await model.takeContaining("BASELINE"); + model.replyText(baselineRequest, "BASELINE-ANSWER"); + await waitFor( + "baseline peer turn completed", + () => peer!.turnEnds("completed").length >= 1, + 15_000, + ); + + const { createTestPaseoDaemon } = await loadConsumer( + "packages/server/src/server/test-utils/paseo-daemon.ts", + ); + const { DaemonClient } = await loadConsumer( + "packages/server/src/server/test-utils/daemon-client.ts", + ); + + // The daemon (in-process) passes its environment to `jcode acp`, so expose + // the isolated jcode environment before the daemon starts. + for (const [key, value] of Object.entries(daemon.env)) { + if (key.startsWith("JCODE_") || key === "OPENAI_COMPAT_API_KEY" || key === "PATH") { + setEnv(key, value === undefined ? undefined : String(value)); + } + } + setEnv("JCODE_SOCKET", workspace.socketPath); + + paseo = await createTestPaseoDaemon({ + pluginsEnabled: true, + paseoHomeRoot: path.join(workspace.root, "paseo"), + }); + client = new DaemonClient({ url: `ws://127.0.0.1:${paseo.port}/ws` }); + await client.connect(); + await client.fetchAgents({ subscribe: {} }); + await client.installDirectoryPlugin(path.join(CONSUMER_ROOT, "plugins/jcode-provider")); + const imported = await client.importAgent({ + providerId: "jcode", + providerHandleId: `plugin:${JSON.stringify({ version: 1, data: { sessionId } })}`, + cwd: workspace.cwd, + }); + let agentId: string = imported.id; + + const timeline = async () => { + const page = await client.fetchAgentTimeline(agentId, { + direction: "tail", + limit: 0, + projection: "canonical", + }); + return page.entries as Array<{ item: any }>; + }; + const agent = async () => (await client.fetchAgent({ agentId })).agent; + const waitIdle = () => + waitFor("agent idle", async () => (await agent()).status === "idle", 30_000); + const nextRequest = (text?: string) => + text ? model.takeContaining(text) : waitFor("next model request", () => model.pending[0], 20_000); + const send = (text: string, options?: Record) => { + const promise: Promise = client.sendMessage(agentId, text, options); + // The harness releases the model response before awaiting the send. + // Mark a rejection handled immediately so an early busy rejection cannot + // kill the process before the scenario's await attaches. + promise.catch(() => undefined); + return promise; + }; + + // --- s01: real connector import and replay ----------------------------- + await runScenario(evidence, "s01-import-and-replay", async () => { + const entries = await timeline(); + const texts = userTexts(entries); + if (JSON.stringify(texts) !== JSON.stringify(["BASELINE"])) { + throw new Error(`unexpected replayed user messages: ${safeJson(texts)}`); + } + const snapshot = await agent(); + const baselineIdentity = peer!.identityFor("BASELINE"); + if (!baselineIdentity) throw new Error("no daemon turn identity for BASELINE"); + return { + replayedUserMessages: texts, + baselineTurnIdentity: baselineIdentity, + model: snapshot.model, + capabilities: snapshot.capabilities, + providerHandle: snapshot.persistence ?? null, + }; + }); + + // --- s02: forward steering through the connector ----------------------- + await runScenario(evidence, "s02-forward-steer", async () => { + peer!.message(20, "PEER-TURN"); + const peerRequest = await nextRequest("PEER-TURN"); + const steerPromise = send("FORWARD-STEER", { + activeTurnBehavior: "steer", + }); + model.replyText(peerRequest, "PEER-INTERMEDIATE"); + await steerPromise; + const steerRequest = await nextRequest("FORWARD-STEER"); + model.replyText(steerRequest, "FORWARD-ANSWER"); + await waitIdle(); + + const texts = userTexts(await timeline()); + const expected = ["BASELINE", "PEER-TURN", "FORWARD-STEER"]; + if (JSON.stringify(texts) !== JSON.stringify(expected)) { + throw new Error(`unexpected user messages: ${safeJson(texts)}`); + } + const identity = peer!.identityFor("PEER-TURN"); + const starts = peer!.turnStarts().filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)); + const completed = peer!.turnEnds("completed").filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)); + const cancelled = peer!.turnEnds("cancelled").filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)); + if (starts.length !== 1) throw new Error(`expected one turn start, got ${starts.length}`); + if (completed.length !== 1) throw new Error(`expected one completion, got ${completed.length}`); + if (cancelled.length !== 0) throw new Error("forward steer cancelled the turn"); + return { + steerJoinedIdentity: identity, + turnStarts: starts.length, + turnCompletions: completed.length, + turnCancellations: cancelled.length, + request: requestSummary(steerRequest), + texts, + }; + }); + + // --- s03: reverse steering from the peer ------------------------------ + await runScenario(evidence, "s03-reverse-steer", async () => { + const prompt = send("FOREGROUND"); + const foregroundRequest = await nextRequest("FOREGROUND"); + peer!.softInterrupt(21, "REVERSE-STEER"); + model.replyText(foregroundRequest, "FOREGROUND-INTERMEDIATE"); + await prompt; + await waitFor( + "peer steer text in Paseo timeline", + async () => userTexts(await timeline()).includes("REVERSE-STEER"), + 15_000, + ); + const steerRequest = await nextRequest("REVERSE-STEER"); + model.replyText(steerRequest, "REVERSE-ANSWER"); + await waitIdle(); + + const texts = userTexts(await timeline()); + const expected = ["BASELINE", "PEER-TURN", "FORWARD-STEER", "FOREGROUND", "REVERSE-STEER"]; + if (JSON.stringify(texts) !== JSON.stringify(expected)) { + throw new Error(`unexpected user messages: ${safeJson(texts)}`); + } + const identity = peer!.identityFor("FOREGROUND"); + if (!identity) throw new Error("no daemon identity for the Paseo-origin turn"); + const completed = peer!.turnEnds("completed").filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)); + const cancelled = peer!.turnEnds("cancelled").filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)); + if (completed.length !== 1 || cancelled.length !== 0) { + throw new Error(`turn terminal mismatch completed=${completed.length} cancelled=${cancelled.length}`); + } + return { foregroundTurnIdentity: identity, texts, request: requestSummary(steerRequest) }; + }); + + // --- s04: one execution, several prompt responses, settled once ------ + // The contract does not require the consumer to replace its internal + // activeTurn id with the daemon identity. What it requires is that several + // prompt responses naming one execution settle completion, display and + // usage once. This drives an origin prompt and a concurrent steer into the + // same execution while the model backend is deliberately held, so the + // daemon evidence shows one turn identity taking the injected steer and + // the consumer shows one visible terminal, one answer pair, and one usage + // total covering both model calls. + await runScenario(evidence, "s04-turn-correlation-single-settlement", async () => { + const ORIGIN = "CORRELATED-TURN"; + const STEER = "CORRELATED-STEER"; + const originUsage = { promptTokens: 1000, completionTokens: 20 }; + const steerUsage = { promptTokens: 300, completionTokens: 7 }; + const expectedInput = originUsage.promptTokens + steerUsage.promptTokens; + const expectedOutput = originUsage.completionTokens + steerUsage.completionTokens; + const requestsBefore = model.requests.length; + const snapshots: Array<{ + lifecycle: string; + activeTurnId: string | null; + usage: Record | null; + }> = []; + const unsubscribe = paseo.daemon.agentManager.subscribe( + (event: any) => { + if (event.type !== "agent_state") return; + if (event.agent.id !== agentId) return; + snapshots.push({ + lifecycle: String(event.agent.lifecycle), + activeTurnId: event.agent.activeTurnId ?? null, + usage: event.agent.lastUsage ?? null, + }); + }, + { agentId, replayState: false }, + ); + try { + const origin = send(ORIGIN); + const originRequest = await nextRequest(ORIGIN); + const daemonIdentity = await waitFor( + "daemon identity for CORRELATED-TURN", + () => peer!.identityFor(ORIGIN), + 15_000, + ); + const during = await waitFor( + "Paseo active turn during CORRELATED-TURN", + async () => (await agent()).activeTurn, + 10_000, + ); + + // Queue the steer into the same execution while the origin's model + // call is still held. Nothing has been answered yet. + const steer = send(STEER, { activeTurnBehavior: "steer" }); + await waitFor( + "Paseo accepted the steer into the active turn", + async () => { + const snapshot = await agent(); + const texts = userTexts(await timeline()); + return ( + snapshot.activeTurn?.turnId === during.turnId && + texts.includes(STEER) && + model.pending.length === 1 && + model.pending[0] === originRequest + ); + }, + 10_000, + ); + + model.replyText(originRequest, "CORRELATION-ANSWER", originUsage); + const steerRequest = await nextRequest(STEER); + if (steerRequest === originRequest) { + throw new Error("steer reused the origin model request"); + } + // The first answer has streamed, but the execution is not terminal: + // the injected steer still owes a model call and an answer. + const midExecution = await agent(); + if (midExecution.status !== "running" || midExecution.activeTurn?.turnId !== during.turnId) { + throw new Error( + `Paseo settled after the first answer: ${safeJson({ status: midExecution.status, activeTurn: midExecution.activeTurn })}`, + ); + } + model.replyText(steerRequest, "STEER-ANSWER", steerUsage); + await origin; + await steer; + await waitIdle(); + + if (model.requests.length - requestsBefore !== 2) { + throw new Error( + `expected exactly two model calls in one execution, got ${model.requests.length - requestsBefore}`, + ); + } + if (completedFor(daemonIdentity) !== 1) { + throw new Error("shared turn did not complete exactly once"); + } + const otherStarts = peer! + .turnStarts() + .filter((event) => event.user_content === STEER); + if (otherStarts.length !== 0) { + throw new Error("the steer opened a second daemon turn"); + } + const injection = peer!.events + .filter( + (event) => + event.type === "turn_event" && + turnIdentityKey(event.identity) === turnIdentityKey(daemonIdentity), + ) + .map((event) => event.event) + .find( + (inner: any) => + inner?.type === "soft_interrupt_injected" && inner.content === STEER, + ); + if (!injection) { + throw new Error("the steer was not injected into the origin's daemon turn"); + } + if (!Array.isArray(injection.client_ids) || injection.client_ids.length === 0) { + throw new Error("the injected steer carried no client ids"); + } + + const entries = await timeline(); + const answers = entries.filter( + (entry) => + entry.item?.type === "assistant_message" && + (String(entry.item.text ?? "").includes("CORRELATION-ANSWER") || + String(entry.item.text ?? "").includes("STEER-ANSWER")), + ); + const answerTexts = answers.map((entry) => String(entry.item.text ?? "")); + const expectedAnswers = ["CORRELATION-ANSWER", "STEER-ANSWER"]; + if (JSON.stringify(answerTexts) !== JSON.stringify(expectedAnswers)) { + throw new Error(`unexpected visible answers: ${safeJson(answerTexts)}`); + } + const premature = snapshots.filter( + (entry) => entry.lifecycle !== "running" && entry.activeTurnId !== null, + ); + if (premature.length > 0) { + throw new Error(`Paseo kept a turn active outside running: ${safeJson(premature)}`); + } + const transitions = snapshots + .map((entry) => entry.lifecycle) + .filter((lifecycle, index, all) => index === 0 || lifecycle !== all[index - 1]); + if (JSON.stringify(transitions) !== JSON.stringify(["running", "idle"])) { + throw new Error(`unexpected Paseo lifecycle transitions: ${safeJson(transitions)}`); + } + const turnIds = Array.from( + new Set( + snapshots + .map((entry) => entry.activeTurnId) + .filter((turnId): turnId is string => turnId !== null), + ), + ); + if (turnIds.length !== 1 || turnIds[0] !== during.turnId) { + throw new Error(`the execution used several local turn ids: ${safeJson(turnIds)}`); + } + const after = await agent(); + if (after.activeTurn) { + throw new Error(`active turn survived completion: ${safeJson(after.activeTurn)}`); + } + const usage = after.lastUsage ?? null; + if ( + usage?.inputTokens !== expectedInput || + usage?.outputTokens !== expectedOutput + ) { + throw new Error( + `usage settled as ${safeJson(usage)}; expected ${expectedInput} input / ${expectedOutput} output once`, + ); + } + const doubled = snapshots.filter( + (entry) => + entry.usage?.inputTokens === expectedInput * 2 || + entry.usage?.outputTokens === expectedOutput * 2, + ); + if (doubled.length > 0) { + throw new Error(`usage was applied twice: ${safeJson(doubled)}`); + } + return { + paseoLocalTurnId: during.turnId, + daemonTurnIdentity: daemonIdentity, + daemonCompletions: completedFor(daemonIdentity), + modelCalls: model.requests.length - requestsBefore, + injectedSteer: { + point: injection.point, + clientIds: injection.client_ids.length, + scopedToOrigin: true, + }, + visibleAnswers: answerTexts, + lifecycle: snapshots, + usage: { + inputTokens: usage.inputTokens, + outputTokens: usage.outputTokens, + expectedInput, + expectedOutput, + }, + singleSettlement: true, + peerTerminals: peerTurnIdentities(peer!), + }; + } finally { + unsubscribe(); + } + }); + + function completedFor(identity: string | undefined) { + return peer!.turnEnds("completed").filter((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)).length; + } + + // --- s05: Stop followed immediately by a new prompt ------------------- + await runScenario(evidence, "s05-stop-then-prompt", async () => { + const stopping = send("STOPPED-PROMPT").catch((error) => { + evidence.notes.push(`stop prompt rejected: ${String(error)}`); + }); + const stopRequest = await nextRequest("STOPPED-PROMPT"); + model.replyToolCall(stopRequest, { + id: "live-stop-tool", + name: "bash", + arguments: { command: "sleep 30" }, + }); + const stoppedIdentity = await waitFor( + "stopped turn identity", + () => peer!.identityFor("STOPPED-PROMPT"), + 15_000, + ); + await waitFor( + "tool execution for stopped turn", + () => + peer!.events.some( + (event) => + event.type === "turn_event" && + turnIdentityKey(event.identity) === turnIdentityKey(stoppedIdentity) && + event.event?.type === "tool_exec" && + event.event?.id === "live-stop-tool", + ), + 10_000, + ); + await client.cancelAgent(agentId); + const successor = send("SUCCESSOR-PROMPT", { + activeTurnBehavior: "interrupt", + }); + const successorRequest = await nextRequest("SUCCESSOR-PROMPT"); + model.replyText(successorRequest, "SUCCESSOR-ANSWER"); + await successor; + await waitIdle(); + const successorIdentity = await waitFor( + "successor turn identity", + () => peer!.identityFor("SUCCESSOR-PROMPT"), + 15_000, + ); + const cancelled = peer! + .turnEnds("cancelled") + .map((event) => event.identity); + const cancelledKeys = cancelled.map(turnIdentityKey); + if (!cancelledKeys.includes(turnIdentityKey(stoppedIdentity))) { + throw new Error(`stopped turn ${safeJson(stoppedIdentity)} was not cancelled; cancelled=${safeJson(cancelled)}`); + } + if (cancelledKeys.includes(turnIdentityKey(successorIdentity))) { + throw new Error("a delayed stop cancelled the successor turn"); + } + if ( + peer! + .turnEnds("completed") + .every((event) => turnIdentityKey(event.identity) !== turnIdentityKey(successorIdentity)) + ) { + throw new Error("successor turn did not complete"); + } + if (turnIdentityKey(successorIdentity) === turnIdentityKey(stoppedIdentity)) { + throw new Error("successor reused the stopped turn identity"); + } + void stopping; + return { + stoppedIdentity, + successorIdentity, + cancelled, + successorCompleted: true, + texts: userTexts(await timeline()), + }; + }); + + // --- s06: Stop an observed peer turn (stopObservedTurn) --------------- + await runScenario(evidence, "s06-stop-observed-peer-turn", async () => { + peer!.message(22, "PEER-STOP-TURN"); + const request = await nextRequest("PEER-STOP-TURN"); + model.replyToolCall(request, { + id: "live-peer-stop-tool", + name: "bash", + arguments: { command: "sleep 30" }, + }); + const identity = await waitFor( + "peer stop turn identity", + () => peer!.identityFor("PEER-STOP-TURN"), + 15_000, + ); + await waitFor( + "peer tool execution", + () => + peer!.events.some( + (event) => + event.type === "turn_event" && + turnIdentityKey(event.identity) === turnIdentityKey(identity) && + event.event?.type === "tool_exec" && + event.event?.id === "live-peer-stop-tool", + ), + 10_000, + ); + await client.cancelAgent(agentId); + await waitFor( + "observed peer turn cancelled", + () => peer!.turnEnds("cancelled").some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)), + 10_000, + ); + return { + observedTurnIdentity: identity, + cancelled: true, + snapshotStatus: (await agent()).status, + }; + }); + + // --- s07: model and effort propagate both directions ------------------ + await runScenario(evidence, "s07-model-effort", async () => { + await client.setAgentModel(agentId, ALT_MODEL); + await waitFor( + "peer model_changed", + () => peer!.events.find((event) => event.type === "model_changed" && event.model === ALT_MODEL), + 10_000, + ); + await client.setAgentThinkingOption(agentId, "high"); + await waitFor( + "peer reasoning_effort_changed high", + () => + peer!.events.find( + (event) => + event.type === "reasoning_effort_changed" && + (event.effort === "high" || event.reasoning_effort === "high"), + ), + 10_000, + ); + const after = await agent(); + if (after.model !== ALT_MODEL) throw new Error(`Paseo model is ${safeJson(after.model)}`); + const effortHigh = after.thinkingOptionId === "high" || after.effectiveThinkingOptionId === "high"; + if (!effortHigh) { + throw new Error( + `Paseo thinking option is ${safeJson({ thinkingOptionId: after.thinkingOptionId, effective: after.effectiveThinkingOptionId })}`, + ); + } + const prompt = send("EFFORT-MODEL-PROBE"); + const request = await nextRequest("EFFORT-MODEL-PROBE"); + const summary = requestSummary(request); + model.replyText(request, "EFFORT-MODEL-ANSWER"); + await prompt; + await waitIdle(); + if (summary?.model !== ALT_MODEL) { + throw new Error(`provider request model ${safeJson(summary?.model)} != ${ALT_MODEL}`); + } + if (!summary?.effort || summary.effort[1] !== "high") { + throw new Error(`provider request effort is ${safeJson(summary?.effort)}`); + } + + // Reverse direction uses the real protocol setters, not a textual + // `/effort` message: only the setters fan out to other attachments. + peer!.setReasoningEffort(31, "low"); + const peerEffort = await waitFor( + "peer set_reasoning_effort reply", + () => + peer!.events.find( + (event) => + event.type === "reasoning_effort_changed" && + event.id === 31 && + (event.effort === "low" || event.reasoning_effort === "low"), + ), + 10_000, + ); + const afterPeer = await waitFor( + "Paseo reflects peer effort", + async () => { + const snapshot = await agent(); + return snapshot.effectiveThinkingOptionId === "low" || snapshot.thinkingOptionId === "low" + ? snapshot + : undefined; + }, + 10_000, + ); + peer!.setModel(32, DEFAULT_MODEL); + const peerModel = await waitFor( + "peer set_model reply", + () => + peer!.events.find( + (event) => + event.type === "model_changed" && + event.id === 32 && + event.model === DEFAULT_MODEL, + ), + 10_000, + ); + const afterPeerModel = await waitFor( + "Paseo reflects peer model", + async () => { + const snapshot = await agent(); + return snapshot.model === DEFAULT_MODEL ? snapshot : undefined; + }, + 10_000, + ); + return { + model: summary.model, + effort: summary.effort, + passthrough: { model: after.model, thinkingOptionId: after.thinkingOptionId }, + peerSetterEffort: peerEffort, + peerSetterModel: peerModel, + paseoAfterPeerSetters: { + model: afterPeerModel.model, + thinkingOptionId: afterPeer.thinkingOptionId, + effectiveThinkingOptionId: afterPeer.effectiveThinkingOptionId, + }, + }; + }); + + // --- s08: textual skill selection through the connector --------------- + await runScenario(evidence, "s08-skill-selection-textual", async () => { + const prompt = send("/acceptance-skill SKILL-SELECT"); + const request = await nextRequest("SKILL-SELECT"); + const body = bodyText(request.body); + if (!body.includes(SKILL_MARKER)) { + throw new Error("skill instructions were not present in the provider request"); + } + model.replyText(request, "SKILL-ANSWER"); + await prompt; + await waitIdle(); + + const sticky = send("STICKY-AFTER-SKILL"); + const stickyRequest = await nextRequest("STICKY-AFTER-SKILL"); + const stickyBody = bodyText(stickyRequest.body); + model.replyText(stickyRequest, "STICKY-ANSWER"); + await sticky; + await waitIdle(); + return { + markerPresentOnSelection: true, + markerPresentOnStickyPrompt: stickyBody.includes(SKILL_MARKER), + note: "explicit selection and null clear are scored by s10-explicit-skill-select-clear and jcode-acp-probe.ts", + }; + }); + + // --- s09: owner-aware close (delete, not archive) --------------------- + // Paseo archive intentionally cancels the active run before archiving, so + // it is not the owner-aware close path. Deleting the agent closes the + // provider session without a prior cancel: the ACP view must close while a + // peer-owned turn keeps running. + await runScenario(evidence, "s09-owner-aware-close", async () => { + peer!.message(24, "CLOSE-PEER-TURN"); + const request = await nextRequest("CLOSE-PEER-TURN"); + model.replyToolCall(request, { + id: "live-close-tool", + name: "bash", + arguments: { command: "sleep 2" }, + }); + const identity = await waitFor( + "close peer turn identity", + () => peer!.identityFor("CLOSE-PEER-TURN"), + 15_000, + ); + await client.deleteAgent(agentId); + // The peer turn must survive Paseo's close and complete afterwards. + const completionRequest = await waitFor( + "provider request after close", + () => model.pending[0], + 20_000, + ); + model.replyText(completionRequest, "PEER-SURVIVED-CLOSE"); + await waitFor( + "peer turn completed after Paseo close", + () => peer!.turnEnds("completed").some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)), + 20_000, + ); + if (peer!.turnEnds("cancelled").some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity))) { + throw new Error("Paseo close cancelled the peer turn"); + } + peer!.message(25, "AFTER-CLOSE"); + const afterRequest = await nextRequest("AFTER-CLOSE"); + model.replyText(afterRequest, "AFTER-CLOSE-ANSWER"); + await waitFor( + "peer turn after close", + () => peer!.identityFor("AFTER-CLOSE"), + 15_000, + ); + await waitFor( + "post-close peer completion", + () => + peer! + .turnEnds("completed") + .some( + (event) => + turnIdentityKey(event.identity) === + turnIdentityKey(peer!.identityFor("AFTER-CLOSE")), + ), + 15_000, + ); + const reopened = await client.importAgent({ + providerId: "jcode", + providerHandleId: `plugin:${JSON.stringify({ version: 1, data: { sessionId } })}`, + cwd: workspace.cwd, + }); + agentId = reopened.id; + return { + deleted: true, + peerTurnSurvivedClose: true, + peerTurnIdentity: identity, + sessionAliveAfterClose: true, + reopenedAgentId: reopened.id, + }; + }); + + // --- s10: explicit skill selection and null clear -------------------- + await runScenario(evidence, "s10-explicit-skill-select-clear", async () => { + const markerMessages = (body: Record) => + (Array.isArray(body.messages) ? (body.messages as Array>) : []).filter( + (message) => JSON.stringify(message).includes(SKILL_MARKER), + ).length; + + // s08 selected the skill textually, so start from a known cleared state. + const reset = send("RESET-SKILL", { skill: null }); + const resetRequest = await nextRequest("RESET-SKILL"); + const baseline = markerMessages(resetRequest.body); + model.replyText(resetRequest, "RESET-SKILL-ANSWER"); + await reset; + await waitIdle(); + + const selected = send("EXPLICIT-SKILL", { skill: "acceptance-skill" }); + const selectedRequest = await nextRequest("EXPLICIT-SKILL"); + const afterSelect = markerMessages(selectedRequest.body); + if (afterSelect !== baseline + 1) { + throw new Error( + `explicit skill selection did not append a reminder (${baseline} -> ${afterSelect})`, + ); + } + model.replyText(selectedRequest, "EXPLICIT-SKILL-ANSWER"); + await selected; + await waitIdle(); + + const sticky = send("EXPLICIT-STICKY"); + const stickyRequest = await nextRequest("EXPLICIT-STICKY"); + const afterSticky = markerMessages(stickyRequest.body); + model.replyText(stickyRequest, "EXPLICIT-STICKY-ANSWER"); + await sticky; + await waitIdle(); + if (afterSticky !== afterSelect) { + throw new Error(`sticky explicit selection appended ${afterSticky - afterSelect} extra reminders`); + } + + const cleared = send("EXPLICIT-CLEARED", { skill: null }); + const clearedRequest = await nextRequest("EXPLICIT-CLEARED"); + const afterClear = markerMessages(clearedRequest.body); + const clearedText = lastUserText(clearedRequest.body); + model.replyText(clearedRequest, "EXPLICIT-CLEARED-ANSWER"); + await cleared; + await waitIdle(); + if (afterClear !== afterSelect) { + throw new Error(`clear prompt appended ${afterClear - afterSelect} extra reminders`); + } + if (!clearedText.endsWith("EXPLICIT-CLEARED")) { + throw new Error(`clear prompt text was modified: ${safeJson(clearedText)}`); + } + + // Re-selecting after the clear must append a new reminder. If the clear + // had not changed the attachment's sticky state, this would be a no-op. + const reselect = send("EXPLICIT-RESELECT", { skill: "acceptance-skill" }); + const reselectRequest = await nextRequest("EXPLICIT-RESELECT"); + const afterReselect = markerMessages(reselectRequest.body); + model.replyText(reselectRequest, "EXPLICIT-RESELECT-ANSWER"); + await reselect; + await waitIdle(); + if (afterReselect !== afterSelect + 1) { + throw new Error( + `re-selection produced ${afterReselect} reminders; the connector clear did not clear the skill`, + ); + } + return { + reminders: { baseline, afterSelect, afterSticky, afterClear, afterReselect }, + clearedText, + explicitSelectAndClear: true, + }; + }); + + // --- s12: observer detach stops updates, not peer work ---------------- + // A peer-originated turn is already running when Paseo detaches. The view + // never owned that turn, so detach must close the attachment without + // cancelling the work, stop delivering updates to the closed record, and + // leave the native session reloadable. + await runScenario(evidence, "s12-observer-detach", async () => { + const OBSERVED = "OBSERVED-TURN"; + const ANSWER = "OBSERVER-SURVIVED-DETACH"; + const handle = (await agent()).persistence ?? null; + peer!.message(27, OBSERVED); + const request = await nextRequest(OBSERVED); + const identity = await waitFor( + "observed peer turn identity", + () => peer!.identityFor(OBSERVED), + 15_000, + ); + // Hold the turn in a tool while the attachment is live, then detach. + model.replyToolCall(request, { + id: "live-observer-tool", + name: "bash", + arguments: { command: "sleep 3" }, + }); + const observed = await waitFor( + "Paseo observes the peer turn", + async () => { + const snapshot = await agent(); + const turnId = String(snapshot.activeTurn?.turnId ?? ""); + return snapshot.status === "running" && turnId.startsWith("acp-vendor:") + ? snapshot + : null; + }, + 15_000, + ); + + // Watch the push channel, not a post-hoc timeline read: fetching a closed + // record can hydrate history from the provider again. + const pushed: any[] = []; + const stopPushes = client.subscribeRawMessages((message: any) => pushed.push(message)); + await client.detachProviderSession(agentId); + const pushBaseline = pushed.length; + const closed = await agent().catch(() => null); + if (closed && closed.status !== "closed") { + throw new Error(`Paseo kept the observed agent live: ${safeJson(closed.status)}`); + } + const closedTimeline = await timeline().catch(() => null); + + // The next model call happens after the held tool, i.e. after detach. + const completionRequest = await waitFor( + "provider request after observer detach", + () => model.pending[0], + 20_000, + ); + model.replyText(completionRequest, ANSWER); + await waitFor( + "peer turn completed after observer detach", + () => + peer! + .turnEnds("completed") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)), + 20_000, + ); + if ( + peer! + .turnEnds("cancelled") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)) + ) { + throw new Error("observer detach cancelled the peer turn"); + } + + const afterTimeline = await timeline().catch(() => null); + stopPushes(); + const postDetachPushes = pushed.slice(pushBaseline); + const leaked = postDetachPushes.filter((message) => { + const encoded = JSON.stringify(message); + return ( + message.type === "agent_stream" && + encoded.includes(agentId) && + encoded.includes(ANSWER) + ); + }); + if (leaked.length > 0) { + throw new Error("the detached attachment received post-detach updates"); + } + const closedReadSurfacedAnswer = (afterTimeline ?? []).some((entry) => + JSON.stringify(entry).includes(ANSWER), + ); + + // Reload the same native session through a fresh agent record. + const resumed = await client.resumeAgent(handle, { cwd: workspace.cwd }); + agentId = resumed.id; + await waitFor( + "resumed agent replays the peer turn and answer", + async () => { + const entries = await timeline(); + const text = JSON.stringify(userTexts(entries)); + return text.includes(OBSERVED) && JSON.stringify(entries).includes(ANSWER); + }, + 20_000, + ); + const reloaded = send("AFTER-RELOAD"); + const reloadRequest = await nextRequest("AFTER-RELOAD"); + model.replyText(reloadRequest, "AFTER-RELOAD-ANSWER"); + await reloaded; + await waitIdle(); + + return { + observedTurnIdentity: identity, + observedLifecycle: observed.status, + observedActiveTurnId: observed.activeTurn?.turnId ?? null, + detachedStatus: closed?.status ?? null, + closedTimelineLength: closedTimeline?.length ?? null, + postDetachAgentStreams: postDetachPushes.filter( + (message) => message.type === "agent_stream", + ).length, + // A fresh timeline read for the closed record can load provider + // history; this is a read, not a delivered update. + closedReadSurfacedAnswer, + readTimelineLength: afterTimeline?.length ?? null, + detachedAttachmentLeakedAnswer: leaked.length, + peerCompletedAfterDetach: true, + peerCancelledAfterDetach: false, + resumedAgentId: resumed.id, + reloadReplayedPeerWork: true, + promptAfterReload: "AFTER-RELOAD-ANSWER", + }; + }); + + // --- s11: explicit detach preserves inserted work -------------------- + await runScenario(evidence, "s11-detach-preserves-inserted-work", async () => { + const prompt = send("DETACH-INSERTED"); + const request = await nextRequest("DETACH-INSERTED"); + const identity = await waitFor( + "detach turn identity", + () => peer!.identityFor("DETACH-INSERTED"), + 15_000, + ); + await client.detachProviderSession(agentId); + // The inserted turn belongs to the shared daemon session and must finish. + model.replyText(request, "DETACH-ANSWER"); + await waitFor( + "inserted work completed after detach", + () => + peer! + .turnEnds("completed") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)), + 20_000, + ); + if ( + peer! + .turnEnds("cancelled") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(identity)) + ) { + throw new Error("connector detach cancelled inserted work"); + } + await prompt.catch(() => undefined); + const closedSnapshot = await agent().catch(() => null); + if (closedSnapshot && closedSnapshot.status !== "closed") { + throw new Error(`Paseo kept the detached agent live: ${safeJson(closedSnapshot.status)}`); + } + + // Paseo keeps the closed record, so a second import of the same native + // session is rejected by design. Prove survival the way the shared + // session is actually used: another attached client keeps working. + peer!.message(26, "AFTER-DETACH-PEER"); + const peerRequest = await nextRequest("AFTER-DETACH-PEER"); + model.replyText(peerRequest, "AFTER-DETACH-PEER-ANSWER"); + const afterIdentity = await waitFor( + "peer turn after detach", + () => peer!.identityFor("AFTER-DETACH-PEER"), + 15_000, + ); + await waitFor( + "peer completion after detach", + () => + peer! + .turnEnds("completed") + .some((event) => turnIdentityKey(event.identity) === turnIdentityKey(afterIdentity)), + 15_000, + ); + const replayed = userTexts(await timeline()); + return { + detachedTurnIdentity: identity, + insertedWorkCompleted: true, + agentClosed: true, + replayedUserMessages: replayed, + peerContinuesAfterDetach: true, + }; + }); + } catch (error) { + evidence.notes.push(`setup or teardown failure: ${error instanceof Error ? error.stack ?? error.message : String(error)}`); + throw error; + } finally { + try { + await client?.close?.(); + } catch { + // ignore + } + try { + await paseo?.close?.(); + } catch { + // ignore + } + peer?.destroy(); + try { + await daemon?.stop(); + } catch { + // ignore + } + try { + await model.close(); + } catch { + // ignore + } + for (const restore of restoreEnv.reverse()) restore(); + const evidencePath = + process.env.JCODE_ACCEPTANCE_OUT ?? + path.join(process.env.JCODE_SCRATCH_DIR ?? "/tmp", "jcode-live-acceptance.evidence.json"); + evidence.notes.push(`daemon log tail: ${(daemon?.logs.join("").slice(-2000) ?? "").replace(/\s+/g, " ")}`); + evidence.notes.push(`isolated workspace: ${workspace.root}`); + writeEvidence(evidencePath, evidence); + console.log(`evidence: ${evidencePath}`); + console.log(`scenarios: ${summaryLine(evidence)}`); + if (process.env.JCODE_ACCEPTANCE_KEEP === "1") { + console.log(`kept isolated workspace: ${workspace.root}`); + } else { + await workspace.cleanup().catch(() => undefined); + } + const failed = Object.values(evidence.scenarios).some((scenario) => scenario.status === "fail"); + if (failed) process.exitCode = 1; + } +} + +main().catch((error) => { + console.error(error instanceof Error ? error.stack ?? error.message : error); + process.exitCode = 1; +}); diff --git a/scripts/live-acceptance/jcode-live-common.ts b/scripts/live-acceptance/jcode-live-common.ts new file mode 100644 index 00000000000..c065b748e95 --- /dev/null +++ b/scripts/live-acceptance/jcode-live-common.ts @@ -0,0 +1,642 @@ +// Shared plumbing for the live Paseo/jcode v1 acceptance harness. +// +// Every ACP or daemon byte in these harnesses is produced by the real jcode +// binary. Only the model backend is scripted: a local OpenAI-compatible HTTP +// server whose responses the harness releases deliberately, so turn order is +// deterministic and no provider tokens are spent. +import { spawn, spawnSync, type ChildProcess } from "node:child_process"; +import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http"; +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { connect, type Socket } from "node:net"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { createInterface } from "node:readline"; + +export const SKILL_MARKER = "LIVE_ACCEPTANCE_SKILL_MARKER_7c1e"; +export const DEFAULT_MODEL = "gpt-5.2"; +export const ALT_MODEL = "gpt-5.2-mini"; + +export const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +export async function waitFor( + label: string, + probe: () => T | undefined | null | false | Promise, + timeoutMs = 20_000, + intervalMs = 50, +): Promise { + const deadline = Date.now() + timeoutMs; + let last: unknown; + while (Date.now() < deadline) { + try { + const value = await probe(); + if (value) return value as T; + last = value; + } catch (error) { + last = error; + } + await sleep(intervalMs); + } + throw new Error( + `timed out after ${timeoutMs}ms waiting for ${label}; last=${safeJson(last).slice(0, 800)}`, + ); +} + +export function safeJson(value: unknown): string { + if (value === undefined) return "undefined"; + try { + return JSON.stringify(value) ?? String(value); + } catch { + return String(value); + } +} + +/// The daemon wire carries turn identity as `{runtime_generation, turn_id}` +/// while the ACP adapter renders the same identity as `":"`. +/// Compare the two by a single normalized key, never by raw shape. +export function turnIdentityKey(identity: unknown): string { + if (typeof identity === "string") return identity; + if (identity && typeof identity === "object") { + const value = identity as { runtime_generation?: unknown; turn_id?: unknown }; + if (value.runtime_generation !== undefined && value.turn_id !== undefined) { + return `${String(value.runtime_generation)}:${String(value.turn_id)}`; + } + } + return safeJson(identity); +} + +export function messageText(content: unknown): string { + if (typeof content === "string") return content; + if (Array.isArray(content)) { + return content + .map((part) => + typeof part === "string" + ? part + : typeof part === "object" && part + ? String((part as Record).text ?? "") + : "", + ) + .join(""); + } + return ""; +} + +export function bodyText(body: unknown): string { + const messages = (body as { messages?: Array<{ content?: unknown }> } | undefined)?.messages; + return Array.isArray(messages) ? messages.map((m) => messageText(m.content)).join("\n") : ""; +} + +export function lastUserText(body: unknown): string { + const messages = (body as { messages?: Array<{ role?: string; content?: unknown }> } | undefined) + ?.messages; + if (!Array.isArray(messages)) return ""; + for (let index = messages.length - 1; index >= 0; index -= 1) { + if (messages[index].role === "user") return messageText(messages[index].content); + } + return ""; +} + +/** Finds a nested key/value that looks like an effort setting. Returns [key, value]. */ +export function findEffort(value: unknown, depth = 0): [string, string] | null { + if (depth > 6 || value === null || typeof value !== "object") return null; + for (const [key, child] of Object.entries(value as Record)) { + if ( + typeof child === "string" && + /effort|reasoning|verbosity/i.test(key) && + child.length > 0 + ) { + return [key, child]; + } + } + for (const child of Object.values(value as Record)) { + const found = findEffort(child, depth + 1); + if (found) return found; + } + return null; +} + +export interface ModelRequest { + body: Record; + response: ServerResponse; + /** True once a reply/abort has been written. */ + settled: boolean; +} + +export interface MockUsage { + promptTokens: number; + completionTokens: number; + cachedTokens?: number; +} + +export interface MockModel { + baseUrl: string; + requests: ModelRequest[]; + pending: ModelRequest[]; + close(): Promise; + takeContaining(text: string, timeoutMs?: number): Promise; + replyText(request: ModelRequest, text: string, usage?: MockUsage): void; + replyToolCall(request: ModelRequest, options: { + id: string; + name: string; + arguments: Record; + }): void; + abort(request: ModelRequest): void; +} + +function sse(res: ServerResponse, events: Array | "[DONE]">) { + res.writeHead(200, { + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + Connection: "keep-alive", + }); + for (const event of events) { + res.write(`data: ${typeof event === "string" ? event : JSON.stringify(event)}\n\n`); + } + res.end(); +} + +function chunk( + model: string, + delta: Record, + finishReason: string | null, + usage?: MockUsage, +): Record { + const frame: Record = { + id: "live-acceptance", + object: "chat.completion.chunk", + model, + choices: [{ index: 0, delta, finish_reason: finishReason }], + }; + if (usage) { + frame.usage = { + prompt_tokens: usage.promptTokens, + completion_tokens: usage.completionTokens, + total_tokens: usage.promptTokens + usage.completionTokens, + ...(usage.cachedTokens === undefined + ? {} + : { prompt_tokens_details: { cached_tokens: usage.cachedTokens } }), + }; + } + return frame; +} + +export async function startMockModel( + models: string[] = [DEFAULT_MODEL, ALT_MODEL], +): Promise { + const requests: ModelRequest[] = []; + const pending: ModelRequest[] = []; + + const handler = async (request: IncomingMessage, response: ServerResponse) => { + let raw = ""; + for await (const part of request) raw += part; + const url = request.url ?? ""; + if (url.includes("/models")) { + response.setHeader("Content-Type", "application/json"); + response.end( + JSON.stringify({ object: "list", data: models.map((id) => ({ id, object: "model" })) }), + ); + return; + } + let body: Record = {}; + try { + body = raw ? (JSON.parse(raw) as Record) : {}; + } catch { + body = {}; + } + const model = typeof body.model === "string" ? body.model : DEFAULT_MODEL; + const tools = Array.isArray(body.tools) ? body.tools : []; + // jcode also asks for background session summaries. Those tool-free + // completions must not consume a foreground turn gate. + if (tools.length === 0) { + if (body.stream === false) { + response.setHeader("Content-Type", "application/json"); + response.end( + JSON.stringify({ + id: "auxiliary", + object: "chat.completion", + model, + choices: [ + { index: 0, message: { role: "assistant", content: "E2E session" }, finish_reason: "stop" }, + ], + }), + ); + } else { + sse(response, [ + chunk(model, { role: "assistant", content: "E2E session" }, null), + chunk(model, {}, "stop"), + "[DONE]", + ]); + } + return; + } + const entry: ModelRequest = { body, response, settled: false }; + requests.push(entry); + pending.push(entry); + }; + + const server: Server = createServer((request, response) => { + void handler(request, response).catch((error) => { + try { + response.destroy(error instanceof Error ? error : undefined); + } catch { + // already gone + } + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("mock model has no TCP port"); + + const take = (entry: ModelRequest) => { + entry.settled = true; + const index = pending.indexOf(entry); + if (index >= 0) pending.splice(index, 1); + }; + + return { + baseUrl: `http://127.0.0.1:${address.port}/v1`, + requests, + pending, + async close() { + for (const entry of pending.splice(0)) { + try { + entry.response.destroy(); + } catch { + // already gone + } + } + server.closeAllConnections(); + await new Promise((resolve) => server.close(() => resolve())); + }, + async takeContaining(text, timeoutMs = 20_000) { + return waitFor( + `model request containing ${JSON.stringify(text)}`, + () => pending.find((entry) => bodyText(entry.body).includes(text)), + timeoutMs, + ); + }, + replyText(entry, text, usage) { + if (entry.settled) throw new Error("model request already settled"); + take(entry); + const model = typeof entry.body.model === "string" ? entry.body.model : DEFAULT_MODEL; + sse(entry.response, [ + chunk(model, { role: "assistant", content: text }, null), + chunk(model, {}, "stop", usage), + "[DONE]", + ]); + }, + replyToolCall(entry, options) { + if (entry.settled) throw new Error("model request already settled"); + take(entry); + const model = typeof entry.body.model === "string" ? entry.body.model : DEFAULT_MODEL; + sse(entry.response, [ + chunk( + model, + { + role: "assistant", + tool_calls: [ + { + index: 0, + id: options.id, + type: "function", + function: { name: options.name, arguments: JSON.stringify(options.arguments) }, + }, + ], + }, + null, + ), + chunk(model, {}, "tool_calls"), + "[DONE]", + ]); + }, + abort(entry) { + if (entry.settled) return; + take(entry); + try { + entry.response.destroy(); + } catch { + // already gone + } + }, + }; +} + +export interface LiveWorkspace { + root: string; + cwd: string; + home: string; + runtime: string; + socketPath: string; + binDir: string; + cleanup(): Promise; +} + +export async function createWorkspace(prefix = "jcode-live-"): Promise { + const root = await mkdtemp(path.join(tmpdir(), prefix)); + const cwd = path.join(root, "work"); + const home = path.join(root, "home"); + const runtime = path.join(root, "runtime"); + const binDir = path.join(root, "bin"); + await Promise.all([mkdir(cwd, { recursive: true }), mkdir(home, { recursive: true }), mkdir(runtime, { recursive: true }), mkdir(binDir, { recursive: true })]); + const skillDir = path.join(cwd, ".jcode", "skills", "acceptance-skill"); + await mkdir(skillDir, { recursive: true }); + await writeFile( + path.join(skillDir, "SKILL.md"), + `---\nname: acceptance-skill\ndescription: Live acceptance probe skill.\n---\n\nWhen this skill is active, include the marker ${SKILL_MARKER} in your answer.\n`, + "utf8", + ); + return { + root, + cwd, + home, + runtime, + socketPath: path.join(root, "jcode.sock"), + binDir, + async cleanup() { + await rm(root, { recursive: true, force: true }); + }, + }; +} + +export interface JcodeDaemon { + child: ChildProcess; + socketPath: string; + logs: string[]; + env: NodeJS.ProcessEnv; + stop(): Promise; +} + +export function daemonEnv(workspace: LiveWorkspace, modelBaseUrl: string, model = DEFAULT_MODEL): NodeJS.ProcessEnv { + return { + ...process.env, + HOME: process.env.HOME, + JCODE_NO_TELEMETRY: "1", + JCODE_HOME: workspace.home, + JCODE_RUNTIME_DIR: workspace.runtime, + JCODE_SOCKET: workspace.socketPath, + JCODE_OPENAI_COMPAT_API_BASE: modelBaseUrl, + JCODE_OPENAI_COMPAT_DEFAULT_MODEL: model, + OPENAI_COMPAT_API_KEY: "isolated-live-acceptance-key", + PATH: `${workspace.binDir}${path.delimiter}${process.env.PATH ?? ""}`, + }; +} + +export async function startJcodeDaemon(options: { + binary: string; + workspace: LiveWorkspace; + modelBaseUrl: string; + model?: string; +}): Promise { + const env = daemonEnv(options.workspace, options.modelBaseUrl, options.model); + const logs: string[] = []; + const child = spawn( + options.binary, + [ + "--socket", + options.workspace.socketPath, + "--provider", + "openai-compatible", + "--model", + options.model ?? DEFAULT_MODEL, + "serve", + ], + { cwd: options.workspace.cwd, env, stdio: ["ignore", "pipe", "pipe"] }, + ); + child.stdout?.on("data", (data) => logs.push(String(data))); + child.stderr?.on("data", (data) => logs.push(String(data))); + await waitFor( + "jcode daemon socket", + () => { + if (child.exitCode !== null) { + throw new Error(`jcode daemon exited ${child.exitCode}\n${logs.join("").slice(-4000)}`); + } + return existsSync(options.workspace.socketPath); + }, + 30_000, + ); + return { + child, + socketPath: options.workspace.socketPath, + logs, + env, + async stop() { + if (child.exitCode !== null || child.signalCode !== null) return; + const closed = new Promise((resolve) => child.once("close", () => resolve())); + child.kill("SIGTERM"); + const settled = await Promise.race([ + closed.then(() => true), + sleep(3_000).then(() => false), + ]); + if (!settled) { + child.kill("SIGKILL"); + await Promise.race([closed, sleep(2_000)]); + } + }, + }; +} + +export interface PeerEvent { + [key: string]: any; +} + +export interface LivePeer { + socket: Socket; + events: PeerEvent[]; + send(message: Record): void; + subscribe(workingDir: string, clientInstanceId: string, targetSessionId?: string): Promise; + message(id: number, content: string): void; + softInterrupt(id: number, content: string, urgent?: boolean): void; + cancel(id: number): void; + /// Protocol request that mutates shared config and fans out to other attachments. + setReasoningEffort(id: number, effort: string): void; + setModel(id: number, model: string): void; + destroy(): void; + turnStarts(): PeerEvent[]; + turnEnds(end: "completed" | "cancelled" | "failed"): PeerEvent[]; + identityFor(content: string): string | undefined; +} + +export async function connectPeer(socketPath: string): Promise { + const socket = connect(socketPath); + await new Promise((resolve, reject) => { + socket.once("connect", resolve); + socket.once("error", reject); + }); + const events: PeerEvent[] = []; + const lines = createInterface({ input: socket }); + lines.on("line", (line) => { + try { + events.push(JSON.parse(line) as PeerEvent); + } catch { + events.push({ type: "unparsed", line }); + } + }); + const peer: LivePeer = { + socket, + events, + send(message) { + socket.write(`${JSON.stringify(message)}\n`); + }, + async subscribe(workingDir, clientInstanceId, targetSessionId) { + peer.send({ + type: "subscribe", + id: 1, + supports_turn_identity: true, + working_dir: workingDir, + client_instance_id: clientInstanceId, + ...(targetSessionId ? { target_session_id: targetSessionId } : {}), + }); + return waitFor( + "peer session id", + () => events.find((event) => event.type === "session"), + 15_000, + ); + }, + message(id, content) { + peer.send({ type: "message", id, content }); + }, + softInterrupt(id, content, urgent = true) { + peer.send({ type: "soft_interrupt", id, content, urgent }); + }, + cancel(id) { + peer.send({ type: "cancel", id }); + }, + setReasoningEffort(id, effort) { + peer.send({ type: "set_reasoning_effort", id, effort }); + }, + setModel(id, model) { + peer.send({ type: "set_model", id, model }); + }, + destroy() { + lines.close(); + socket.destroy(); + }, + turnStarts() { + return events.filter((event) => event.type === "turn_started"); + }, + turnEnds(end) { + return events.filter((event) => event.type === "turn_event" && event.end === end); + }, + identityFor(content) { + return events.find( + (event) => event.type === "turn_started" && event.user_content === content, + )?.identity; + }, + }; + return peer; +} + +export function binaryProvenance(binary: string) { + const version = spawnSync(binary, ["--version"], { encoding: "utf8" }); + const digest = spawnSync("sha256sum", [binary], { encoding: "utf8" }); + return { + binary, + resolved: existsSync(binary) ? binary : null, + version: (version.stdout ?? "").trim() || (version.stderr ?? "").trim(), + sha256: (digest.stdout ?? "").split(/\s+/)[0] ?? null, + stat: version.status, + }; +} + +export function linkBinary(binary: string, binDir: string, name = "jcode"): string { + const target = path.join(binDir, name); + spawnSync("ln", ["-sf", binary, target]); + return target; +} + +export interface Evidence { + startedAt: string; + provenance: Record; + environment: Record; + scenarios: Record; + notes: string[]; +} + +export function newEvidence(): Evidence { + return { + startedAt: new Date().toISOString(), + provenance: {}, + environment: {}, + scenarios: {}, + notes: [], + }; +} + +export function recordScenario( + evidence: Evidence, + name: string, + status: "pass" | "fail" | "skip", + payload?: { detail?: unknown; error?: unknown }, +) { + evidence.scenarios[name] = { + status, + ...(payload?.detail !== undefined ? { detail: payload.detail } : {}), + ...(payload?.error !== undefined ? { error: String(payload.error) } : {}), + }; +} + +export async function runScenario( + evidence: Evidence, + name: string, + body: () => Promise, +): Promise { + const selected = (process.env.JCODE_ACCEPTANCE_SCENARIOS ?? "") + .split(",") + .map((entry) => entry.trim()) + .filter(Boolean); + if (selected.length > 0 && !selected.some((entry) => name === entry || name.startsWith(entry))) { + recordScenario(evidence, name, "skip"); + persistIncremental(evidence); + return true; + } + try { + const detail = await body(); + recordScenario(evidence, name, "pass", { detail }); + persistIncremental(evidence); + return true; + } catch (error) { + const message = + error instanceof Error + ? error.stack ?? error.message + : `${safeJson(error)}${(error as { code?: number })?.code !== undefined ? ` code=${(error as { code?: number }).code}` : ""}`; + recordScenario(evidence, name, "fail", { error: message }); + persistIncremental(evidence); + return false; + } +} + +/// Keep per-scenario evidence durable even when a later floating promise or +/// shutdown path kills the process before the final write. +function persistIncremental(evidence: Evidence) { + const file = process.env.JCODE_ACCEPTANCE_OUT; + if (!file) return; + try { + writeEvidence(file, evidence); + } catch { + // Evidence must never change scenario behavior. + } +} + +export function writeEvidence(file: string, evidence: Evidence) { + mkdirSync(path.dirname(file), { recursive: true }); + writeFileSync(file, `${JSON.stringify(evidence, null, 2)}\n`, "utf8"); +} + +export function summaryLine(evidence: Evidence): string { + const entries = Object.entries(evidence.scenarios); + const passed = entries.filter(([, value]) => value.status === "pass").length; + const failed = entries.filter(([, value]) => value.status === "fail").length; + const skipped = entries.filter(([, value]) => value.status === "skip").length; + return `${passed} passed, ${failed} failed, ${skipped} skipped`; +} + +export function requireBinary(): string { + const binary = process.env.JCODE_E2E_BINARY; + if (!binary || !path.isAbsolute(binary)) { + throw new Error( + "Set JCODE_E2E_BINARY to an absolute, current jcode executable (real build, not target/debug from Nov 9)", + ); + } + if (!existsSync(binary)) throw new Error(`JCODE_E2E_BINARY does not exist: ${binary}`); + return binary; +} diff --git a/scripts/live-acceptance/run-live-acceptance.sh b/scripts/live-acceptance/run-live-acceptance.sh new file mode 100755 index 00000000000..6a1e0f7da91 --- /dev/null +++ b/scripts/live-acceptance/run-live-acceptance.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# One entry point for live v1 acceptance of jcode through the real Paseo ACP +# connector plus a direct live ACP companion probe. +# +# JCODE_E2E_BINARY=/abs/path/to/jcode scripts/live-acceptance/run-live-acceptance.sh +# +# Optional: +# PASEO_CONSUMER_ROOT consumer checkout (default: sibling checkout +# ../paseo-v1-acceptance next to this repository) +# JCODE_ACCEPTANCE_OUT_DIR evidence directory (default: +# $JCODE_SCRATCH_DIR/live-acceptance/... when set, +# otherwise /target/live-acceptance/...) +# JCODE_EXPECTED_PROVENANCE substring that must appear in `jcode --version` +# JCODE_ACCEPTANCE_ONLY connector|probe +# +# The harness creates its own daemon socket, JCODE_HOME, JCODE_RUNTIME_DIR and +# Paseo home. It never touches the shared daemon or user config. +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +consumer_root="${PASEO_CONSUMER_ROOT:-$(cd "$repo_root/.." && pwd)/paseo-v1-acceptance}" +binary="${JCODE_E2E_BINARY:-}" +only="${JCODE_ACCEPTANCE_ONLY:-all}" + +case "$only" in + all | connector | probe) ;; + *) printf 'live-acceptance: JCODE_ACCEPTANCE_ONLY must be all, connector or probe (got %s)\n' "$only" >&2; exit 2 ;; +esac + +fail() { + printf 'live-acceptance: %s\n' "$*" >&2 + exit 1 +} + +[[ -n "$binary" ]] || fail "set JCODE_E2E_BINARY to an absolute current jcode executable" +[[ "$binary" = /* ]] || fail "JCODE_E2E_BINARY must be an absolute path: $binary" +[[ -x "$binary" ]] || fail "JCODE_E2E_BINARY is not executable: $binary" +[[ -d "$consumer_root" ]] || fail "consumer checkout not found: $consumer_root" +[[ -x "$consumer_root/node_modules/.bin/tsx" ]] || fail "consumer checkout has no tsx: $consumer_root" +[[ -f "$consumer_root/plugins/jcode-provider/index.server.ts" ]] || fail "consumer jcode provider plugin missing" + +stamp="$(date -u +%Y%m%dT%H%M%SZ)" +if [[ -n "${JCODE_ACCEPTANCE_OUT_DIR:-}" ]]; then + out_dir="$JCODE_ACCEPTANCE_OUT_DIR" +elif [[ -n "${JCODE_SCRATCH_DIR:-}" ]]; then + out_dir="$JCODE_SCRATCH_DIR/live-acceptance/live-run-$stamp" +else + out_dir="$repo_root/target/live-acceptance/live-run-$stamp" +fi +mkdir -p "$out_dir" + +version="$("$binary" --version 2>&1 | head -1)" +if command -v sha256sum >/dev/null 2>&1; then + digest="$(sha256sum "$binary" | awk '{print $1}')" +else + digest="$(shasum -a 256 "$binary" | awk '{print $1}')" +fi +consumer_head="$(git -C "$consumer_root" rev-parse HEAD)" +consumer_branch="$(git -C "$consumer_root" branch --show-current)" +consumer_dirty="$(git -C "$consumer_root" status --porcelain | wc -l | tr -d ' ')" + +if [[ -n "${JCODE_EXPECTED_PROVENANCE:-}" && "$version" != *"$JCODE_EXPECTED_PROVENANCE"* ]]; then + fail "binary provenance mismatch: '$version' does not contain '$JCODE_EXPECTED_PROVENANCE'" +fi +if [[ "$version" == *"900265ac7"* ]]; then + printf 'live-acceptance: WARNING: binary is the stale 2026-11-09 build (900265ac7); evidence is not current\n' >&2 +fi + +{ + printf 'repo_root=%s\n' "$repo_root" + printf 'binary=%s\n' "$binary" + printf 'binary_version=%s\n' "$version" + printf 'binary_sha256=%s\n' "$digest" + printf 'consumer_root=%s\n' "$consumer_root" + printf 'consumer_head=%s\n' "$consumer_head" + printf 'consumer_branch=%s\n' "$consumer_branch" + printf 'consumer_dirty_files=%s\n' "$consumer_dirty" + printf 'out_dir=%s\n' "$out_dir" + printf 'node=%s\n' "$(node --version)" + printf 'node_conditions=source\n' + printf 'started_utc=%s\n' "$stamp" +} | tee "$out_dir/provenance.txt" + +run_one() { + local name="$1" + local harness="$2" + printf '\n== %s ==\n' "$name" + set +e + ( + cd "$consumer_root" + PASEO_CONSUMER_ROOT="$consumer_root" \ + JCODE_ACCEPTANCE_OUT="$out_dir/$name.evidence.json" \ + node --conditions=source --import tsx "$harness" + ) 2>&1 | tee "$out_dir/$name.log" + local status="${PIPESTATUS[0]}" + set -e + printf '%s exit=%s\n' "$name" "$status" | tee -a "$out_dir/summary.txt" + return "$status" +} + +status=0 +if [[ "$only" == "all" || "$only" == "connector" ]]; then + run_one connector "$repo_root/scripts/live-acceptance/jcode-live-acceptance.ts" || status=1 +fi +if [[ "$only" == "all" || "$only" == "probe" ]]; then + run_one probe "$repo_root/scripts/live-acceptance/jcode-acp-probe.ts" || status=1 +fi + +printf '\nevidence directory: %s\n' "$out_dir" +printf 'connector evidence: %s\n' "$out_dir/connector.evidence.json" +printf 'probe evidence: %s\n' "$out_dir/probe.evidence.json" +exit "$status" diff --git a/tests/e2e/acp_conformance.rs b/tests/e2e/acp_conformance.rs index 98014d4cb07..6cbb522bf0b 100644 --- a/tests/e2e/acp_conformance.rs +++ b/tests/e2e/acp_conformance.rs @@ -992,3 +992,5 @@ mod tool_wait; mod clause_index; mod retirement; + +mod commands_v1; diff --git a/tests/e2e/acp_conformance/TRACEABILITY.md b/tests/e2e/acp_conformance/TRACEABILITY.md index 4d6400d7a3c..7a0ace7a685 100644 --- a/tests/e2e/acp_conformance/TRACEABILITY.md +++ b/tests/e2e/acp_conformance/TRACEABILITY.md @@ -51,7 +51,7 @@ behavior of the clause, not a browser/client UI promise. | C-DISCOVERY-P08 | acp_conformance_negotiation_legacy_v1; acp_capability_metadata_identifies_each_attachment; unit: initialize_advertises_jcode_capability_meta_in_every_profile | | C-OPERATIONS-P01 | acp_conformance_create_list_load; acp_conformance_list_metadata_and_pagination; acp_conformance_close_owner_aware_v1; acp_conformance_config_commands_v1; UI/pagination cursor handling is consumer-side | | C-OPERATIONS-P02 | acp_conformance_create_list_load; acp_conformance_list_metadata_and_pagination; acp_conformance_close_owner_aware_v1; acp_conformance_config_commands_v1; UI/pagination cursor handling is consumer-side | -| C-SHARED-P01 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; known-fail: acp_conformance_transport_loss_withdraws_waiting_steer | +| C-SHARED-P01 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; acp_conformance_transport_loss_withdraws_waiting_steer | | C-V2CLOSE-P01 | Deferred v2/SDK track; no v1 wire requirement | | C-V2CLOSE-P02 | Deferred v2/SDK track; no v1 wire requirement | | C-V2CLOSE-P03 | Deferred v2/SDK track; no v1 wire requirement | @@ -69,7 +69,7 @@ behavior of the clause, not a browser/client UI promise. | C-CONFIG-P01 | acp_conformance_config_commands_v1; acp_conformance_config_peer_update_contract; acp_conformance_busy_config_errors_are_atomic; acp_conformance_peer_config_serializes_after_active_turn; acp_skill_clear_round_trip; unit: advertised_commands_parse_to_real_dispatch_variants, unknown_escaped_and_path_like_slash_text_is_untouched, a_busy_session_rejects_an_explicit_skill_selection_before_insertion | | C-CONFIG-P02 | acp_conformance_config_commands_v1; acp_conformance_config_peer_update_contract; acp_conformance_busy_config_errors_are_atomic; acp_conformance_peer_config_serializes_after_active_turn; acp_skill_clear_round_trip; unit: advertised_commands_parse_to_real_dispatch_variants, unknown_escaped_and_path_like_slash_text_is_untouched, a_busy_session_rejects_an_explicit_skill_selection_before_insertion | | C-CONFIG-P03 | acp_conformance_config_commands_v1; acp_conformance_config_peer_update_contract; acp_conformance_busy_config_errors_are_atomic; acp_conformance_peer_config_serializes_after_active_turn; acp_skill_clear_round_trip; unit: advertised_commands_parse_to_real_dispatch_variants, unknown_escaped_and_path_like_slash_text_is_untouched, a_busy_session_rejects_an_explicit_skill_selection_before_insertion | -| C-CONFIG-P04 | acp_conformance_config_commands_v1; acp_conformance_config_peer_update_contract; acp_conformance_busy_config_errors_are_atomic; acp_conformance_peer_config_serializes_after_active_turn; acp_skill_clear_round_trip; unit: advertised_commands_parse_to_real_dispatch_variants, unknown_escaped_and_path_like_slash_text_is_untouched, a_busy_session_rejects_an_explicit_skill_selection_before_insertion | +| C-CONFIG-P04 | acp_conformance_compact_command_compacts_real_history; acp_conformance_subagent_command_spawns_live_worker; acp_conformance_config_commands_v1; unit: advertised_commands_parse_to_real_dispatch_variants | | C-CONFIG-P05 | acp_conformance_config_commands_v1; acp_conformance_config_peer_update_contract; acp_conformance_busy_config_errors_are_atomic; acp_conformance_peer_config_serializes_after_active_turn; acp_skill_clear_round_trip; unit: advertised_commands_parse_to_real_dispatch_variants, unknown_escaped_and_path_like_slash_text_is_untouched, a_busy_session_rejects_an_explicit_skill_selection_before_insertion | | C-CONFIG-P06 | Deferred v2 command insertion/echo | | C-CONFIG-P07 | acp_skill_clear_round_trip; unit: skill_selection_reads_the_canonical_metadata_field | @@ -86,10 +86,10 @@ behavior of the clause, not a browser/client UI promise. | C-ERRORS-P03 | acp_conformance_framing_ids_and_errors; acp_ignored_mcp_servers_notice_follows_the_response; acp_conformance_negotiation_legacy_v1; unit: initialize_advertises_jcode_capability_meta_in_every_profile; authentication/approval UI is consumer-side | | C-ERRORS-P04 | acp_conformance_framing_ids_and_errors; acp_ignored_mcp_servers_notice_follows_the_response; acp_conformance_negotiation_legacy_v1; unit: initialize_advertises_jcode_capability_meta_in_every_profile; authentication/approval UI is consumer-side | | C-ERRORS-P05 | acp_conformance_framing_ids_and_errors; acp_ignored_mcp_servers_notice_follows_the_response; acp_conformance_negotiation_legacy_v1; unit: initialize_advertises_jcode_capability_meta_in_every_profile; authentication/approval UI is consumer-side | -| C-ACCEPTANCE-P01 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P01 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P03 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-ACCEPTANCE-P01 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P01 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P03 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | | C-SDK-P01 | Deferred v2/SDK track; no v1 wire requirement | | C-SDK-P02 | Deferred v2/SDK track; no v1 wire requirement | | C-SDK-P03 | Deferred v2/SDK track; no v1 wire requirement | @@ -132,10 +132,10 @@ behavior of the clause, not a browser/client UI promise. | C-OPERATIONS-P01-R8 | acp_conformance_create_list_load; acp_conformance_list_metadata_and_pagination; acp_conformance_close_owner_aware_v1; acp_conformance_config_commands_v1; UI/pagination cursor handling is consumer-side | | C-OPERATIONS-P01-R9 | acp_conformance_create_list_load; acp_conformance_list_metadata_and_pagination; acp_conformance_close_owner_aware_v1; acp_conformance_config_commands_v1; UI/pagination cursor handling is consumer-side | | C-OPERATIONS-P01-R10 | acp_conformance_create_list_load; acp_conformance_list_metadata_and_pagination; acp_conformance_close_owner_aware_v1; acp_conformance_config_commands_v1; UI/pagination cursor handling is consumer-side | -| C-SHARED-P01-R1 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; known-fail: acp_conformance_transport_loss_withdraws_waiting_steer | -| C-SHARED-P01-R2 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; known-fail: acp_conformance_transport_loss_withdraws_waiting_steer | -| C-SHARED-P01-R3 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; known-fail: acp_conformance_transport_loss_withdraws_waiting_steer | -| C-SHARED-P01-R4 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; known-fail: acp_conformance_transport_loss_withdraws_waiting_steer | +| C-SHARED-P01-R1 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; acp_conformance_transport_loss_withdraws_waiting_steer | +| C-SHARED-P01-R2 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; acp_conformance_transport_loss_withdraws_waiting_steer | +| C-SHARED-P01-R3 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; acp_conformance_transport_loss_withdraws_waiting_steer | +| C-SHARED-P01-R4 | acp_conformance_shared_observe_steer; acp_conformance_detach_withdraws_pending_steer; acp_conformance_seeded_three_client_invariants; acp_conformance_transport_loss_preserves_inserted_work; acp_conformance_transport_loss_withdraws_waiting_steer | | C-CORRELATION-P01-I1 | acp_conformance_shared_observe_steer; acp_conformance_config_commands_v1; consumer completion deduplication is consumer-side | | C-CORRELATION-P01-I2 | acp_conformance_shared_observe_steer; acp_conformance_config_commands_v1; consumer completion deduplication is consumer-side | | C-CORRELATION-P01-I3 | acp_conformance_shared_observe_steer; acp_conformance_config_commands_v1; consumer completion deduplication is consumer-side | @@ -149,18 +149,18 @@ behavior of the clause, not a browser/client UI promise. | C-ERRORS-P04-I2 | acp_conformance_framing_ids_and_errors; acp_ignored_mcp_servers_notice_follows_the_response; acp_conformance_negotiation_legacy_v1; unit: initialize_advertises_jcode_capability_meta_in_every_profile; authentication/approval UI is consumer-side | | C-ERRORS-P04-I3 | acp_conformance_framing_ids_and_errors; acp_ignored_mcp_servers_notice_follows_the_response; acp_conformance_negotiation_legacy_v1; unit: initialize_advertises_jcode_capability_meta_in_every_profile; authentication/approval UI is consumer-side | | C-ERRORS-P04-I4 | acp_conformance_framing_ids_and_errors; acp_ignored_mcp_servers_notice_follows_the_response; acp_conformance_negotiation_legacy_v1; unit: initialize_advertises_jcode_capability_meta_in_every_profile; authentication/approval UI is consumer-side | -| C-CORPUS-P01-I1 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P01-I2 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P01-I3 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P01-I4 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I1 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I2 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I3 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I4 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I5 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I6 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I7 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | -| C-CORPUS-P02-I8 | Corpus scenarios 01–05, 07–11 through acp_conformance tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P01-I1 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P01-I2 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P01-I3 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P01-I4 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I1 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I2 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I3 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I4 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I5 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I6 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | +| C-CORPUS-P02-I7 | Corpus recordings 07 and 12; acp_conformance_compact_command_compacts_real_history; acp_conformance_subagent_command_spawns_live_worker; consumer replay is Paseo-side | +| C-CORPUS-P02-I8 | Corpus recordings 01–05, 07–12 through acp_conformance and acp_shared_session tests; v2 scenario 06 deferred; consumer replay is Paseo-side | | C-SDK-P03-R1 | Deferred v2/SDK track; no v1 wire requirement | | C-SDK-P03-R2 | Deferred v2/SDK track; no v1 wire requirement | | C-SDK-P03-R3 | Deferred v2/SDK track; no v1 wire requirement | @@ -205,10 +205,9 @@ The original atlas report was observational and its hypotheses are superseded. - **QA-STOP**, confirmed jcode defect: the 500 ms cancel reset races bash's 750 ms handoff. `acp_conformance_stop_observed_bash_wait` runs a real bash file gate and asserts canceled retirement without further model output. - Ignored on this lane's baseline; fix d46434b52 belongs to the separate lane. - Enabled companion `acp_known_defect_stop_observed_peer` pins the working - provider-stream case. We do not make wrong cancellation behavior a passing - requirement. The ignored regression can be enabled after integrating the fix. + The repair is integrated and this regression is enabled. It passed in + starting-head Fork CI `42738049168`, as did the companion + `acp_known_defect_stop_observed_peer` for the provider-stream case. - **QA-STEER**, consumer defect: jcode emits the reverse steer. The corrected, enabled `acp_known_defect_reverse_steer_is_mirrored` sends an actual daemon SoftInterrupt, observes admission, releases the provider safe point and checks @@ -220,12 +219,11 @@ The original atlas report was observational and its hypotheses are superseded. session with two deliberate equal user prompts, verifies one replay per row, grouping/order, distinct IDs and repeat-load stability. Deduplicating equal text would fail this test and hide legitimate repeated submissions. -- **ACP-LOAD-BRACKET**, newly found jcode defect: repeated mid-turn load opens - nested observation brackets. Ignored regression - `acp_conformance_repeated_midturn_load_has_one_bracket` preserves the failing - schedule. The ordinary generator explicitly detaches before loading an - already-attached view. This restriction is a known coverage gap, not a proposed - contract rule. Fix is assigned to the separate defect lane. +- **ACP-LOAD-BRACKET**, repaired jcode defect: repeated mid-turn load previously + opened nested observation brackets. The retained-attachment repair is integrated, + and `acp_conformance_repeated_midturn_load_has_one_bracket` is enabled and + passed in starting-head Fork CI `42738049168`. The earlier baseline restriction + is not a current contract requirement or a remaining missing implementation. ## Generator and corpus accounting @@ -244,9 +242,11 @@ The original atlas report was observational and its hypotheses are superseded. running; origin/steer response correlation; terminal identity; no unanswered requests; exact origin output before/after steer; targeted cancellation and successor completion; detach preserves the authoritative witness's completion. -- Corpus files 01–05 and 07–11: ten files. 06 is the deferred v2 delivery scenario. - New 09 is active-turn attach, 10 reverse steer, 11 clean imported repeated user - rows. All are live recorded and structurally replayed via the existing helper. +- The corpus contains twelve files: 01–05 and 07–12, with separate form and + mid-turn attach recordings numbered 09. Scenario 06 is deferred v2 delivery. + Recording 12 adds explicit skill selection and null clear through + `acp_skill_clear_round_trip`. Every file is recorded from the real adapter + and structurally replayed via the existing helper. ## Additional retirement probes and precise coverage limits @@ -266,22 +266,24 @@ The original atlas report was observational and its hypotheses are superseded. - `acp_conformance_transport_loss_preserves_inserted_work`: abrupt origin ACP child loss closes the daemon socket; a surviving witness receives normal completion of the same inserted turn. -- **ACP-LOST-STEER**, new known defect: +- **ACP-LOST-STEER**, repaired jcode defect: `acp_conformance_transport_loss_withdraws_waiting_steer` confirms queue admission before killing the steering ACP child, while the provider remains gated before - its insertion safe point. After allowing cleanup and releasing the gate, the - baseline makes a second provider call for the disconnected attachment's input. - This execution assertion (2 calls instead of 1), not the cleanup timeout, - demonstrates the failed pending-input withdrawal. The assertion is ignored on - this baseline and assigned to the separate fix lane. Already-inserted steers - are intentionally outside this withdrawal expectation. + its insertion safe point. The baseline made a second provider call for the + disconnected attachment's input. Connection-owned cleanup is now integrated, + the regression is enabled, and it passed in starting-head Fork CI + `42738049168`. Already-inserted steers are intentionally outside this + withdrawal expectation. Indexed does not mean independently tested end to end. Remaining limits: -- C-CONFIG-P04: compact/subagent discovery and parsing are pinned; full daemon - compaction scheduling and spawned-worker lifetime are not exercised by this - ACP suite. Those mechanisms are outside the new boundary harness; no claim - that launch completion equals worker completion is tested here. +- C-CONFIG-P04: `commands_v1.rs` now exercises real daemon compaction and + worker lifetime. The compaction test verifies the summarization input and + that the next provider call receives the applied summary instead of older + history, retaining recent turns. The subagent test distinguishes the launch + tool receipt from a child session's live delta and completed turn. Native + provider compaction, visible-window spawn and worker failure/cancellation + remain outside these two focused command tests. - C-CONTENT-P04 and the client-side halves of list/config/capability clauses: consumer rendering, own-echo suppression, diagnostic deduplication, reconnect UI and capability gating belong to the Paseo consumer tests. @@ -290,7 +292,7 @@ Indexed does not mean independently tested end to end. Remaining limits: observable-but-forbidden-Stop daemon capability combination is invented. - The generator covers short, valid bounded schedules, not all arbitrary unbounded histories. Abrupt transport loss is tested separately; repeated load - without detach is retained as a known-fail schedule. Provider faults and many + without detach is covered by an enabled regression. Provider faults and many queued steers remain in existing focused tests rather than randomized here. - Source/version governance, deferred v2/SDK and consumer-only clauses have explicit non-wire dispositions in the inventory. Wording proposals above diff --git a/tests/e2e/acp_conformance/commands_v1.rs b/tests/e2e/acp_conformance/commands_v1.rs new file mode 100644 index 00000000000..60652be7417 --- /dev/null +++ b/tests/e2e/acp_conformance/commands_v1.rs @@ -0,0 +1,435 @@ +//! End-to-end command scheduling: `/compact` and `/subagent` must do real work +//! through the daemon, not only parse in the adapter. +use super::*; +use std::collections::VecDeque; +use std::sync::atomic::{AtomicUsize, Ordering}; + +/// A provider that supports jcode compaction and records every completion, so +/// the test can witness the daemon summarizing actual prompt history and then +/// applying the summary to the next turn. +#[derive(Clone)] +struct CompactionProbeProvider { + responses: Arc>>>, + calls: Arc>>>, + finished: Arc, +} + +impl CompactionProbeProvider { + fn new() -> Self { + Self { + responses: Arc::new(Mutex::new(VecDeque::new())), + calls: Arc::new(Mutex::new(Vec::new())), + finished: Arc::new(AtomicUsize::new(0)), + } + } + + fn queue(&self, events: Vec) { + self.responses.lock().unwrap().push_back(events); + } + + fn calls(&self) -> Vec> { + self.calls.lock().unwrap().clone() + } + + fn finished_calls(&self) -> usize { + self.finished.load(Ordering::SeqCst) + } +} + +#[async_trait::async_trait] +impl Provider for CompactionProbeProvider { + async fn complete( + &self, + messages: &[Message], + _tools: &[ToolDefinition], + _system: &str, + _resume_session_id: Option<&str>, + ) -> Result { + self.calls.lock().unwrap().push(messages.to_vec()); + let events = self + .responses + .lock() + .unwrap() + .pop_front() + .unwrap_or_default(); + let items: Vec> = events.into_iter().map(Ok).collect(); + let finished = Arc::clone(&self.finished); + // Mark the call complete only once the consumer polls the stream to its + // end, so the test does not treat "provider called" as "summary ready". + let stream = stream::unfold( + (items.into_iter(), finished), + |(mut items, finished)| async move { + match items.next() { + Some(item) => Some((item, (items, finished))), + None => { + finished.fetch_add(1, Ordering::SeqCst); + None + } + } + }, + ); + Ok(Box::pin(stream)) + } + + fn name(&self) -> &str { + "compaction-probe" + } + + fn model(&self) -> String { + "probe-model".to_string() + } + + fn set_model(&self, _model: &str) -> Result<()> { + Ok(()) + } + + fn available_models(&self) -> Vec<&'static str> { + vec!["probe-model"] + } + + fn supports_compaction(&self) -> bool { + true + } + + fn context_window(&self) -> usize { + 6_000 + } + + fn fork(&self) -> Arc { + Arc::new(self.clone()) + } +} + +#[tokio::test] +async fn acp_conformance_compact_command_compacts_real_history() -> Result<()> { + let _env = setup_test_env()?; + let runtime = tempfile::tempdir_in("/tmp")?; + let _runtime = EnvVarGuard::set("JCODE_RUNTIME_DIR", runtime.path()); + let socket = runtime.path().join("jcode.sock"); + let _socket_env = EnvVarGuard::set("JCODE_SOCKET", socket.clone()); + let provider = Arc::new(CompactionProbeProvider::new()); + for turn in 1..=6 { + provider.queue(vec![ + StreamEvent::TextDelta(history_answer(turn)), + StreamEvent::MessageEnd { + stop_reason: Some("end_turn".to_string()), + }, + ]); + } + provider.queue(vec![ + StreamEvent::TextDelta("compaction-summary-marker".to_string()), + StreamEvent::MessageEnd { + stop_reason: Some("end_turn".to_string()), + }, + ]); + provider.queue(vec![ + StreamEvent::TextDelta("after-compaction-answer".to_string()), + StreamEvent::MessageEnd { + stop_reason: Some("end_turn".to_string()), + }, + ]); + let daemon = server::Server::new_with_paths( + provider.clone(), + socket.clone(), + runtime.path().join("debug.sock"), + ); + let task = tokio::spawn(async move { daemon.run().await }); + let result = async { + wait_for_socket(&socket).await?; + let mut client = AcpClient::spawn()?; + initialize(&mut client).await?; + let cwd = std::env::current_dir()?; + let created = client.request("session/new", json!({"cwd":cwd})).await?; + let session = created["sessionId"] + .as_str() + .context("session/new returned no session id")? + .to_string(); + + for turn in 1..=6 { + let question = format!("history-question-{turn}"); + let response = client + .request( + "session/prompt", + json!({"sessionId":session,"prompt":[{"type":"text","text":question}]}), + ) + .await?; + assert_eq!(response["stopReason"], "end_turn"); + let expected = history_answer(turn); + let updates = client.take_notifications(); + assert!( + updates + .iter() + .any(|message| notification_text(message) == Some(expected.as_str())), + "turn {turn} did not stream its answer: {updates:?}" + ); + } + assert_eq!( + provider.calls().len(), + 6, + "each prompt must reach the provider exactly once" + ); + + let response = client + .request( + "session/prompt", + json!({"sessionId":session,"prompt":[{"type":"text","text":"/compact"}]}), + ) + .await?; + assert_eq!(response["stopReason"], "end_turn"); + timeout( + Duration::from_secs(10), + client.wait_for_notification(|message| { + notification_text(message).is_some_and(|text| { + text.contains("Context Status") && text.contains("Compacting context") + }) + }), + ) + .await + .context("manual compact status")??; + wait_for_finished_calls(&provider, 7).await?; + + let calls = provider.calls(); + assert_eq!( + calls.len(), + 7, + "manual compaction must issue one provider summarization call" + ); + let summarized = messages_text(&calls[6]); + assert!( + summarized.contains("history-question-1"), + "compaction must read the oldest retained prompt: {summarized}" + ); + assert!( + summarized.contains(&history_answer(1)), + "compaction must read the oldest retained answer: {summarized}" + ); + assert!( + !summarized.contains("history-question-6"), + "compaction must leave the recent turns out of the summary: {summarized}" + ); + + let response = client + .request( + "session/prompt", + json!({"sessionId":session,"prompt":[{"type":"text","text":"after-compaction-question"}]}), + ) + .await?; + assert_eq!(response["stopReason"], "end_turn"); + let updates = client.take_notifications(); + assert!( + updates + .iter() + .any(|message| notification_text(message) == Some("after-compaction-answer")), + "the turn after compaction did not complete: {updates:?}" + ); + let calls = provider.calls(); + assert_eq!( + calls.len(), + 8, + "the post-compaction prompt must reach the provider once" + ); + let after = messages_text(&calls[7]); + assert!( + after.contains("compaction-summary-marker"), + "the applied summary must replace older history: {after}" + ); + assert!( + after.contains("history-question-6"), + "recent turns must survive compaction: {after}" + ); + assert!( + !after.contains("history-question-1") && !after.contains(&history_answer(1)), + "summarized turns must not remain in the next provider context: {after}" + ); + client.shutdown().await; + Ok(()) + } + .await; + task.abort(); + result +} + +#[tokio::test] +async fn acp_conformance_subagent_command_spawns_live_worker() -> Result<()> { + let _env = setup_test_env()?; + let runtime = tempfile::tempdir_in("/tmp")?; + let _runtime = EnvVarGuard::set("JCODE_RUNTIME_DIR", runtime.path()); + let socket = runtime.path().join("jcode.sock"); + let _socket_env = EnvVarGuard::set("JCODE_SOCKET", socket.clone()); + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let provider = Arc::new(MockProvider::gated(gate.clone())); + provider.queue_response(vec![ + StreamEvent::TextDelta("child-part-1".to_string()), + StreamEvent::TextDelta("child-part-2".to_string()), + StreamEvent::MessageEnd { + stop_reason: Some("end_turn".to_string()), + }, + ]); + let daemon = server::Server::new_with_paths( + provider.clone(), + socket.clone(), + runtime.path().join("debug.sock"), + ); + let task = tokio::spawn(async move { daemon.run().await }); + let result = async { + wait_for_socket(&socket).await?; + let mut parent = AcpClient::spawn()?; + initialize(&mut parent).await?; + let cwd = std::env::current_dir()?; + let created = parent.request("session/new", json!({"cwd":cwd})).await?; + let session = created["sessionId"] + .as_str() + .context("session/new returned no session id")? + .to_string(); + + let launch = parent + .send_request( + "session/prompt", + json!({"sessionId":session,"prompt":[{"type":"text","text":"/subagent child-scout"}]}), + ) + .await?; + let tool_done = timeout( + Duration::from_secs(10), + parent.wait_for_notification(|message| terminal_tool_update(message).is_some()), + ) + .await + .context("subagent tool completion")??; + let update = terminal_tool_update(&tool_done).context("tool update")?; + assert_eq!( + update["status"], "completed", + "the /subagent swarm tool failed: {}", + update["rawOutput"] + ); + let output = update["rawOutput"]["output"] + .as_str() + .context("tool output")?; + let child = spawned_child_id(output) + .with_context(|| format!("unexpected /subagent output: {output}"))? + .to_string(); + let response = parent.wait_for_response(launch).await?; + assert_eq!(response["result"]["stopReason"], "end_turn"); + + // The worker is still inside its first provider call: one delta is on + // the wire and the response is parked at the mock gate. Attaching now + // must observe the live turn rather than a completed transcript. + let mut worker = AcpClient::spawn()?; + initialize(&mut worker).await?; + worker + .request("session/resume", json!({"sessionId":child,"cwd":cwd})) + .await?; + let started = timeout( + Duration::from_secs(10), + worker.wait_for_notification(|message| { + message["method"] == "_jcode/turn" && message["params"]["state"] == "started" + }), + ) + .await + .context("worker turn attachment")??; + assert_eq!(started["params"]["sessionId"], json!(child)); + let turn = started["params"]["turnId"].clone(); + let before_release = worker.take_notifications(); + assert!( + !before_release + .iter() + .any(|message| notification_text(message) == Some("child-part-2")), + "the gated worker streamed past its gate before release: {before_release:?}" + ); + + gate.add_permits(1); + let live = timeout( + Duration::from_secs(10), + worker.wait_for_notification(|message| { + notification_text(message) == Some("child-part-2") + }), + ) + .await + .context("worker live delta")??; + assert_eq!(live["params"]["sessionId"], json!(child)); + let finished = timeout( + Duration::from_secs(10), + worker.wait_for_notification(|message| { + message["method"] == "_jcode/turn" && message["params"]["state"] != "started" + }), + ) + .await + .context("worker turn terminal")??; + assert_eq!(finished["params"]["turnId"], turn); + assert_eq!(finished["params"]["state"], "completed"); + + // The child's own live delta and terminal above are the request/outcome + // proof. Do not constrain the global provider call count: a parent + // title or follow-up completion is unrelated housekeeping. + let listed = parent.request("session/list", json!({"cwd":cwd})).await?; + assert!( + listed["sessions"] + .as_array() + .is_some_and(|sessions| sessions.iter().any(|entry| entry["sessionId"] == json!(child))), + "the spawned worker must be a registered session: {listed}" + ); + + parent.shutdown().await; + worker.shutdown().await; + Ok(()) + } + .await; + task.abort(); + result +} + +fn history_answer(turn: usize) -> String { + format!("history-answer-{turn}-{}", "x".repeat(600)) +} + +async fn wait_for_finished_calls( + provider: &CompactionProbeProvider, + expected: usize, +) -> Result<()> { + timeout(Duration::from_secs(10), async { + loop { + if provider.finished_calls() >= expected { + break Ok::<(), anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await + .with_context(|| { + format!( + "provider finished {} of {expected} expected completions", + provider.finished_calls() + ) + })??; + Ok(()) +} + +fn messages_text(messages: &[Message]) -> String { + messages + .iter() + .map(flatten_text_blocks) + .collect::>() + .join("\n---\n") +} + +fn notification_text(message: &Value) -> Option<&str> { + message["params"]["update"]["content"]["text"].as_str() +} + +fn terminal_tool_update(message: &Value) -> Option<&Value> { + let update = &message["params"]["update"]; + if update["sessionUpdate"] != "tool_call_update" { + return None; + } + match update["status"].as_str()? { + "completed" | "failed" => Some(update), + _ => None, + } +} + +fn spawned_child_id(output: &str) -> Option<&str> { + output + .strip_prefix("Spawned new agent: ")? + .lines() + .next() + .map(str::trim) + .filter(|id| !id.is_empty()) +} diff --git a/tests/e2e/acp_shared_session/extensions.rs b/tests/e2e/acp_shared_session/extensions.rs index 529edc9fc7d..7bcdcaabe66 100644 --- a/tests/e2e/acp_shared_session/extensions.rs +++ b/tests/e2e/acp_shared_session/extensions.rs @@ -333,6 +333,10 @@ async fn acp_skill_clear_round_trip() -> Result<()> { "a bare clear starts no model turn: {updates:#?}" ); + // Keep explicit selection and null-clear semantics in the shared + // consumer corpus, not only in this agent-side assertion. + crate::test_support::acp_fixture::check("12-skill-selection-clear-v1.jsonl", &acp.tape)?; + // Clear with text: the text runs as a normal, unskilled turn. let prompt = "Reply with exactly the word: alpha"; let ran = acp diff --git a/tests/e2e/test_support/acp_fixture.rs b/tests/e2e/test_support/acp_fixture.rs index 111eb565f41..9b8618361cf 100644 --- a/tests/e2e/test_support/acp_fixture.rs +++ b/tests/e2e/test_support/acp_fixture.rs @@ -118,6 +118,153 @@ fn acp_conformance_comparator_identity_drift_and_order() { assert!(compare(&tape("a", "a"), &reversed).is_err()); } +/// The corpus is the shared executable contract, so the replay comparison is +/// not enough: it only reads adapter-to-client messages. A consumer also +/// replays the client half, and an orphaned or truncated file would silently +/// stop being a contract. Validate the envelope and both directions for every +/// checked-in file, and require each file to be replayed by a real test. +#[test] +fn acp_corpus_files_are_well_formed_and_referenced() -> Result<()> { + let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")); + let fixture_dir = root.join("tests/fixtures/acp"); + let mut files = Vec::new(); + for entry in std::fs::read_dir(&fixture_dir)? { + let path = entry?.path(); + if path.extension().and_then(|extension| extension.to_str()) == Some("jsonl") { + files.push(path); + } + } + files.sort(); + assert!(!files.is_empty(), "the corpus has files to replay"); + + let mut referenced = std::collections::BTreeSet::new(); + let mut pending = vec![root.join("tests/e2e")]; + while let Some(dir) = pending.pop() { + for entry in std::fs::read_dir(dir)? { + let path = entry?.path(); + if path.is_dir() { + pending.push(path); + } else if path.extension().and_then(|extension| extension.to_str()) == Some("rs") { + let source = std::fs::read_to_string(&path)?; + for fragment in source.split("acp_fixture::check(\"").skip(1) { + if let Some((name, _)) = fragment.split_once('"') { + referenced.insert(name.to_string()); + } + } + } + } + } + + for path in &files { + let name = path + .file_name() + .and_then(|name| name.to_str()) + .with_context(|| format!("corpus path is not UTF-8: {}", path.display()))? + .to_string(); + assert!( + referenced.contains(&name), + "corpus file {name} is not replayed by any test" + ); + let mut client_requests = std::collections::BTreeSet::new(); + let mut client_responses = std::collections::BTreeSet::new(); + let mut adapter_requests = std::collections::BTreeSet::new(); + let mut adapter_responses = std::collections::BTreeSet::new(); + let mut saw_client = false; + let mut saw_adapter = false; + for (index, line) in std::fs::read_to_string(path)?.lines().enumerate() { + let record: Value = serde_json::from_str(line) + .with_context(|| format!("{name}:{} is not JSON", index + 1))?; + let object = record + .as_object() + .with_context(|| format!("{name}:{} is not an object", index + 1))?; + assert_eq!( + object.len(), + 3, + "{name}:{} carries exactly seq, dir and msg", + index + 1 + ); + assert_eq!( + record["seq"].as_u64(), + Some(index as u64 + 1), + "{name}:{} has a monotonic sequence number", + index + 1 + ); + let dir = record["dir"] + .as_str() + .with_context(|| format!("{name}:{} has a string dir", index + 1))?; + let msg = record["msg"] + .as_object() + .with_context(|| format!("{name}:{} has an object msg", index + 1))?; + assert_eq!( + msg.get("jsonrpc").and_then(Value::as_str), + Some("2.0"), + "{name}:{} is a framed JSON-RPC message", + index + 1 + ); + let id = msg + .get("id") + .and_then(Value::as_str) + .map(str::to_string) + .or_else(|| { + msg.get("id") + .and_then(Value::as_u64) + .map(|id| id.to_string()) + }); + let is_request = msg.contains_key("method") && id.is_some(); + let is_notification = msg.contains_key("method") && id.is_none(); + let is_response = !msg.contains_key("method") && id.is_some(); + assert!( + is_request || is_notification || is_response, + "{name}:{} is a request, notification or response", + index + 1 + ); + match dir { + "c2a" => { + saw_client = true; + if is_request { + client_requests.insert(id.clone().expect("requests carry an id")); + } + if is_response { + client_responses.insert(id.clone().expect("responses carry an id")); + } + } + "a2c" => { + saw_adapter = true; + if is_request { + adapter_requests.insert(id.clone().expect("requests carry an id")); + } + if is_response { + adapter_responses.insert(id.clone().expect("responses carry an id")); + } + } + other => panic!("{name}:{} has an unknown direction {other}", index + 1), + } + } + assert!(saw_client && saw_adapter, "{name} records both directions"); + for id in &client_requests { + assert!( + adapter_responses.contains(id), + "{name}: client request {id} has no adapter response" + ); + } + for id in &adapter_requests { + assert!( + client_responses.contains(id), + "{name}: adapter request {id} has no client response" + ); + } + } + for name in &referenced { + assert!( + files + .iter() + .any(|path| path.file_name().and_then(|file| file.to_str()) == Some(name.as_str())), + "a test references missing corpus file {name}" + ); + } + Ok(()) +} + #[test] fn acp_conformance_comparator_normalizes_only_agent_info_version() { let tape = |version: Value| { diff --git a/tests/fixtures/acp/12-skill-selection-clear-v1.jsonl b/tests/fixtures/acp/12-skill-selection-clear-v1.jsonl new file mode 100644 index 00000000000..ef3ffca6ecd --- /dev/null +++ b/tests/fixtures/acp/12-skill-selection-clear-v1.jsonl @@ -0,0 +1,13 @@ +{"dir":"c2a","msg":{"id":1,"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":1}},"seq":1} +{"dir":"a2c","msg":{"id":1,"jsonrpc":"2.0","result":{"agentCapabilities":{"_meta":{"jcode":{"extensions":["shared_control","peer_turns","turn_correlation","skill_selection"],"interfaceVersion":1,"profile":"standard","sessionMcpServers":false}},"loadSession":true,"mcpCapabilities":{"http":false,"sse":false},"promptCapabilities":{"audio":false,"embeddedContext":true,"image":true},"sessionCapabilities":{"close":{},"list":{},"resume":{}}},"agentInfo":{"name":"jcode","title":"Jcode","version":"0.99.1"},"authMethods":[],"protocolVersion":1}},"seq":2} +{"dir":"c2a","msg":{"id":2,"jsonrpc":"2.0","method":"session/new","params":{"cwd":"/home/ctaylor/.jcode/scratch/.tmpVVZK1b","mcpServers":[]}},"seq":3} +{"dir":"a2c","msg":{"id":2,"jsonrpc":"2.0","result":{"_meta":{"jcode":{"clientInstanceId":"acp-241c19ce-55ca-4118-a3d1-ca49ea16edc6/0","sharedControl":{"detach":true,"implicitSteering":true,"observe":true,"stopObservedTurn":true}}},"configOptions":[{"category":"model","currentValue":"unknown","id":"model","name":"Model","options":[{"name":"unknown","value":"unknown"}],"type":"select"}],"models":{"availableModels":[{"modelId":"unknown","name":"unknown"}],"currentModelId":"unknown"},"sessionId":"session_frog_1794346608281_a006c58f7068c040"}},"seq":4} +{"dir":"c2a","msg":{"id":3,"jsonrpc":"2.0","method":"session/prompt","params":{"_meta":{"jcode":{"skill":"interrogate"}},"prompt":[{"text":"","type":"text"}],"sessionId":"session_frog_1794346608281_a006c58f7068c040"}},"seq":5} +{"dir":"a2c","msg":{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"session_frog_1794346608281_a006c58f7068c040","update":{"availableCommands":[{"description":"Switch the model for this session, or show the current model","input":{"hint":"model id (optional)"},"name":"model"},{"description":"List models available from the active provider","name":"models"},{"description":"Set reasoning effort, or show the current effort","input":{"hint":"none|minimal|low|medium|high|xhigh|max (optional)"},"name":"effort"},{"description":"Summarize older messages to free context (applied on the next prompt)","name":"compact"},{"description":"Launch a background swarm worker with a prompt","input":{"hint":"[--model ] [--type ] "},"name":"subagent"},{"description":"Test interrogate","input":{"hint":"request for this skill (optional)"},"name":"interrogate"}],"sessionUpdate":"available_commands_update"}}},"seq":6} +{"dir":"a2c","msg":{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"session_frog_1794346608281_a006c58f7068c040","update":{"content":{"text":"Activated skill: interrogate - Test interrogate\n\n","type":"text"},"messageId":"session_frog_1794346608281_a006c58f7068c040:skill-notice:3","sessionUpdate":"agent_message_chunk"}}},"seq":7} +{"dir":"a2c","msg":{"id":3,"jsonrpc":"2.0","result":{"_meta":{"jcode":{"role":"none","turnId":null}},"stopReason":"end_turn"}},"seq":8} +{"dir":"c2a","msg":{"id":4,"jsonrpc":"2.0","method":"session/prompt","params":{"prompt":[{"text":"/interrogate","type":"text"}],"sessionId":"session_frog_1794346608281_a006c58f7068c040"}},"seq":9} +{"dir":"a2c","msg":{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"session_frog_1794346608281_a006c58f7068c040","update":{"content":{"text":"Activated skill: interrogate - Test interrogate\n\n","type":"text"},"messageId":"session_frog_1794346608281_a006c58f7068c040:skill-notice:4","sessionUpdate":"agent_message_chunk"}}},"seq":10} +{"dir":"a2c","msg":{"id":4,"jsonrpc":"2.0","result":{"_meta":{"jcode":{"role":"none","turnId":null}},"stopReason":"end_turn"}},"seq":11} +{"dir":"c2a","msg":{"id":5,"jsonrpc":"2.0","method":"session/prompt","params":{"_meta":{"jcode":{"skill":null}},"prompt":[],"sessionId":"session_frog_1794346608281_a006c58f7068c040"}},"seq":12} +{"dir":"a2c","msg":{"id":5,"jsonrpc":"2.0","result":{"_meta":{"jcode":{"role":"none","turnId":null}},"stopReason":"end_turn"}},"seq":13} diff --git a/tests/fixtures/acp/README.md b/tests/fixtures/acp/README.md index 1f68957a2e5..cfe256f6dc8 100644 --- a/tests/fixtures/acp/README.md +++ b/tests/fixtures/acp/README.md @@ -22,6 +22,13 @@ that resembles a placeholder. Missing fields, changed content, identity splits or collapses and notification reordering fail comparison. The checked-in records contain original emitted values. +The harness also validates the corpus itself, not only the replay: every +`.jsonl` file must be referenced by an `acp_fixture::check` call, each record +must carry exactly `seq`, `dir` and `msg` with a monotonic sequence number, +`msg` must be framed JSON-RPC 2.0, and every request in either direction must +have its response. That keeps the client half, which the replay comparison +does not read, usable as the consumer's executable contract. + Run on a development host with sufficient memory, following AGENTS.md: ```sh @@ -57,11 +64,8 @@ No timer sleeps establish protocol ordering. - `08-negotiation-legacy-v1.jsonl`: future-version negotiation, full adapter metadata, extension errors, ignored prompt metadata, legacy shared-control refusal and omitted correlation. Uses the existing test-only Unix daemon stub, with no runtime capability switch. -Wire tests also exercise observer detach, queued-input withdrawal, adopted-steer detach, -parameter errors and observer-close brackets. All contract-gap tests are enabled. - -Number 06 is intentionally absent: acceptance scenario 6 in the contract is -the deferred v2 delivery track, not a missing v1 recording. +- `09-elicitation-form-v1.jsonl`: opted-in form elicitation. This pulled-forward + optional feature has its own transcript, separate from the mid-turn attach case. - `09-midturn-attach-v1.jsonl`: observer attaches to a gated active turn; exactly one start/terminal pair preserves its identity. Replayed by the @@ -71,3 +75,13 @@ the deferred v2 delivery track, not a missing v1 recording. ACP-owned turn; peer user chunk and its message identity precede affected output. - `11-import-repeated-user-replay-v1.jsonl`: two deliberately equal stored user rows imported and loaded twice; distinct IDs and grouping stay stable. +- `12-skill-selection-clear-v1.jsonl`: explicit metadata selection, textual + selection and an explicit `skill: null` clear. Each selection-only response + has `turnId: null` and `role: none`, and clearing emits no activation notice + or model turn. Recorded and replayed by `acp_skill_clear_round_trip`. + +Wire tests also exercise observer detach, queued-input withdrawal, adopted-steer detach, +parameter errors and observer-close brackets. All contract-gap tests are enabled. + +Number 06 is intentionally absent: acceptance scenario 6 in the contract is +the deferred v2 delivery track, not a missing v1 recording.