diff --git a/docs/SKILL-ROUTER.md b/docs/SKILL-ROUTER.md index 722087fca..f1704a831 100644 --- a/docs/SKILL-ROUTER.md +++ b/docs/SKILL-ROUTER.md @@ -2,7 +2,7 @@ # Skill Router -34 live skills. Choose guidance for a concrete task need; no skill is mandatory. +35 live skills. Choose guidance for a concrete task need; no skill is mandatory. A clear task can proceed in the native agent. Read a skill only when its description fits. Names and descriptions below come from each source SKILL.md; explicit invocation remains available. @@ -61,6 +61,7 @@ Names and descriptions below come from each source SKILL.md; explicit invocation | [ms](../skills/ms/SKILL.md) | Find and load guidance with the meta_skill search engine. Use when: searching a skill corpus; CASS owns past sessions and Skill Builder owns package authoring. | | [ntm](../skills/ntm/SKILL.md) | Operate selected NTM agent panes and inspect native state. Use when: persistent tmux roles are requested; pane liveness and prompt delivery are not validation. | | [rch](../skills/rch/SKILL.md) | Offload one build through RCH or diagnose its remote compiler. Use when: remote compilation is selected; report errors without creating a retry controller. | +| [run-forensics](../skills/run-forensics/SKILL.md) | Answer questions about an agent run that already happened by reading its recording instead of the agent's memory. Triggers: "why did it do that", "which step changed this file", "reproduce that run", "replay the failure". | | [sbh](../skills/sbh/SKILL.md) | Inspect disk pressure with SBH and perform an authorized recovery action. Use when: storage diagnosis or SBH recovery is requested; inspection does not authorize deletion. | | [using-flywheel](../skills/using-flywheel/SKILL.md) | Operate the Agentic Coding Flywheel through its native workflow. Use when: the caller explicitly selects this factory; convergence and closed work do not prove semantic acceptance. | | [using-gc](../skills/using-gc/SKILL.md) | Operate Gas City through its Mayor, registry packs and native run state. Use when: the caller explicitly selects Gas City; factory completion does not replace independent judgment. | @@ -95,6 +96,7 @@ Names and descriptions below come from each source SKILL.md; explicit invocation | `research` | execution | `keep_specialist` | - | `research`, `codebase_recon`, `pattern_mining` | `write_research_report`, `write_recon_pack`, `write_pattern_evidence` | | `reverse-engineer` | execution | `keep_specialist` | - | `reverse_engineer` | `clone_upstream_repo`, `authorized_binary_execution`, `write_teardown_artifacts` | | `rpi` | meta | `keep_strategy` | `plan`, `implement`, `validate` | `own_authorized_outcome`, `report` | `dispatch_core_phases` | +| `run-forensics` | execution | `keep_optional_adapter` | - | `run-forensics` | `read_recording`, `replay_recorded_run` | | `sbh` | execution | `keep_optional_adapter` | - | `sbh` | `delete_reclaimable_files`, `release_disk_ballast`, `modify_host_storage_config` | | `security` | product | `keep_specialist` | - | `security` | `write_scan_artifacts` | | `skill-builder` | meta | `keep_specialist` | - | `skill_builder`, `heal_skill`, `export_skill`, `distill_expertise` | `write_skill_source`, `write_build_report`, `regenerate_skill_projections`, `repair_skill_projections`, `write_converted_skill_projection`, `write_advisory_proposal` | diff --git a/docs/SKILLS.md b/docs/SKILLS.md index 722087fca..f1704a831 100644 --- a/docs/SKILLS.md +++ b/docs/SKILLS.md @@ -2,7 +2,7 @@ # Skill Router -34 live skills. Choose guidance for a concrete task need; no skill is mandatory. +35 live skills. Choose guidance for a concrete task need; no skill is mandatory. A clear task can proceed in the native agent. Read a skill only when its description fits. Names and descriptions below come from each source SKILL.md; explicit invocation remains available. @@ -61,6 +61,7 @@ Names and descriptions below come from each source SKILL.md; explicit invocation | [ms](../skills/ms/SKILL.md) | Find and load guidance with the meta_skill search engine. Use when: searching a skill corpus; CASS owns past sessions and Skill Builder owns package authoring. | | [ntm](../skills/ntm/SKILL.md) | Operate selected NTM agent panes and inspect native state. Use when: persistent tmux roles are requested; pane liveness and prompt delivery are not validation. | | [rch](../skills/rch/SKILL.md) | Offload one build through RCH or diagnose its remote compiler. Use when: remote compilation is selected; report errors without creating a retry controller. | +| [run-forensics](../skills/run-forensics/SKILL.md) | Answer questions about an agent run that already happened by reading its recording instead of the agent's memory. Triggers: "why did it do that", "which step changed this file", "reproduce that run", "replay the failure". | | [sbh](../skills/sbh/SKILL.md) | Inspect disk pressure with SBH and perform an authorized recovery action. Use when: storage diagnosis or SBH recovery is requested; inspection does not authorize deletion. | | [using-flywheel](../skills/using-flywheel/SKILL.md) | Operate the Agentic Coding Flywheel through its native workflow. Use when: the caller explicitly selects this factory; convergence and closed work do not prove semantic acceptance. | | [using-gc](../skills/using-gc/SKILL.md) | Operate Gas City through its Mayor, registry packs and native run state. Use when: the caller explicitly selects Gas City; factory completion does not replace independent judgment. | @@ -95,6 +96,7 @@ Names and descriptions below come from each source SKILL.md; explicit invocation | `research` | execution | `keep_specialist` | - | `research`, `codebase_recon`, `pattern_mining` | `write_research_report`, `write_recon_pack`, `write_pattern_evidence` | | `reverse-engineer` | execution | `keep_specialist` | - | `reverse_engineer` | `clone_upstream_repo`, `authorized_binary_execution`, `write_teardown_artifacts` | | `rpi` | meta | `keep_strategy` | `plan`, `implement`, `validate` | `own_authorized_outcome`, `report` | `dispatch_core_phases` | +| `run-forensics` | execution | `keep_optional_adapter` | - | `run-forensics` | `read_recording`, `replay_recorded_run` | | `sbh` | execution | `keep_optional_adapter` | - | `sbh` | `delete_reclaimable_files`, `release_disk_ballast`, `modify_host_storage_config` | | `security` | product | `keep_specialist` | - | `security` | `write_scan_artifacts` | | `skill-builder` | meta | `keep_specialist` | - | `skill_builder`, `heal_skill`, `export_skill`, `distill_expertise` | `write_skill_source`, `write_build_report`, `regenerate_skill_projections`, `repair_skill_projections`, `write_converted_skill_projection`, `write_advisory_proposal` | diff --git a/docs/contracts/context-map.md b/docs/contracts/context-map.md index 17848f9ad..034be668f 100644 --- a/docs/contracts/context-map.md +++ b/docs/contracts/context-map.md @@ -94,6 +94,7 @@ | `rpi` | consumes | `implement` | | `rpi` | consumes | `validate` | | `rpi` | produces | `rpi-report.v1` | +| `run-forensics` | produces | `run-forensics-report` | | `security` | consumes | `repo-context` | | `security` | produces | `security-gate-summary.json` | | `security` | produces | `suite-summary.json` | diff --git a/docs/reference/agentops-skill-domain-map.md b/docs/reference/agentops-skill-domain-map.md index 6eeefbf64..bd08ebc16 100644 --- a/docs/reference/agentops-skill-domain-map.md +++ b/docs/reference/agentops-skill-domain-map.md @@ -16,7 +16,7 @@ ## supporting -`account-rotation`, `agent-mail`, `agent-native`, `cass`, `cc-hooks`, `craft-goal`, `dcg`, `doc`, `memory`, `ms`, `ntm`, `rch`, `refactor`, `reverse-engineer`, `sbh`, `skill-builder`, `skill-eval`, `test` +`account-rotation`, `agent-mail`, `agent-native`, `cass`, `cc-hooks`, `craft-goal`, `dcg`, `doc`, `memory`, `ms`, `ntm`, `rch`, `refactor`, `reverse-engineer`, `run-forensics`, `sbh`, `skill-builder`, `skill-eval`, `test` ## Inventory @@ -48,6 +48,7 @@ | `research` | execution | `keep_specialist` | - | `research`, `codebase_recon`, `pattern_mining` | `write_research_report`, `write_recon_pack`, `write_pattern_evidence` | | `reverse-engineer` | execution | `keep_specialist` | - | `reverse_engineer` | `clone_upstream_repo`, `authorized_binary_execution`, `write_teardown_artifacts` | | `rpi` | meta | `keep_strategy` | `plan`, `implement`, `validate` | `own_authorized_outcome`, `report` | `dispatch_core_phases` | +| `run-forensics` | execution | `keep_optional_adapter` | - | `run-forensics` | `read_recording`, `replay_recorded_run` | | `sbh` | execution | `keep_optional_adapter` | - | `sbh` | `delete_reclaimable_files`, `release_disk_ballast`, `modify_host_storage_config` | | `security` | product | `keep_specialist` | - | `security` | `write_scan_artifacts` | | `skill-builder` | meta | `keep_specialist` | - | `skill_builder`, `heal_skill`, `export_skill`, `distill_expertise` | `write_skill_source`, `write_build_report`, `regenerate_skill_projections`, `repair_skill_projections`, `write_converted_skill_projection`, `write_advisory_proposal` | diff --git a/docs/reference/agentops-skill-graph.md b/docs/reference/agentops-skill-graph.md index 642758d19..0e5a2510b 100644 --- a/docs/reference/agentops-skill-graph.md +++ b/docs/reference/agentops-skill-graph.md @@ -30,6 +30,7 @@ graph LR research["research"] reverse_engineer["reverse-engineer"] rpi["rpi"] + run_forensics["run-forensics"] sbh["sbh"] security["security"] skill_builder["skill-builder"] diff --git a/images/claude/manifest.json b/images/claude/manifest.json index 15bdb1ec1..d770e807b 100644 --- a/images/claude/manifest.json +++ b/images/claude/manifest.json @@ -1,7 +1,7 @@ { "image": "claude", "schema_version": "skill-image.v1", - "skill_count": 34, + "skill_count": 35, "skills": [ { "disposition": "keep_optional_adapter", @@ -133,6 +133,11 @@ "path": "skills/rpi/", "slug": "rpi" }, + { + "disposition": "keep_optional_adapter", + "path": "skills/run-forensics/", + "slug": "run-forensics" + }, { "disposition": "keep_optional_adapter", "path": "skills/sbh/", diff --git a/images/codex/manifest.json b/images/codex/manifest.json index 96569bd12..b7472a31c 100644 --- a/images/codex/manifest.json +++ b/images/codex/manifest.json @@ -1,7 +1,7 @@ { "image": "codex", "schema_version": "skill-image.v1", - "skill_count": 34, + "skill_count": 35, "skills": [ { "disposition": "keep_optional_adapter", @@ -159,6 +159,12 @@ "source_path": "skills/rpi/", "twin_path": "skills-codex/rpi/" }, + { + "disposition": "keep_optional_adapter", + "slug": "run-forensics", + "source_path": "skills/run-forensics/", + "twin_path": "skills-codex/run-forensics/" + }, { "disposition": "keep_optional_adapter", "slug": "sbh", diff --git a/images/gemini/plugin.json b/images/gemini/plugin.json index e48333042..353fe3068 100644 --- a/images/gemini/plugin.json +++ b/images/gemini/plugin.json @@ -1,6 +1,6 @@ { "agents": "./agents", - "description": "AgentOps 34-skill metadata-derived bundle for Google Antigravity and Gemini.", + "description": "AgentOps 35-skill metadata-derived bundle for Google Antigravity and Gemini.", "hooks": "./hooks/hooks.json", "mcpServers": { "agent-mail": { diff --git a/images/gemini/skills/run-forensics/SKILL.md b/images/gemini/skills/run-forensics/SKILL.md new file mode 100644 index 000000000..4359a6ca8 --- /dev/null +++ b/images/gemini/skills/run-forensics/SKILL.md @@ -0,0 +1,119 @@ +--- +name: run-forensics +description: 'Answer questions about an agent run that already happened by reading its recording instead of the agent''s memory. Triggers: "why did it do that", "which step changed this file", "reproduce that run", "replay the failure".' +practices: +- sre +hexagonal_role: supporting +consumes: [] +produces: +- run-forensics-report +context_rel: [] +skill_api_version: 1 +user-invocable: true +metadata: + capabilities: [run-forensics] + effects: [read_recording, replay_recorded_run] + canonical_status: canonical + disposition: keep_optional_adapter + tier: execution + dependencies: [] +context: + window: fork + intent: + mode: task + sections: + exclude: [HISTORY] +--- + +# run-forensics + +## Purpose + +An agent asked "why did you do that?" answers from a summary of its own context window. The tool +results, the shell exit codes and the files that changed without anyone mentioning them are already +gone from it. The answer comes out fluent, confident and occasionally wrong — worse than "I don't +know", because it gets believed and written into a commit message. + +This skill enforces one rule: when a question is about something that already happened, read the +recording before answering. It uses [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) +(Apache-2.0, npm, Node 20+), which records an agent at the HTTP boundary to its model provider from +outside the process and can serve that recording back with the provider unreachable. + +## When to Use + +- A past run changed a file, ran a command or broke a build, and nobody knows which step did it. +- A colleague reports a failure you cannot reproduce, and you do not have their key or their machine. +- A failed session should become a regression test rather than a paragraph in an issue. +- Someone is treating an agent's own explanation as a conclusion and you need to know whether + evidence supports it. + +Do **not** use this for planning the next change, reviewing a diff, or debugging code no agent ran. +If there is no recording, say so and offer to start one — do not substitute recall. + +## Inputs + +- A run identifier, or "the most recent run". +- The specific thing to explain: a file change, a command, a build failure. +- Whether the ask is explanation, reproduction, or model comparison. +- The acceptable blast radius for a replay: may it reach a database, a container, another host? + +## Instructions + +1. **Confirm a recording exists.** Run `orca list`. If it is empty, state that plainly and offer + `orca record -- `; stop rather than reconstructing from memory. +2. **Classify the question and read only what it needs.** + - *What happened?* → `orca show `: model turns with token counts and stop reasons, tool + calls with arguments and results, shell commands with exit codes, files changed. + - *Why did this happen?* → `orca graph --to `: the causal chain to that one event. + Reading a 200-event timeline and reasoning over it is slower, costs more context, and invites + the confident guess this skill exists to prevent. + - *Does it still reproduce?* → `orca replay `. +3. **Label every causal claim** as `recorded` (the recorder watched it) or `inferred` (derived at + query time from a named rule). Never merge the two. +4. **Read the recorded shell commands before any replay.** A replay is not a dry run: the agent + process runs again, so every command it issued runs again. List what will repeat first. +5. **Replay into a scratch worktree** (`orca replay --worktree`). Otherwise the recorded file + tree is restored over the working tree, and uncommitted work is absent meanwhile. +6. **Report the verdict line verbatim**, then the residual uncertainty. + +## Output + +- One sentence answering the question asked. +- The specific events behind it: sequence numbers, types, key arguments, exit codes. +- A `recorded` / `inferred` label on every causal claim. +- If replayed: the verdict line quoted exactly (`reused` / `exact` / `divergences` / `unmatched`), + plus which side effects actually repeated. +- What the recording does not cover. + +## Examples + +```text +Why did the build break in run_4f2a? Use the recording, not your memory. +Which step deleted config.yaml in the last run? +Replay run_4f2a into a worktree and tell me whether it still fails. +``` + +## Troubleshooting + +- Symptom: `orca list` is empty after a run you watched happen. + Fix: the agent pins its own provider origin and reads no base-URL variable, so nothing was + captured. Record it with `orca attach --port ` and point the agent's config at that port. +- Symptom: `reused=3/5` looks like a partial failure. + Fix: it usually is not. Harnesses make calls for themselves — a quota probe, a session-naming + request — and a replay does not repeat them. +- Symptom: a vision agent replays with divergences instead of `exact`. + Fix: expected. A re-rendered screenshot is different bytes. +- Symptom: replay fails to start in a Node project. + Fix: a scratch worktree is built from tracked files, so `node_modules` is absent. Replay in place + for those, and say so in the report. + +## Boundaries + +Three things this skill must never claim: + +- **A matching replay is not a determinism result.** It shows the recorded run reproduces, not that + the model is stable across calls. +- **`egress=blocked` means model-provider egress only.** Recorded tool calls still execute for real + on replay — a recorded `curl` reaches the network. Replay is not a sandbox. +- **Embedding calls are not captured** by the default adapter, so a RAG step's retrieval is absent + from the trace even when the chat turns are complete. diff --git a/registry.json b/registry.json index c09ec9589..3caf44e41 100644 --- a/registry.json +++ b/registry.json @@ -433,6 +433,15 @@ "path": "skills/rpi/SKILL.md", "type": "skill" }, + { + "driven_by_skills": [ + "run-forensics" + ], + "id": "skill:run-forensics:run-forensics", + "name": "run-forensics", + "path": "skills/run-forensics/SKILL.md", + "type": "skill" + }, { "driven_by_skills": [ "sbh" @@ -627,19 +636,19 @@ "cli_commands": 0, "gates": 0, "reference_impls": 0, - "skills": 69, - "total": 69 + "skills": 70, + "total": 70 }, "cli_top_level_commands": [], "schema_version": 3, "summary": { - "capabilities": 69, + "capabilities": 70, "cli_commands": 0, "eval_files": 0, "hooks": 0, "job_types": 0, "knowledge_stores": 0, - "skills": 34, + "skills": 35, "workflows": 0 }, "surfaces": { @@ -1083,6 +1092,22 @@ "reference_count": 4, "tier": "meta" }, + { + "capabilities": [ + "run-forensics" + ], + "disposition": "keep_optional_adapter", + "effects": [ + "read_recording", + "replay_recorded_run" + ], + "has_references": false, + "has_skill_md": true, + "name": "run-forensics", + "path": "skills/run-forensics/", + "reference_count": 0, + "tier": "execution" + }, { "capabilities": [ "sbh" diff --git a/skills-codex-overrides/catalog.json b/skills-codex-overrides/catalog.json index 1999deceb..b0cd6b48a 100644 --- a/skills-codex-overrides/catalog.json +++ b/skills-codex-overrides/catalog.json @@ -211,6 +211,12 @@ "treatment": "parity_only", "wave": "catalog-parity", "reason": "Auto-generated parity twin (codex-sync): skills/memory is the source of truth; no durable Codex-specific divergence yet." + }, + { + "name": "run-forensics", + "treatment": "parity_only", + "wave": "catalog-parity", + "reason": "Auto-generated parity twin (codex-sync): skills/run-forensics is the source of truth; no durable Codex-specific divergence yet." } ] } diff --git a/skills-codex/.agentops-manifest.json b/skills-codex/.agentops-manifest.json index 9b6264c34..ee13a1c1e 100644 --- a/skills-codex/.agentops-manifest.json +++ b/skills-codex/.agentops-manifest.json @@ -2,7 +2,7 @@ "generator": "manual-maintained", "source_root": "skills", "layout": "modular", - "codex_override_catalog_hash": "cfde05a564c19e506c9bf62c8a54c676f802f9b2a63550a25b9590f15a9a65de", + "codex_override_catalog_hash": "6902b2b7867fd770c55c96151b5551d938824086ee405c19397f337837594585", "codex_override_catalog": { "version": 1, "description": "Machine-readable Codex treatment map for the full skill catalog.", @@ -236,6 +236,12 @@ "treatment": "parity_only", "wave": "catalog-parity", "reason": "Auto-generated parity twin (codex-sync): skills/memory is the source of truth; no durable Codex-specific divergence yet." + }, + { + "name": "run-forensics", + "treatment": "parity_only", + "wave": "catalog-parity", + "reason": "Auto-generated parity twin (codex-sync): skills/run-forensics is the source of truth; no durable Codex-specific divergence yet." } ] }, @@ -396,6 +402,12 @@ "source_hash": "47ba75a0bba866ec0db1210e309ad40ede89f48c65ec90a00cb6d1e2e606505e", "generated_hash": "e693c85afa9523379659186a8713724f30566ebae71dd864927dd434e089de38" }, + { + "name": "run-forensics", + "source_skill": "skills/run-forensics", + "source_hash": "62b723e2152df1c0a0f484befcf644ea969eaa41dd690be8ff8aa579366dd339", + "generated_hash": "38fcd1476376b86820cc63161babd92c3dd0f9b6dcbe6725a6eb162c44e8e172" + }, { "name": "sbh", "source_skill": "skills/sbh", @@ -445,5 +457,5 @@ "generated_hash": "c88209fb2cca50df76edb8395b6b2ccaa038b6d6cc4c5bba81dd4c8ee917bd09" } ], - "package_count": 34 + "package_count": 35 } diff --git a/skills-codex/run-forensics/.agentops-generated.json b/skills-codex/run-forensics/.agentops-generated.json new file mode 100644 index 000000000..a1273103c --- /dev/null +++ b/skills-codex/run-forensics/.agentops-generated.json @@ -0,0 +1,7 @@ +{ + "generator": "codex-sync", + "source_skill": "skills/run-forensics", + "layout": "modular", + "source_hash": "62b723e2152df1c0a0f484befcf644ea969eaa41dd690be8ff8aa579366dd339", + "generated_hash": "38fcd1476376b86820cc63161babd92c3dd0f9b6dcbe6725a6eb162c44e8e172" +} diff --git a/skills-codex/run-forensics/SKILL.md b/skills-codex/run-forensics/SKILL.md new file mode 100644 index 000000000..042ad85d3 --- /dev/null +++ b/skills-codex/run-forensics/SKILL.md @@ -0,0 +1,96 @@ +--- +name: run-forensics +description: 'Answer questions about an agent run that already happened by reading its recording instead of the agent''s memory. Triggers: "why did it do that", "which step changed this file", "reproduce that run", "replay the failure".' +--- +# run-forensics + +## Purpose + +An agent asked "why did you do that?" answers from a summary of its own context window. The tool +results, the shell exit codes and the files that changed without anyone mentioning them are already +gone from it. The answer comes out fluent, confident and occasionally wrong — worse than "I don't +know", because it gets believed and written into a commit message. + +This skill enforces one rule: when a question is about something that already happened, read the +recording before answering. It uses [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) +(Apache-2.0, npm, Node 20+), which records an agent at the HTTP boundary to its model provider from +outside the process and can serve that recording back with the provider unreachable. + +## When to Use + +- A past run changed a file, ran a command or broke a build, and nobody knows which step did it. +- A colleague reports a failure you cannot reproduce, and you do not have their key or their machine. +- A failed session should become a regression test rather than a paragraph in an issue. +- Someone is treating an agent's own explanation as a conclusion and you need to know whether + evidence supports it. + +Do **not** use this for planning the next change, reviewing a diff, or debugging code no agent ran. +If there is no recording, say so and offer to start one — do not substitute recall. + +## Inputs + +- A run identifier, or "the most recent run". +- The specific thing to explain: a file change, a command, a build failure. +- Whether the ask is explanation, reproduction, or model comparison. +- The acceptable blast radius for a replay: may it reach a database, a container, another host? + +## Instructions + +1. **Confirm a recording exists.** Run `orca list`. If it is empty, state that plainly and offer + `orca record -- `; stop rather than reconstructing from memory. +2. **Classify the question and read only what it needs.** + - *What happened?* → `orca show `: model turns with token counts and stop reasons, tool + calls with arguments and results, shell commands with exit codes, files changed. + - *Why did this happen?* → `orca graph --to `: the causal chain to that one event. + Reading a 200-event timeline and reasoning over it is slower, costs more context, and invites + the confident guess this skill exists to prevent. + - *Does it still reproduce?* → `orca replay `. +3. **Label every causal claim** as `recorded` (the recorder watched it) or `inferred` (derived at + query time from a named rule). Never merge the two. +4. **Read the recorded shell commands before any replay.** A replay is not a dry run: the agent + process runs again, so every command it issued runs again. List what will repeat first. +5. **Replay into a scratch worktree** (`orca replay --worktree`). Otherwise the recorded file + tree is restored over the working tree, and uncommitted work is absent meanwhile. +6. **Report the verdict line verbatim**, then the residual uncertainty. + +## Output + +- One sentence answering the question asked. +- The specific events behind it: sequence numbers, types, key arguments, exit codes. +- A `recorded` / `inferred` label on every causal claim. +- If replayed: the verdict line quoted exactly (`reused` / `exact` / `divergences` / `unmatched`), + plus which side effects actually repeated. +- What the recording does not cover. + +## Examples + +```text +Why did the build break in run_4f2a? Use the recording, not your memory. +Which step deleted config.yaml in the last run? +Replay run_4f2a into a worktree and tell me whether it still fails. +``` + +## Troubleshooting + +- Symptom: `orca list` is empty after a run you watched happen. + Fix: the agent pins its own provider origin and reads no base-URL variable, so nothing was + captured. Record it with `orca attach --port ` and point the agent's config at that port. +- Symptom: `reused=3/5` looks like a partial failure. + Fix: it usually is not. Harnesses make calls for themselves — a quota probe, a session-naming + request — and a replay does not repeat them. +- Symptom: a vision agent replays with divergences instead of `exact`. + Fix: expected. A re-rendered screenshot is different bytes. +- Symptom: replay fails to start in a Node project. + Fix: a scratch worktree is built from tracked files, so `node_modules` is absent. Replay in place + for those, and say so in the report. + +## Boundaries + +Three things this skill must never claim: + +- **A matching replay is not a determinism result.** It shows the recorded run reproduces, not that + the model is stable across calls. +- **`egress=blocked` means model-provider egress only.** Recorded tool calls still execute for real + on replay — a recorded `curl` reaches the network. Replay is not a sandbox. +- **Embedding calls are not captured** by the default adapter, so a RAG step's retrieval is absent + from the trace even when the chat turns are complete. diff --git a/skills-codex/run-forensics/prompt.md b/skills-codex/run-forensics/prompt.md new file mode 100644 index 000000000..90e785c78 --- /dev/null +++ b/skills-codex/run-forensics/prompt.md @@ -0,0 +1,8 @@ +# run-forensics + +Answer questions about an agent run that already happened by reading its recording instead of the agent's memory. Triggers: "why did it do that", "which step changed this file", "reproduce that run", "replay the failure". + +## Instructions + +Load and follow the skill instructions from the sibling `SKILL.md` file for this skill. +Then read local files in `references/` and `scripts/` when needed. diff --git a/skills/SKILL-TIERS.md b/skills/SKILL-TIERS.md index 21b9dc1d8..284f8bfda 100644 --- a/skills/SKILL-TIERS.md +++ b/skills/SKILL-TIERS.md @@ -8,7 +8,7 @@ ## execution -`account-rotation`, `agent-mail`, `cass`, `cc-hooks`, `dcg`, `idea-genie`, `implement`, `memory`, `ms`, `ntm`, `plan`, `rch`, `refactor`, `research`, `reverse-engineer`, `sbh`, `test`, `using-flywheel`, `using-gc` +`account-rotation`, `agent-mail`, `cass`, `cc-hooks`, `dcg`, `idea-genie`, `implement`, `memory`, `ms`, `ntm`, `plan`, `rch`, `refactor`, `research`, `reverse-engineer`, `run-forensics`, `sbh`, `test`, `using-flywheel`, `using-gc` ## judgment @@ -60,6 +60,7 @@ | `research` | execution | `keep_specialist` | - | `research`, `codebase_recon`, `pattern_mining` | `write_research_report`, `write_recon_pack`, `write_pattern_evidence` | | `reverse-engineer` | execution | `keep_specialist` | - | `reverse_engineer` | `clone_upstream_repo`, `authorized_binary_execution`, `write_teardown_artifacts` | | `rpi` | meta | `keep_strategy` | `plan`, `implement`, `validate` | `own_authorized_outcome`, `report` | `dispatch_core_phases` | +| `run-forensics` | execution | `keep_optional_adapter` | - | `run-forensics` | `read_recording`, `replay_recorded_run` | | `sbh` | execution | `keep_optional_adapter` | - | `sbh` | `delete_reclaimable_files`, `release_disk_ballast`, `modify_host_storage_config` | | `security` | product | `keep_specialist` | - | `security` | `write_scan_artifacts` | | `skill-builder` | meta | `keep_specialist` | - | `skill_builder`, `heal_skill`, `export_skill`, `distill_expertise` | `write_skill_source`, `write_build_report`, `regenerate_skill_projections`, `repair_skill_projections`, `write_converted_skill_projection`, `write_advisory_proposal` | diff --git a/skills/catalog.json b/skills/catalog.json index 2f367ec84..00bc4bf3a 100644 --- a/skills/catalog.json +++ b/skills/catalog.json @@ -1,6 +1,6 @@ { "schema_version": "3", - "skill_count": 34, + "skill_count": 35, "skills": [ { "canonical_status": "canonical", @@ -885,6 +885,34 @@ "tier": "meta", "user_invocable": true }, + { + "canonical_status": "canonical", + "capabilities": [ + "run-forensics" + ], + "codex_override_present": false, + "consumes": [], + "context_rel": [], + "dependencies": [], + "description": "Answer questions about an agent run that already happened by reading its recording instead of the agent's memory. Triggers: \"why did it do that\", \"which step changed this file\", \"reproduce that run\", \"replay the failure\".", + "disposition": "keep_optional_adapter", + "effects": [ + "read_recording", + "replay_recorded_run" + ], + "graph_root": false, + "hexagonal_role": "supporting", + "name": "run-forensics", + "practices": [ + "sre" + ], + "produces": [ + "run-forensics-report" + ], + "references_count": 0, + "tier": "execution", + "user_invocable": true + }, { "canonical_status": "canonical", "capabilities": [ diff --git a/skills/run-forensics/SKILL.md b/skills/run-forensics/SKILL.md new file mode 100644 index 000000000..4359a6ca8 --- /dev/null +++ b/skills/run-forensics/SKILL.md @@ -0,0 +1,119 @@ +--- +name: run-forensics +description: 'Answer questions about an agent run that already happened by reading its recording instead of the agent''s memory. Triggers: "why did it do that", "which step changed this file", "reproduce that run", "replay the failure".' +practices: +- sre +hexagonal_role: supporting +consumes: [] +produces: +- run-forensics-report +context_rel: [] +skill_api_version: 1 +user-invocable: true +metadata: + capabilities: [run-forensics] + effects: [read_recording, replay_recorded_run] + canonical_status: canonical + disposition: keep_optional_adapter + tier: execution + dependencies: [] +context: + window: fork + intent: + mode: task + sections: + exclude: [HISTORY] +--- + +# run-forensics + +## Purpose + +An agent asked "why did you do that?" answers from a summary of its own context window. The tool +results, the shell exit codes and the files that changed without anyone mentioning them are already +gone from it. The answer comes out fluent, confident and occasionally wrong — worse than "I don't +know", because it gets believed and written into a commit message. + +This skill enforces one rule: when a question is about something that already happened, read the +recording before answering. It uses [OrcaReplay](https://github.com/Continuum-AI-Corp/OrcaReplay) +(Apache-2.0, npm, Node 20+), which records an agent at the HTTP boundary to its model provider from +outside the process and can serve that recording back with the provider unreachable. + +## When to Use + +- A past run changed a file, ran a command or broke a build, and nobody knows which step did it. +- A colleague reports a failure you cannot reproduce, and you do not have their key or their machine. +- A failed session should become a regression test rather than a paragraph in an issue. +- Someone is treating an agent's own explanation as a conclusion and you need to know whether + evidence supports it. + +Do **not** use this for planning the next change, reviewing a diff, or debugging code no agent ran. +If there is no recording, say so and offer to start one — do not substitute recall. + +## Inputs + +- A run identifier, or "the most recent run". +- The specific thing to explain: a file change, a command, a build failure. +- Whether the ask is explanation, reproduction, or model comparison. +- The acceptable blast radius for a replay: may it reach a database, a container, another host? + +## Instructions + +1. **Confirm a recording exists.** Run `orca list`. If it is empty, state that plainly and offer + `orca record -- `; stop rather than reconstructing from memory. +2. **Classify the question and read only what it needs.** + - *What happened?* → `orca show `: model turns with token counts and stop reasons, tool + calls with arguments and results, shell commands with exit codes, files changed. + - *Why did this happen?* → `orca graph --to `: the causal chain to that one event. + Reading a 200-event timeline and reasoning over it is slower, costs more context, and invites + the confident guess this skill exists to prevent. + - *Does it still reproduce?* → `orca replay `. +3. **Label every causal claim** as `recorded` (the recorder watched it) or `inferred` (derived at + query time from a named rule). Never merge the two. +4. **Read the recorded shell commands before any replay.** A replay is not a dry run: the agent + process runs again, so every command it issued runs again. List what will repeat first. +5. **Replay into a scratch worktree** (`orca replay --worktree`). Otherwise the recorded file + tree is restored over the working tree, and uncommitted work is absent meanwhile. +6. **Report the verdict line verbatim**, then the residual uncertainty. + +## Output + +- One sentence answering the question asked. +- The specific events behind it: sequence numbers, types, key arguments, exit codes. +- A `recorded` / `inferred` label on every causal claim. +- If replayed: the verdict line quoted exactly (`reused` / `exact` / `divergences` / `unmatched`), + plus which side effects actually repeated. +- What the recording does not cover. + +## Examples + +```text +Why did the build break in run_4f2a? Use the recording, not your memory. +Which step deleted config.yaml in the last run? +Replay run_4f2a into a worktree and tell me whether it still fails. +``` + +## Troubleshooting + +- Symptom: `orca list` is empty after a run you watched happen. + Fix: the agent pins its own provider origin and reads no base-URL variable, so nothing was + captured. Record it with `orca attach --port ` and point the agent's config at that port. +- Symptom: `reused=3/5` looks like a partial failure. + Fix: it usually is not. Harnesses make calls for themselves — a quota probe, a session-naming + request — and a replay does not repeat them. +- Symptom: a vision agent replays with divergences instead of `exact`. + Fix: expected. A re-rendered screenshot is different bytes. +- Symptom: replay fails to start in a Node project. + Fix: a scratch worktree is built from tracked files, so `node_modules` is absent. Replay in place + for those, and say so in the report. + +## Boundaries + +Three things this skill must never claim: + +- **A matching replay is not a determinism result.** It shows the recorded run reproduces, not that + the model is stable across calls. +- **`egress=blocked` means model-provider egress only.** Recorded tool calls still execute for real + on replay — a recorded `curl` reaches the network. Replay is not a sandbox. +- **Embedding calls are not captured** by the default adapter, so a RAG step's retrieval is absent + from the trace even when the chat turns are complete.