Skip to content

skills: add run-forensics - #1151

Open
xizhuomengcontin wants to merge 1 commit into
boshu2:mainfrom
xizhuomengcontin:skill/run-forensics
Open

xizhuomengcontin wants to merge 1 commit into
boshu2:mainfrom
xizhuomengcontin:skill/run-forensics

Conversation

@xizhuomengcontin

Copy link
Copy Markdown

Adds one execution-tier skill: run-forensics — answer questions about a run that already happened by reading its recording rather than the agent's memory.

Disclosure: I maintain OrcaReplay, the tool this skill drives. Apache-2.0, npm, Node 20+, no paid tier, no account, no hosted service.

Why this one

An agent asked "why did you do that?" answers from a summary of its own context window. The tool results, the shell exit codes and the files that changed without anyone mentioning them are already gone from it. The answer comes out fluent, confident and occasionally wrong — worse than "I don't know", because it gets believed and written into a commit message.

The skill's whole job is to refuse that path: when the question is about something that already happened, read the recording first. It fits the guide's three criteria — narrow (one intent), reusable (invoked whenever a past run is in question), observable (every claim is traceable to a recorded event).

Boundaries the skill states rather than hides

These are in the ## Boundaries section because they are the three things a forensics skill is most likely to overclaim:

  • A matching replay is not a determinism result. It shows the recorded run reproduces, not that the model is stable.
  • egress=blocked is model-provider egress only. Recorded tool calls still execute for real on replay — a recorded curl reaches the network. Replay is not a sandbox.
  • Embedding calls are not captured by the default adapter, so a RAG step's retrieval is absent from the trace even when the chat turns are complete.

It also requires reading the recorded shell commands before the first replay, and replaying into a scratch worktree, because a replay re-runs the agent process and therefore re-runs its commands.

Gates

Ran everything the contributing guide lists:

bash skills/skill-builder/scripts/heal.sh --strict     # exit 0
python3 scripts/generate-skill-mesh.py                 # skill mesh: generated (35 skills)
bash scripts/codex-sync.sh --force --only run-forensics # generated 1 twin
bash tests/docs/validate-doc-release.sh
  PASS: metadata inventory covers 35 source and Codex skills exactly once
  PASS: release message freeze intact
  FAIL: Link validation

About that one FAIL, and about the diff size

The link failure is pre-existing and not mine. It is two links to a file that does not exist:

BROKEN: docs/index.md:27 -> cli/commands.md
BROKEN: docs/reference/index.md:9 -> ../cli/commands.md

Neither file is touched by this PR. I left them alone because fixing them is an infra change and infra is maintainer territory — happy to send a separate PR if you want it.

On the generated files: generate-skill-mesh.py rewrites 45 files on a clean checkout, before any change of mine — the committed projections have drifted from what the generator produces. Committing that wholesale would have buried one skill under forty-odd unrelated files.

So I diffed the generator's output on a clean tree against its output with this skill present, and applied only that delta onto the committed versions. The result is the 11 files below, all pure additions except where a count or a trailing-comma line had to move:

 3 +  1 -  docs/SKILL-ROUTER.md
 3 +  1 -  docs/SKILLS.md
 1 +  0 -  docs/contracts/context-map.md
 2 +  1 -  docs/reference/agentops-skill-domain-map.md
 1 +  0 -  docs/reference/agentops-skill-graph.md
 6 +  1 -  images/claude/manifest.json
 7 +  1 -  images/codex/manifest.json
 1 +  1 -  images/gemini/plugin.json
29 +  4 -  registry.json
 2 +  1 -  skills/SKILL-TIERS.md
29 +  1 -  skills/catalog.json

plus the new skills/run-forensics/, skills-codex/run-forensics/ and images/gemini/skills/run-forensics/, and the two codex manifest/catalog files codex-sync.sh owns.

If you would rather I just run the generator and let the drift land in this PR, say so and I will push that instead — it is a one-line change on my side. I chose the narrow diff because the drift is yours to decide about, not something a first-time contributor should quietly fold into a skill PR.

Not included

No references/ or scripts/ — the skill is one file, so there is nothing to link and nothing for the dead-reference check to catch. No TODO/FIXME, no symlinks.

Answers questions about a past agent run by reading its OrcaReplay recording
instead of the agent's own memory, and replays that run offline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant