skills: add run-forensics - #1151
Open
xizhuomengcontin wants to merge 1 commit into
Open
xizhuomengcontin wants to merge 1 commit into
xizhuomengcontin wants to merge 1 commit into
Conversation
Answers questions about a past agent run by reading its OrcaReplay recording instead of the agent's own memory, and replays that run offline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds one
execution-tier skill: run-forensics — answer questions about a run that already happened by reading its recording rather than the agent's memory.Disclosure: I maintain OrcaReplay, the tool this skill drives. Apache-2.0, npm, Node 20+, no paid tier, no account, no hosted service.
Why this one
An agent asked "why did you do that?" answers from a summary of its own context window. The tool results, the shell exit codes and the files that changed without anyone mentioning them are already gone from it. The answer comes out fluent, confident and occasionally wrong — worse than "I don't know", because it gets believed and written into a commit message.
The skill's whole job is to refuse that path: when the question is about something that already happened, read the recording first. It fits the guide's three criteria — narrow (one intent), reusable (invoked whenever a past run is in question), observable (every claim is traceable to a recorded event).
Boundaries the skill states rather than hides
These are in the
## Boundariessection because they are the three things a forensics skill is most likely to overclaim:egress=blockedis model-provider egress only. Recorded tool calls still execute for real on replay — a recordedcurlreaches the network. Replay is not a sandbox.It also requires reading the recorded shell commands before the first replay, and replaying into a scratch worktree, because a replay re-runs the agent process and therefore re-runs its commands.
Gates
Ran everything the contributing guide lists:
About that one FAIL, and about the diff size
The link failure is pre-existing and not mine. It is two links to a file that does not exist:
Neither file is touched by this PR. I left them alone because fixing them is an infra change and infra is maintainer territory — happy to send a separate PR if you want it.
On the generated files:
generate-skill-mesh.pyrewrites 45 files on a clean checkout, before any change of mine — the committed projections have drifted from what the generator produces. Committing that wholesale would have buried one skill under forty-odd unrelated files.So I diffed the generator's output on a clean tree against its output with this skill present, and applied only that delta onto the committed versions. The result is the 11 files below, all pure additions except where a count or a trailing-comma line had to move:
plus the new
skills/run-forensics/,skills-codex/run-forensics/andimages/gemini/skills/run-forensics/, and the two codex manifest/catalog filescodex-sync.showns.If you would rather I just run the generator and let the drift land in this PR, say so and I will push that instead — it is a one-line change on my side. I chose the narrow diff because the drift is yours to decide about, not something a first-time contributor should quietly fold into a skill PR.
Not included
No
references/orscripts/— the skill is one file, so there is nothing to link and nothing for the dead-reference check to catch. NoTODO/FIXME, no symlinks.