-
Notifications
You must be signed in to change notification settings - Fork 2
feat: wire Claude Workflow gap readiness + ship real D1 receipts demo #3438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
4c9d776
59744a3
4c63e3d
89e54ba
056c91e
8a7b4d9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Non-claims | ||
|
|
||
| - This packet does not prove every sentence in PR #3282's body is independently verified -- only the four claims explicitly checked in report.md are scored; claim 4 is explicitly scored `not_supported`. | ||
| - This packet does not establish that `session_refs` correctly resolves every PR reference archive-wide -- only that it resolves this one case with structural evidence. | ||
| - This packet does not reproduce on the public seed corpus (seed 1843); it requires read-only access to the live archive and the `Sinity/polylogue` GitHub history. |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,67 @@ | ||||||
| # D1 "The Receipts": Claim-vs-Evidence on a Real Merged PR | ||||||
|
|
||||||
| Predeclaration receipt: `artifact:d1-receipts-predeclaration`. | ||||||
|
|
||||||
| Pick a real merged, agent-authored PR from this repository. Resolve it to | ||||||
| its authoring/dispatch session **structurally** — via `session_refs` | ||||||
| (kind=`pull_request`), not by regex-scanning message prose or a time-window | ||||||
| heuristic. Then check specific sentences from the PR body against that | ||||||
| session's own recorded tool_use/tool_result blocks: does the evidence | ||||||
| actually support the claim, or is the claim resting on the PR body's own | ||||||
| prose with nothing underneath it? | ||||||
|
|
||||||
| Product primitives only: `session_refs` (the typed evidence table wired by | ||||||
| PR #3425/#3431), `polylogue read --view correlation`, and structural SQL | ||||||
| reads over `blocks`/`session_refs` for citation (mirroring the exact | ||||||
| read-only query style PR #3392 and PR #3282 themselves used in their own | ||||||
| Verification sections — this demo does not invent a new access pattern). | ||||||
|
|
||||||
| ## Steps | ||||||
|
|
||||||
| 1. Resolve PR → session structurally: | ||||||
| ```sql | ||||||
| SELECT session_id, repo, ref_number, url | ||||||
| FROM session_refs | ||||||
| WHERE kind = 'pull_request' AND repo = 'Sinity/polylogue' AND ref_number = 3282; | ||||||
| ``` | ||||||
| Cross-check the same resolution through the CLI's own read surface: | ||||||
| `polylogue find "id:<session_id>" then read --view correlation --format json` | ||||||
| (this is the surface PR #3425/#3431 wired `session_refs` into — | ||||||
| `insights/session_commit.py:build_correlation_result` and | ||||||
| `insights/correlation_view.py`). | ||||||
|
|
||||||
| 2. Fetch the PR body from GitHub (`gh pr view 3282 --json body`) and pull | ||||||
| out individually falsifiable sentences — not the whole prose block, each | ||||||
| claim on its own. | ||||||
|
|
||||||
| 3. For each claim, search the resolved session's own `blocks` rows | ||||||
| (`tool_use`/`tool_result`, joined by `tool_id`) for structural evidence: | ||||||
| an exact command, an exact exit code, an exact pytest summary line. A | ||||||
| claim with no matching block is marked **not independently verified in | ||||||
| this session** — never silently upgraded to "supported" because the PR | ||||||
| body asserts it. | ||||||
|
|
||||||
| 4. Render the two columns: claimed sentence | observed block evidence | ||||||
| (drillable via the cited `block:` ref), with an explicit status per row. | ||||||
|
|
||||||
| ## Note on this run | ||||||
|
|
||||||
| This demo's session turned out to be a **merge-conductor** session: its own | ||||||
| `blocks` are almost entirely `Bash` (53 of 56 tool_use blocks) plus 3 `Read` | ||||||
| calls — zero `Edit`/`Write` tool_use. The actual file edits for PR #3282 | ||||||
| happened in separately dispatched worker sessions across multiple git | ||||||
| worktrees (`/realm/worktrees/polylogue-membership-head*`); this session | ||||||
| orchestrates `git`, `gh pr create`, and `devtools test`/`devtools verify` | ||||||
| invocations across those worktrees and stitches the result into one PR. | ||||||
|
|
||||||
| This is itself a real, useful finding, not a inconvenience to hide: the PR | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Fix the article before “inconvenience.” Use “an inconvenience” because Proposed fix-This is itself a real, useful finding, not a inconvenience to hide:
+This is itself a real, useful finding, not an inconvenience to hide:📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||
| body's own Verification section names a 7-file `devtools test` invocation | ||||||
| ("`devtools test tests/unit/sources/test_live_batch_support.py ...` — all | ||||||
| passing, **see individual commit messages for per-commit pass counts**") — | ||||||
| its own parenthetical admits the aggregate command was never run as one | ||||||
| shot. Searching this session's blocks confirms it: the 7-file string only | ||||||
| appears inside the `gh pr create --body` tool_input (i.e. inside the PR body | ||||||
| text itself), never as an actual invoked command. That specific claim is | ||||||
| marked **not independently verified in this session** in `report.md` and | ||||||
| `checks.json` — precisely the honesty discipline this packet exists to | ||||||
| enforce, applied to itself. | ||||||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| {"ref": "artifact:d1-receipts-evidence", "cited_for": "Demo Packet v2 receipt root", "verified_via": "committed evidence.ndjson"} | ||
| {"ref": "session:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc", "cited_for": "session_refs kind=pull_request resolves this session to Sinity/polylogue#3282", "verified_via": "sqlite3 index.db: SELECT session_id,repo,ref_number,url FROM session_refs WHERE kind='pull_request' AND repo='Sinity/polylogue' AND ref_number=3282"} | ||
| {"ref": "block:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc:ae9a4788-4bf6-4b89-b01e-90f10e622981:0", "cited_for": "the gh pr create tool_use --body text is byte-identical to the PR body later fetched live via gh pr view 3282, including the exact bullet claims checked below", "verified_via": "diff of tool_input command against `gh pr view 3282 --repo Sinity/polylogue --json body`"} | ||
| {"ref": "block:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc:a98d94d0-831f-4cf6-946b-40bf99179283:0", "cited_for": "gh pr create tool_result: https://github.com/Sinity/polylogue/pull/3282 -- confirms this session actually opened PR #3282, not merely referenced it", "verified_via": "tool_result text (single URL line)"} | ||
| {"ref": "block:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc:bdacfb31-ecc7-4491-be2f-891f8bfb888b:0", "cited_for": "tool_use invoking `timeout 180 devtools verify --quick`, checking the PR claim 'devtools verify --quick -- pass'", "verified_via": "tool_input command text"} | ||
| {"ref": "block:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc:9786b513-9ad0-4e25-ad3e-e18a0f60220f:0", "cited_for": "tool_result: structured verify-run JSON, every step exit=0, total_duration_s=32.99, exit_code=0 -- SUPPORTS the claim via structure, not a trusted pass/fail word in prose", "verified_via": "tool_result JSON body (per-step name/duration_s/exit array)"} | ||
| {"ref": "block:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc:ebe785a2-de91-4de3-9b28-43bd1a8e9596:0", "cited_for": "tool_use invoking devtools test against tests/unit/maintenance/test_rebuild_index_bulk_build.py (+4 more files), checking the Solution-section claim 'rebuild_index bulk FTS materialization checkpoints progress'", "verified_via": "tool_input command text"} | ||
| {"ref": "block:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc:59a1be2d-7881-4d64-997d-d055dd61aa74:0", "cited_for": "tool_result: pytest summary '123 passed in 8.05s', ok (12.2s) -- SUPPORTS the rebuild_index claim", "verified_via": "tool_result text (pytest summary line)"} | ||
| {"ref": "polylogue-6mvg", "cited_for": "the PR body's own tracking-item reference ('Ref polylogue-6mvg'), cited verbatim inside the gh pr create --body text", "verified_via": "block:...ae9a4788... tool_input"} | ||
| {"ref": "action.tool_use.count=53,Bash", "cited_for": "56 total tool_use blocks in this session: 53 Bash, 3 Read, 0 Edit, 0 Write -- this is a merge-conductor/orchestration session, not the direct file-editing session", "verified_via": "sqlite3 index.db: SELECT tool_name, count(*) FROM blocks WHERE session_id=... AND block_type='tool_use' GROUP BY tool_name"} | ||
| {"ref": "action.negative-control.missing-7-file-devtools-test-invocation", "cited_for": "counterexample: the PR body's exact 7-file `devtools test tests/unit/sources/test_live_batch_support.py ...` string appears ONLY inside the gh-pr-create --body text (i.e. inside the PR body itself), never as an actually-invoked command in this session's blocks -- 0 matching tool_use rows when the gh-pr-create block is excluded", "verified_via": "sqlite3 index.db: SELECT count(*) FROM blocks WHERE session_id=... AND block_type='tool_use' AND tool_input LIKE '%test_live_batch_support.py%' AND tool_input NOT LIKE '%gh pr create%' -- returns 0"} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| archive_cursor: "live-archive:/realm/db/polylogue (read-only, file:...?mode=ro)" | ||
| measure_version: demo-packet-v2 | ||
| commit_sha: 59744a30bf461a587cc679ee62a432e8cd2cf82a | ||
| sample_frame_predicate: "session_refs WHERE kind='pull_request' AND repo='Sinity/polylogue' AND ref_number=3282, resolving to claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc" | ||
| run_date: "2026-07-31" | ||
| claim: "session_refs typed pull_request evidence resolves a real merged PR to its authoring/dispatch session, and specific PR-body verification sentences can be checked against that session's own recorded blocks -- with unsupported claims marked as such, not silently believed" |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| {"text": "SELECT session_id, repo, ref_number, url FROM session_refs WHERE kind='pull_request' AND repo='Sinity/polylogue' AND ref_number=3282", "lowered_spec": {"unit": "session_ref", "predicate_kind": "structural_equality", "table": "session_refs"}} | ||
| {"text": "find \"id:claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc\" then read --view correlation --format json", "lowered_spec": {"unit": "session", "entry": "id", "view": "correlation"}} | ||
| {"text": "SELECT tool_name, count(*) FROM blocks WHERE session_id='claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc' AND block_type='tool_use' GROUP BY tool_name ORDER BY 2 DESC", "lowered_spec": {"unit": "block", "pipeline_stages": ["group:tool_name", "count"]}} | ||
| {"text": "SELECT tu.block_id, tr.text FROM blocks tu JOIN blocks tr ON tr.tool_id=tu.tool_id AND tr.block_type='tool_result' AND tr.session_id=tu.session_id WHERE tu.session_id='claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc' AND tu.tool_input LIKE '%devtools verify --quick%'", "lowered_spec": {"unit": "block", "predicate_kind": "join_tool_use_to_tool_result"}} | ||
| {"text": "SELECT count(*) FROM blocks WHERE session_id='claude-code-session:5ecdb160-495a-4d9b-b80a-3a24886af8cc' AND block_type='tool_use' AND tool_input LIKE '%test_live_batch_support.py%' AND tool_input NOT LIKE '%gh pr create%'", "lowered_spec": {"unit": "block", "predicate_kind": "negative_control_count", "expected_result": 0}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add blank lines around the fenced SQL block.
Markdownlint reports
MD031for this fence. Add a blank line before and after the fence.Proposed fix
📝 Committable suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.23.1)
[warning] 22-22: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
[warning] 26-26: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
🤖 Prompt for AI Agents
Source: Linters/SAST tools