feat(audit): stream audit progress as areas on the headless path - #1263
Conversation
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| if (typeof value !== 'string') return null; | ||
| const trimmed = value.trim(); | ||
| if (trimmed === '') return null; | ||
| return trimmed.slice(0, MAX_AREA_TITLE_LENGTH); |
There was a problem hiding this comment.
Low: Agent-controlled content is sent without security scanning
A malicious repository or downloaded skill can instruct the agent to place readable source or configuration content in a check's area; this function only truncates that value, and TaskStreamPush then sends it to the PostHog sessions endpoint. Audit MCP arguments are not covered by the Write/Edit or publish_handoff output scans. Restrict streamed titles to trusted, predefined area identifiers, or apply the same output scanning and redaction used for other agent-authored outbound content.
PR overviewThis pull request changes audit task progress streaming to report audit areas rather than individual checks. The implementation updates the audit-area task stream payloads sent during an audit. One issue remains open: agent-controlled audit area text can be transmitted to the session analytics endpoint without content scanning or redaction. A malicious repository or downloaded skill could use this path to expose readable source or configuration data, although exploitation depends on untrusted instructions influencing the agent. No issues have yet been addressed. Open issues (1)
Fixed/addressed: 0 · PR risk: 4/10 |
gewenyu99
changed the title
An audit resolved tens of ledger checks and none of it left the machine: `tasks` holds the program's step rows, which a headless or CI run does not have, so a five-minute audit pushed `tasks: []`. Shipping every check would be noise, and each row carries a file path and a details string derived from the user's code. - `rollUpAuditAreas` folds the ledger into one row per audit area, appended after the program rows. No wire-schema change, so no backend change. A finding is progress, not a failure, so an area never reports `failed`. - `startAuditLedgerWatcher` mirrors the ledger into the session through the UI's frameworkContext seam, owned by `runAgent` so every path gets it, and started before the run recipe seeds the ledger. `HeadlessUI` tees that setter into the store, without which a headless run mirrors nothing. - pi gets native mirrors of the three `audit_*` tools. They existed only on the MCP facade, and pi mounts no MCP server, so an audit on the harness the switchboard binds it to could not move a check off pending. Descriptions and write semantics move to shared helpers so the facades cannot drift. - `--task-stream-log` dumps every attempted sync as JSONL. `--ci` always dumps and never pushes, since a synthetic run would create a session row in a real project. Consent now gates the push, not the dump. - Seeds `init-not-duplicated`, which the skill resolves in a batch with `init-correct`; a batch resolve rejects atomically, so its absence discarded both. - The audit program declares an e2e profile, and the snapshot harness signs framework-context by value so a screen rerendering from the ledger is captured more than once. Generated-By: PostHog Desktop Task-Id: 6490a5fd-5ee9-4203-b9c4-0b4ff180c45b
2a8d3af to
35bc8b3
Compare
2 participants
Audit runs publish live state that nothing could see. This makes the run legible — in the terminal, in a local dump, and on the wire — and fixes the two defects that made the ledger inert.
What was wrong
TaskStreamUpdate.tasksholds the program's step rows, which a headless or CI run does not have. A five-minute audit pushedtasks: [].filepath and adetailsstring derived from the user's code.audit_*tools existed only on the MCP facade, and pi mounts no MCP server. The switchboard binds audit toharness=pi, so the agent searched, found nothing, and completed the audit without resolving a single check. The Audit plan tab sat at 13 pending rows for the whole run and the report was composed from the agent's own notes instead of the ledger it documents as its source of truth.audit_resolve_checksrejects a batch atomically, so the absentinit-not-duplicateddiscardedinit-correctwith it and cost a retry.What this does
Area rollup.
rollUpAuditAreasfolds the ledger into one task row per audit area, in seed order, and those rows extend the program rows intaskswhile continuing their integer ids. No wire-schema change, so no backend change. A finding is progress, not a failure:error,warning, andsuggestionall resolve an area tocompleted, and an area never reportsfailed. Check labels,file, anddetailsnever leave the machine.One ledger reader.
startAuditLedgerWatchermirrors.posthog-audit-checks.jsoninto the session throughgetUI().setFrameworkContext, andrunAgentowns the lifecycle, so the interactive runner, the headless runner, and the e2e host all get it. It starts before the run recipe resolves, because an audit seeds the ledger from insiderun().HeadlessUInow tees that setter into the store; without it a headless run mirrors nothing.pi ledger tools. Native mirrors of
audit_seed_checks/audit_add_checks/audit_resolve_checks, opt-in per program like the skill-menu pair. Descriptions, write semantics, and error strings moved into shared helpers so the two facades cannot drift —mcp.tsgot 105 lines lighter. They register asparallelbehind the ledger mutex, which is what lets the skill fan checks out across subagents without losing an update.Destinations follow the surface.
--task-stream-log--task-stream-log--ci/tmp/posthog-wizard-task-stream.jsonlA CI run is synthetic, so a push would create a session row in a real project. The dump records every attempted sync, one JSON line per push, truncated per run. Dev and CI builds only:
createFileDestinationreturns null in a published build and the flag is undeclared there. Telemetry consent now governs the PostHog destination only, so--no-telemetrywith a dump still logs.Harness sampling.
tui-snapshotssigned framework-context by key, so a screen that rerenders from an artifact updated in place was captured once, empty. It digests the values now.Verified on a real run
PROGRAM=auditthrough the e2e host against a Next.js app with PostHog already installed, project 228144, real gateway:unknown check id(s): init-not-duplicated)pending → in_progress → completedgrep -c '"details"\|"file"' → 0Installation (4/4)and per-check glyphsrunPhase: completed, notebook uploaded, skills cleaned uppnpm test— 174 files, 2804 tests. 20 new tests cover the rollup, the destination, the pi tools (seed/resolve/append, unknown-id refusal, duplicate refusal, concurrent fan-out), the per-program gate, and the seed invariant.Known gap, outside this repo
The app subscribes per
workflow_id, andwizardActiveSessionDetectorLogicwatchesposthog-integrationplus conditionallyself-driving. Anauditsession is stored and streamed correctly — writes are confirmed201/200, no allowlist, flag, or team gate drops it, and task titles render generically — but no surface watches that channel. Making it visible is two lines inposthog: anAUDIT_WORKFLOW_IDconstant inwizard-sync/workflows.tsand that id in the detector's base watched set.Separately, the narrow audit leaves push
workflow_id: 'agent-skill', because that is the config id behindwizard audit events. Onlyaudit allpushesaudit.Created with PostHog Desktop