Turn TaskTape into an agent-operated debugging instrument

.github/workflows/ci.yml

@@ -18,5 +18,6 @@ jobs:
           node-version: 22
           cache: pnpm
       - run: pnpm install --frozen-lockfile
+      - run: pnpm exec playwright install chromium
       - run: pnpm check
       - run: pnpm test:e2e

README.md

@@ -1,43 +1,61 @@
 # TaskTape
 
-**Show your work once. Replay it safely.**
+**Turn a narrated bug recording into a living regression check.**
 
-TaskTape is a desktop automation builder that learns from a recorded demonstration and the user's own words. It turns both into an editable workflow recipe that can be reviewed, scheduled, and executed.
+TaskTape Replay is a macOS desktop tool for teaching repeatable regression checks. Record the bug, explain the expected result, and TaskTape builds an editable check that can be run again after the product changes.
+
+Claude Code and Codex can also operate TaskTape directly through its built-in local MCP server. An agent can launch an instrumented browser, reproduce a bug, collect screenshots, DOM snapshots, console messages, network failures, and an action trace, then compile that evidence into the same Replay check a human can review and run.
 
 ## Current status
 
-TaskTape turns a recording into a saved task with editable instructions, an execution capability, and an optional schedule. A user can type or dictate the intended result, review what TaskTape learned, edit the goal and instructions, run it now, or choose hourly, daily, weekday, or weekly timing. Manual and scheduled results appear in Run history.
+The Build Week reference flow records a browser bug, captures the expected outcome by voice or text, replays the interaction with OpenAI computer use, and judges the final screen against that outcome. Each run stores a passed, failed, or inconclusive result with the final screenshot and a concise evidence trail.
+
+The agent-operated reference flow is limited to local HTTP and HTTPS development servers. TaskTape creates a headed isolated Chrome session and exposes 12 bounded MCP tools for observation, interaction, evidence capture, check creation, and explicit execution. Finishing a capture never runs or schedules the check.
+
+Checks have editable instructions and can run now or on hourly, daily, weekday, or weekly timing. Manual and scheduled results appear in Run history, so a team can see when a previously fixed behavior regresses.
 
 A spoken schedule such as "every Monday at 9 AM" prefills the save controls. It never becomes active until the user confirms unattended execution and saves the task. The Scheduled view shows the next run, last result, pause or resume, and Run now controls.
 
 Recording starts with TaskTape's visual source gallery, which shows full displays and every currently available application window as named thumbnail tiles. The selected source is revalidated in Electron's main process immediately before capture.
 
 Users can add or replace their own OpenAI API key from Settings. App-managed keys are encrypted through the operating system's secure storage and never exposed back to the renderer; `.env.local` remains a development-only fallback.
 
-The initial release is macOS-first and is being built for OpenAI Build Week. The product vision is broader than the hackathon implementation, but the demo will prove one complete, reliable workflow rather than simulate universal desktop control.
+The initial release is macOS-first and is being built for OpenAI Build Week. The hackathon demo proves one complete browser regression loop deeply. It does not claim broad, deterministic automation across every desktop application.
 
 The reliable local-file capability learns extension groups and child-folder destinations from the demonstration, then organizes top-level files inside one approved folder. The user does not configure separate video or image fields. Tasks that need an application or website use OpenAI computer use through a native macOS input adapter. Model safety checks stop the run for review, and the agent is bounded to 25 turns. Scheduled runs require TaskTape to be open and the Mac to be awake. Background launch, history search, rollback, and safety-check resumption are not yet implemented.
 
 ## Product loop
 
-1. Record a real desktop workflow.
-2. Describe the result by voice or text.
-3. Review the inferred goal, learned actions, and schedule.
-4. Save the task and confirm any recurring execution.
-5. Run it now or let TaskTape run it on schedule while the app is open.
-6. Inspect the result, run history, or start another workflow.
+1. Record the bug yourself or ask Claude Code or Codex to reproduce it through TaskTape.
+2. Capture the expected result by voice, text, or the MCP session request.
+3. Review the learned replay steps, trace evidence, and success condition.
+4. Save the check and choose whether it should run on a schedule.
+5. Replay it against the current build.
+6. Inspect the verdict, final screenshot, evidence, and run history.
+
+## Agent connection
+
+TaskTape shows the current local endpoint and copyable setup commands in Settings. With the app open, the default commands are:
+
+```bash
+claude mcp add --transport http tasktape http://127.0.0.1:19790/mcp
+codex mcp add tasktape --url http://127.0.0.1:19790/mcp
+```
+
+Ask the connected agent to reproduce a bug on a local development URL and turn it into a Replay check. Agent evidence is stored under TaskTape's local application data, never in the repository.
 
 ## Documentation
 
 - [Product brief](docs/product-brief.md)
 - [Architecture](docs/architecture.md)
+- [Agent-operated replay milestone](docs/agent-mcp-milestone.md)
 - [Voice and scheduling research](docs/voice-and-scheduling-research.md)
 - [Milestone roadmap](docs/roadmap.md)
 - [Verification log](docs/verification.md)
 
 ## Development
 
-Prerequisites: Node.js 22+, pnpm 11.7.0, Xcode Command Line Tools, and macOS for desktop capture and computer-control verification.
+Prerequisites: Node.js 22+, pnpm 11.7.0, Xcode Command Line Tools, Google Chrome, and macOS for desktop capture, browser instrumentation, and computer-control verification.
 
 ```bash
 cd /Users/rohitpurkait/Documents/codex_build_week

docs/agent-mcp-milestone.md

@@ -0,0 +1,74 @@
+# Agent-operated replay milestone
+
+## Product promise
+
+TaskTape is a debugging instrument that records itself while an agent reproduces a bug. Claude Code, Codex, or another MCP client can operate a headed browser through TaskTape, collect synchronized evidence, and turn the completed reproduction into a saved Replay check.
+
+The first release proves this deeply for browser bugs. General desktop observation remains a later capability.
+
+## Palmier reference
+
+Palmier demonstrates the interaction model TaskTape will follow:
+
+- The desktop application owns the canonical project state.
+- A local MCP server exposes native project operations to external agents.
+- The server binds only to the loopback interface.
+- The same tools and state power both the application and external clients.
+- The application shows direct setup instructions for Claude Code and Codex.
+
+TaskTape independently implements these ideas for debugging under its MIT license. Palmier's GPL implementation is an architectural reference only.
+
+## MCP boundary
+
+The embedded server listens on `127.0.0.1` and rejects non-local browser origins. It exposes compact debugging operations instead of unrestricted shell or filesystem access.
+
+Initial tools:
+
+- `start_bug_session`: launch a headed instrumented browser at an HTTP or HTTPS URL and begin trace capture.
+- `observe_page`: return the current URL, title, concise accessibility snapshot, and screenshot.
+- `click`: click an element selected by accessible role and name, label, text, or CSS.
+- `fill`: fill a labeled or CSS-selected input.
+- `select_option`: choose an option in a labeled or CSS-selected select.
+- `press_key`: send one validated key chord.
+- `wait_for`: wait for visible text or a bounded duration.
+- `add_note`: attach agent reasoning or issue context without executing it.
+- `finish_bug_session`: stop capture, persist evidence, and compile the actions into a saved computer Replay check.
+- `run_check`: run a saved check through TaskTape's existing computer-use and visual-verification pipeline.
+- `get_run_result`: retrieve a persisted verdict and evidence.
+- `list_checks`: list saved computer Replay checks.
+
+## Evidence bundle
+
+Each session remains local and contains:
+
+- Session metadata and expected outcome.
+- Ordered agent actions and notes.
+- Initial and final screenshots.
+- Playwright trace with DOM snapshots, screenshots, sources, and action timing.
+- Console messages.
+- Failed requests and HTTP error responses.
+- Compiled replay instructions and saved workflow identifier.
+
+The model receives compact summaries and resource links. Large recordings and traces are not inserted into the conversation.
+
+## Safety
+
+- Only one active browser session is allowed.
+- URLs must use HTTP or HTTPS, cannot contain credentials, and are limited to loopback development servers for this milestone.
+- The browser profile is temporary and isolated.
+- Tool arguments are schema-validated and text lengths are bounded.
+- Browser actions are limited to the TaskTape-created page.
+- Finishing a session saves a check but does not run or schedule it.
+- Running a saved check remains an explicit separate tool call.
+- The server shuts down with TaskTape and is never exposed to the LAN.
+
+## Exit gate
+
+The milestone is complete only when all of the following pass:
+
+1. Unit tests cover schemas, URL restrictions, selectors, action compilation, persistence, and cleanup.
+2. An MCP client discovers the server and its tools through the protocol.
+3. An end-to-end MCP journey launches the disposable broken browser target, observes it, clicks Save asset, records the Uncategorized result, and finishes the session.
+4. The evidence directory contains valid metadata, initial and final screenshots, a non-empty Playwright trace, and the captured console or network records.
+5. The compiled workflow persists the expected outcome and replay instructions and appears in TaskTape.
+6. Existing unit, Electron, live Replay, and packaging gates remain green, with paid live tests run only once after deterministic checks pass.

docs/architecture.md

@@ -15,6 +15,22 @@ Electron is used because its desktop capture and process isolation APIs let the
 
 `contextIsolation`, renderer sandboxing, and disabled Node integration are mandatory. New IPC operations require a typed contract and input validation.
 
+## Reference product path
+
+The Build Week product is TaskTape Replay: a narrated bug recording becomes a reusable regression check. The reference target is a disposable browser application with a broken and a fixed mode. This keeps the demo observable and repeatable while exercising the same capture, analysis, computer-use, evidence, scheduling, and history boundaries used by the wider product.
+
+TaskTape supports two inputs into that shared model. A person can record a screen or window and describe the intended result, or an external agent can reproduce a bug through TaskTape's MCP browser tools. Both paths produce a versioned computer Replay check with editable instructions and an expected outcome.
+
+## Agent connection
+
+The Electron main process starts an MCP Streamable HTTP server on `127.0.0.1:19790`. It uses the official TypeScript MCP SDK and the SDK's localhost host validation to protect against DNS rebinding. Requests with non-local browser origins are rejected. Automated Electron runs bind an ephemeral loopback port to avoid test collisions.
+
+The MCP server and desktop renderer share the same browser evidence manager, workflow persistence, execution boundary, and run history. The renderer never hosts the server and receives only a narrow status object through preload IPC. Claude Code and Codex connection commands are shown in Settings, following Palmier's pattern of keeping setup beside the canonical local application state.
+
+The first MCP capability intentionally targets local web applications. `start_bug_session` accepts only loopback HTTP or HTTPS URLs, launches a temporary headed Chrome context, starts Playwright tracing, and allows one active session. Accessible role, label, text, or CSS selectors drive bounded click, fill, select, key, and wait operations. Password fields are refused.
+
+Every operation stores an ordered action plus a screenshot. The session also records console messages, page errors, failed requests, HTTP error responses, initial and final screenshots, and a Playwright trace containing DOM snapshots and action timing. Finishing compiles the recorded actions into a review-required computer workflow and closes the temporary browser. Running or scheduling remains a separate explicit action.
+
 ## Capture source selection
 
 TaskTape lists full displays and open application windows through Electron `desktopCapturer`, then renders a grouped thumbnail gallery in the sandboxed UI. A source ID selected by the user is validated against a freshly enumerated main-process source list. The display-media handler consumes that one pending selection and rejects requests without one; the renderer cannot nominate an arbitrary capture target.
@@ -39,13 +55,19 @@ The deterministic filesystem adapter supports inspect, classify, create-director
 
 The file recipe scans regular files at the top level of one user-selected folder. It contains a dynamic list of model-inferred file groups, extension matchers, and child-folder destinations. This supports demonstrated structures such as footage, documents, archives, or project assets without hard-coded video and image fields. It creates only schema-valid child folders and either moves or copies with exclusive destination creation. Unmatched files and collisions stay unchanged. A persisted plan records file size and modification time; execution refuses an action if the source changed after review.
 
-The computer agent activates the saved target application, requests actions from `gpt-5.6`, executes supported mouse and keyboard events through pinned `@nut-tree-fork/nut-js`, and returns a screenshot after each action batch. Running input inside TaskTape gives macOS a stable Accessibility identity. Coordinates, key names, drag paths, typed-text length, provider request time, and turn count are bounded. On-screen text is treated as untrusted content. Any pending model safety check stops execution before its actions run.
+The computer agent activates the saved target application, requests actions from `gpt-5.6`, executes supported mouse and keyboard events through pinned `@nut-tree-fork/nut-js`, and returns a screenshot after each action batch. Running input inside TaskTape gives macOS a stable Accessibility identity. Coordinates, key names, drag paths, typed-text length, 60-second provider requests, and a 25-turn limit are bounded. On-screen text is treated as untrusted content. Any pending model safety check stops execution before its actions run.
+
+## Outcome verification
+
+Computer checks persist a plain-language expected outcome. After replay, TaskTape sends the final screenshot and that condition to a separate schema-bound visual evaluator. The evaluator returns `passed`, `failed`, or `inconclusive`, plus a short summary and evidence list. The run log stores that verdict and final screenshot so completion and history can explain what was observed instead of treating successful mouse clicks as proof that the product worked.
+
+Execution and evaluation remain separate boundaries. A completed interaction can still produce a failed regression check. Missing or ambiguous visual evidence produces an inconclusive result rather than a fabricated pass.
 
 Version 1 and version 2 file recipes are migrated on read into equivalent version 3 tasks. The source-folder permission boundary and deterministic executor remain unchanged during migration.
 
 ## Persistence
 
-Recordings, versioned workflow recipes, schedules, approved plans, and activity logs use filesystem-backed local metadata. Workflow JSON files are written with mode `0600`. The history view reads immutable run logs across saved workflows. SQLite remains deferred until history search or larger run volumes require indexed, transactional state.
+Recordings, agent evidence sessions, versioned workflow recipes, schedules, approved plans, and activity logs use filesystem-backed local metadata. Workflow and agent-session JSON files are written with mode `0600`. Browser traces and screenshots remain inside the local application-data directory. The history view reads immutable run logs across saved workflows. SQLite remains deferred until history search or larger run volumes require indexed, transactional state.
 
 ## Scheduling
 
@@ -58,6 +80,7 @@ The current scheduler runs only while TaskTape is open. Operating-system backgro
 - API credentials remain in the main process and are never exposed back to the renderer. Keys entered in Settings are encrypted through Electron `safeStorage`, persisted with mode `0600`, and take precedence over the development-only environment fallback.
 - The main process allows only trusted-renderer display capture and microphone requests. Electron 43 on macOS reports `getDisplayMedia` as a media request with no camera or microphone type, while microphone requests contain only `audio`; camera-bearing requests remain denied.
 - Recordings and extracted frames are ignored by Git and local by default.
+- The MCP server is loopback-only, validates host and origin boundaries, and limits browser sessions to local development URLs without embedded credentials.
 - Manual actions require explicit review and approval. Scheduled actions require separate unattended-run consent. File tasks remain limited to the saved folder and collision-safe executor. Computer tasks stop on model safety checks. Rollback and safety-check resumption are not yet implemented.
 - External links are opened through the operating system after the application denies new in-app windows.
 
@@ -68,3 +91,5 @@ The current scheduler runs only while TaskTape is open. Operating-system backgro
 - Playwright's Electron support for packaged user journeys.
 - Manual macOS verification for screen-recording and microphone permission behavior.
 - Live OpenAI tests labeled separately from deterministic mocked tests.
+- A real SDK MCP client for protocol discovery and tool invocation.
+- A packaged-app MCP journey that captures the broken browser fixture and persists a visible Replay check.