test: establish a focused process and persistence integration suite

[Flare576]

Problem

Ei has core Vitest tests, TUI unit tests, browser Playwright E2E, and PTY-based TUI E2E, but no first-class integration-test lane. The slow Claude Code/Codex/Cursor installer files are already integration-style tests embedded in root Vitest; they install generated artifacts into a temporary HOME and run those artifacts in real Bun subprocesses. They should remain covered, but merely moving them from tests/unit/cli to tests/integration/cli would be a directory rename: root Vitest currently includes every tests/**/*.test.ts file.

This is distinct from #47. #47 covers the UI → mocked LLM → extraction/ceremony chain. This issue covers host-process, filesystem persistence, external-source, and remote-sync contracts that neither browser nor TUI E2E can faithfully own.

Decision sought

Create a focused integration-test lane. It must test only seams where real process, generated-artifact, filesystem, source-schema, or remote-API behavior changes the outcome. It must not become a duplicate E2E suite.

Existing coverage to keep where it is

• Claude Code, Codex, Cursor, Pi, and OMP generated installer artifacts already execute against fake ei/bunx binaries and real filesystem/Bun runtime under root Vitest.
• Web Playwright already owns browser ↔ IndexedDB/localStorage, mock LLM/ComfyUI, chat, rooms, and UI persistence flows.
• TUI PTY E2E already owns terminal interaction, editor subprocesses, rooms, onboarding, upgrade flows, and mock-LLM UI behavior.

Neither existing E2E runner can host an external harness hook protocol, two independently owned filesystem processes, or a deterministic external-source schema fixture.

Recommended lane contract

1. Add a named npm run test:integration command and a dedicated discovery boundary such as tests/integration/**; do not rely on a directory name while root Vitest still glob-includes it.
2. Provide isolated temporary HOME and EI_DATA_PATH, real filesystem storage, bounded child-process lifecycle/cleanup, and deterministic local fakes only. No live LLM, OAuth provider, harness CLI, or production sync API.
3. Permit the lane to launch real Bun-generated artifacts and multiple local processes when that is the contract under test.
4. Keep ordinary pure logic in unit tests and visible browser/terminal user journeys in their existing E2E suites.
5. Decide explicitly whether this lane gates release/deploy. Recommendation: make it a required CI gate once the initial P0 cases are stable; include it in test:all at that point.

Initial coverage backlog

P0 — live corrections drain across independent processes

Start a real filesystem-backed Processor that holds ei.lock; invoke a CLI/MCP correction writer from another process; then observe the live processor loop drain corrections.json.

Oracle: a valid correction appears in durable state only after the processor drain, the queue becomes [], and a malformed record is isolated while a following valid record still applies. Existing tests simulate this with same-process/self-drain behavior; they do not prove the lock-owner/process-loop contract.

P0 — source reader → importer → persisted Ei state

Seed real external transcript fixtures (at minimum Pi and Claude Code JSONL; add OpenCode JSON and Bun/SQLite Codex where practical), execute the real reader and importer against real storage, then reload state.

Oracle: qualified message IDs, roles, persona attribution, external replacement semantics, extraction-point/processed-session state, and queued-scan source IDs survive a reload. Current reader and importer tests mostly use separate doubles at this seam.

P1 — document import and FileStorage persistence lifecycle

Drive Processor.importDocument() through queue serialization, segmentation handling, document-batch completion, and a second storage load. Cover the FileStorage save/load and backup/restore behavior that this pipeline relies on.

Oracle: Emmett-attributed segment content and provenance, processed_documents entry, queued chunk metadata, and recovered persisted state are exact after reload/backup restoration.

P1 — generated OpenCode plugin host contract

Install the OpenCode plugin, load the generated module in Bun, and invoke its experimental.chat.system.transform callback through a minimal exact host fixture.

Oracle: supported quoted/unquoted agent names inject one relationship block, malformed prompts are no-ops, repeated calls do not duplicate injection or re-run the CLI fetch for a cached persona, and the local ei → bunx fallback remains observable. This generated plugin has no execution coverage today.

P1 — remote sync client ↔ local API contract

Run RemoteSync against a local API fixture rather than the production endpoint. This may first require making the API base injectable.

Oracle: encrypted/compressed payload only crosses the wire; HEAD/GET/POST and ETag/412 conflict sequencing are correct; restored state matches the source; retry/error paths remain honest. Current browser tests cover local backup UI but not remote sync transport or conflict resolution.

P2 — concurrent ownership, startup scheduling, and restart durability

Start two isolated app/process instances against one data path; add focused restart assertions for a state-changing command; and run one boot with real tool bootstrap, short autosave, and overdue ceremony/heartbeat settings.

Oracle: the second owner receives the documented lock outcome, state JSON remains valid, and no message/update is silently lost or duplicated; a fresh process observes the expected persisted result. The background-work case records the expected mock-LLM request types/order, drains its queue, persists last_ceremony/last_heartbeat and tool providers, and does not duplicate work after restart. Current TUI fixtures intentionally disable this startup work, so their PTY coverage does not exercise it.

Non-goals

• Do not move existing installer tests solely for taxonomy or to hide their runtime.
• Do not duplicate browser flows in a Node/Bun process suite.
• Do not call real third-party APIs, spend money, or depend on a developer's local LLM/harness installation.
• Do not subsume feat: E2E tests for ceremony pipeline (dedup, rewrite, validate) #47's extraction/ceremony browser E2E work.

Related existing-suite work (not new integration-lane work)

Add deterministic Web E2E coverage for successful local-provider onboarding, document import/generation/unsource/download, targeted capture/knowledge-search results, and OAuth callback state handling. Add TUI PTY assertions for persisted state after restart. These have valid existing runner homes.

Acceptance criteria

• A dedicated discovery boundary and test:integration command exist.
• The runner creates and cleans isolated data paths and terminates all children on failure.
• P0 correction-drain and reader→importer→reload contracts pass with deterministic fixtures.
• At least one P1 generated-host or document/FileStorage contract passes before the lane is release-gating.
• CI policy is explicit: required gate or intentionally non-gating, with rationale recorded.
• Existing installer tests remain protected by an equivalent or stronger runtime oracle.