feat(agui): AG-UI protocol adapter + generative UI (L1, default-off) — supersedes #387

[plauzy]

AG-UI protocol adapter + generative UI (Phase-A)

Implements the AG-UI L1 adapter proposed in #386: a thin, default-off surface that maps CAO's normalized fleet events onto AG-UI typed SSE events, plus a generative-UI producer path. Supersedes #387.

Every new surface is additive and default-off — with the flags unset, the existing localhost dashboard, MCP servers, and HTTP API are byte-identical.

What ships here vs. follow-ups (issue #386)

Layer / concern	This PR	Follow-up
L1 AG-UI SSE adapter — /agui/v1/stream, typed events, STATE_SNAPSHOT/STATE_DELTA, ?since= replay	✅	—
L1 Generative-UI producer — emit_ui, server-validated frozen allow-list	✅	—
OpenTelemetry dispatch instrumentation — execute_tool span + cao.orchestration.dispatches counter + traceparent (opt-in)	✅	—
mock_cli credentials-free provider (deterministic CI)	✅	—
L2 Dependency-free EventSource viewer example (+ demo recording)	✅	—
L2 A2A transport + signed Agent Card	—	stacked PR (in review)
L3 Standalone dashboard PWA / stock-client interop demo (+ recording)	—	follow-up PR
CI devcontainer	—	follow-up PR

• L1 AG-UI adapter + generative UI
• OpenTelemetry dispatch instrumentation
• mock_cli provider
• Dependency-free EventSource viewer example + demo recording (this PR)
• L2 A2A transport (stacked follow-up)
• L3 standalone dashboard PWA / stock-client interop demo + recording (follow-up)
• CI devcontainer (follow-up)

On the demo videos: this PR originally bundled the standalone PWA and its walkthrough; per review that was split out so this stays a reviewable L1 adapter. The more meaningful proof for an adapter is a stock client rendering this stream, so this PR now ships that directly: a dependency-free EventSource viewer (below, "what is shipped now"). The heavier stock-client interop (React + @ag-ui/client/CopilotKit through a run-endpoint shim) is previewed below under "coming in a future phase" and ships with the integrations/ follow-up (#386).

Demo recordings

What is shipped now — dependency-free EventSource viewer (Path A). A stock
browser EventSource client — one HTML file, no npm/framework/build — renders
this PR's /agui/v1/stream: the fleet projection (STATE_SNAPSHOT +
RFC-6902 STATE_DELTA) and all six allow-listed generative-UI components,
driven live by examples/agui-dashboard/showcase.sh. Off-list components render
an inert placeholder (never executed). Source: examples/agui-eventsource-viewer/.

This GIF is generated by the build, not hand-recorded — the AG-UI demo (shift-left recording) CI job runs examples/agui-eventsource-viewer/tools/record-demo.mjs, which drives the live viewer, asserts each component renders and the off-list component is refused (CI fails on drift), and exports the GIF.

Coming in a future phase — stock AG-UI client interop (Path B). A minimal
React + @ag-ui/client (CopilotKit-style) app renders the same stream through a
thin run-endpoint shim that bridges AG-UI's per-run RunAgentInput POST to
CAO's one-way ambient fleet stream (and rewrites CAO's named SSE frames into
the stock AG-UI wire format). This is the in-repo seed of the
integrations/cli-agent-orchestrator package proposed in #386 and ships in a
follow-up PR.

Preview only — the Path B app (React + @ag-ui/client + run-endpoint shim) ships in the integrations/cli-agent-orchestrator follow-up (#386); this GIF is a walkthrough of that work-in-progress.

Scope

• AG-UI streaming — GET /agui/v1/stream maps CAO's normalized events to AG-UI typed events (STATE_SNAPSHOT on connect, RFC-6902 STATE_DELTA, ?since= replay), behind CAO_AGUI_ENABLED + scope auth. The generator registers its live subscription before replaying history and dedupes the overlap by event id (with an SSE id: cursor), so a ?since= reconnect resumes with neither a gap nor a duplicate. Metadata-only: message bodies never travel on the wire.
• Generative UI — agents author UI via a frozen, server-validated component allow-list (emit_ui / POST /agui/v1/emit_ui); off-list components and oversized/non-serializable props are refused server-side (no HTML/script/eval).
• OpenTelemetry (opt-in, [otel] extra) — the inter-agent dispatch seam emits a GenAI execute_tool span and a cao.orchestration.dispatches counter, and propagates W3C traceparent into plugin events; base install degrades to no-ops.
• Credentials-free mock_cli provider for deterministic CI.

Testing

• Backend — AG-UI stream mapping, the streaming generator (snapshot + replay dedup + STATE_DELTA), emit_ui validation/refusal, enablement gating (route and publisher share one helper), auth hardening, and telemetry instrumentation (test/api/test_agui_*, test/services/test_agui_stream_mapping.py, test/telemetry/*). Full non-e2e suite green; black / isort / mypy clean.
• Live proof — examples/agui-dashboard/showcase.sh gates on the six GENERATIVE_UI frames arriving on the live SSE stream (doubles as a deployment smoke test; non-zero exit on any miss).

Live evidence (Option 1 — the L1 wire, credentials-free)

examples/agui-dashboard/run.sh + showcase.sh against a live cao-server (CAO_AGUI_ENABLED=1) — no tokens, no tmux:

live showcase transcript

[showcase] emitting the six allow-listed components:
  approval_card  -> HTTP 200  {"ok":true,"event_id":"eb49c65b-…","component":"approval_card"}
  choice_prompt  -> HTTP 200  {"ok":true,"event_id":"a9fb0ac3-…","component":"choice_prompt"}
  diff_summary   -> HTTP 200  {"ok":true,"event_id":"eefb8a67-…","component":"diff_summary"}
  progress       -> HTTP 200  {"ok":true,"event_id":"59bd036e-…","component":"progress"}
  metric         -> HTTP 200  {"ok":true,"event_id":"0ec20e09-…","component":"metric"}
  agent_card     -> HTTP 200  {"ok":true,"event_id":"a45bfb4d-…","component":"agent_card"}
[showcase] emitting an OFF-LIST component (must be refused 400):
  iframe         -> HTTP 400  {"detail":"Unknown UI component 'iframe'. Allowed: ['agent_card', 'approval_card', 'choice_prompt', 'diff_summary', 'metric', 'progress']"}

[showcase] AG-UI frames captured from http://localhost:9891/agui/v1/stream:
event: STATE_SNAPSHOT
event: GENERATIVE_UI
event: GENERATIVE_UI
event: GENERATIVE_UI
event: GENERATIVE_UI
event: GENERATIVE_UI
event: GENERATIVE_UI

[showcase] PASS: 6 components accepted (HTTP 200), iframe refused (HTTP 400), 6 GENERATIVE_UI frames on the live stream.

Reproduce:

./examples/agui-dashboard/run.sh          # starts cao-server with the AG-UI surface on
./examples/agui-dashboard/showcase.sh     # drives emit_ui + gates on the live frames

Implements the AG-UI L1 adapter proposed in #386.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@673b2d0). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #436   +/-   ##
=======================================
  Coverage        ?   89.35%           
=======================================
  Files           ?      156           
  Lines           ?    18510           
  Branches        ?        0           
=======================================
  Hits            ?    16539           
  Misses          ?     1971           
  Partials        ?        0           

Flag	Coverage Δ
unittests	89.35% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[fanhongy]

PR #436 Code Review

PR: #436
Reviewed commit: 94b63a48832bc19b3d0a392ca12439a82da43b77

Must Fix

1. CAO_AGUI_ENABLED exposes a stream that never receives lifecycle events

Location: src/cli_agent_orchestrator/plugins/builtin/event_log_publisher.py:84

api.main._agui_enabled() accepts either CAO_AGUI_ENABLED or
CAO_MCP_APPS_ENABLED, but EventLogPublisher._emit() only checks
apps.enabled. The documented quick start uses only CAO_AGUI_ENABLED=true,
so session, terminal, and handoff events are discarded before reaching the
AG-UI bus.

Reproduced on the PR:

agui_route_enabled=True
lifecycle_publisher_enabled=False

Fix: use one shared surface-enablement helper for both the API and publisher,
or make the publisher gate true when either flag is enabled. Add an integration
test that enables only CAO_AGUI_ENABLED, emits a real lifecycle hook, and
observes the corresponding AG-UI frame.

2. The PWA discards the authoritative state channel

Location: cao_pwa/src/components/InstanceTab.tsx:73

The server emits STATE_SNAPSHOT on every connection and STATE_DELTA
after changes, but the reducer has no branch for either event and returns the
existing state. Consequently, sessions and terminals that predate the browser
connection never appear. Switching tabs also remounts the component and loses
all locally reduced state.

Fix: retain a typed dashboard snapshot in reducer state, hydrate it from
data.snapshot, and apply RFC 6902 patches from data.delta with a proven JSON
Patch library. Derive the visible session/terminal maps from that state. Add a
component test that starts from a non-empty snapshot and then applies a delta.

3. Replay has an event-loss window before live subscription

Location: src/cli_agent_orchestrator/api/main.py:919

The generator reads history, builds/yields a fleet snapshot, and only subscribes
to the live bus at line 936. Any event published after history() returns but
before subscribe() registers is absent from both sources and is permanently
lost. This contradicts the documented "resumes without a gap" contract.
Queue overflow has the same silent-gap problem because the connection remains
open and the client has no sequence-gap signal.

Fix: register a live subscription before taking the replay snapshot, then
replay history and drain queued events with event-id deduplication. Add a
monotonic cursor or SSE id: field; on queue overflow, signal/close the stream
so the client is forced through replay. Add a concurrency test that publishes
during the history/live handoff.

4. OpenTelemetry is initialized but no production operation is instrumented

Locations: src/cli_agent_orchestrator/api/main.py:456,
src/cli_agent_orchestrator/telemetry/otel.py:88,
docs/otel-deployment.md:120

The PR installs exporters and defines span helpers, but searches under src/
find only helper definitions. No handoff, assign, send-message, provider, or
workflow path calls invoke_agent_span, execute_tool_span, or chat_span.
No metric instrument is created either. Enabling telemetry therefore exports no
application spans or metrics, despite the changelog and deployment guide saying
those dispatches emit GenAI telemetry.

Fix: instrument the actual orchestration boundaries and propagate
traceparent when constructing plugin events. Create and update the promised
metrics, or narrow the feature/docs to traces only. Add an integration test that
executes one dispatch through a real service seam and asserts an exported span,
instead of testing helper functions in isolation.

5. A clean default development install cannot collect the new tests

Locations: pyproject.toml:58, test/telemetry/conftest.py:12

The OTel SDK is correctly optional, but test/telemetry/conftest.py
unconditionally imports it while the dev dependency group does not install the
otel extra. In a fresh PR worktree, uv run pytest ... failed during
collection with:

ModuleNotFoundError: No module named 'opentelemetry.sdk'

The same tests pass only after uv sync --extra otel.

Fix: include the OTel SDK/exporter in the dev dependency group, or make the
SDK-specific suite skip cleanly when the extra is absent and add a separate
extra-enabled CI job. Keep the existing subprocess test for the base-install
fallback.

Suggested Fixes

6. Token redaction depends on using the cao-server wrapper

Location: src/cli_agent_orchestrator/api/main.py:2943

The access-log filter is installed only in main(). Running the valid ASGI form
uvicorn cli_agent_orchestrator.api.main:app bypasses it and logs the
access_token query parameter. Install the filter during app setup/lifespan,
before requests are accepted, and test the imported-app deployment path.

7. The new devcontainer commands target a nonexistent root JS project

Locations: .devcontainer/devcontainer.json:19,
.devcontainer/test-ci.sh:17

The repository root has no package.json or Bun lockfile, yet both files run
bun install and root-level bun run lint/typecheck/build/test. The optional
E2E path also references nonexistent packages/cao-playwright-utils.

Fix: run the Python commands at the root and run npm commands explicitly in
web/, cao_mcp_apps/, and cao_pwa/, matching the actual workflows.

8. The documented IndexedDB fallback does not exist

Locations: cao_pwa/src/instances.ts:14, cao_pwa/src/App.tsx:13,
docs/pwa.md:199

IndexedDB errors reject directly; App has no rejection handler and remains on
Loading... forever. The troubleshooting guide claims an in-memory fallback.

Fix: implement a process-local storage adapter fallback and catch initial
load failures, or remove the claim and render an actionable storage error.

9. The offline-instance confirmation is not a confirmation

Location: cao_pwa/src/components/InstancePicker.tsx:38

On a failed health probe the UI sets "add anyway?" and immediately persists the
instance and closes the dialog, so the operator cannot answer.

Fix: return after showing the warning and require a second explicit submit,
or provide separate Cancel and Add anyway actions.

Good To Fix

1. npm audit reports four dev-only advisories, including critical Vitest and
   high-severity Vite advisories. Production dependencies are clean
   (npm audit --omit=dev). Refresh the lockfile to patched versions.
2. git diff --check origin/main...HEAD fails on seven Antigravity fixture files
   because of added blank lines at EOF.
3. cao_pwa is described as a PWA but has no web app manifest or service worker.
   Either add the installability/offline plumbing or call it a standalone web app.
4. The PR combines AG-UI, telemetry, a provider, broad test-fixture replacement,
   devcontainer work, and unrelated terminology edits. Splitting independent
   changes would materially reduce regression risk and review cost.

Verification

• GitHub checks: all reported checks pass.
• Targeted backend suite with [otel]: 85 passed.
• PWA unit/component suite: 24 passed.
• Fresh default backend environment: test collection fails without [otel].
• PWA production dependency audit: no vulnerabilities.

[Copilot]

Pull request overview

This PR introduces an additive, default-off AG-UI surface (including generative UI production), along with supporting infrastructure for safety, testing, observability (optional OpenTelemetry), and a standalone dashboard PWA that consumes the AG-UI stream.

Changes:

• Add AG-UI-related safety/ops hardening (log redaction for query-string bearer tokens, default-off regression tests, emit-ui coverage).
• Add optional OpenTelemetry “GenAI semconv” instrumentation with a strict opt-in gate and “no-extra” no-op fallback.
• Add a standalone cao_pwa/ dashboard plus examples and a credentials-free mock_cli provider to enable deterministic CI paths.

Reviewed changes

Copilot reviewed 126 out of 135 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/utils/test_log_redaction.py	Adds tests asserting query-token redaction for access logs.
test/test_constants.py	Updates CORS origin defaults test for the PWA dev port.
test/telemetry/test_spans.py	Verifies GenAI span helper naming/attributes.
test/telemetry/test_otel_init.py	Pins telemetry init/shutdown opt-in behavior and degrade paths.
test/telemetry/test_optional_extra.py	Ensures telemetry degrades cleanly when OTel isn’t installed.
test/telemetry/test_context_propagation.py	Tests W3C trace-context inject/extract helpers.
test/telemetry/conftest.py	Provides shared in-memory exporter and tracer provider fixture.
test/telemetry/init.py	Declares telemetry test package.
test/services/test_workflow_spec_service.py	Updates workflow version terminology in docstring.
test/services/test_workflow_service.py	Updates workflow version terminology in docstring.
test/services/test_step_output_store.py	Updates workflow version terminology in docstring.
test/services/test_sse_bus.py	Adds edge-case coverage and reset-bus singleton test.
test/services/test_settings_service.py	Resets settings-service module caches for deterministic tests.
test/services/test_event_log_service.py	Adds edge-case coverage and reset-event-log singleton test.
test/scripts/init.py	Declares scripts test package (empty marker).
test/providers/test_native_status_shared.py	Trims outdated provider-scope text in docstring.
test/providers/test_mock_cli_unit.py	Adds unit tests for the new mock_cli provider and fixture binary.
test/providers/test_mock_cli_coverage.py	Covers async initialize branches + cleanup for mock_cli.
test/providers/fixtures/bin/mock_cli	Adds the deterministic bash “agent” used by mock_cli provider tests.
test/providers/fixtures/antigravity_cli_tool_call.txt	Adds fixture capture for Antigravity tool-call output.
test/providers/fixtures/antigravity_cli_response.txt	Adds fixture capture for Antigravity completed response output.
test/providers/fixtures/antigravity_cli_processing.txt	Adds fixture capture for Antigravity processing output.
test/providers/fixtures/antigravity_cli_print_mode.txt	Adds fixture capture for Antigravity print-mode output.
test/providers/fixtures/antigravity_cli_permission.txt	Adds fixture capture for Antigravity permission prompt state.
test/providers/fixtures/antigravity_cli_interactive_clean.txt	Adds fixture capture for Antigravity interactive-mode states.
test/providers/fixtures/antigravity_cli_idle.txt	Adds fixture capture for Antigravity idle/prompt output.
test/providers/fixtures/antigravity_cli_error.txt	Adds fixture capture for Antigravity error/interrupt output.
test/models/test_workflow.py	Updates workflow terminology and clarifies docstrings.
test/mcp_server/test_workflow_tools.py	Updates workflow version terminology in docstring.
test/mcp_server/test_workflow_return.py	Updates workflow version terminology in docstring.
test/mcp_server/test_handoff_equivalence.py	Updates workflow version terminology in docstring.
test/mcp_server/test_emit_ui_coverage.py	Adds MCP tool coverage for emit_ui behavior (200/400/404).
test/fixtures/init.py	Declares managed-subprocess fixture package and intent.
test/examples/test_example_profiles.py	Validates example profiles reference valid ProviderType values.
test/examples/init.py	Declares examples test package (empty marker).
test/e2e/test_headless_ci.py	Adds an e2e test for the headless CI runner example script.
test/e2e/conftest.py	Switches e2e server requirement to managed subprocess fixture + adds Gemini skip fixture.
test/conftest.py	Adds PATH wiring for mock_cli fixture + exposes shared auth fixtures/plugins.
test/clients/test_workflow_index_migration.py	Updates workflow version terminology in docstring.
test/cli/commands/test_workflow.py	Updates workflow version terminology in docstring.
test/api/test_workflow_runs.py	Updates workflow version terminology in docstring.
test/api/test_workflow_endpoints.py	Updates workflow version terminology in docstring + clarifies regression note.
test/api/test_default_off_listeners.py	Adds default-off regression tests for disabled surfaces.
test/api/test_agui_enablement.py	Pins AG-UI enablement contract across env flags.
test/api/test_agui_emit_ui.py	Adds end-to-end producer endpoint tests for allow-list + size bounds + 404 when off.
test/api/test_agui_auth_hardening.py	Ensures token-parse failures map to 401, not 500.
src/cli_agent_orchestrator/utils/logging.py	Adds access-log query-token redaction + stderr WARNING handler.
src/cli_agent_orchestrator/telemetry/spans.py	Adds GenAI semantic-convention span helper context managers.
src/cli_agent_orchestrator/telemetry/semconv.py	Defines frozen GenAI semconv keys + CAO extensions.
src/cli_agent_orchestrator/telemetry/otel.py	Adds opt-in OTel SDK init/shutdown with OTLP exporters and missing-SDK degrade behavior.
src/cli_agent_orchestrator/telemetry/context.py	Adds W3C trace-context inject/extract helpers.
src/cli_agent_orchestrator/telemetry/init.py	Adds optional-extra import strategy and no-op fallback API surface.
src/cli_agent_orchestrator/skills/workflow-author/SKILL.md	Updates “Bolt” phrasing to “version” in skill text.
src/cli_agent_orchestrator/skills/agui-author/SKILL.md	Adds a skill documenting safe generative UI authoring via emit_ui.
src/cli_agent_orchestrator/services/workflow_spec_service.py	Updates workflow version terminology and clarifies reserved/execution references.
src/cli_agent_orchestrator/services/workflow_service.py	Updates workflow version terminology in docstring.
src/cli_agent_orchestrator/services/step_output_store.py	Updates workflow version terminology in docstring.
src/cli_agent_orchestrator/services/status_monitor.py	Tightens typing for pyte screen tracking and casts provider status returns.
src/cli_agent_orchestrator/services/sse_bus.py	Adds SSEBus alias + reset helper for tests.
src/cli_agent_orchestrator/services/event_log_service.py	Adds reset helper for tests.
src/cli_agent_orchestrator/providers/mock_cli.py	Adds deterministic provider wrapping the fixture mock_cli binary.
src/cli_agent_orchestrator/providers/manager.py	Registers mock_cli provider in provider factory.
src/cli_agent_orchestrator/plugins/events.py	Extends event DTOs with optional traceparent for telemetry propagation.
src/cli_agent_orchestrator/plugins/builtin/event_log_publisher.py	Adds backwards-compatible alias for plugin class name.
src/cli_agent_orchestrator/models/workflow.py	Updates workflow version terminology and clarifies “grammar vs engine” boundary.
src/cli_agent_orchestrator/models/workflow_runtime.py	Updates workflow version terminology across runtime DTOs.
src/cli_agent_orchestrator/models/provider.py	Adds ProviderType for mock_cli.
src/cli_agent_orchestrator/mcp_server/server.py	Adds emit_ui MCP tool that POSTs to /agui/v1/emit_ui.
src/cli_agent_orchestrator/ext_apps/static/topology.js	Simplifies comments; keeps topology widget minimal.
src/cli_agent_orchestrator/ext_apps/init.py	Updates comments around topology widget export list.
src/cli_agent_orchestrator/constants.py	Adds OTEL service name constant and extends CORS defaults for PWA port.
src/cli_agent_orchestrator/cli/commands/workflow.py	Updates workflow version terminology in docstring.
skills/workflow-author/SKILL.md	Syncs “Bolt” phrasing to “version” in repo-root skill copy.
skills/plugin.json	Adds skills plugin manifest for shipped skills.
skills/agui-author/SKILL.md	Adds repo-root copy of the agui-author skill content.
scripts/sync_skills.py	Adds agui-author to shipped skills allowlist.
pyproject.toml	Adds [otel] optional extra deps and adds authlib to dev deps for JWT fixtures.
examples/headless-ci/run.sh	Adds headless CI runner script that maps terminal states to exit codes.
examples/headless-ci/README.md	Documents the headless CI example and usage in CI environments.
examples/headless-ci/ci_developer.md	Adds a CI-focused agent profile for non-interactive runs.
examples/cross-provider/README.md	Updates cross-provider README to include Antigravity profile entry.
examples/agui-dashboard/showcase.sh	Adds live-path AG-UI generative UI showcase script (curl-driven).
examples/agui-dashboard/run.sh	Adds demo runner to start server + optionally launch demo fleet + run showcase.
examples/agui-dashboard/README.md	Documents the AG-UI dashboard demo and safety/refusal behaviors.
examples/agui-dashboard/fleet_worker.md	Adds demo agent profile for optional mock fleet.
docs/otel-deployment.md	Adds OTel collector deployment guidance and backend recipes.
docs/mock-cli-provider.md	Documents motivation/design/CI strategy for the mock_cli provider.
CHANGELOG.md	Adds changelog entries for AG-UI stream, generative UI, PWA, telemetry, and mock provider.
cao_pwa/vite.config.ts	Adds Vite config for standalone PWA (dev port 5174) with vitest config.
cao_pwa/tsconfig.json	Adds strict TS config for PWA package.
cao_pwa/src/types.ts	Defines shared PWA types (instances, events, summaries).
cao_pwa/src/test/setup.ts	Sets up vitest environment with fake-indexeddb and dialog stubs.
cao_pwa/src/test/instances.test.ts	Adds IndexedDB CRUD tests for instance storage.
cao_pwa/src/test/InstancePicker.test.tsx	Pins accessible instance picker structure and interactions.
cao_pwa/src/test/GenerativeUI.test.tsx	Tests allow-listed rendering + refusal placeholder behavior.
cao_pwa/src/test/api.test.ts	Tests connectAGUI URL construction, dispatch, reconnection behavior, and health ping.
cao_pwa/src/main.tsx	Adds PWA entry point mounting the React app.
cao_pwa/src/instances.ts	Implements IndexedDB persistence for CAO instance list.
cao_pwa/src/components/InstanceTab.tsx	Implements per-instance stream view + summary panels + generative UI feed.
cao_pwa/src/components/InstancePicker.tsx	Implements instance selection/add/remove UI with basic reachability check.
cao_pwa/src/components/GenerativeUI.tsx	Adds safe client-side renderer with explicit allow-list and inert fallback.
cao_pwa/src/App.tsx	Composes instance picker + active tab and performs initial instance load.
cao_pwa/src/api.ts	Implements EventSource connection and reconnect logic for AG-UI stream.
cao_pwa/playwright.live.config.ts	Adds live-path Playwright config for real-server e2e runs.
cao_pwa/playwright.config.ts	Adds deterministic replay Playwright config for recording artifacts.
cao_pwa/package.json	Adds PWA package manifest, scripts, and dev deps (vitest/playwright/vite).
cao_pwa/index.html	Adds PWA HTML entrypoint.
cao_pwa/e2e/generative-ui.spec.ts	Adds deterministic replay e2e spec validating rendering + refusal.
cao_pwa/.gitignore	Adds PWA-specific ignores for node/build/test artifacts.
.github/workflows/cao-pwa-generative-ui.yml	Adds CI workflow to build/test/record PWA proof artifacts (replay + live).
.devcontainer/test-ci.sh	Adds a devcontainer script to run CI-equivalent steps locally.
.devcontainer/Dockerfile	Adds devcontainer image aligned with Playwright + installs tmux/uv/bun.
.devcontainer/devcontainer.json	Adds devcontainer configuration and post-create commands/port forwards.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[plauzy]

Thanks @fanhongy and @Copilot — all findings addressed. Phase-A was force-pushed and all checks are green. Summary:

Scope (Good-To-Fix #4, "split the PR"): narrowed this PR to the L1 adapter. The standalone dashboard PWA (cao_pwa/) and the CI devcontainer have been removed and will return as focused follow-up PRs — so the PWA findings below are resolved by removal rather than half-fixing them here.

Must-Fix

1. CAO_AGUI_ENABLED starves the stream — route and publisher now share one agui_surface_enabled() helper, so CAO_AGUI_ENABLED=true alone feeds the bus. Added a regression test (publisher emits under the AG-UI flag with MCP Apps off).
2. PWA discards the state channel — resolved by removing the PWA (out of scope for the L1 adapter).
3. Replay event-loss window — the generator now registers the live subscription before the history snapshot and dedupes the replay/live overlap by event id, with an SSE id: cursor; the "resumes without a gap" doc now matches the code. New streaming test covers snapshot + replay + STATE_DELTA.
4. OTel initialized but nothing instrumented — instrumented the send-message dispatch seam (execute_tool span), added a cao.orchestration.dispatches counter, and now propagate W3C traceparent into plugin events. New integration test drives a real dispatch and asserts an exported span. Docs/CHANGELOG narrowed to exactly what ships.
5. Clean install can't collect telemetry tests — test/telemetry/conftest.py imports the SDK lazily and the SDK-dependent modules importorskip, so a base install (no [otel]) skips cleanly; the subprocess fallback test is retained.

Suggested

6. Log redaction bypassed via uvicorn …:app — install_access_log_redaction() moved into the app lifespan, so both cao-server and the imported-app path are covered.
7. Devcontainer targets a nonexistent root JS project — removed the added .devcontainer/{devcontainer.json,Dockerfile,test-ci.sh} (out of scope; the pre-existing features/cao/ is untouched).
   8 & 9. IndexedDB fallback / offline-instance confirmation — resolved by removing the PWA.

Good-To-Fix

1. npm dev-only advisories — moot (PWA removed).
2. Antigravity fixture EOF whitespace — fixed (git diff --check is clean).
3. PWA manifest / service worker — moot (PWA removed).

Copilot inline

• telemetry/__init__.py broad except ImportError — now re-raises unless the missing module is part of the opentelemetry stack, so real bugs in CAO's telemetry modules surface instead of silently degrading.
• telemetry/context.py extract_traceparent docstring — corrected (returns a fresh empty Context, not the active one).
• telemetry/spans.py import-time tracer cache (×3) — the tracer is now resolved at call time, so spans record after init_telemetry() installs the provider in the lifespan. (traceparent also uses an explicit W3C propagator now, so it's independent of the app's global propagator.)
• cao_pwa/src/api.ts dedupe comment — moot (PWA removed); the server-side stream now performs the dedup for real.

[fanhongy]

thanks for the commits. Clean fix, few gaps remaining:

1. The replacement viewer does not replay events missed during reconnect

New regression introduced by the remediation.

Locations: examples/agui-eventsource-viewer/index.html:407-426 , src/cli_agent_orchestrator/api/main.py:830-847 ,
src/cli_agent_orchestrator/api/main.py:944-955 , examples/agui-dashboard/README.md:14

The viewer always opens the fixed URL /agui/v1/stream and relies on native EventSource automatic reconnect. Native reconnect sends the most
recent SSE ID in the Last-Event-ID request header. The endpoint does not accept that header and only replays history when the caller explicitly
supplies ?since= . Consequently, events produced while the browser or proxy connection is down are not replayed. The viewer's seenIds set only
deduplicates delivered events; it cannot recover events it never received.

A focused endpoint reproduction sent Last-Event-ID: missed and observed:

status=200
history_calls=0
missed_event_replayed=False

This contradicts the demo's advertised "SSE reconnect ... with no gap" behavior and the resilience guidance in docs/agui.md:80-87 .

Fix: Prefer an ID-based server cursor. Accept Last-Event-ID , add an event-log lookup that returns records after that ID, and replay them before
draining the pre-registered live queue. Keep ?since= as an explicit fallback. Alternatively, disable native retries in this viewer and manually
reopen with a tracked ?since= , although an event ID cursor is safer than an exclusive timestamp when two records share a
timestamp. Restore an end-to-end test that disconnects the client, emits an event, reconnects, and asserts that the missed event arrives exactly
once.

2. Queue overflow still creates the silent gap from the original finding

Unresolved portion of previous must-fix finding 3.

Locations: src/cli_agent_orchestrator/services/sse_bus.py:52-56 , src/cli_agent_orchestrator/services/sse_bus.py:103-113 ,
src/cli_agent_orchestrator/api/main.py:968-980

Registering the queue before history replay correctly closes the handoff race. However, when that bounded queue fills, _deliver still drops the
new event and leaves the subscriber and HTTP stream open. The AG-UI generator receives no gap signal, so it never closes the connection or
backfills the event from history.

A focused overflow reproduction observed:

queue_size=256
dropped_event_delivered=False
subscriber_still_open=True

This still violates the documented no-gap contract in docs/agui.md:82-87 and docs/agui.md:148-149 .

Fix: Mark the subscriber overflowed and terminate its stream, or deliver a gap sentinel that makes the endpoint close. The client's reconnect must
then resume from its last delivered event ID. Add a test that fills the queue, publishes one more event, and verifies that recovery replays the
dropped record exactly once.

Suggested Fix

3. The new viewer cannot connect to an auth-enabled server as shipped

Location: examples/agui-eventsource-viewer/index.html:407-414 , examples/agui-eventsource-viewer/README.md:64-72

The server requires ?access_token= in auth-enabled mode, but the viewer only accepts a base URL and appends /agui/v1/stream to it. The
README tells the operator to edit the source to add token support. A supplied example should not require source modification for a documented
server mode.

Fix: Add an optional token input, build the stream URL with URL and searchParams , and keep the token in memory only. Cover URL construction
with a small browser test.

Previous other findings:

All addressed.

Verification

• Current GitHub checks: passing.
• AG-UI, publisher, stream, mapping, and SSE bus tests: 77 passed .
• Telemetry tests: 28 passed .
• Lifespan/access-log redaction tests: 9 passed .
• git diff --check origin/main...HEAD : passed.
• Focused Last-Event-ID and queue-overflow reproductions: both confirmed the

[plauzy]

@fanhongy acknowledged and thanks for the multi-turn reviews - working on those fixes now

[plauzy]

Thanks for the review, @fanhongy. Pushed a follow-up (force-with-lease onto the same single Verified commit, 786447c) that resolves the three open findings + confirms the Code Quality status. Each fix was written RED-first (a test reproducing your exact repro) then made GREEN.

finding → fix → proving test

#	Finding	Fix	Proving test(s)
F2	Queue-overflow silent gap (must-fix): a full bounded SSE queue dropped the event but left the subscriber + HTTP stream open — no gap signal, no backfill.	services/sse_bus.py: a full queue on an overflow-close subscriber is now an explicit gap signal — the subscriber is marked overflowed and an OVERFLOW_SENTINEL is enqueued so the drain loop closes the stream; the AG-UI endpoint registers with overflow_close=True. The browser reconnects (F1) and the dropped records replay exactly once. /events (MCP Apps) keeps legacy drop-on-slow.	test/services/test_sse_bus_overflow.py::test_overflow_marks_and_closes (subscriber marked + drain closes, no silent open gap) · test/api/test_agui_stream_overflow.py::test_overflow_triggers_reconnect_backfill (stream closes → subscriber_still_open=False; reconnect replays the dropped record exactly once)
F1	Reconnect event loss (must-fix): the viewer relied on native EventSource reconnect (Last-Event-ID), but the endpoint only replayed on ?since=, so events missed during a drop were never replayed.	GET /agui/v1/stream now reads the Last-Event-ID header and replays event-log records after that id (EventLog.after_id) before the live drain; ?since= keeps precedence; replay is failure-isolated. Dedup-by-id preserved.	test/api/test_agui_stream_reconnect.py (after_id invoked with the header id, missed record replayed exactly once, ?since= precedence, no-cursor no-replay) · test/services/test_event_log_service.py::TestAfterId (strictly-after / unknown-id / TTL) · end-to-end no-gap reconnect in record-demo.mjs (proveReconnectNoGap)
F3	Viewer can't reach an auth-enabled server as shipped (suggested): only a base URL; README said edit the source to add ?access_token=.	examples/agui-eventsource-viewer/index.html gains an Access token field and builds the stream URL via new URL() + searchParams — in-memory only (never localStorage/sessionStorage, redacted from the on-screen status, seeded from the page URL then scrubbed via replaceState). README "edit the source" note dropped.	URL-construction assertion in record-demo.mjs (token attached as ?access_token= with a token, no query param without one, and never persisted to web storage)
F4	Code Quality / mypy RED (CI blocker): pre-existing, repo-wide.	Verified mypy src/ is not green on current upstream/main (d7c1cd0) — 10 errors in agent_scaffold.py / cli/commands/profile.py / memory_service.py (jsonschema stub + real type errors), unrelated to AG-UI. Not bundling those into this feature commit. This PR adds zero new mypy errors; it'll go green once the repo-wide fix lands on main and this commit is rebased.	uv run mypy src/ — same 10 pre-existing errors before and after this change.

Docs (docs/agui.md "Connection resilience" + troubleshooting, examples/agui-*/README.md) are reconciled to the Last-Event-ID reconnect and the overflow→reconnect→backfill contract.

Local verification: targeted AG-UI + services suites GREEN (112 tests), black/isort clean, patch coverage 100% on the changed sse_bus.py / event_log_service.py and the agui_stream endpoint region. The reconnect no-gap + token-URL assertions gate in the AG-UI demo CI job; F1/F2/F3 unit + API proofs gate in Unit Tests.

[plauzy]

CI note: the red Unit Tests / CAO MCP Apps jobs on this PR are inherited from a pre-existing breakage on main, not from this branch. main has been red since #441 added wait_until_input_ready() to ClaudeCodeProvider.initialize() while four test_provider_init_timeout.py tests left get_history() mocked as a MagicMock (→ TypeError in strip_terminal_escapes).

I've opened a dedicated fix: #452 (test-only; isolates the settle-check in those tests). Once #452 lands on main, this PR is already rebased onto latest main and will go green with no further changes. This branch's own gates — Code Quality, AG-UI demo, MCP Apps E2E, Web UI, Security — are green, and the AG-UI unit/API suites pass.

[plauzy]

✅ Resolved (re: this note).

The main regression fix landed as #452 (568d132). I've rebased this PR onto the latest main, and the test-only fix I had temporarily folded in dropped cleanly — this branch no longer touches test_provider_init_timeout.py (its diff vs main for that file is empty). The single Verified commit is now 6e8bb6d.

Full CI is green on the rebased head: Unit Tests (3.10/3.11/3.12), Code Quality, CAO MCP Apps + E2E, AG-UI demo (shift-left recording), Web UI Build, and Security Scan all pass. This PR is back to purely the AG-UI L1 feature + the F1/F2/F3 review remediations.

[plauzy]

@fanhongy @haofeif Let me know if any remaining issues here - i believe i've addressed the full scope at this point and hoping to land this on main to pick up the remaining phases scoped out in #386

[haofeif]

@fanhongy can you please comment ? @plauzy please address the conflicts.

[plauzy]

✅ Done — rebased onto latest main, reconstructed as a clean additive AG-UI diff, CI fully green

(This comment has been edited to the final resolved state.)

1. Main's CI regression — fixed at the source. The red CI was inherited from main, not caused by this PR: #441 added wait_until_input_ready() (which calls strip_terminal_escapes(get_history(...))) while the init-timeout tests mocked get_history as a MagicMock, so strip_terminal_escapes raised TypeError. That was fixed and merged as #452 (test-only isolation, zero production impact). A subsequent main CI run went red purely on transient GitHub Actions infra (##[error]<!DOCTYPE html> during action/tool downloads, across unrelated jobs) — I re-ran it and it is now green, confirming main's code at 673b2d0 is healthy.

2. This branch was reconstructed to be purely additive AG-UI on top of main. The previous commit had accumulated scope creep that diverged from main; removed so the diff only adds AG-UI and never rewrites main's code:

• "Bolt → v" docstring rewordings across the workflow feature (models/workflow*.py, services/workflow*.py, cli/commands/workflow.py, api/main.py, constants.py, mcp_server/server.py, and workflow tests) — restored to main's wording.
• Cosmetic comment rewrites (ext_apps/*), an unrelated examples/cross-provider/README.md row, a status_monitor.py typing change (+ its added test), and a settings-cache test fixture — reset to main.

Result: one atomic, GPG-signed commit (769dce4, Verified) whose diff vs main is additive AG-UI / telemetry / mock_cli only — no reverts or rewordings of main's code.

3. Addressed @fanhongy's overflow no-loss finding. On SSE overflow the bus dropped the oldest buffered event, leaving a hole before the client's Last-Event-ID that replay could never fill. It now drops the newest buffered event, keeping delivered events a contiguous prefix so delivered + EventLog.after_id(last_delivered) == published. test/services/test_sse_bus_overflow.py was strengthened to assert both the contiguous prefix and full no-loss recovery. (Full detail in my reply to @fanhongy below.)

Verification — all green: Unit Tests 3.10/3.11/3.12 ✓, Code Quality (black/isort/mypy — 0 new mypy errors) ✓, AG-UI demo ✓, CAO MCP Apps + E2E ✓, Web UI Build ✓, Trivy/Security/Dependency Review ✓. mergeable: MERGEABLE. Ready for review/merge.

[fanhongy]

Thanks for addressing the reconnect behavior. The stream now closes correctly on overflow, which fixes the original silent-open-gap problem.

With branch plauzy:kiro/pr387-phase-a-agui-core-signed, I can do

  import asyncio

  from cli_agent_orchestrator.services import sse_bus
  from cli_agent_orchestrator.services.event_log_service import EventLog


  async def reproduce():
      sse_bus.SSE_MAX_QUEUE_SIZE = 3

      bus = sse_bus.SseBus()
      log = EventLog()
      subscriber = bus.register(overflow_close=True)

      published = []
      for i in range(4):
          event = log.append("launch", f"t{i}", None, {})
          event["id"] = f"e{i}"
          published.append(event["id"])
          bus.publish(event)

      # Let call_soon_threadsafe deliveries execute.
      for _ in range(4):
          await asyncio.sleep(0)

      delivered = []
      async for event in bus.drain(subscriber):
          delivered.append(event["id"])

      replayed = [
          event["id"]
          for event in log.after_id(delivered[-1])
      ]

      observed = delivered + replayed

      print("published:", published)
      print("delivered:", delivered)
      print("replayed:", replayed)
      print("missing:", sorted(set(published) - set(observed)))

      assert observed == published, "overflow recovery lost an event"


  asyncio.run(reproduce())
  

with result:

  published: ['e0', 'e1', 'e2', 'e3']
  delivered: ['e1', 'e2']
  replayed: ['e3']
  missing: ['e0']
  AssertionError: overflow recovery lost an event

I can see this is covered with unit test, but might worth looking at: /test/services/test_sse_bus_overflow.py

which you have:

   # Some pre-gap events were delivered before the stream closed.
    assert len(received) >= 1
    # And the stream actually CLOSED (the async-for completed) rather than
    # hanging with the gap unsignalled — that is the whole point of the fix.

    bus.unregister(sub)
    assert bus.subscriber_count == 0

This test means:

Let's say if you a queue with capacity 3, publishing e0, e1, e2, e3 produces:

After e0: [e0]
After e1: [e0, e1]
After e2: [e0, e1, e2]

When e3 arrives, the queue is full, and the first item e0 will be removed then inserts the sentinel. Leave the queue like:

[e1, e2, OVERFLOW_SENTINEL]

e0 is now missing. The assertion only asks whether at least one event arrived. It does not ask whether all expected events arrived.

This assertion also passes:

assert bus.subscriber_count == 0

and it only verifies that unregister() removed the subscriber. It says nothing about event delivery.

What Is Missing

The test needs to verify both properties:

1. The stream closes on overflow.
2. Every event is either delivered before closure or replayed after reconnect.

Long-in-short: the existing test correctly proves that overflow triggers closure, but it does not prove the no-event-loss recovery.

It would also be helpful to strengthen test/services/test_sse_bus_overflow.py

Thank you!

[plauzy]

@fanhongy — great catch, and thank you for the precise repro. You're right: closing the stream on overflow fixed the silent-open-gap, but the recovery still lost an event, and your reproduction pinned exactly why.

Root cause. The overflow branch made room for the sentinel by evicting the oldest buffered event:

sub.queue.get_nowait()                 # drops e0 (front)
sub.queue.put_nowait(OVERFLOW_SENTINEL)

With a cap-3 queue fed e0,e1,e2,e3, that left [e1, e2, SENTINEL]. The client delivered [e1, e2], reconnected with Last-Event-ID=e2, and replayed after_id(e2)=[e3] — so e0 was gone forever, because it sat before the id the client anchored replay on. Replay can only recover events after the last delivered id, so a hole at the front is unrecoverable. Exactly your missing: ['e0'].

Fix (services/sse_bus.py). On overflow we now drop the newest buffered event instead of the oldest, keeping the delivered events an unbroken prefix of the stream:

buffered = []
while not sub.queue.empty():
    buffered.append(sub.queue.get_nowait())
for item in buffered[:-1]:              # keep the contiguous prefix; drop newest
    sub.queue.put_nowait(item)
sub.queue.put_nowait(OVERFLOW_SENTINEL)

Now the client delivers [e0, e1], reconnects at e1, and after_id(e1)=[e2, e3] — the dropped/newer events are durably in the EventLog and replayed. No hole, no loss.

Your exact repro now prints:

published: ['e0', 'e1', 'e2', 'e3']
delivered: ['e0', 'e1']
replayed:  ['e2', 'e3']
missing:   []
PASS: observed == published, no event lost

Strengthened test/services/test_sse_bus_overflow.py per your suggestion — it now proves both properties you called out:

1. test_overflow_marks_and_closes asserts the delivered events are a contiguous prefix starting at e0 (received_ids == ["e0", "e1", …]), so the oldest events are provably preserved, not just len >= 1.
2. New test_overflow_recovery_replays_every_event reproduces your scenario end-to-end and asserts delivered + EventLog.after_id(delivered[-1]) == published — the no-event-loss recovery property.

All checks are green on the latest push (Unit Tests 3.10/3.11/3.12, Code Quality, AG-UI demo, MCP Apps + E2E, Web UI Build, Trivy/Security). Thanks again for the thorough review!

[plauzy]

This isn't good - figuring out how these leaked in here and removing

[fanhongy]

wow. This is a very qquick fix. Thanks for responding this fast! LGTM.