Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Browser diagnostics

on:
workflow_dispatch:

permissions:
contents: read

concurrency:
group: browser-diagnostics-${{ github.ref }}
cancel-in-progress: false

jobs:
diagnostics:
name: Sequential browser measurements
runs-on: ubuntu-latest
timeout-minutes: 45
env:
NODE_OPTIONS: --max-old-space-size=4096
CDB_WORKFLOW_BENCHMARK_SAMPLES: '30'
CDB_WORKFLOW_PERFORMANCE_GATE: '1'
steps:
- name: 📥 Checkout selected revision
uses: actions/checkout@v7

- name: 🏗️ Set up workspace
uses: ./.github/actions/setup-workspace
with:
install-chromium: 'true'

- name: 🔨 Build before measuring
run: pnpm exec turbo run build --concurrency=1

- name: 🖥️ Record runner environment
run: |
git rev-parse HEAD
node --version
pnpm exec playwright --version
uname -a
lscpu
uptime
echo 'Linux CI baseline. Compare only matching fixtures, browser versions and runner environments.' >> "$GITHUB_STEP_SUMMARY"
git rev-parse HEAD >> "$GITHUB_STEP_SUMMARY"

# Keep transports and lifecycle work sequential, including after a failed gate.
- name: 📊 Measure WebSocket workflows
id: websocket
continue-on-error: true
env:
CDB_WEBSOCKET_BENCHMARK_OUTPUT: ${{ runner.temp }}/cdb-websocket.json
run: pnpm exec vitest run --project extension-e2e --maxWorkers=1 tests/e2e/native-agent-benchmark.test.ts

- name: 📊 Measure Devframe workflows
id: devframe
if: ${{ !cancelled() && steps.websocket.outcome != 'skipped' }}
continue-on-error: true
env:
CDB_DEVFRAME_BENCHMARK_OUTPUT: ${{ runner.temp }}/cdb-devframe.json
run: pnpm exec vitest run --project extension-e2e --maxWorkers=1 tests/e2e/devframe-native.test.ts

- name: 🔄 Exercise 30 lifecycle cycles
id: lifecycle
if: ${{ !cancelled() && steps.websocket.outcome != 'skipped' }}
continue-on-error: true
env:
CDB_LIFECYCLE_DIAGNOSTICS_OUTPUT: ${{ runner.temp }}/cdb-lifecycle.json
run: pnpm exec vitest run --project extension-e2e --maxWorkers=1 tests/e2e/devframe-native.test.ts

- name: 📝 Report complete and partial results
if: always()
run: node tests/e2e/fixtures/report-validation.ts "$RUNNER_TEMP/cdb-websocket.json" "$RUNNER_TEMP/cdb-devframe.json" "$RUNNER_TEMP/cdb-lifecycle.json"

- name: ✅ Require all diagnostic suites to pass
if: ${{ !cancelled() }}
env:
WEBSOCKET_OUTCOME: ${{ steps.websocket.outcome }}
DEVFRAME_OUTCOME: ${{ steps.devframe.outcome }}
LIFECYCLE_OUTCOME: ${{ steps.lifecycle.outcome }}
run: test "$WEBSOCKET_OUTCOME/$DEVFRAME_OUTCOME/$LIFECYCLE_OUTCOME" = success/success/success
23 changes: 21 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,24 @@ or resume-window expiry deletes the authority record and ends the logical sessio
Session `metadata` is optional JSON-compatible data. CDB validates and stores it but never interprets
it as identity, authority, or policy.

### Optional host orchestration

`@dvcol/cdb-broker` composes the core stores, pairing, logical sessions, grant coordinator and
per-principal tool sessions. `defineBroker` is declarative; `createBroker` creates the runtime.
Management state is a projection of these components. Persistent identity storage preserves the
broker ID and existing provider pairings at a host-selected location.

Navigation presets belong to this optional package, outside the protocol kernel. Each approval binds
the selected preset to a principal and target. Disallowed presets fail without downgrading. Target
generation renewal invalidates document-bound references independently of the navigation policy.

`@dvcol/cdb-devframe` installs this runtime as a context-lifetime service. Provider messages, pairing
proofs and operation cancellation use the host's authenticated Devframe RPC peer. CDB channel closure
does not close that shared connection. Hosts forward peer lifecycle callbacks and dispose the service
on startup failure and shutdown. The optional panel consumes an existing management client and never
instantiates a broker. Registry aggregation, tool ownership/routing, MCP lifecycle and WebMCP
publication remain with the embedding application.

### Grant request coordinator

`createGrantRequestCoordinator` stores the authenticated logical session, principal, requested
Expand All @@ -72,8 +90,9 @@ for the contract and runnable public example.
### Extension helpers

`@dvcol/cdb-extension` provides publication and recovery mechanics that remain useful to a browser
extension but do not import Chrome APIs. The extension host supplies the Chrome adapter and user
approval policy. The separate `presentation` entry is an opt-in content-script helper. It translates
extension but do not import Chrome APIs. The optional `@dvcol/cdb-extension/chrome` entry composes
Chrome debugger, tab, navigation, alarm and storage bindings. The embedding application owns
permissions, installation, tab-selection policy and final approval trust. The separate `presentation` entry is an opt-in content-script helper. It translates
successful CDP pointer commands into sanitized visual events and renders an isolated pointer plus a
temporary favicon. The host owns installation, messaging, current grant state, and navigation
reinjection.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ See [ARCHITECTURE.md](./ARCHITECTURE.md) for the protocol and failure model. See
| ---------------------- | ---------------------------------------------------------------------------------------- |
| `@dvcol/cdb` | Broker, client facade, protocol types, target authorization, leases, and command routing |
| `@dvcol/cdb-birpc` | RPC transport adapter |
| `@dvcol/cdb-broker` | Shared pairing, grant, recovery and per-principal tool orchestration for hosts |
| `@dvcol/cdb-devframe` | Devframe service, shared provider RPC transport, and mountable browser-control panel |
| `@dvcol/cdb-extension` | Browser-extension helpers for publication, recovery, and opt-in control presentation |
| `@dvcol/cdb-mcp` | Principal-scoped semantic and raw MCP tool sessions over a CDB client |
| `@dvcol/cdb-automation-playwright` | Experimental Playwright semantic provider over a broker-authorized CDB executor |
Expand Down
57 changes: 57 additions & 0 deletions docs/benchmarks/2026-09-08-latency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Native action deadlines and discovery measurements

Recorded on 8 September 2026 with Chromium 151.0.7922.34, Node 24.20.0, macOS arm64, Apple M3 Pro (11 logical processors). The normal public fixture contains 10,000 DOM nodes, 200 levels, 20 closed shadow roots and three cross-origin frame levels. Each reported scenario has 30 warm samples. Browser startup, fixture loading and model time are excluded; CDB processing and readiness checks are included. This summary includes failed and successful runs.

## Devframe RPC workflow

Warmup: one fill, one click, then one fill/click batch with observation. Each measured iteration uses exact role/name locators for fill and click, an individual compact snapshot, then the same fill/click batch with `observe: true`. Persistent-overlay samples are collected separately afterward.

| Scenario | Before p95 | Final p95 | Target |
| --- | ---: | ---: | ---: |
| Fill | 793 ms | 433 ms | 500 ms |
| Click | 813 ms | 445 ms | 500 ms |
| Compact snapshot | 930 ms | 935 ms | 1,000 ms |
| Fill/click batch + observation | 2,370 ms | 1,896 ms | — |
| Individual fill + click + snapshot | — | 2,125 ms | — |
| Covered click | Not sampled | 2,131 ms | 2,000 ms input deadline |

All 30 covered clicks returned the covered-element error without input. The deadline bounds CDB execution; transport delivery and event-loop scheduling add observable overhead. Shorter blocked waits are not counted as faster successful actions.

A separate five-sample phase probe before optimization averaged 611 ms discovery, 80 ms actionability, 10 ms input and 3 ms verification. Final successful actions averaged 247, 92, 16 and 2 ms respectively. Discovery now uses filtered accessibility queries at session roots and full trees for in-process child frames. After failed readiness it refreshes the full AX view before renewing the locator. No DOM state is cached across actions. Ancestor-frame and fresh geometry/hit checks remain enabled.

Obtaining query roots through `Accessibility.getRootAXNode` avoids `DOM.getDocument` initialization and its unnecessary DOM event burst. The existing first-use batch and scrolling-replacement Chromium regressions exercise these cases. Final fill/click diagnostics average 47/43 CDP commands, 13,521/12,451 serialized request characters, and 21,341/18,312 response-envelope characters. Full diagnostic traces remain opt-in through `cdb.mcp.action`.

## WebSocket reference workflow

This harness uses fresh element references rather than role/name locators. Its results cannot isolate a transport regression against the Devframe harness. Warmup and sample counts are unchanged between runs.

| Run | Action p95 | Snapshot p95 | Action errors | Gate |
| --- | ---: | ---: | ---: | --- |
| Before | 122 ms | 730 ms | 0 | Pass |
| Intermediate | 107 ms | 657 ms | 0 | Pass |
| Final | 292 ms | 1,877 ms | 0 | **Snapshot gate failed** |

The final WebSocket snapshot run exceeds the one-second target. The variation is unresolved; the earlier passing run does not establish acceptance for the final run. The native behavioral suite passes all 18 scenarios, and the separate Devframe run passes its action and snapshot targets.

## Follow-up validation

The same revision and normal fixture were rerun sequentially later on 8 September, with 30 warm samples per scenario and no concurrent test suites or builds. The first rerun reproduced the snapshot failure. Subsequent runs added timing at the actual client command, artifact HTTP read, and Chrome debugger boundaries without changing production behavior. All samples, including the failed rerun, are retained in the CSV.

| Run | Action p95 | Snapshot p95 | Gate |
| --- | ---: | ---: | --- |
| WebSocket reproduction | 181 ms | 1,183 ms | Snapshot failed |
| WebSocket command/artifact probe | 124 ms | 791 ms | Pass |
| WebSocket confirmation | 111 ms | 702 ms | Pass |
| WebSocket final, including Chrome timing | 111 ms | 712 ms | Pass |
| Devframe confirmation, fill/click | 427/422 ms | 752 ms | Pass |
| Devframe final explicit gate, fill/click | 444/401 ms | 995 ms | Pass |

OS load averages during the failing reproduction rose from approximately 15.2/9.2/10.5 to 18.8/11.1/11.2. Process inspection showed GitKraken and filesystem activity alongside Chromium. The final WebSocket run recorded 5.8/7.8/9.6 at the start and 5.7/7.6/9.4 at the end. Devframe's final run recorded 5.8/7.4/9.3 and 6.5/7.4/9.0. This association supports host contention as a contributor, but does not isolate it as the sole cause of the earlier variation.

Final WebSocket snapshots averaged 642 ms and transferred about 5.6 MB of accessibility artifacts. Each of four AX command round trips averaged 145 ms; direct Chrome debugger execution averaged 67 ms, including the warmup. Artifact HTTP reads totaled 11 ms per snapshot. The command-time difference includes provider/broker serialization, transport and processing; it is not a network-only measurement. These measurements do not justify changing artifact delivery or weakening snapshot completeness.

The final gates pass in this recorded environment, with no production optimization applied during follow-up. Devframe's 995 ms snapshot p95 is close to the budget and should not be described as having substantial performance margin. Its final batch p95 was 1,815 ms; covered clicks returned the readiness error without input at 2,011 ms p95. The benchmark now exposes an explicit Devframe performance gate and both harnesses record machine load for future comparisons.

## Agent payload costs

The final Devframe catalogue is 75,906 serialized characters. Individual fill/click/snapshot arguments and responses total approximately 805 characters per workflow; the batch with observation totals 869. At characters divided by four, these are approximately 19,000 catalogue tokens, 201 individual-workflow tokens and 217 batch-workflow tokens. These are estimates, not measurements from a model tokenizer. The batch trades about 16 estimated payload tokens for one agent round trip instead of three and a shared lease. Observation remains opt-in.
Loading