Skip to content

fix(connection): require hello-ok before operator RPCs - #1425

Open
eleqtrizit wants to merge 4 commits into
mainfrom
fix/issue-1418
Open

eleqtrizit wants to merge 4 commits into
mainfrom
fix/issue-1418

Conversation

@eleqtrizit

@eleqtrizit eleqtrizit commented Sep 15, 2026

Copy link
Copy Markdown

What Problem This Solves

Fixes an issue where users operating the Windows Companion over a Gateway connection would have operator requests sent on the wire before the Gateway handshake completed: the gateway rejects early tracked requests with a 1008 socket close, so session resets and wizard RPCs raced against handshaking or reconnecting and knocked out the connection. Fixes #1418.

Why This Change Was Made

The operator client's readiness signal admitted tracked requests on transport connectivity alone. The fix holds a handshake snapshot at the shared request boundary: tracked mutations submitted before hello-ok are withheld and reported as not-sent instead of false success; the gate reopens once a fresh hello-ok lands, and suppressed mutations are not replayed automatically (callers resubmit). Wizard RPCs surface the pending-handshake error without closing the transport. Post-handshake behavior is unchanged.

User Impact

Tray and wizard sessions no longer lose their Gateway connection by racing the handshake: early requests are withheld (zero frames on the wire) and reported as not-sent, so callers resubmit once the session is ready instead of the Gateway rejecting them on the wire. Callers observe an accurate not-sent result rather than a false success while disconnected.

Evidence

  • Captured wire proof: docs/gateway-handshake-gate-proof.md (redacted TRX-embedded trace from a real dotnet test run of the production client over loopback WebSockets).
  • Focused test counts and exact commands under Validation.

Change Type

  • Bug fix
  • Feature
  • Refactor
  • Docs or instructions
  • Tests or validation
  • Security hardening
  • Chore or infrastructure

Scope

  • Tray or WinUI UX
  • Windows node capability
  • Gateway, connection, or pairing
  • Setup or onboarding
  • Permissions, privacy, or security
  • Tests, CI, or docs

Required proof pools

  • windows-wsl-gateway-e2e: reconnect and operator-request gating on the Gateway connection changed, which matches this pool's requiredWhen. Run 2026-09-17 on a maintainer-provided Windows interactive-desktop host (ARM64, Windows 10.0.28000, dotnet 10.0.401 win-x64, WSL 2.7.12): the manifest command run-proof-tests.ps1 -Project tests\OpenClaw.E2ETests\OpenClaw.E2ETests.csproj -Filter FullyQualifiedName~OpenClaw.E2ETests.Setup.SetupAndConnectTests -ResultName gateway-setup-connect -RuntimeIdentifier win-x64 with OPENCLAW_RUN_E2E=1 at head 17cfc72. Result: total=20, passed=19, skipped=1 (opt-in Ollama permission test, expected skip), failed=0, exit=0 in 5.6 min. Evidence: secret gist with gateway-setup-connect.trx plus redacted loopback diagnostics (gateway-port-allocation.json, service-owned-gateway-start.jsonl): https://gist.github.com/eleqtrizit/8ed9ae54e4832375221afbc6297d3de7 . The fixture provisions an isolated scratch distro OpenClawE2E-{runId}; a first attempt produced two transient failures ("managed WSL gateway is not listening yet", an emulated-x64 WSL startup race), and both tests passed on re-run with no code changes.

Validation

Environment: Linux dev box, .NET SDK from ~/.dotnet (GitVersion MSBuild target requires it on PATH).

  • Capture run (tip 5c02ab5; code unchanged on 6edabb1): export PATH="$HOME/.dotnet:$PATH"; dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --filter "FullyQualifiedName~HandshakeGate" --logger "trx;LogFileName=proof.trx" -> 2/2 passed (both live round-trip gate tests).
  • Affected filter set (GatewayClient + live round-trip + AssistantMedia + ShutdownClassification): 278/278 passed.
  • Protocol drift/contract/closure safety filters: 30/30 passed.
  • OpenClaw.Tray.Tests: unchanged versus a clean base run (6466a04; 18 pre-existing Linux-env failures, identical on base; CI runs this suite on Windows).
  • Required full ./build.ps1 plus full Shared/Tray validation execute as required CI checks on this push; the prior head's only CI failure was Validate documentation (em dashes in the new proof doc, removed in this push).
  • Run separately: the windows-wsl-gateway-e2e pool on a maintainer-provided Windows host 2026-09-17 (see Required proof pools above).

Real Behavior Proof

  • Environment tested: production OpenClawGatewayClient over real loopback WebSockets against an in-process test gateway server; no external gateway or private endpoint contacted.
  • PR head or commit tested: 5c02ab5 (captured); production and test code identical on 6edabb1 (docs-only delta).
  • Exact steps or command run: dotnet test --filter FullyQualifiedName~HandshakeGate --logger "trx;LogFileName=proof.trx".
  • Evidence after fix: docs/gateway-handshake-gate-proof.md (full redacted trace, wire-level assertions from the same run).
  • Observed result: pre-handshake tracked RPCs never reach the wire (loopback server observes no frame; socket stays open with auto-reconnect armed); wizard RPCs throw the pending-handshake error without closing the transport; readiness flips on hello-ok and clears on disconnect; suppressed mutations report not-sent instead of false success; after a server-initiated one-way Close the gate holds, the real auto-reconnect path re-runs challenge -> connect -> hello-ok, and only a fresh submission made after the fresh hello-ok reaches the wire (suppressed mutations are not replayed automatically).
  • Screenshot or artifact links verified? N/A (redacted captured trace committed in-repo).
  • Verified after capture: live Windows Gateway<->operator session under the windows-wsl-gateway-e2e pool (run 2026-09-17 on a maintainer-provided Windows host; total=20, passed=19, skipped=1, failed=0; TRX and redacted diagnostics in the evidence gist linked under Required proof pools).

Security Impact

  • New permissions or capabilities? (No)
  • Secrets or tokens handling changed? (No)
  • New or changed network calls? (No — the guard only withholds frames the gateway would reject; strictly fewer early frames on the wire)
  • Command or tool execution surface changed? (No)
  • Data access scope changed? (No)
  • If any answer is Yes, explain the risk and mitigation: n/a

Compatibility and Migration

  • Backward compatible? (Yes — the only behavior change is inside the handshake window, where previously-sent RPCs would get the socket 1008-closed; callers' existing degrade paths handle the new null/throw shapes)
  • Config or environment changes? (No)
  • Migration needed? (No)
  • If yes, list the exact upgrade steps: n/a

Review Conversations

  • I replied to or resolved every bot review conversation addressed by this PR.
  • I left unresolved only conversations that still need maintainer judgment.

The gateway 1008-closes (PolicyViolation) any non-connect frame sent
before hello-ok (#1418). Gate operator readiness (IsConnectedToGateway)
on the handshake snapshot and guard the application send paths:

- SendTrackedRequestAsync: fire-and-forget requests are silently dropped
  pre-handshake (mirrors the socket-closed contract; the post-handshake
  refresh burst re-requests the state).
- SendWizardRequestAsync: throws InvalidOperationException carrying the
  shared HandshakePendingError marker; tolerant callers (pairing, models
  fallback, media, payload reads) already degrade gracefully.
- GetUpdateStatusAsync: null pre-handshake, keeping its no-info contract.
- TryRequestPayloadAsync: handshake-pending reads return null, same shape
  as unknown-method.
- connect itself is exempt by construction: the handshake rides
  SendConnectMessageAsync/SendRawAsync, never the tracked/wizard paths.

Tests: mechanical wizard/media suites satisfy the gate via a
CompleteHandshakeForTest flag-set (real challenge/connect/hello-ok path
is exercised by the round-trip suite, which now performs the full dance
against the loopback gateway); three new guard tests pin suppression,
wizard rejection, and readiness lifecycle.
@clawsweeper

clawsweeper Bot commented Sep 15, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P1 Urgent regression or broken agent/channel workflow affecting real users now. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 15, 2026
@clawsweeper

clawsweeper Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed September 17, 2026, 11:57 AM ET / 15:57 UTC (Revision 7).

ClawSweeper review

What this changes

The PR withholds operator requests until the Gateway handshake completes, reports suppressed mutations as unsuccessful, and adds regression tests and captured WebSocket traces.

Merge readiness

Ready for maintainer review

This remains a useful fix: current main and v2026.9.4 lack the handshake guard. Prior findings are resolved, and the new Windows evidence resolves the previous product-path proof blocker. No introduced correctness or security defect remains identified.

Priority: P1
Reviewed head: 17cfc728cf40a9f65632f860f5a6f6812c0e1826

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused repair with direct transport proof, Windows product-path evidence, and no remaining actionable findings.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (logs): The unchanged production operator client is exercised over real WebSockets with withheld handshakes and a server-initiated disconnect, showing suppression and recovery after fresh hello-ok. The inspected Windows/WSL artifact adds successful real Gateway connection and restart coverage, resolving the prior scoped proof gap.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (logs): The unchanged production operator client is exercised over real WebSockets with withheld handshakes and a server-initiated disconnect, showing suppression and recovery after fresh hello-ok. The inspected Windows/WSL artifact adds successful real Gateway connection and restart coverage, resolving the prior scoped proof gap.
Evidence reviewed 10 items Current main still needs the fix: At fetched main, readiness remains transport-only and the tracked request path lacks the handshake guard. The same revision is tagged v2026.9.4.
Release identity: The release tag v2026.9.4 points to the inspected main revision, which does not contain this fix.
Introduced request admission changes: The pinned merge-base delta guards wizard and tracked requests with the existing handshake snapshot. Suppressed tracked mutations now propagate false through their wrappers, resolving the earlier finding.
Findings None None.
Security None None.

How this fits together

The Windows Companion's shared operator client carries session, model, pairing, and wizard requests to the Gateway. Its readiness signal tells consumers when the connection can accept application requests.

flowchart TD
 A[Companion requests] --> B[Operator client]
 B --> C{Socket open and handshake complete?}
 C -->|No| D[Report unavailable or not sent]
 C -->|Yes| E[Send request to Gateway]
 F[Gateway hello acknowledgement] --> C
 G[Disconnect] --> H[Clear handshake readiness]
 H --> C
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test growth production +52 net lines; tests +334 net lines Production growth implements the shared readiness guard; test growth covers suppression, mutation results, and reconnect behavior.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #1418
Summary: This PR is the explicit candidate repair for the reported operator handshake race.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Technical review

Best possible solution:

Keep handshake readiness at the shared operator request boundary, preserving explicit unsuccessful submission results and requiring fresh submissions after reconnect.

Do we have a high-confidence way to reproduce the issue?

Yes, source provides a high-confidence path: submit an operator request while its WebSocket is open but hello-ok is pending. Current main admits that request; this review did not execute a failing current-main reproduction.

Is this the best way to solve the issue?

Yes. Reusing the existing handshake snapshot at the shared send boundaries addresses the reported race without adding configuration, changing persisted state, or duplicating connection ownership.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 3c43751b2bac.

Labels

Label justifications:

  • P1: The linked report documents repeated operator connection failures during startup and reconnect in an existing user workflow.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The unchanged production operator client is exercised over real WebSockets with withheld handshakes and a server-initiated disconnect, showing suppression and recovery after fresh hello-ok. The inspected Windows/WSL artifact adds successful real Gateway connection and restart coverage, resolving the prior scoped proof gap.
  • proof: sufficient: Contributor real behavior proof is sufficient. The unchanged production operator client is exercised over real WebSockets with withheld handshakes and a server-initiated disconnect, showing suppression and recovery after fresh hello-ok. The inspected Windows/WSL artifact adds successful real Gateway connection and restart coverage, resolving the prior scoped proof gap.

Evidence

What I checked:

  • Current main still needs the fix: At fetched main, readiness remains transport-only and the tracked request path lacks the handshake guard. The same revision is tagged v2026.9.4. (src/OpenClaw.Shared/OpenClawGatewayClient.cs:288, 3c43751b2bac)
  • Release identity: The release tag v2026.9.4 points to the inspected main revision, which does not contain this fix. (3c43751b2bac)
  • Introduced request admission changes: The pinned merge-base delta guards wizard and tracked requests with the existing handshake snapshot. Suppressed tracked mutations now propagate false through their wrappers, resolving the earlier finding. (src/OpenClaw.Shared/OpenClawGatewayClient.cs:2144, 17cfc728cf40)
  • Existing handshake lifecycle: Disconnect clears the snapshot; correlated, current-generation hello-ok processing publishes it before handshake success and the initial refresh burst. The patch preserves the dedicated connect path and existing credential handling. (src/OpenClaw.Shared/OpenClawGatewayClient.cs:2402, 17cfc728cf40)
  • Direct behavior proof and review continuity: The captured production-client WebSocket trace shows zero early mutation frames, false submission results, disconnect invalidation, automatic reconnect, and a fresh mutation reaching the wire after hello-ok. Production and test files are unchanged between captured commit 5c02ab5 and the reviewed head; the proof document's prior punctuation finding is also resolved. (docs/gateway-handshake-gate-proof.md:27, 17cfc728cf40)
  • Windows product-path artifact inspected: Read the linked gist through the GitHub API and parsed gateway-setup-connect.trx: 20 total, 19 passed, zero failed, and one unrelated opt-in Ollama test not executed. Passing scenarios include fresh operator/node connection, shared-token and setup-code flows, Gateway restart/reconnect, and real Gateway invocation. Runtime output includes hello-ok and completed operator handshakes. The captured PR body and September 17 contributor comment identify this as the 17cfc72 Windows/WSL run: https://gist.github.com/eleqtrizit/8ed9ae54e4832375221afbc6297d3de7.

Likely related people:

  • karkarl: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Scott Hanselman: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (6 earlier review cycles)
  • reviewed 2026-09-15T01:57:10.536Z sha 234a926 :: needs real behavior proof before merge. :: [P2] Return an unsuccessful result for suppressed mutation requests
  • reviewed 2026-09-15T02:28:56.174Z sha 5c02ab5 :: needs real behavior proof before merge. :: none
  • reviewed 2026-09-15T02:53:40.277Z sha 6edabb1 :: blocked before merge. :: [P2] [P2] Replace em dashes in the new proof document
  • reviewed 2026-09-15T14:03:20.464Z sha 17cfc72 :: blocked before merge. :: none
  • reviewed 2026-09-15T15:38:26.177Z sha 17cfc72 :: blocked before merge. :: none
  • reviewed 2026-09-15T15:44:55.926Z sha 17cfc72 :: blocked before merge. :: none

SendTrackedRequestAsync returns Task<bool> now: false when the socket is
closed or the hello-ok handshake is still pending (#1418), true only after
the frame hit SendRawAsync successfully. Boolean mutation wrappers
(ResetSessionAsync et al.) propagate the not-sent result instead of
reporting a false success for frames that never reached the wire;
read-style fire-and-forget callers ignore the result and rely on the
post-handshake refresh burst, unchanged.

Adds two live round-trip tests capturing the wire-level trace:
withheld handshake suppresses sessions.reset (no frame on the wire,
submission=false), and a server-initiated one-way Close drives the
client's real auto-reconnect path, re-running the challenge -> connect
-> hello-ok dance before the gate reopens and the mutation is sent.
Redacted [trace] lines captured from the actual green dotnet test run
(TRX-embedded ITestOutputHelper output) of both HandshakeGate live
round-trip tests: pre-hello-ok suppression with zero frames on the wire,
gate held while disconnected after a server-initiated one-way Close, and
the real auto-reconnect path re-running challenge -> connect -> hello-ok
before the mutation is sent. Auth tokens, device ids, and device key
material redacted; all values are test-only.
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Sep 15, 2026
scripts/validate-docs.ps1 rejects U+2014 in maintained Markdown; replace
the six em dashes in the proof document with permitted punctuation. The
captured trace blocks are unchanged.
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Sep 15, 2026
@eleqtrizit

Copy link
Copy Markdown
Author

Body restored: Required proof pools, Validation, and Real Behavior Proof sections are filled in (captured artifact linked, exact commands and pass counts, explicit blockers: windows-wsl-gateway-e2e pool maintainer-scheduled; full build via CI). @clawsweeper re-review

@eleqtrizit
eleqtrizit marked this pull request as ready for review September 15, 2026 15:40
@eleqtrizit

Copy link
Copy Markdown
Author

windows-wsl-gateway-e2e pool recorded: maintainer-provided Windows interactive-desktop host (ARM64, dotnet 10.0.401 win-x64, WSL 2.7.12), manifest command at head 17cfc72 with OPENCLAW_RUN_E2E=1. Result total=20 passed=19 skipped=1 (opt-in Ollama test, expected skip) failed=0 exit=0. TRX and redacted loopback diagnostics in the evidence gist linked in the PR body. Both previously transient failures (emulated-x64 WSL gateway startup race) passed on re-run with no code changes. @clawsweeper re-review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P1 Urgent regression or broken agent/channel workflow affecting real users now. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Operator RPC can be sent before connect/hello-ok, causing 1008 PolicyViolation

1 participant