Skip to content

fix: recover protocol switches and keep user permissions current - #80

Open
marvinli001 wants to merge 2 commits into
cedar2025:devfrom
marvinli001:fix/protocol-switch-recovery
Open

marvinli001 wants to merge 2 commits into
cedar2025:devfrom
marvinli001:fix/protocol-switch-recovery

Conversation

@marvinli001

@marvinli001 marvinli001 commented Sep 8, 2026 •

Copy link
Copy Markdown

Problem

Changing a node's protocol changes more than its inbound type. A machine reassigned from a VLESS REALITY node to Hysteria2 can successfully stop the old node, then fail the replacement with:

initial setup: protocol "hysteria" requires TLS certificate files

The certificate manager defaults to none, while the machine registry can retain the exited service's handle. Subsequent discovery then skips it. Related paths also advance the received config hash before application succeeds, validate a new transport against the previous kernel, retain obsolete sing-box inbound tags, or leave Xray listeners accepting while shutdown drains asynchronously.

This PR makes protocol changes a retryable reconciliation between the desired snapshot and the successfully applied snapshot. It covers both an existing node ID changing configuration and a machine binding moving to a different node ID.

There are intentional compatibility changes to certificate defaults and Hysteria2 user updates; these are described below rather than hidden in the implementation.

What changes

One preparation/application path

Initialization, WS configuration messages, REST snapshots and retries now feed the same sequence:

  1. Clone and validate the target's shape.
  2. Select a compatible kernel using the target protocol/version, transport, TLS/REALITY mode, multiplexing, proxy protocol and custom configuration.
  3. Resolve certificate policy and prepare material without replacing the certificate currently serving.
  4. Validate the complete kernel configuration before binding.
  5. Apply the affected inbound/kernel and commit the applied state only after success.

The original configured kernel remains the preference. Automatically selecting Xray for XHTTP does not permanently change it. Unsupported combinations are reported instead of silently ignoring settings. No new underlying protocol or dependency version is introduced.

Recoverable lifecycle

  • Failed applications do not advance the applied hash. Duplicate messages and 304 responses therefore do not suppress recovery.
  • Bootstrap and apply failures retry in process with capped exponential backoff. Initial machine discovery also retries transient panel failures.
  • Machine handles track generations and exited services become restartable. Late exits cannot remove a newer instance.
  • Removed/disabled bindings stop before their replacements start, and are never restored as rollback.
  • sing-box removes inbound tags absent from the new target. Xray closes old inbound handlers synchronously before a replacement binds, with bounded draining.
  • Rollback of a still-authorized binding uses the latest user set. Cancellation is checked before restoring or committing.
  • Switching to a target without ordinary TLS deactivates obsolete certificate work after successful application.
  • Zero-user nodes remain synchronized and start when users become available.

Authorization is independent of a failed target

Retaining the old listener during a bad configuration update must not retain old permissions. User removals/rotations are now applied to the kernel actually serving, even if the new target cannot be prepared. If that listener cannot enforce the current user set, it stops and retries instead of indefinitely accepting revoked credentials.

A stale REST user response can no longer undo a newer WS revocation just because the config response was 304/nil. ETag invalidation is synchronized and generation-aware, so an older in-flight HTTP response cannot restore invalidated tags. Bootstrap clears both tags before retrying an incomplete initial snapshot.

UserDiff also removes the old identity when the same user ID acquires a new UUID, preserving Xray's remove-before-add requirement.

Snapshot ownership and Xray user-update failures

The service captures and owns the desired configuration before hashing it. The prepared target hashes the snapshot it actually validates and applies. Deep copies preserve the JSON representation, including empty containers and nested typed nil slices, so duplicate WS messages or REST/304 responses do not trigger repeated application through mismatched hashes.

Xray converts all credentials to add before its first native change. Native additions/removals stop at the first failure, and bookkeeping records the completed operations rather than the requested whole set. A failed native update rebuilds the instance from the latest desired users; if recovery also fails, an error reaches the service, which stops the listener and retains that desired set for retry. Successful recovery is reported as a completed transition; unrecovered failures never advance the applied user hash. Removal errors are considered idempotent only when GetUser confirms the account is absent.

UUID rotation removes the old identity once before adding its replacement. Service-level delta rotation uses the kernels' full user-set replacement rather than a separate pre-removal whose error could be lost. Limit-only changes retain their existing non-restarting path.

This covers both the snapshot-hash inline comment and the Xray user-update error accounting concern in Copilot's review. Fault-injection tests exercise the production UserManager lookup, partial changes, recovery into a real Xray instance, and failed recovery. Real VLESS clients verify credential rotation and service recovery after rejected credentials. The service-level real-kernel failure case exercises credential conversion; mid-operation AddUser/RemoveUser failures are injected at the Xray adapter layer.

Certificate behavior and compatibility

Configuration Result
Managed TLS target, no explicit certificate policy Prepare/reuse a self-signed certificate
Supported REALITY target Validate REALITY settings; no ordinary certificate
Plaintext/non-TLS target No certificate issuance
Explicit self/file/content/http/dns Honor that policy; errors are not downgraded to self-signed
Explicit none combined with required/requested TLS Report a configuration conflict
Legacy auto_tls: true Remains ACME HTTP-01
Standalone TLS target Still requires an explicit certificate strategy

Automatic certificates use a namespace derived from panel URL and node ID beneath the configured cache root. This prevents entries inheriting one cert_dir from overwriting each other. Explicit certificate paths remain unchanged. Reuse validates the key pair, time validity and names/SANs; keys are persisted with mode 0600.

Important consequences:

  • Self-signed material solves server startup, not client trust. Strict clients must trust/pin it or operators must configure a trusted certificate. No subscription or client verification setting is changed.
  • tls=1 never silently becomes a plaintext listener merely because a certificate was omitted. Existing deployments relying on implicit TLS offload need an explicit backend TLS contract. This PR does not infer the presence of a CDN/nginx; panels that couple public and backend TLS may need a separate offload setting.
  • If a panel omits cert_config when its UI says none, the node cannot distinguish that omission from an absent policy. An explicit opt-out must actually be sent.
  • REALITY and ECH keys are not ordinary TLS certificates.

Hysteria2 concurrency tradeoff

The pinned sing-quic v0.6.0 updater writes authentication maps concurrently with its HTTP/3 authentication reader. A lock around only the caller does not protect those readers.

Hysteria2 user changes therefore replace that inbound instead of calling the unsafe updater. Existing QUIC sessions may be interrupted and clients may need to reconnect. Other supported inbounds keep their native user-update path. The regression test permits this documented reconnect window while checking the final authorization state and running under the race detector.

This avoids introducing an additional dependency fork in this PR. A future concurrency-safe dependency API could restore an in-place update for Hysteria2.

Review guide

Area Files / focus
Desired/applied state, retries, authorization internal/service/service.go, target.go
Capability/TLS classification and kernel selection internal/model/capability.go, validate.go
Staged certificate material and persistence internal/cert/cert.go
Machine lifecycle and generations internal/machine/machine.go
Listener removal, replacement and rollback internal/kernel/singbox/singbox.go, internal/kernel/xray/xray.go
Snapshot normalization and ETags internal/controlplane/*, internal/panel/client.go
Extended behavior/compatibility notes docs/protocol-switching.md

The patch deliberately excludes downstream provisioning, installation-source, monitoring and release integrations.

Local validation

The final review revision (dbf1970) was independently verified on macOS arm64 / Go 1.26.8:

  • 477 tests/subtests passed, with the race detector and full production tags; zero failed tests, skipped tests, or data races.
  • Changed Go files pass formatting checks; git diff --check is clean.
  • go vet with production tags and go mod verify passed.
  • make build-linux and make build-linux-arm64 passed for both the node and xbctl binaries. These are cross-builds, not VPS acceptance tests.
  • The nested nil-slice regression failed before the final guard and passes after it. The Xray error-injection tests also exposed failures in the original implementation before the repair.

Linux runtime validation of the final commit is recorded separately under CI below. Temporary local build binaries were removed; this independent review did not create a VM or container.

Full suite command (the package set and tags match make test):

go test -race -count=1 -json \
  -tags "with_quic with_utls with_wireguard with_acme with_clash_api" ./...
go vet -tags "with_quic with_utls with_wireguard with_acme with_clash_api" ./...
go mod verify
make build-linux
make build-linux-arm64

The real-kernel regressions exercise:

  • Automatic Hysteria2 certificates, QUIC authentication and an HTTPS request through the proxy on IPv4 and IPv6 loopback.
  • REALITY → Hysteria2 → REALITY through WS/REST, and machine node-ID rebinding.
  • sing-box → Xray XHTTP → sing-box, including obsolete-listener removal.
  • Bad targets, port conflicts, cancellation, empty-user recovery, and initial HTTP/config/user snapshots across retries.
  • Rejection of plaintext VLESS when TLS is requested.
  • Fresh authentication attempts by revoked users while the desired target is invalid, through WS delta/full and REST.
  • A late REST user list after a WS revocation.
  • Concurrent Hysteria2 authentication/user refresh, certificate-cache isolation, and ETag invalidation while responses are in flight.
  • Real Xray/VLESS credential replacement, rejection of the previous credential, and listener shutdown/recovery when the desired credentials cannot be constructed.

Fixtures use local servers, temporary certificates, test users and random ports. No production subscription, VPS credentials or external ACME/DNS account was used.

CI

Verified fork CI for dbf1970: run 34229043987 passed 477 tests/subtests with the race detector on Go 1.26.7 / linux/amd64, plus both Linux amd64 and arm64 build jobs. No individual tests failed or were skipped. Image publication and release jobs were correctly skipped on the fork.

The upstream PR run for the same commit is action_required, with no jobs executed. That workflow requires maintainer approval; it has not run the tests or builds.

  • Run the suite with the same feature tags as shipped binaries, including QUIC, uTLS and ACME.
  • Read the Go version from go.mod.
  • Include dev PRs and run tests/builds on feature branches.
  • Gate image/release publication to pushes in the canonical repository and its release branches/tags; fork and PR validation cannot publish canonical artifacts.

Related work

Copilot AI lite review requested due to automatic review settings September 8, 2026 08:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are correctness issues in the updated hashing and Xray user-update success accounting that can desynchronize applied state or report successful authorization changes when underlying operations failed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR restructures node protocol switching into a reconciliation flow between “desired” and “successfully applied” snapshots, aiming to make protocol changes retryable while keeping authorization and certificates consistent during failures.

Changes:

  • Introduces a unified target preparation pipeline (shape validation → kernel selection → cert planning/prep → kernel config validation) to avoid partially applied protocol switches.
  • Hardens lifecycle recovery: retry/backoff for failed applies and machine node restarts, plus generation-aware handle tracking.
  • Tightens certificate defaults and ETag invalidation behavior, with expanded regression coverage and documentation.
File summaries
File Description
README.md Adds link to new protocol-switching/recovery documentation.
Makefile Centralizes build/test tags and runs full-suite tests with tags.
internal/service/target.go Adds target preparation pipeline including cert planning and kernel validation pre-bind.
internal/service/target_test.go Tests cert policy matrix, identity naming, and auto-cert isolation.
internal/service/realkernel_test.go Adds real-kernel end-to-end regression scenarios for switching and recovery.
internal/service/bootstrap_http_test.go Regression test for bootstrap retry semantics and user refetch behavior.
internal/service/authorization_regression_test.go Regressions for TLS downgrade prevention and revocation correctness under failed switches.
internal/service/authentication_concurrency_test.go Race-oriented test around HY2 auth vs user refresh replacement.
internal/panel/client.go Adds generation-aware ETag caching to prevent stale in-flight restores.
internal/panel/cache_test.go Tests ETag invalidation vs in-flight responses and concurrency.
internal/model/validate.go Splits validation into kernel-independent “shape” vs kernel-specific validation.
internal/model/types.go Adds deep cloning helpers for NodeSpec and user slices.
internal/model/panel.go Validates panel snapshots with shape-only validation to allow kernel reselection later.
internal/model/panel_test.go Updates tests for shape-vs-kernel validation behavior.
internal/model/capability.go Adds capability/TLS requirement classification and kernel selection logic.
internal/model/capability_test.go Pins compatibility tables and tests kernel selection and config hints.
internal/machine/machine.go Adds restartable backoff lifecycle, serialized discovery triggers, and generation-aware exits.
internal/machine/machine_test.go Covers rebinding, backoff restarts, wakeups, and stale-exit handling.
internal/kernel/xray/xray.go Adds Validate(), synchronous retire/bind sequencing, and UUID-rotation removal handling.
internal/kernel/xray/config.go Avoids mutating desired state when forcing Trojan TLS; always applies stream settings for HTTP.
internal/kernel/xray/config_test.go Adds tests for non-mutation and proxy-protocol behavior on plain HTTP.
internal/kernel/singbox/singbox.go Adds Validate(), inbound removal on switch, rollback logic, and HY2 user replacement strategy.
internal/kernel/singbox/reload_test.go Tests stale inbound removal, rollback on reload failure, and Validate behavior.
internal/kernel/singbox/export_test.go Exposes inbound manager to tests.
internal/kernel/kernel.go Adds Kernel.Validate() and improves UserDiff for UUID rotation.
internal/kernel/kernel_test.go Tests UserDiff rotation vs limit-only changes.
internal/controlplane/types.go Adds Source.ResetCache() for ETag invalidation.
internal/controlplane/panel.go Implements ResetCache via panel client ETag reset.
internal/controlplane/machine.go Implements ResetCache for machine control plane.
internal/controlplane/local.go Implements no-op ResetCache for local control plane.
internal/controlplane/panel_test.go Updates WS translation tests to be kernel-independent where appropriate.
internal/controlplane/mailbox.go Switches mailbox cloning to shared model.CloneNodeSpec().
internal/cert/cert.go Refactors cert manager into Prepare/Commit workflow; adds reusable self-signed behavior and ACME instance tracking.
internal/cert/selfsigned_test.go Adds comprehensive self-signed persistence/reuse/rotation tests.
docs/protocol-switching.md Documents protocol switching semantics, certificate defaults, lifecycle behavior, and test guidance.
.github/workflows/ci.yml Uses go.mod Go version; broadens push branches and gates publish steps to canonical pushes.
Review details

Suppressed comments (1)

internal/kernel/xray/xray.go:563

  • UpdateUsers logs and commits x.users = users even when AddUser fails (and it also reports added/removed based on the diff, not actual successful operations). That can leave the in-memory snapshot out of sync with the real xray UserManager state and reintroduce the misleading “users updated” success scenario from #53 when an add fails.
	for _, u := range toAdd {
		mu, err := toMemoryUser(proto, nc, u)
		if err != nil {
			nlog.Core().Warn("xray: skip user in UpdateUsers", "user", u.ID, "error", err)
			continue
  • Files reviewed: 38/38 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +98 to +102
kernelType: selection.Kernel,
kernelReason: selection.Reason,
plan: plan,
configHash: computeConfigHash(spec),
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants