fix(core): disconnect the live session of a user who was just removed - #198
Merged
Merged
Conversation
Protocols that authenticate once per session keep serving a client after its user is gone from the inbound. Swapping the user table only decides who may start a *new* session, and ConnTracker closes the routed connections but not the session carrying them -- so the client opens another stream on the one it already has and is served. That is issue #175: a client DepleteJob disabled for running out of quota kept running, and the same hole covers every multiplex session, where one authenticated carrier connection serves every stream after it. core/usersession records, per inbound, which user the session at a source address authenticated as. Removing a user closes the sessions there is a closer to reach -- anytls, and the sing-mux carrier behind vless, vmess and trojan -- and mutes the ones there is not, which is all the QUIC protocols offer: their session lives inside sing-quic with no handle out. A muted source has nothing routed for it any more, so the traffic stops either way, and the block ages out after ten minutes rather than becoming a lockout. This sits at the inbound layer rather than in ConnTracker because a tracker-level gate only sees a connection after routing: it misses the ones the router answers itself, and refusing there still costs one real dial to the destination first. The two layers stay separate -- IP limits keep their gate in ConnTracker, whose ban state has to outlive a core restart. The hook is a router wrapper, not a field plus a block in every handler, so each copy under core/protocol carries a single added line: inbound.router = withUserSessions(inbound.router) Everything that line reaches lives in users.go, which the copy check skips, so the expected diffs grow by one each. anytls costs three instead: it holds its session in NewConnection, which the router never sees, so that call is redirected through users.go as well. Cutting hangs off each protocol's UpdateUsers, which leaves the service layer untouched and still covers all three paths that reach it -- DepleteJob, a panel save, and a node push. Two details are easy to undo by accident: - Idle entries are only dropped when they have no closer. A tracked session's lastSeen moves only when it opens another connection, so one carrying a single long-lived stream looks idle while it is perfectly alive; sweeping it would discard the only handle on it and the next removal would find nothing to cut. Tracked entries are cleaned up by their own deferred Untrack. - The user and the closer are recorded under one lock. Split apart, a removal landing in between sees a user with no closer, files the session as unclosable and mutes it -- and a mux carrier is never gated, so that mute does nothing at all. Verified on the test server, three protocol tunnels driven by a probe that reconnects every second, with the client expired and DepleteJob disabling it. Attempts made strictly after the disable: anytls hysteria2 vless+mux 1.8.2, no change 11 served 11 served 11 served with this change 0 served 0 served 0 served The control was built from main at the same version and sing-box, so the only variable is this change.
The backstop was measured from when the block was written, so a mute expired ten minutes later whatever the client was doing. But a client that keeps retrying is a session that is still alive -- and for QUIC, still authenticated, because the streams it opens afterwards never consult the user table again. A client DepleteJob had disabled therefore got its traffic back after ten minutes, on the very session the mute existed to stop. Both windows now run from the last refused attempt, so quiet is what lifts a block, not the clock. The sweep was a second road to the same place. Allowed only ever touched the block, never the entry, so a source being refused once a second looked idle and the sweep dropped its entry -- taking the block with it. Allowed now keeps the entry alive for as long as it is refusing it. A block also no longer changes kind behind the caller's back. A kick is about a user who is still enabled and lifts after thirty seconds of quiet; a removal is not, and must not be downgraded to that by kicking the same name, nor cleared by an unrelated save that happens to list the user in keep. `block.at` has no readers left and is gone. Found by re-reviewing the previous commit. The test meant to cover the backstop aged the block rather than the last attempt -- exactly the distinction that was wrong -- so it passed either way; it now ages the attempt, and a companion test pins the case it was missing. Verified on the test server across thirteen minutes of retries at one per second, spanning the ten-minute mark that used to end the mute: anytls 0 served / 695 refused hy2 0 served / 698 refused vless 0 served / 698 refused The hysteria2 inbound logged 718 arrivals in that window, so the session was alive throughout and being refused -- not quietly dead, which would have proved nothing.
…e cut Replaces the source-address mute of the two commits before this one, which could not be made correct. anytls and the sing-mux carrier arrive as a net.Conn that lives exactly as long as the session does, so those are tracked by address and closed outright. That part stands. The QUIC protocols give this layer no handle on the session and no usable name for one either. sing-quic keeps its session list unexported and the per-stream ctx it hands the handler is the Service's own, shared by every session. All three services set quic-go's DisablePathManager, which rewrites a connection's remote address on the first decryptable packet from a new one with no path validation, so the address moves under a NAT rebind; and it is an ephemeral UDP port that is recycled to somebody else afterwards. There is also no moment at which a QUIC session can be declared gone, because quic-go keeps it alive with a 10s PING whether or not a single byte is routed -- which is what defeated both the mute's quiet window and, in review, an attempt to age a per-session entry out after ten idle minutes. Either one silently let a removed user keep an idle session. So for QUIC the registry no longer tracks sessions. It records which users have been seen on the inbound: a set keyed by user rather than address, bounded by the client count rather than the session count, and therefore needing no expiry at all. CloseUsers reports those users as Unclosable, the QUIC inbounds turn that into ErrRestartRequired, and InboundService.UpdateInboundsUsers rebuilds the inbound -- which destroys every QUIC session on it, the removed user's included. Being wrong is one-sided on purpose: a user who connected and then left for good still costs one restart that bought nothing, while the other direction is issue #175 itself. The cost is only paid when a removed user is actually connected, so adding a user or rotating a credential still updates in place. Closes #175.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #175.
The hole
Protocols that authenticate once per session keep serving a client after its
user is gone from the inbound. Swapping the user table only decides who may
start a new session;
ConnTrackercloses the routed connections but not thesession carrying them, so the client opens another stream on the one it already
has and is served. Every multiplex session has the same shape — one
authenticated carrier connection serves every stream opened after it.
Measured on the test server against
main, a clientDepleteJobhad justdisabled for running out of quota was served 11 out of 11 retries on anytls,
hysteria2 and vless+mux alike.
Two kinds of session
anytls and the sing-mux carrier arrive as a
net.Connthat lives exactly aslong as the session does.
core/usersessiontracks those by source address andcloses them outright — the key is one TCP connection's address, it cannot move,
and
Untrackruns when the session ends. That is the whole story for anytls,vless, vmess and trojan, and it disturbs nobody else on the inbound.
The QUIC protocols give this layer no handle on the session, and no usable
name for one either:
sing-quickeeps its session list unexported, and the ctx it hands thehandler per stream is the
Service's own (serverSession{ctx: s.ctx}),shared by every session on the listener;
DisablePathManager, which rewrites aconnection's remote address the moment a decryptable packet arrives from a new
one, with no path validation — one NAT rebind and the address is different;
session ends, and nothing tells this layer that it has been;
quic-go keeps one alive with a 10s PING whether or not a single byte is
routed, so an idle session and a dead one are indistinguishable from here.
Two earlier revisions of this PR tried to do better and could not. Muting the
source address is defeated by the second and third points. Ageing a per-session
entry out after ten idle minutes — which is what the first review round of this
PR produced — is defeated by the fourth: a client idle past the window was
dropped from the registry, no restart was asked for, and its still-authenticated
session went on being served. That is #175 verbatim. Both versions looked fine
because a probe that reconnects every second is never idle and never moves.
What this does
For QUIC the registry does not track sessions at all. It records something
weaker and completely reliable: which users have been seen on this inbound.
That set is keyed by user rather than by address, so a client whose address
moves is still one entry; and it is bounded by the client count rather than the
session count, which is what lets it carry no expiry at all. A set with no
expiry is the only kind that cannot be wrong about a session it cannot see.
CloseUserscuts every session it holds a transport for and reports the rest asUnclosable. The QUIC inbounds turn that intoErrRestartRequired, andInboundService.UpdateInboundsUsersrebuilds the inbound — the sameRemoveInbound/AddInboundpath protocols with no in-place update alreadytake. That destroys every QUIC session on the listener, the removed user's
included.
Nothing in
core/usersessionrefuses a connection any more, in any mode.What it costs
On a QUIC inbound, disabling a user who has been connected disconnects
everyone on that inbound once. They reconnect on their own.
Being wrong here is one-sided on purpose: a user who connected and then left for
good is still in the seen set, so removing them costs one restart that bought
nothing. The other direction — deciding a session is gone when it is not — is
issue #175, so the cost is paid on that side.
It is not paid otherwise: adding a user, rotating a UUID or password, or
removing a user who never connected all still update in place.
DepleteClientsdisables every expired client in one transaction and calls
UpdateInboundsUsersonce with the union of their inbounds, so a deplete roundcosts at most one restart per inbound, not one per client.
One line per copy
Unchanged, and
scripts/check-protocol-copies.shstill reports 8/8 ok againstsing-box v1.14.1 with the same
expect_diffcounts: everything here lives inusers.go(exempt) or in the header comment above thepackageclause (whichthe script strips). Six copies carry the single line
and anytls costs three, because it holds its session in
NewConnection, whichthe router never sees.
Verification
The hole is measured; this implementation is not yet. The 11/11 figure above
was taken against
mainand stands as evidence that the bug is real. What hasbeen verified for the code in this PR is everything below the network:
check-protocol-copies.sh(8/8),go vet,go buildandgo test ./...with the CI tag set, all green;core/usersessionclean under-race, including a stress run driving everyexported method from eight goroutines at once, which also asserts that the
seen set grows per user and not per address;
out one at a time and the matching test confirmed to go red — including
"expire an idle user out of the seen set", which is the defect described
above, and "key the seen set by address instead of user".
Mutation testing is also what makes the review trail below worth trusting
rather than just worth reading: the ten-minute-idle defect was found by a code
review that ran after a 12/12 mutation score, because the suite at that point
had the wrong behaviour written into it as an assertion
(
TestSweptSourceIsNotReportedUnclosable). That test is gone, replaced byTestIdleUserIsNeverForgotten.Not covered by anything automatic: the one line in each QUIC
users.gothatturns
UnclosableintoErrRestartRequired. Constructing an*Inboundneedsthe whole of
NewInbound(TLS config, listener), which is too much to stand upin a unit test.
Result.RestartRequireditself is table-tested, and the sevencall sites were checked by hand — the three QUIC protocols call it, the other
four deliberately do not and say why.
Still to do before merge: a run on the test server — deplete a connected
client on a hysteria2 inbound and confirm both halves, that it is actually cut
off and that the inbound comes back. Worth doing with an idle client too,
since that is the case both earlier revisions got wrong.
Known edges
sing-box's own inbound; forking it belongs with the live-traffic work.
KickUserSessionsexists and is tested, but has no caller until thepanel-side disconnect lands. On a QUIC inbound it can only report
Unclosable— whether disconnecting one user is worth restarting the inboundeveryone else is on is a decision for that caller, not this layer.
RemoveInboundthen a failingAddInboundleaves the inbound gone from thecore while the transaction rolls back, and
checkCoreJobwill not noticebecause the core itself is still up. That is pre-existing behaviour for every
protocol without an in-place update, but QUIC deplete now takes this path
routinely, so the exposure is larger than it was. Not addressed here.
fix: close sessions of removed users on UpdateUsers anytls/sing-anytls#4 are still open. A per-session handle would turn the
restart back into a targeted close.