From 0375f05233d896a48014d95b0a8b6e4d30ae034c Mon Sep 17 00:00:00 2001 From: Danathar Date: Wed, 11 Nov 2026 09:17:24 -0500 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B=20[v6]=20claude=20auth:=20never=20?= =?UTF-8?q?discard=20an=20agent's=20refreshed=20credential=20the=20server?= =?UTF-8?q?=20cannot=20read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In a per-UID deployment the hive server cannot read the 0600 file an agent's Claude Code token refresh leaves in place of its credential symlink. claude.LoadTokens returned nil on that read error, so the bridge repair treated the fleet's only fresh token (the refresh had rotated the shared grant) as "no token": it kept the dead shared file, moved the fresh file aside and relinked the agent, and every agent logged out when the shared access token expired. - entrypoint root perm guard: hive_rebridge_claude_credentials runs the bridge repair every 5 s as root, with the Go promotion rule. It judges credentials with jq output that is only flags and expiresAt, reads the private file with dd iflag=nofollow from a cd -P pinned directory, and writes the promoted file as the shared file's owner and group (setpriv) with the shared mode minus world bits, via temp file and rename. - Go fail-safe: claude.ReadTokens keeps the read error. The repair no longer moves aside or relinks a private file it cannot read; it logs one ERROR per agent and file version and leaves the agent on its copy. - Only a complete credential (access token, refresh token, expiresAt > 0) that is usable can win, in both repairs; promoteClaudeCredential re-checks the bytes it writes; WriteCredentials refuses a credential with no access token. - Tests (Go, not run locally) and perm guard script cases; docs and changelog. Refs #17253, #17590 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar --- ...aude-bridge-unreadable-agent-credential.md | 1 + src/crux/agent/claude_credential_bridge.go | 136 ++++++++++- .../claude_credential_bridge_internal_test.go | 213 +++++++++++++++++ src/crux/claude/oauth.go | 41 +++- src/crux/claude/oauth_read_tokens_test.go | 77 +++++++ src/deploy/entrypoint.sh | 214 +++++++++++++++++- src/deploy/test_entrypoint_perm_guard.sh | 205 +++++++++++++++++ src/docs/agent-configuration.md | 17 +- src/docs/agent-watchdog.md | 7 +- 9 files changed, 891 insertions(+), 20 deletions(-) create mode 100644 changelog.d/fixed-claude-bridge-unreadable-agent-credential.md create mode 100644 src/crux/claude/oauth_read_tokens_test.go diff --git a/changelog.d/fixed-claude-bridge-unreadable-agent-credential.md b/changelog.d/fixed-claude-bridge-unreadable-agent-credential.md new file mode 100644 index 0000000000..7fa7556a7e --- /dev/null +++ b/changelog.d/fixed-claude-bridge-unreadable-agent-credential.md @@ -0,0 +1 @@ +- Claude credential bridge: on a per-UID hive the hive server cannot read the private `0600` file an agent's token refresh leaves in place of its credential link, and the repair took that read error for "no token". It kept the shared file, whose grant the refresh had just rotated, and moved the agent's fresh token aside, so every Claude agent logged out when the shared access token expired. The container entrypoint's root permission guard now runs the same repair every 5 seconds: it judges both files without printing any token, promotes the private copy only when it is complete (access token, refresh token, `expiresAt`), usable and newer, writes it as the shared file's owner and group with the shared file's mode (world bits removed), and restores the link. The hive server's own repair no longer moves aside or relinks a private file it cannot read; it logs one ERROR per agent and file version instead. A failed refresh's empty-token, `expiresAt` 0 file never replaces the shared login, and `claude.WriteCredentials` refuses a credential with no access token (#17253, #17590). diff --git a/src/crux/agent/claude_credential_bridge.go b/src/crux/agent/claude_credential_bridge.go index 4ca94d05f0..60a71ec2e7 100644 --- a/src/crux/agent/claude_credential_bridge.go +++ b/src/crux/agent/claude_credential_bridge.go @@ -39,10 +39,11 @@ import ( // whichever of the two credentials is newer AND usable in the shared file, and // restores the symlink: // -// - the private copy wins only when it is usable (claude.TokensUsable: a live -// access token, or a refresh grant that has not expired) and either the -// shared credential is not usable or the private access token expires -// LATER than the shared one. A refresh mints a later expiresAt, so this is +// - the private copy wins only when it is complete and usable +// (claudeBridgePromotable: access and refresh tokens present, an expiresAt +// recorded, and claude.TokensUsable: a live access token, or a refresh +// grant that has not expired) and either the shared credential is not +// usable or the private access token expires LATER than the shared one. A refresh mints a later expiresAt, so this is // what promotes one agent's fresh refresh to the whole fleet instead of // stranding everyone else on the grant it just rotated; // - otherwise the shared file stays as it is (ties keep the shared file, so @@ -59,6 +60,34 @@ import ( // to /data/home/.claude): that "private file" is the fleet's login. Such an // agent is skipped with one warning; its next launch retires the directory // link (setupPerAgentCLIHome) before the repair runs. +// +// Who can judge the private file. In a per-UID deployment the hive server runs +// as its own uid (dev), and the private file is the AGENT's, mode 0600, so the +// server cannot read it. It can still rename it (the agent's .claude directory +// is writable to it), and it used to: a read error looked exactly like "no +// token", the shared copy was kept, and the only fresh token in the fleet (a +// refresh rotates the grant the shared file still holds) was moved aside +// while every other agent was relinked to the dead grant. Two things now +// stop that: +// +// - The root perm guard in src/deploy/entrypoint.sh +// (hive_rebridge_claude_credentials) runs this same repair, with the same +// promotion rule, every 5 seconds. As root it can read the private file; +// it writes the promoted credential with the shared file's own owner, +// group and mode, so no token becomes readable by anyone who could not +// already read the shared login. It is also fast enough to publish one +// agent's refresh before the other agents' CLIs reach for the old grant. +// - This Go repair never moves aside or relinks a private file it cannot +// read (any read error other than "not there"): it logs one ERROR per +// agent and file version and leaves the agent on its private copy, which +// is at worst the token it was already running on. The dashboard keeps +// showing the bridge as broken until one of the repairs succeeds. +// +// Neither repair lets an incomplete credential win: the private copy must hold +// an access token, a refresh token and a recorded expiresAt +// (claudeBridgePromotable). A failed refresh leaves exactly such an +// incomplete file (empty tokens, expiresAt 0), and promoting it would sign +// the whole fleet out. // Repair triggers, recorded in the audit detail and the log. const ( @@ -79,9 +108,13 @@ const claudeSharedCredentialMode = 0o660 // Test seams. chownClaudeCredential applies the previous shared file's owner // to the promoted replacement; it is a seam so tests run without root or real // agent uids. claudeBridgeNow stamps the moved-aside file names. +// claudeBridgeReadTokens reads a credential for the repair; tests replace it to +// stand in for another uid's 0600 file, which a test running as one user (or +// as root) cannot produce for real. var ( - chownClaudeCredential = os.Chown - claudeBridgeNow = time.Now + chownClaudeCredential = os.Chown + claudeBridgeNow = time.Now + claudeBridgeReadTokens = claude.ReadTokens ) // claudeBridgeMu serializes repairs: the launch path, the watchdog and the @@ -193,10 +226,37 @@ func claudeBridgeLinkIsTarget(link, target string) (bool, error) { return os.SameFile(li, ti), nil } +// errClaudeBridgeAgentUnreadable: the agent's private credential exists but +// this process cannot read it (in a per-UID deployment, the agent's 0600 file +// and the hive server's own uid). Its token cannot be judged, so the repair +// leaves it exactly where it is. +var errClaudeBridgeAgentUnreadable = errors.New("agent's private credential cannot be read by the hive process") + +// errClaudeBridgeNotPromotable: promoteClaudeCredential re-checks the bytes it +// is about to write and refuses an incomplete or unusable credential. +var errClaudeBridgeNotPromotable = errors.New("refusing to promote an incomplete or unusable credential over the shared login") + +// claudeBridgeUnreadableWarned holds, per credential link, the modification +// time of the unreadable private file last logged, so the watchdog logs once +// per agent and file version instead of every cycle. A new refresh (a new +// mtime) is logged again. +var claudeBridgeUnreadableWarned sync.Map + +// claudeBridgePromotable reports whether t may replace the shared login: a +// complete credential (access token, refresh token and a recorded expiresAt) +// that is usable now. A failed refresh leaves an empty-token, expiresAt-0 +// file, which never qualifies; neither does a token with no recorded expiry, +// which cannot be judged newer than anything. The root perm guard in +// src/deploy/entrypoint.sh applies the same rule. +func claudeBridgePromotable(t *claude.OAuthTokens, now time.Time) bool { + return t != nil && t.AccessToken != "" && t.RefreshToken != "" && t.ExpiresAt > 0 && + claude.TokensUsable(t, now) +} + // preferAgentClaudeCredential decides which credential the shared file keeps. // See the file comment for the rule. func preferAgentClaudeCredential(agent, shared *claude.OAuthTokens, now time.Time) bool { - if !claude.TokensUsable(agent, now) { + if !claudeBridgePromotable(agent, now) { return false } if !claude.TokensUsable(shared, now) { @@ -210,7 +270,9 @@ func preferAgentClaudeCredential(agent, shared *claude.OAuthTokens, now time.Tim // repair (a symlink, an absent link, a directory, which is not this failure, // or a link or target directory that cannot be resolved). It returns an error // wrapping errClaudeBridgeLinkIsTarget, and touches nothing, when the link -// path resolves to the shared credential itself. +// path resolves to the shared credential itself, and one wrapping +// errClaudeBridgeAgentUnreadable, again touching nothing, when this process +// cannot read the private file. func (m *Manager) repairClaudeCredentialBridge(agentName, link, target string) (*claudeBridgeRepair, error) { claudeBridgeMu.Lock() defer claudeBridgeMu.Unlock() @@ -235,6 +297,17 @@ func (m *Manager) repairClaudeCredentialBridge(agentName, link, target string) ( if same { return nil, fmt.Errorf("%w: %s", errClaudeBridgeLinkIsTarget, link) } + + // Never move aside a file whose token cannot be judged. An unreadable + // private copy is, in a per-UID deployment, the normal case right after a + // refresh, and it may be the only fresh token in the fleet. + if _, rerr := claudeBridgeReadTokens(link); rerr != nil { + if errors.Is(rerr, fs.ErrNotExist) { + // Gone since the Lstat: the CLI is mid-rename. Next cycle. + return nil, nil + } + return nil, fmt.Errorf("%w: %w", errClaudeBridgeAgentUnreadable, rerr) + } now := claudeBridgeNow() // Move the private file aside FIRST. After the rename the CLI can no @@ -244,9 +317,22 @@ func (m *Manager) repairClaudeCredentialBridge(agentName, link, target string) ( if err != nil { return nil, fmt.Errorf("move private credential aside: %w", err) } + agentTokens, rerr := claudeBridgeReadTokens(staleAgent) + if rerr != nil { + // The file renamed is not the one the check above read (the CLI + // replaced it in between) and this one cannot be read either. Put it + // back unless the CLI has already written yet another file there. + if _, lerr := os.Lstat(link); errors.Is(lerr, fs.ErrNotExist) { + if berr := os.Rename(staleAgent, link); berr != nil { + m.logger.Error("claude credential bridge: an unreadable private credential could not be put back; it is kept at the stale path", + "agent", agentName, "stale", staleAgent, "error", berr) + } + } + return nil, fmt.Errorf("%w: %w", errClaudeBridgeAgentUnreadable, rerr) + } rep := &claudeBridgeRepair{agent: agentName, kept: "shared", staleAgentPath: staleAgent} - if preferAgentClaudeCredential(claude.LoadTokens(staleAgent), claude.LoadTokens(target), now) { + if preferAgentClaudeCredential(agentTokens, claude.LoadTokens(target), now) { staleShared, perr := promoteClaudeCredential(staleAgent, target, now) if perr != nil { // Put the private file back: the agent's copy was the only usable @@ -298,12 +384,17 @@ func moveAsideClaudeCredential(path string, now time.Time) (string, error) { // promoteClaudeCredential replaces the shared credential with the bytes of src, // atomically and at claudeSharedCredentialMode, owned like the file it // replaces. The replaced file is preserved beside it first; its path is -// returned ("" when there was no shared file). +// returned ("" when there was no shared file). The bytes written are checked +// again with claudeBridgePromotable, so a src that changed after it was judged +// can never put an incomplete credential over the shared login. func promoteClaudeCredential(src, target string, now time.Time) (string, error) { data, err := os.ReadFile(src) if err != nil { return "", err } + if !claudeBridgePromotable(claude.ParseTokens(data), now) { + return "", errClaudeBridgeNotPromotable + } uid, gid := -1, -1 staleShared := "" if info, err := os.Stat(target); err == nil { @@ -386,6 +477,11 @@ func (m *Manager) rebridgeClaudeCredential(agentName, link, target, trigger stri // Any other outcome clears the mark, so a legacy layout that shows up // again later is logged again. claudeBridgeLegacyWarned.Delete(link) + if errors.Is(err, errClaudeBridgeAgentUnreadable) { + m.logClaudeBridgeUnreadable(agentName, link, trigger, err) + return nil + } + claudeBridgeUnreadableWarned.Delete(link) if err != nil { // Logged, not audited: a repair that keeps failing (an unprivileged // hive that cannot enter the agent's 0700 home) would otherwise add an @@ -415,6 +511,26 @@ func (m *Manager) rebridgeClaudeCredential(agentName, link, target, trigger stri return rep } +// logClaudeBridgeUnreadable says, once per agent and private-file version, +// that the repair left an agent on a private credential it could not read. +// Not audited: nothing changed. +func (m *Manager) logClaudeBridgeUnreadable(agentName, link, trigger string, err error) { + var stamp int64 + if info, lerr := os.Lstat(link); lerr == nil { + stamp = info.ModTime().UnixNano() + } + if prev, loaded := claudeBridgeUnreadableWarned.Swap(link, stamp); loaded && prev == stamp { + return + } + errno := "" + var en syscall.Errno + if errors.As(err, &en) { + errno = fmt.Sprintf("%d (%s)", int(en), en.Error()) + } + m.logger.Error("claude credential bridge: the hive process cannot read this agent's private credential, so its token cannot be judged; leaving the agent on its private copy (not moved aside, not re-linked). The root perm guard in the container entrypoint repairs it within seconds; if this repeats for the same agent, that guard is not running", + "agent", agentName, "path", link, "trigger", trigger, "errno", errno, "error", err) +} + // RepairClaudeCredentialBridges checks every per-agent Claude credential // bridge and repairs each one a regular file has replaced. It returns the // names of the agents it re-bridged, sorted. Agents are snapshotted under m.mu diff --git a/src/crux/agent/claude_credential_bridge_internal_test.go b/src/crux/agent/claude_credential_bridge_internal_test.go index 9d4d9ce5e3..451c852297 100644 --- a/src/crux/agent/claude_credential_bridge_internal_test.go +++ b/src/crux/agent/claude_credential_bridge_internal_test.go @@ -3,6 +3,7 @@ package agent import ( "bytes" "errors" + "io/fs" "log/slog" "os" "path/filepath" @@ -335,3 +336,215 @@ func TestSetupInteractiveHome_LegacyDirLinkLaunchRetiresBeforeRebridge(t *testin } assertBridgeRestored(t, link, target) } + +// withUnreadableClaudeCredential makes the repair's credential reader fail +// with EACCES for every path unreadable(path) accepts, as the hive server's +// uid does on an agent's 0600 file in a per-UID deployment. A test cannot +// produce that for real without root and a second uid. +func withUnreadableClaudeCredential(t *testing.T, unreadable func(path string) bool) { + t.Helper() + orig := claudeBridgeReadTokens + claudeBridgeReadTokens = func(path string) (*claude.OAuthTokens, error) { + if unreadable(path) { + return nil, &fs.PathError{Op: "open", Path: path, Err: syscall.EACCES} + } + return orig(path) + } + t.Cleanup(func() { claudeBridgeReadTokens = orig }) +} + +// TestRepairClaudeCredentialBridge_UnreadablePrivateCredentialLeftInPlace is +// the live per-UID incident: an agent's refresh left a fresh 0600 private +// file (the only live token, its refresh rotated the shared grant) that the +// hive server cannot read. On the v6 tip the read error looked like "no +// token": the file was moved aside, kept=shared, and the agent was relinked +// to the dead grant. Now nothing moves, nothing is linked, the dashboard +// still reports the bridge as broken, and the ERROR is logged once per file +// version rather than every watchdog cycle. +func TestRepairClaudeCredentialBridge_UnreadablePrivateCredentialLeftInPlace(t *testing.T) { + shared := withSharedAgentHome(t) + m, sink := testManagerWithSink(t) + var logs bytes.Buffer + m.logger = slog.New(slog.NewTextHandler(&logs, nil)) + now := time.Now() + sharedJSON := fakeClaudeCredential("fake-rotated-access", now.Add(3*time.Minute), "fake-rotated-refresh") + agentJSON := fakeClaudeCredential("fake-fresh-access", now.Add(8*time.Hour), "fake-fresh-refresh") + link, target := brokenBridgeFixture(t, shared, "quality", sharedJSON, agentJSON) + agentDir := filepath.Dir(link) + withUnreadableClaudeCredential(t, func(path string) bool { return filepath.Dir(path) == agentDir }) + + rep, err := m.repairClaudeCredentialBridge("quality", link, target) + if rep != nil || !errors.Is(err, errClaudeBridgeAgentUnreadable) || !errors.Is(err, fs.ErrPermission) { + t.Fatalf("repair = (%+v, %v), want (nil, errClaudeBridgeAgentUnreadable wrapping EACCES)", rep, err) + } + for cycle := 0; cycle < 3; cycle++ { + if got := m.rebridgeClaudeCredential("quality", link, target, claudeBridgeTriggerWatchdog); got != nil { + t.Fatalf("cycle %d: rebridge = %+v, want nil (left in place)", cycle, got) + } + } + + info, err := os.Lstat(link) + if err != nil || !info.Mode().IsRegular() { + t.Fatalf("private credential must stay a regular file at the link path: info=%v err=%v", info, err) + } + if got := readFileString(t, link); got != agentJSON { + t.Errorf("private credential content changed: %s", got) + } + if got := staleFiles(t, agentDir); len(got) != 0 { + t.Errorf("an unreadable private credential was moved aside: %v", got) + } + if got := readFileString(t, target); got != sharedJSON { + t.Errorf("shared credential changed: %s", got) + } + if got := staleFiles(t, filepath.Dir(target)); len(got) != 0 { + t.Errorf("shared credential was moved aside: %v", got) + } + if !claudeCredentialBridgeBroken("quality", 2007, "claude") { + t.Error("the dashboard must keep reporting the bridge as broken while the private copy is in place") + } + if ev := sink.find("claude_credential_rebridged"); ev != nil { + t.Errorf("nothing changed, so nothing may be audited as a re-bridge: %+v", ev) + } + const msg = "cannot read this agent's private credential" + if n := strings.Count(logs.String(), msg); n != 1 { + t.Fatalf("unreadable-credential error logged %d times over 3 cycles, want 1:\n%s", n, logs.String()) + } + if !strings.Contains(logs.String(), "level=ERROR") || !strings.Contains(logs.String(), "permission denied") || + !strings.Contains(logs.String(), link) { + t.Errorf("log must be an ERROR naming the path and the errno:\n%s", logs.String()) + } + if strings.Contains(logs.String(), "fake-") { + t.Errorf("log carries token material:\n%s", logs.String()) + } + + // The CLI refreshes again: a new file version is worth one more line. + later := time.Now().Add(time.Hour) + if err := os.Chtimes(link, later, later); err != nil { + t.Fatal(err) + } + m.rebridgeClaudeCredential("quality", link, target, claudeBridgeTriggerWatchdog) + if n := strings.Count(logs.String(), msg); n != 2 { + t.Errorf("a new private file version must be logged again: got %d lines, want 2", n) + } +} + +// TestRepairClaudeCredentialBridge_UnreadableAfterMoveIsPutBack: the file at +// the link was readable when checked, but the one actually renamed (the CLI +// replaced it in between) is not. It is put back at the link path, not left +// at a stale name with the agent relinked to the shared file. +func TestRepairClaudeCredentialBridge_UnreadableAfterMoveIsPutBack(t *testing.T) { + shared := withSharedAgentHome(t) + m, _ := testManagerWithSink(t) + now := time.Now() + sharedJSON := fakeClaudeCredential("fake-shared-access", now.Add(time.Hour), "fake-shared-refresh") + agentJSON := fakeClaudeCredential("fake-fresh-access", now.Add(8*time.Hour), "fake-fresh-refresh") + link, target := brokenBridgeFixture(t, shared, "scanner", sharedJSON, agentJSON) + withUnreadableClaudeCredential(t, func(path string) bool { + return strings.Contains(filepath.Base(path), claudeCredentialStaleInfix) + }) + + rep, err := m.repairClaudeCredentialBridge("scanner", link, target) + if rep != nil || !errors.Is(err, errClaudeBridgeAgentUnreadable) { + t.Fatalf("repair = (%+v, %v), want (nil, errClaudeBridgeAgentUnreadable)", rep, err) + } + info, err := os.Lstat(link) + if err != nil || !info.Mode().IsRegular() || readFileString(t, link) != agentJSON { + t.Fatalf("private credential must be back at the link path: info=%v err=%v", info, err) + } + if got := staleFiles(t, filepath.Dir(link)); len(got) != 0 { + t.Errorf("nothing may stay at a stale name: %v", got) + } + if got := readFileString(t, target); got != sharedJSON { + t.Errorf("shared credential changed: %s", got) + } +} + +// TestRepairClaudeCredentialBridge_EmptyTokenPrivateNeverWins: a failed refresh +// writes a private file with empty tokens and expiresAt 0. It must never +// replace the shared login, usable or not; the agent is relinked to it. +func TestRepairClaudeCredentialBridge_EmptyTokenPrivateNeverWins(t *testing.T) { + const emptyJSON = `{"claudeAiOauth":{"accessToken":"","refreshToken":"","expiresAt":0,"scopes":[]}}` + now := time.Now() + cases := map[string]string{ + "usable shared": fakeClaudeCredential("fake-live-access", now.Add(4*time.Hour), "fake-live-refresh"), + "unusable shared": fakeClaudeCredential("fake-dead-access", now.Add(-time.Hour), ""), + } + for name, sharedJSON := range cases { + t.Run(name, func(t *testing.T) { + shared := withSharedAgentHome(t) + m, sink := testManagerWithSink(t) + link, target := brokenBridgeFixture(t, shared, "sec-check", sharedJSON, emptyJSON) + + rep := m.rebridgeClaudeCredential("sec-check", link, target, claudeBridgeTriggerWatchdog) + if rep == nil || rep.kept != "shared" { + t.Fatalf("repair = %+v, want kept=shared", rep) + } + if got := readFileString(t, target); got != sharedJSON { + t.Errorf("an empty-token private file replaced the shared credential: %s", got) + } + if got := staleFiles(t, filepath.Dir(target)); len(got) != 0 { + t.Errorf("shared credential must not be replaced or moved: %v", got) + } + assertBridgeRestored(t, link, target) + if ev := sink.find("claude_credential_rebridged"); ev == nil || ev.Fields["kept"] != "shared" { + t.Errorf("audit entry = %+v, want kept=shared", ev) + } + }) + } +} + +// TestClaudeBridgePromotable_RejectsIncompleteCredentials pins the shared +// promotion rule (also applied by the root perm guard in entrypoint.sh): only a +// complete, usable credential may replace the shared login. On the v6 tip a +// token with no recorded expiresAt counted as usable and newer than an +// expired shared login, and so did one with no refresh token. +func TestClaudeBridgePromotable_RejectsIncompleteCredentials(t *testing.T) { + now := time.Now() + deadShared := &claude.OAuthTokens{AccessToken: "fake", ExpiresAt: now.Add(-time.Hour).UnixMilli()} + cases := []struct { + name string + agent *claude.OAuthTokens + want bool + }{ + {"complete and live", &claude.OAuthTokens{AccessToken: "a", RefreshToken: "r", ExpiresAt: now.Add(time.Hour).UnixMilli()}, true}, + {"empty tokens, expiresAt 0", &claude.OAuthTokens{}, false}, + {"tokens but no expiresAt", &claude.OAuthTokens{AccessToken: "a", RefreshToken: "r"}, false}, + {"no refresh token", &claude.OAuthTokens{AccessToken: "a", ExpiresAt: now.Add(time.Hour).UnixMilli()}, false}, + {"no access token", &claude.OAuthTokens{RefreshToken: "r", ExpiresAt: now.Add(time.Hour).UnixMilli()}, false}, + {"nil", nil, false}, + } + for _, c := range cases { + if got := claudeBridgePromotable(c.agent, now); got != c.want { + t.Errorf("%s: claudeBridgePromotable = %v, want %v", c.name, got, c.want) + } + if got := preferAgentClaudeCredential(c.agent, deadShared, now); got != c.want { + t.Errorf("%s: preferAgentClaudeCredential over a dead shared login = %v, want %v", c.name, got, c.want) + } + } +} + +// TestPromoteClaudeCredential_RefusesUnusableBytes: the promotion re-checks the +// bytes it is about to write, so a source that changed after it was judged +// can never blank the shared login. Nothing is preserved or written. +func TestPromoteClaudeCredential_RefusesUnusableBytes(t *testing.T) { + dir := t.TempDir() + src := filepath.Join(dir, "private.json") + target := filepath.Join(dir, ".credentials.json") + now := time.Now() + sharedJSON := fakeClaudeCredential("fake-live-access", now.Add(time.Hour), "fake-live-refresh") + if err := os.WriteFile(target, []byte(sharedJSON), 0o660); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(src, []byte(`{"claudeAiOauth":{"accessToken":"","refreshToken":"","expiresAt":0}}`), 0o600); err != nil { + t.Fatal(err) + } + if _, err := promoteClaudeCredential(src, target, now); !errors.Is(err, errClaudeBridgeNotPromotable) { + t.Fatalf("promote error = %v, want errClaudeBridgeNotPromotable", err) + } + if got := readFileString(t, target); got != sharedJSON { + t.Errorf("shared credential changed: %s", got) + } + if got := staleFiles(t, dir); len(got) != 0 { + t.Errorf("a refused promotion must not preserve or move anything: %v", got) + } +} diff --git a/src/crux/claude/oauth.go b/src/crux/claude/oauth.go index 8b262a88fa..9e726aa60c 100644 --- a/src/crux/claude/oauth.go +++ b/src/crux/claude/oauth.go @@ -6,6 +6,7 @@ import ( "crypto/sha256" "encoding/base64" "encoding/json" + "errors" "fmt" "io" "net/http" @@ -342,8 +343,20 @@ func ApplyProfile(tokens *OAuthTokens, prof Profile, fetchErr error, previous *O return tokens.SubscriptionType != "" || tokens.RateLimitTier != "" } -// WriteCredentials atomically writes tokens to the credentials file. +// ErrNoAccessToken is returned by WriteCredentials for a token set with no +// access token. Writing one would replace whatever login is on file with a +// signed-out credential. +var ErrNoAccessToken = errors.New("refusing to write a Claude credential with no access token") + +// WriteCredentials atomically writes tokens to the credentials file. It +// refuses a nil token set or one with an empty access token +// (ErrNoAccessToken), so no hive caller can blank the shared login with a +// signed-out credential; the dashboard sign-in, its only caller in the hive, +// never produces one (ExchangeCode rejects an empty access token). func WriteCredentials(tokens *OAuthTokens, path string) error { + if tokens == nil || tokens.AccessToken == "" { + return ErrNoAccessToken + } dir := filepath.Dir(path) if err := os.MkdirAll(dir, 0o2775); err != nil { return fmt.Errorf("create credentials dir: %w", err) @@ -366,13 +379,33 @@ func WriteCredentials(tokens *OAuthTokens, path string) error { } // LoadTokens parses the credentials file and returns the stored token set, or -// nil when the file is absent, malformed, or carries no token block. It never -// applies an expiry rule — callers decide what a stale token means to them. +// nil when the file is absent, unreadable, malformed, or carries no token +// block. It never applies an expiry rule — callers decide what a stale token +// means to them. Callers that must tell "this process cannot read the file" +// apart from "the file holds no usable token" use ReadTokens. func LoadTokens(path string) *OAuthTokens { + tokens, _ := ReadTokens(path) + return tokens +} + +// ReadTokens is LoadTokens with the read error kept. The error is non-nil +// only when the file could not be read at all (absent, or not readable by +// this process, e.g. EACCES on another uid's 0600 file); test it with +// errors.Is(err, fs.ErrNotExist) or fs.ErrPermission. A file that was read +// but is malformed, or carries no token block, returns (nil, nil): it was +// readable, and it holds nothing usable. The error never carries file +// content. +func ReadTokens(path string) (*OAuthTokens, error) { data, err := os.ReadFile(path) if err != nil { - return nil + return nil, err } + return ParseTokens(data), nil +} + +// ParseTokens decodes a credentials file body already in memory, with the +// LoadTokens rules: nil when it is malformed or carries no token block. +func ParseTokens(data []byte) *OAuthTokens { var creds Credentials if err := json.Unmarshal(data, &creds); err != nil { return nil diff --git a/src/crux/claude/oauth_read_tokens_test.go b/src/crux/claude/oauth_read_tokens_test.go new file mode 100644 index 0000000000..a812371e50 --- /dev/null +++ b/src/crux/claude/oauth_read_tokens_test.go @@ -0,0 +1,77 @@ +package claude + +import ( + "errors" + "io/fs" + "os" + "path/filepath" + "testing" + "time" +) + +// TestReadTokensKeepsTheReadError: the credential bridge must tell "this +// process cannot read the file" apart from "the file holds no usable token". +// LoadTokens folds both into nil; ReadTokens keeps the read error and returns +// (nil, nil) for a file it read but could not use. +func TestReadTokensKeepsTheReadError(t *testing.T) { + dir := t.TempDir() + + if tok, err := ReadTokens(filepath.Join(dir, "absent.json")); tok != nil || !errors.Is(err, fs.ErrNotExist) { + t.Errorf("absent file: ReadTokens = (%v, %v), want (nil, ErrNotExist)", tok, err) + } + + // A directory cannot be read as a file, as root or not: the stand-in for + // a read error other than "not there" (EACCES needs a second uid). + if tok, err := ReadTokens(dir); tok != nil || err == nil || errors.Is(err, fs.ErrNotExist) { + t.Errorf("unreadable path: ReadTokens = (%v, %v), want (nil, a non-ErrNotExist error)", tok, err) + } + if LoadTokens(dir) != nil { + t.Error("LoadTokens must keep returning nil for an unreadable path") + } + + malformed := filepath.Join(dir, "malformed.json") + if err := os.WriteFile(malformed, []byte("not json"), 0o600); err != nil { + t.Fatal(err) + } + if tok, err := ReadTokens(malformed); tok != nil || err != nil { + t.Errorf("malformed file: ReadTokens = (%v, %v), want (nil, nil)", tok, err) + } + + good := filepath.Join(dir, "good.json") + exp := time.Now().Add(time.Hour).UnixMilli() + if err := WriteCredentials(&OAuthTokens{AccessToken: "fake-a", RefreshToken: "fake-r", ExpiresAt: exp}, good); err != nil { + t.Fatal(err) + } + if tok, err := ReadTokens(good); err != nil || tok == nil || tok.ExpiresAt != exp { + t.Errorf("good file: ReadTokens = (%+v, %v), want the stored tokens", tok, err) + } +} + +// TestWriteCredentialsRefusesSignedOutTokens: no hive writer may put a +// signed-out credential (no access token) over the login on file. +func TestWriteCredentialsRefusesSignedOutTokens(t *testing.T) { + path := filepath.Join(t.TempDir(), ".credentials.json") + const onFile = `{"claudeAiOauth":{"accessToken":"fake-live"}}` + if err := os.WriteFile(path, []byte(onFile), 0o660); err != nil { + t.Fatal(err) + } + for name, tok := range map[string]*OAuthTokens{ + "nil": nil, + "empty access token": {RefreshToken: "fake-r", ExpiresAt: time.Now().Add(time.Hour).UnixMilli()}, + "empty token set": {}, + } { + if err := WriteCredentials(tok, path); !errors.Is(err, ErrNoAccessToken) { + t.Errorf("%s: WriteCredentials error = %v, want ErrNoAccessToken", name, err) + } + } + got, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(got) != onFile { + t.Errorf("the credential on file changed: %s", got) + } + if _, err := os.Stat(path + ".tmp"); !errors.Is(err, fs.ErrNotExist) { + t.Errorf("a refused write must not leave a temp file: %v", err) + } +} diff --git a/src/deploy/entrypoint.sh b/src/deploy/entrypoint.sh index 111d79e2dd..7cc095f7e0 100644 --- a/src/deploy/entrypoint.sh +++ b/src/deploy/entrypoint.sh @@ -1769,12 +1769,224 @@ if [ "$(id -u)" = "0" ]; then return 0 } + # ── Claude credential bridge repair, as root (hive_rebridge_claude_credentials) + # + # Each per-agent ~/.claude/.credentials.json is a symlink to the shared + # /data/home/.claude/.credentials.json. A Claude Code token refresh writes a + # temp file and renames it over that path, which replaces the link with a + # private file owned by the agent's uid, mode 0600. The refresh also rotates + # the grant, so the shared file's refresh token is now dead and the private + # file holds the only live login in the fleet. + # + # The Go repair (crux/agent/claude_credential_bridge.go) runs as the hive + # server's own uid and cannot read that 0600 file. On a live per-UID hive it + # took the read error for "no token", kept the dead shared file, moved the + # fresh one aside, and the fleet logged out one access-token lifetime later. + # Root can read it, so this guard runs the same repair with the same rule + # every 5 seconds: move the private file aside, promote it into the shared + # file only when it is complete and usable and either the shared login is + # not usable or the private token expires later, then restore the link. + # Five seconds also beats the other agents' CLIs, which refresh within a + # minute or two of each other and would otherwise redeem the rotated grant. + # + # Rules this code keeps: + # - No token text is ever printed. jq prints only 0/1 flags and the + # expiresAt number (hive_claude_cred_judge), and file contents only move + # between files. + # - Access is never widened. The promoted file is written by a process + # running as the shared file's owner and group, in the shared directory, + # with the shared file's mode minus any world bits, then renamed over it + # (atomic). Only what could read the shared login can read the result. + # - Root never follows a path an agent controls. The agent's .claude is + # entered with cd -P and checked with pwd -P, every name after that is + # relative, and the private file is read with dd iflag=nofollow into a + # fresh root-only temp dir. In a directory another uid can write, root + # only renames and creates the link (both act on the name, never + # through it) and writes no file content. + # - Nothing is deleted. The private file becomes + # .credentials.json.stale- beside it (owner and mode kept); a + # replaced shared file is preserved as .credentials.json.stale- + # beside the shared one, mode 0600. When anything is inconclusive the + # private file is put back and the link is left alone. + # - No shared login (signed out, or never signed in) means nothing to + # bridge to: the guard does nothing, as the Go sign-in hook does. + + # hive_claude_cred_judge FILE — print "USABLE PROMOTABLE EXPIRES_AT" as three + # integers: USABLE and PROMOTABLE are 1 or 0, EXPIRES_AT is claudeAiOauth + # .expiresAt in epoch milliseconds (0 when absent). An unreadable, malformed + # or non-credential file prints "0 0 0". The rules are Go's: + # usable = an access token whose expiresAt is absent or in the future, + # or a refresh token whose refreshTokenExpiresAt is absent or + # in the future (claude.TokensUsable); + # promotable = usable, with an access token, a refresh token and an + # expiresAt > 0 (claudeBridgePromotable), so a failed + # refresh's empty-token, expiresAt-0 file never is. + hive_claude_cred_judge() { + _cj="$(jq -r ' + def present: if type == "string" then length > 0 else false end; + def stamp: if type == "number" and . > 0 then floor else 0 end; + (.claudeAiOauth // {}) as $o + | (now * 1000 | floor) as $now + | ($o.accessToken // "" | present) as $acc + | ($o.refreshToken // "" | present) as $ref + | ($o.expiresAt // 0 | stamp) as $exp + | ($o.refreshTokenExpiresAt // 0 | stamp) as $rexp + | (($acc and ($exp == 0 or $exp >= $now)) + or ($ref and ($rexp == 0 or $rexp >= $now))) as $usable + | [ (if $usable then 1 else 0 end), + (if $usable and $acc and $ref and $exp > 0 then 1 else 0 end), + $exp ] + | map(tostring) | join(" ") + ' "$1" 2>/dev/null)" || _cj="" + case "$_cj" in + [01]\ [01]\ [0-9]*) case "${_cj##* }" in *[!0-9]*) _cj="0 0 0" ;; esac ;; + *) _cj="0 0 0" ;; + esac + printf '%s\n' "$_cj" + return 0 + } + + # hive_claude_promote_sh — script that installs the bytes on its stdin as the + # shared credential. It runs as the shared file's owner and group (see + # hive_rebridge_claude_credentials), never as root. Arguments: SHARED STAMP + # MODE. It preserves the current shared file as SHARED.stale-STAMP[-N] + # (0600, never overwriting one), writes a temp file beside SHARED, sets MODE + # and the shared group, and renames it over SHARED. Prints "promoted" only + # when the rename succeeded. + hive_claude_promote_sh() { + # shellcheck disable=SC2016 + printf '%s' 'shared=$1; stamp=$2; mode=$3; umask 077 +tmp=$(mktemp "$shared.rebridge.XXXXXX" 2>/dev/null) || exit 0 +cat > "$tmp" 2>/dev/null && [ -s "$tmp" ] || { rm -f "$tmp"; exit 0; } +old="$shared.stale-$stamp"; n=0 +while [ -e "$old" ] || [ -L "$old" ]; do + n=$((n + 1)); [ "$n" -lt 100 ] || { rm -f "$tmp"; exit 0; } + old="$shared.stale-$stamp-$n" +done +(set -C; cat "$shared" > "$old") 2>/dev/null || { rm -f "$tmp"; exit 0; } +chmod "$mode" "$tmp" 2>/dev/null || { rm -f "$tmp"; exit 0; } +chgrp "$(id -g)" "$tmp" 2>/dev/null || true +mv -fT "$tmp" "$shared" 2>/dev/null || { rm -f "$tmp"; exit 0; } +echo promoted' + } + + # hive_rebridge_claude_credential DIR SHARED STAMP — repair one agent's + # .claude directory DIR (/data/home/agents//.claude). A subshell, so + # its cd stays local. Prints one log line when it re-bridged the agent. + hive_rebridge_claude_credential() ( + _rb_dir="$1"; _rb_shared="$2"; _rb_stamp="$3" + _rb_agent="${_rb_dir%/.claude}"; _rb_agent="${_rb_agent##*/}" + _rb_want="$(cd -P "${_rb_dir%/*}" 2>/dev/null && pwd -P)/.claude" || exit 0 + # A symlinked .claude is the legacy whole-directory layout: its + # .credentials.json IS the shared login, never a private copy. + [ -d "$_rb_dir" ] && [ ! -L "$_rb_dir" ] || exit 0 + cd -P "$_rb_dir" 2>/dev/null || exit 0 + [ "$(pwd -P)" = "$_rb_want" ] || exit 0 + [ -f .credentials.json ] && [ ! -L .credentials.json ] || exit 0 + [ .credentials.json -ef "$_rb_shared" ] && exit 0 + + _rb_stale=".credentials.json.stale-$_rb_stamp"; _rb_n=0 + while [ -e "$_rb_stale" ] || [ -L "$_rb_stale" ]; do + _rb_n=$((_rb_n + 1)); [ "$_rb_n" -lt 100 ] || exit 0 + _rb_stale=".credentials.json.stale-$_rb_stamp-$_rb_n" + done + _rb_work="$(mktemp -d "${TMPDIR:-/tmp}/hive-rebridge.XXXXXX" 2>/dev/null)" || exit 0 + [ -n "$_rb_work" ] || exit 0 + # Move aside FIRST: the CLI's next write lands at the link path, so what + # is judged below cannot change under the judgment. + if ! mv -fT .credentials.json "$_rb_stale" 2>/dev/null; then + rm -rf "$_rb_work"; exit 0 + fi + _rb_kept="" + if dd if="$_rb_stale" of="$_rb_work/agent" iflag=nofollow status=none 2>/dev/null; then + # shellcheck disable=SC2046 + set -- $(hive_claude_cred_judge "$_rb_work/agent") $(hive_claude_cred_judge "$_rb_shared") + # $1 $2 $3: private usable/promotable/expiresAt; $4 $5 $6: shared. + if [ "$2" = 1 ] && { [ "$4" = 0 ] || [ "$3" -gt "$6" ]; }; then + # The shared file's owner, group and mode, read without following a + # link. The mode loses any world bits: match the shared login's + # access, or narrow it, never widen it. + _rb_uid=""; _rb_gid=""; _rb_mode="" + if [ -f "$_rb_shared" ] && [ ! -L "$_rb_shared" ]; then + _rb_uid="$(stat -c '%u' "$_rb_shared" 2>/dev/null)" || _rb_uid="" + _rb_gid="$(stat -c '%g' "$_rb_shared" 2>/dev/null)" || _rb_gid="" + _rb_mode="$(stat -c '%a' "$_rb_shared" 2>/dev/null)" || _rb_mode="" + fi + case "$_rb_mode" in + *[!0-7]*|'') _rb_mode="" ;; + *) _rb_mode="$(printf '%o' $((0$_rb_mode & 0770)))" ;; + esac + _rb_out="" + if [ -z "$_rb_uid" ] || [ -z "$_rb_gid" ] || [ -z "$_rb_mode" ]; then + : + elif [ "$(id -u)" = 0 ]; then + # Never write in the shared directory as root, and never as root's + # identity: a root-owned shared file would lock every agent out. + if [ "$_rb_uid" != 0 ] && command -v setpriv >/dev/null 2>&1; then + _rb_out="$(setpriv --reuid="$_rb_uid" --regid="$_rb_gid" --clear-groups \ + sh -c "$(hive_claude_promote_sh)" sh "$_rb_shared" "$_rb_stamp" "$_rb_mode" \ + < "$_rb_work/agent" 2>/dev/null)" || _rb_out="" + fi + else + _rb_out="$(sh -c "$(hive_claude_promote_sh)" sh "$_rb_shared" "$_rb_stamp" "$_rb_mode" \ + < "$_rb_work/agent" 2>/dev/null)" || _rb_out="" + fi + [ "$_rb_out" = promoted ] && _rb_kept=agent + else + _rb_kept=shared + fi + fi + rm -rf "$_rb_work" + if [ -z "$_rb_kept" ]; then + # Inconclusive (unreadable, unparsable copy, or a promotion that did + # not land): put the private file back, unless the CLI has written a + # newer one meanwhile, and leave the link as it is. + [ -e .credentials.json ] || [ -L .credentials.json ] \ + || mv -fT "$_rb_stale" .credentials.json 2>/dev/null || true + echo "[entrypoint] WARN: claude credential bridge: could not judge or promote the private credential of agent '$_rb_agent'; left it in place" + exit 0 + fi + # EEXIST: the CLI wrote a fresh file since the move; the next pass takes it. + ln -s "$_rb_shared" .credentials.json 2>/dev/null || true + echo "[entrypoint] claude credential bridge: re-bridged agent '$_rb_agent' (kept=$_rb_kept, private copy moved to $_rb_dir/$_rb_stale)" + exit 0 + ) + + # hive_rebridge_claude_credentials — one pass over every per-agent home. + # Cheap when nothing is wrong: one test per agent, no fork. + hive_rebridge_claude_credentials() { + [ "${HIVE_SHARED_AGENT_HOME:-}" = "1" ] && return 0 + command -v jq >/dev/null 2>&1 || return 0 + _rbs_shared=/data/home/.claude/.credentials.json + [ -f "$_rbs_shared" ] && [ ! -L "$_rbs_shared" ] || return 0 + _rbs_stamp="" + for _rbs_dir in /data/home/agents/*/.claude; do + [ -f "$_rbs_dir/.credentials.json" ] && [ ! -L "$_rbs_dir/.credentials.json" ] || continue + if [ -z "$_rbs_stamp" ]; then + # Root promotes only by dropping to the shared file's owner, which + # needs setpriv and a non-root owner. Without them, leave every + # private file alone rather than re-bridge on half a decision; the Go + # repair then logs that it cannot judge the file. + if [ "$(id -u)" = 0 ]; then + command -v setpriv >/dev/null 2>&1 || return 0 + [ "$(stat -c '%u' "$_rbs_shared" 2>/dev/null || echo 0)" != 0 ] || return 0 + fi + _rbs_stamp="$(date -u +%Y%m%dT%H%M%SZ)" + fi + hive_rebridge_claude_credential "$_rbs_dir" "$_rbs_shared" "$_rbs_stamp" || true + done + return 0 + } + # hive_fix_credentials_fast — the 5s polling path. Bounded and cheap: the # files a CLI rewrites owner-only on token/session refresh, and no tree walk # at all. This is the backstop that would have repaired #5730 and #8713 - # within five seconds. + # within five seconds. It also re-bridges any per-agent Claude + # credential a token refresh replaced (hive_rebridge_claude_credentials), + # before the shared file's own group repair. hive_fix_credentials_fast() { hive_fix_copilot_config + hive_rebridge_claude_credentials hive_fix_shared_credential /data/home/.claude/.credentials.json hive_fix_shared_credential /data/home/.gemini/antigravity-cli/antigravity-oauth-token hive_fix_shared_credential /data/home/.gemini/antigravity-cli/cache/onboarding.json diff --git a/src/deploy/test_entrypoint_perm_guard.sh b/src/deploy/test_entrypoint_perm_guard.sh index 857dfce8dd..6c21643485 100755 --- a/src/deploy/test_entrypoint_perm_guard.sh +++ b/src/deploy/test_entrypoint_perm_guard.sh @@ -58,6 +58,7 @@ fi ok "guard functions extracted from entrypoint.sh ($(grep -c '^ hive_.*() {' "$GUARDS") functions)" for fn in hive_fix_shared_credential hive_fix_copilot_config hive_fix_tree \ + hive_claude_cred_judge hive_rebridge_claude_credentials \ hive_fix_claude_instant hive_fix_credentials_fast hive_guard_forever \ hive_watch_once; do if grep -q "^ ${fn}() {" "$GUARDS" || grep -q "^ ${fn}() " "$GUARDS"; then @@ -594,6 +595,210 @@ for repair in hive_fix_tree_recent hive_fix_tree; do done rm -f "$SETTINGS" "$SIBLING" +# ── 14. Root re-bridge of a per-agent Claude credential ───────────────────── +# A Claude Code refresh replaces an agent's ~/.claude/.credentials.json symlink +# with a private 0600 file and rotates the grant, so that file holds the only +# live login. The hive server's uid cannot read it; the root guard must judge +# it, promote it when it is complete, usable and newer, keep the shared file's +# owner and mode, never let an empty-token file win, and never print a token. +# Driven against its own tree, so the cases above are unaffected. +if ! command -v jq >/dev/null 2>&1; then + ok "SKIP: jq not installed; hive_rebridge_claude_credentials is a no-op without it" +else + RBHOME="$WORK/rbhome" + RBTMP="$WORK/rbtmp" + mkdir -p "$RBHOME/.claude" "$RBHOME/agents" "$RBTMP" + sed "s#/data/home#${RBHOME}#g" "$GUARDS" > "$WORK/guards.rebridge.sh" + RB_SHARED="$RBHOME/.claude/.credentials.json" + RB_OUT="$WORK/rebridge.log" + : > "$RB_OUT" + NOW_MS=$(( $(date +%s) * 1000 )) + HOUR_MS=3600000 + # Every token below carries this marker; the guard's output must never. + SENTINEL="TOKSENTINEL" + rb_cred() { + printf '{"claudeAiOauth":{"accessToken":"%s","refreshToken":"%s","expiresAt":%s,"scopes":["user:inference"]}}' "$1" "$2" "$3" + } + rb_shared() { rb_cred "$@" > "$RB_SHARED"; chmod "${RB_SHARED_MODE:-640}" "$RB_SHARED"; } + rb_agent() { + mkdir -p "$RBHOME/agents/$1/.claude" + rm -f "$RBHOME/agents/$1/.claude/.credentials.json" + printf '%s' "$2" > "$RBHOME/agents/$1/.claude/.credentials.json" + chmod 600 "$RBHOME/agents/$1/.claude/.credentials.json" + } + rb_pass() { + TMPDIR="$RBTMP" sh -c 'set -e; . "$1"; hive_rebridge_claude_credentials; echo REBRIDGE_PASS_OK' \ + sh "$WORK/guards.rebridge.sh" >>"$RB_OUT" 2>&1 + } + rb_linked() { + [ -L "$RBHOME/agents/$1/.claude/.credentials.json" ] \ + && [ "$(readlink "$RBHOME/agents/$1/.claude/.credentials.json")" = "$RB_SHARED" ] + } + rb_stale_count() { find "$1" -maxdepth 1 -name '.credentials.json.stale-*' | wc -l | tr -d ' '; } + + # 14a. Newer, complete, usable private copy over an expired shared one: + # promoted atomically, with the shared file's owner and mode. + rb_shared "${SENTINEL}-old-access" "${SENTINEL}-old-refresh" $(( NOW_MS - HOUR_MS )) + OWNER_BEFORE="$(stat -c '%u:%g' "$RB_SHARED")" + INODE_BEFORE="$(stat -c '%i' "$RB_SHARED")" + OLD_SHARED="$(cat "$RB_SHARED")" + FRESH="$(rb_cred "${SENTINEL}-new-access" "${SENTINEL}-new-refresh" $(( NOW_MS + 8 * HOUR_MS )))" + rb_agent fresh "$FRESH" + rb_pass + if [ "$(cat "$RB_SHARED")" = "$FRESH" ]; then + ok "rebridge: a newer usable private credential is promoted into the shared file" + else + bad "rebridge: a newer usable private credential is promoted" "shared file was not replaced" + fi + if [ "$(stat -c '%i' "$RB_SHARED")" != "$INODE_BEFORE" ] \ + && [ -z "$(find "$RBHOME/.claude" -maxdepth 1 -name '.credentials.json.rebridge.*')" ]; then + ok "rebridge: the promotion is a rename of a finished temp file (new inode, no temp left)" + else + bad "rebridge: the promotion is atomic" "inode unchanged or a temp file was left behind" + fi + if [ "$(stat -c '%a' "$RB_SHARED")" = "640" ] && [ "$(stat -c '%u:%g' "$RB_SHARED")" = "$OWNER_BEFORE" ]; then + ok "rebridge: the promoted file keeps the shared file's owner, group and mode (0640)" + else + bad "rebridge: the promoted file keeps the shared owner and mode" \ + "now $(stat -c '%a %u:%g' "$RB_SHARED"), was 640 $OWNER_BEFORE" + fi + OLD_COPY="$(find "$RBHOME/.claude" -maxdepth 1 -name '.credentials.json.stale-*' | head -1)" + if [ -n "$OLD_COPY" ] && [ "$(cat "$OLD_COPY")" = "$OLD_SHARED" ] && [ "$(stat -c '%a' "$OLD_COPY")" = "600" ]; then + ok "rebridge: the replaced shared credential is preserved beside it (0600)" + else + bad "rebridge: the replaced shared credential is preserved" "found: ${OLD_COPY:-none}" + fi + PRIV_COPY="$(find "$RBHOME/agents/fresh/.claude" -maxdepth 1 -name '.credentials.json.stale-*' | head -1)" + if rb_linked fresh && [ -n "$PRIV_COPY" ] && [ "$(stat -c '%a' "$PRIV_COPY")" = "600" ]; then + ok "rebridge: the agent is linked back to the shared login and its private file kept aside (0600)" + else + bad "rebridge: the agent is re-linked and its private file kept" "link or stale copy missing" + fi + grep -q "re-bridged agent 'fresh' (kept=agent" "$RB_OUT" \ + && ok "rebridge: the promotion is logged with kept=agent" \ + || bad "rebridge: the promotion is logged" "$(cat "$RB_OUT")" + + # 14b. A failed refresh's empty-token, expiresAt-0 file never wins, even + # over a shared login that is itself expired and grant-less. + for shared_case in usable unusable; do + if [ "$shared_case" = usable ]; then + rb_shared "${SENTINEL}-live-access" "${SENTINEL}-live-refresh" $(( NOW_MS + 4 * HOUR_MS )) + else + rb_shared "${SENTINEL}-dead-access" "" $(( NOW_MS - HOUR_MS )) + fi + BEFORE="$(cat "$RB_SHARED")"; INODE_BEFORE="$(stat -c '%i' "$RB_SHARED")" + rb_agent "empty-$shared_case" '{"claudeAiOauth":{"accessToken":"","refreshToken":"","expiresAt":0,"scopes":[]}}' + rb_pass + if [ "$(cat "$RB_SHARED")" = "$BEFORE" ] && [ "$(stat -c '%i' "$RB_SHARED")" = "$INODE_BEFORE" ] \ + && rb_linked "empty-$shared_case"; then + ok "rebridge: an empty-token private file never replaces a $shared_case shared file; the agent is re-linked" + else + bad "rebridge: an empty-token private file never replaces a $shared_case shared file" \ + "shared changed or agent not re-linked" + fi + done + # A complete-looking token with no recorded expiry cannot be judged newer. + rb_shared "${SENTINEL}-dead2-access" "" $(( NOW_MS - HOUR_MS )) + BEFORE="$(cat "$RB_SHARED")" + rb_agent noexp "$(rb_cred "${SENTINEL}-noexp-access" "${SENTINEL}-noexp-refresh" 0)" + rb_pass + [ "$(cat "$RB_SHARED")" = "$BEFORE" ] \ + && ok "rebridge: a private file with expiresAt 0 is never promoted" \ + || bad "rebridge: a private file with expiresAt 0 is never promoted" "shared file was replaced" + + # 14c. Not newer: the shared login stays, the private file goes aside. + rb_shared "${SENTINEL}-signin-access" "${SENTINEL}-signin-refresh" $(( NOW_MS + 8 * HOUR_MS )) + BEFORE="$(cat "$RB_SHARED")" + rb_agent older "$(rb_cred "${SENTINEL}-older-access" "${SENTINEL}-older-refresh" $(( NOW_MS + HOUR_MS )))" + rb_pass + if [ "$(cat "$RB_SHARED")" = "$BEFORE" ] && rb_linked older \ + && [ "$(rb_stale_count "$RBHOME/agents/older/.claude")" = 1 ]; then + ok "rebridge: an older private file leaves the shared login alone (kept=shared) and is moved aside" + else + bad "rebridge: an older private file leaves the shared login alone" "shared changed or agent not re-linked" + fi + + # 14d. Promotion never widens access: a world-readable shared file yields a + # promoted file with the world bits removed. + RB_SHARED_MODE=644 rb_shared "${SENTINEL}-exp-access" "${SENTINEL}-exp-refresh" $(( NOW_MS - HOUR_MS )) + rb_agent narrow "$(rb_cred "${SENTINEL}-narrow-access" "${SENTINEL}-narrow-refresh" $(( NOW_MS + 8 * HOUR_MS )))" + rb_pass + if [ "$(stat -c '%a' "$RB_SHARED")" = "640" ] && grep -q "${SENTINEL}-narrow-access" "$RB_SHARED"; then + ok "rebridge: a promoted credential never gets world access (0644 shared -> 0640)" + else + bad "rebridge: a promoted credential never gets world access" "mode $(stat -c '%a' "$RB_SHARED")" + fi + + # 14e. Things that are not a private copy are never touched: the legacy + # whole-directory link, a private symlink, and any agent when no shared + # login exists at all. + rb_shared "${SENTINEL}-keep-access" "${SENTINEL}-keep-refresh" $(( NOW_MS - HOUR_MS )) + BEFORE="$(cat "$RB_SHARED")"; INODE_BEFORE="$(stat -c '%i' "$RB_SHARED")" + STALE_BEFORE="$(rb_stale_count "$RBHOME/.claude")" + mkdir -p "$RBHOME/agents/legacy" + ln -s "$RBHOME/.claude" "$RBHOME/agents/legacy/.claude" + mkdir -p "$RBHOME/agents/planted/.claude" + printf '%s' "$(rb_cred "${SENTINEL}-elsewhere-access" "${SENTINEL}-elsewhere-refresh" $(( NOW_MS + 9 * HOUR_MS )))" > "$WORK/elsewhere.json" + ln -s "$WORK/elsewhere.json" "$RBHOME/agents/planted/.claude/.credentials.json" + rb_pass + if [ "$(cat "$RB_SHARED")" = "$BEFORE" ] && [ "$(stat -c '%i' "$RB_SHARED")" = "$INODE_BEFORE" ] \ + && [ "$(rb_stale_count "$RBHOME/.claude")" = "$STALE_BEFORE" ] \ + && [ "$(readlink "$RBHOME/agents/planted/.claude/.credentials.json")" = "$WORK/elsewhere.json" ]; then + ok "rebridge: a legacy directory link and a planted symlink are left alone" + else + bad "rebridge: a legacy directory link and a planted symlink are left alone" "something was moved or promoted" + fi + rm -rf "$RBHOME/agents/legacy" "$RBHOME/agents/planted" + mv "$RB_SHARED" "$WORK/shared.parked" + rb_agent nologin "$(rb_cred "${SENTINEL}-nologin-access" "${SENTINEL}-nologin-refresh" $(( NOW_MS + 8 * HOUR_MS )))" + rb_pass + if [ -f "$RBHOME/agents/nologin/.claude/.credentials.json" ] && [ ! -L "$RBHOME/agents/nologin/.claude/.credentials.json" ] \ + && [ ! -e "$RB_SHARED" ]; then + ok "rebridge: with no shared login nothing is moved (a logout is not undone)" + else + bad "rebridge: with no shared login nothing is moved" "private file moved or shared file created" + fi + rm -rf "$RBHOME/agents/nologin" + mv "$WORK/shared.parked" "$RB_SHARED" + + # 14f. A private file the guard cannot read is put back, not discarded, + # and the link is not restored over it. (Mode 000 stands in for another + # uid's 0600 file; a root CI runner reads it anyway, so skip there.) + if [ "$(id -u)" != 0 ]; then + BEFORE="$(cat "$RB_SHARED")" + rb_agent unreadable "$(rb_cred "${SENTINEL}-locked-access" "${SENTINEL}-locked-refresh" $(( NOW_MS + 8 * HOUR_MS )))" + chmod 000 "$RBHOME/agents/unreadable/.claude/.credentials.json" + rb_pass + if [ -f "$RBHOME/agents/unreadable/.claude/.credentials.json" ] \ + && [ ! -L "$RBHOME/agents/unreadable/.claude/.credentials.json" ] \ + && [ "$(rb_stale_count "$RBHOME/agents/unreadable/.claude")" = 0 ] \ + && [ "$(cat "$RB_SHARED")" = "$BEFORE" ]; then + ok "rebridge: an unreadable private credential stays in place and the shared login is untouched" + else + bad "rebridge: an unreadable private credential stays in place" "it was moved aside or re-linked" + fi + chmod 600 "$RBHOME/agents/unreadable/.claude/.credentials.json" + rm -rf "$RBHOME/agents/unreadable" + else + ok "SKIP: unreadable-file case needs a non-root runner" + fi + + # 14g. Every pass survived set -e, left no root temp dirs, and printed no + # token text anywhere. + PASSES="$(grep -c REBRIDGE_PASS_OK "$RB_OUT")" + [ "$PASSES" -ge 8 ] \ + && ok "rebridge: every pass returned 0 under set -e ($PASSES passes)" \ + || bad "rebridge: every pass returned 0 under set -e" "only $PASSES passes completed: $(cat "$RB_OUT")" + [ -z "$(ls -A "$RBTMP")" ] \ + && ok "rebridge: no work directory is left behind" \ + || bad "rebridge: no work directory is left behind" "$(ls -A "$RBTMP")" + if grep -q "$SENTINEL" "$RB_OUT"; then + bad "rebridge: no token text in the guard's output" "$(grep -c "$SENTINEL" "$RB_OUT") line(s) carry token text" + else + ok "rebridge: no token text in the guard's output ($(wc -l < "$RB_OUT" | tr -d ' ') lines checked)" + fi +fi + echo "" echo "Results: $PASS passed, $FAIL failed" [ "$FAIL" -eq 0 ] || exit 1 diff --git a/src/docs/agent-configuration.md b/src/docs/agent-configuration.md index af61e9df89..3bde393612 100644 --- a/src/docs/agent-configuration.md +++ b/src/docs/agent-configuration.md @@ -689,19 +689,28 @@ Before [#17031](https://github.com/hivecommons/hive/issues/17031) `.claude`, `.g The `.claude/.credentials.json` bridge is a symlink, and Claude Code does not write through it. About every 8 hours the CLI refreshes its OAuth token by writing a temp file in `~/.claude` and renaming it over `.credentials.json`. The rename replaces the symlink with a private regular file, owned by the agent's uid with mode `0600`. That agent now runs on its own copy: a later dashboard sign-in rewrites the shared file and never reaches it, and if its private token is revoked it answers every kick with `Login expired · Please run /login` or `OAuth token revoked`. -Claude Code has no supported way to keep the link. `CLAUDE_CONFIG_DIR` moves the whole config directory (settings, transcripts, auto-memory), which would undo the per-agent split above, and a bind mount cannot be renamed over. So hive detects and repairs the replacement instead, in three places: +The refresh also matters to every other agent. It rotates the OAuth grant, so the refresh token still in the shared file is dead from that moment, and the agent's private file holds the only live login in the fleet. The other agents refresh within a minute or two of each other, so that file has to reach the shared login quickly. +Claude Code has no supported way to keep the link. `CLAUDE_CONFIG_DIR` moves the whole config directory (settings, transcripts, auto-memory), which would undo the per-agent split above, and a bind mount cannot be renamed over. So hive detects and repairs the replacement instead, in four places: + +- every 5 seconds, in the container entrypoint's root permission guard (`hive_rebridge_claude_credentials` in `src/deploy/entrypoint.sh`); - at every agent launch; -- on every credential-watchdog cycle (`HIVE_CREDENTIAL_WATCHDOG_INTERVAL`, default 5 minutes), so a refresh that happens while the agent runs is caught within one cycle; +- on every credential-watchdog cycle (`HIVE_CREDENTIAL_WATCHDOG_INTERVAL`, default 5 minutes); - right after a successful dashboard Claude sign-in. +The entrypoint guard does most of the work on a per-UID hive. The other three run inside the hive server, which runs as its own uid and cannot read an agent's `0600` file. Root can, so the guard is what judges and promotes a fresh refresh, and it does so within seconds. + The repair never deletes a token: 1. It moves the private file aside to `.credentials.json.stale-` in the agent's `~/.claude`. The file keeps its owner and `0600` mode. -2. It keeps whichever credential is newer **and** usable in the shared `/data/home/.claude/.credentials.json`. "Usable" is the rule the rest of hive uses: a live access token, or a refresh grant that has not expired. "Newer" means the access token's `expiresAt` is later. A fresh refresh in one agent therefore becomes the fleet's login, instead of stranding every other agent on the grant it just rotated. When the agent's copy wins, the old shared file is kept beside it as `.credentials.json.stale-` and the new one is written `0660` with the old file's owner and group. On a tie, or when the agent's copy is not usable, the shared file stays as it is. +2. It keeps whichever credential is newer **and** usable in the shared `/data/home/.claude/.credentials.json`. "Usable" is the rule the rest of hive uses: a live access token, or a refresh grant that has not expired. "Newer" means the access token's `expiresAt` is later. The agent's copy must also be complete: it needs an access token, a refresh token and a recorded `expiresAt`. A failed refresh leaves a file with empty tokens and `expiresAt` 0, and that file never wins. A fresh refresh in one agent therefore becomes the fleet's login, instead of stranding every other agent on the grant it just rotated. When the agent's copy wins, the old shared file is kept beside it as `.credentials.json.stale-` (`0600`), and the new one is written to a temp file and renamed over the shared file. On a tie, or when the agent's copy is not usable, the shared file stays as it is. 3. It restores the symlink. -Each repair writes a `claude_credential_rebridged` Audit Log entry naming the agent, the trigger (`launch`, `watchdog` or `sign-in`), which credential was kept (`shared` or `agent`) and where the moved-aside file went. It never records token material. Until the repair runs, the agent card shows **credential bridge broken (private stale file)** instead of a generic "needs login". Moved-aside files are not cleaned up automatically; remove them by hand once you no longer need them. +A promotion never makes a token readable by anyone who could not already read the shared login. The entrypoint guard writes the promoted file as the shared file's owner and group (it drops root for that step) and gives it the shared file's mode with any world bits removed. The hive server's own repair writes it `0660` with the old file's owner and group. The guard never prints token text: it judges a credential with `jq` output that is only flags and the `expiresAt` number. + +If the hive server cannot read an agent's private file, it does not move the file aside and does not restore the link. The agent keeps running on its private copy, and the hive log has one ERROR per agent and file version: `claude credential bridge: the hive process cannot read this agent's private credential`. That is the normal state for the few seconds before the entrypoint guard repairs it. If the same agent's line keeps coming back, the root guard is not running, for example because the container does not start as root. + +The entrypoint guard logs each repair in the container log as `[entrypoint] claude credential bridge: re-bridged agent '' (kept=agent|shared, private copy moved to )`. It does nothing while there is no shared login (signed out, or never signed in), and nothing under `HIVE_SHARED_AGENT_HOME=1`. Each repair by the hive server writes a `claude_credential_rebridged` Audit Log entry naming the agent, the trigger (`launch`, `watchdog` or `sign-in`), which credential was kept (`shared` or `agent`) and where the moved-aside file went. It never records token material. Until the repair runs, the agent card shows **credential bridge broken (private stale file)** instead of a generic "needs login". Moved-aside files are not cleaned up automatically; remove them by hand once you no longer need them. The repair skips an agent whose whole `~/.claude` is still the old directory symlink to `/data/home/.claude`. That happens to an agent that has been paused or stopped since before the per-agent split, because only a launch replaces the directory link. Through that link the agent's `.credentials.json` is the shared login itself, so there is nothing private to move aside. The hive log says so once per agent (`claude credential bridge: this agent's .claude directory resolves to the shared login directory`), the card does not show a broken bridge, and the agent's next launch replaces the directory link with its own `~/.claude`. diff --git a/src/docs/agent-watchdog.md b/src/docs/agent-watchdog.md index 8b6ee4e26f..0d1825d03c 100644 --- a/src/docs/agent-watchdog.md +++ b/src/docs/agent-watchdog.md @@ -129,7 +129,12 @@ $ podman exec hive ls -l /data/home/agents//.claude/.credentials.json A regular file here, instead of a symlink to `/data/home/.claude/.credentials.json`, is the cause. A Claude Code token refresh replaced the link with a private copy, so no sign-in can reach that agent. The agent card says **credential bridge broken (private stale file)**. -Hive repairs this on its own: at the agent's next launch, on the next credential-watchdog tick, and right after the next dashboard sign-in. It moves the private file aside as `.credentials.json.stale-`, keeps whichever token is newer and usable in the shared file, restores the symlink, and writes a `claude_credential_rebridged` Audit Log entry. See [When a Claude token refresh breaks the credential link](agent-configuration.md#when-a-claude-token-refresh-breaks-the-credential-link) for the rule. You do not need to move files by hand. If the card still shows the broken bridge after a watchdog cycle, the hive log has a `claude credential bridge: repair failed` warning that names the file and the error. +Hive repairs this on its own. The container entrypoint's root permission guard does it within 5 seconds (container log: `[entrypoint] claude credential bridge: re-bridged agent ''`). The hive server also tries at the agent's next launch, on the next credential-watchdog tick, and right after the next dashboard sign-in (`claude_credential_rebridged` in the Audit Log). Either one moves the private file aside as `.credentials.json.stale-`, keeps whichever token is newer, complete and usable in the shared file, and restores the symlink. See [When a Claude token refresh breaks the credential link](agent-configuration.md#when-a-claude-token-refresh-breaks-the-credential-link) for the rule. You do not need to move files by hand. + +If the card still shows the broken bridge after a minute, check the hive log: + +- `claude credential bridge: the hive process cannot read this agent's private credential` means the hive server left the file alone because it cannot judge it. Only the root guard can repair it. If this line repeats for the same agent, that guard is not running. +- `claude credential bridge: repair failed` names the file and the error of a repair that was attempted. ## Configuration