From 0a405ee3756aa70e9f384b6339a53ab72467e288 Mon Sep 17 00:00:00 2001 From: marketing Date: Mon, 17 Aug 2026 17:04:11 +0000 Subject: [PATCH 1/4] docs(claude): scope the CLAUDE_CONFIG_DIR keepVars finding to the daemon-inherit path The 2026-08-15 entry states the daemon 'cannot inject CLAUDE_CONFIG_DIR'. True of inheritance from the daemon's own process.env, which getBaseEnv() allowlists (src/pty/agent-pty.ts:399, keepVars :402). It is not true of the agent .env, a separate unfiltered path: :133-144 writes every key into ptyEnv at :141, passed to node-pty at :184. Setting the var in an agent's .env does reach the child. The unscoped wording was quoted to halt a five-agent remediation that was in fact sound. This file is loaded into every agent's context at boot, so the false blocker regenerated on demand rather than being read once. Also fixes the coordinate: :368 is a comment in onExit, not getBaseEnv -- real code, right file, plausible, not the thing. Adds the no-code-read proof: ten agents have populated private config dirs, only writable by a claude that received the var, and the var is not in keepVars, so it arrived via .env. Confirmed live: adoption's .env gained the var at 16:59:22Z and its config dir was populated at 17:00:26Z. --- CLAUDE.md | 52 +++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 258d8c1265..9d42ea8c63 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -175,11 +175,53 @@ UNVERIFIED. **VERIFIED = measured this day with the command output in hand.** - **WHEN AN ARTIFACT IS UNOBSERVABLE, READ THE CODE THAT CONSTRUCTS IT.** `ps eww` returns nothing readable on darwin, so "does the daemon inject `CLAUDE_CONFIG_DIR`?" looked unanswerable. It is settled - in one read: `agent-pty.ts:368 getBaseEnv()` is an **explicit `keepVars` allowlist** that does *not* - spread `process.env`, and the var is not in it โ€” so the daemon **cannot** inject it even if it carried - it. **Construction RULES OUT cases; observation only FAILS TO FIND them.** Same shape as the detector - bug one level up: `T` was measured on `last_fire_attempted_at`, an axis the failure cannot touch, so it - stayed fresh while everything real froze โ€” **the signal was measured on the wrong side of the event.** + in one read: `agent-pty.ts:399 getBaseEnv()` is an **explicit `keepVars` allowlist** (`:402`) that does + *not* spread `process.env`, and the var is not in it โ€” so the daemon **cannot inherit it from its own + environment.** **Construction RULES OUT cases; observation only FAILS TO FIND them.** Same shape as the + detector bug one level up: `T` was measured on `last_fire_attempted_at`, an axis the failure cannot + touch, so it stayed fresh while everything real froze โ€” **the signal was measured on the wrong side of + the event.** + + ### ๐Ÿ”ด CORRECTED 2026-08-17 17:0xZ โ€” THE SENTENCE ABOVE IS TRUE AND IT WAS READ AS A BLOCKER IT IS NOT + **"The daemon cannot inherit it" is NOT "the agent cannot receive it."** The agent's `.env` is a + **separate, unfiltered path**: `agent-pty.ts:133-144` reads it line-by-line and writes **every** key into + `ptyEnv` at **`:141`** with no allowlist, and `ptyEnv` is handed to node-pty at **`:184`**. โŸน **Setting + `CLAUDE_CONFIG_DIR` in an agent's `.env` DOES reach the child โ€” that is how every agent which has it is + configured.** + โœ… **PROOF THAT NEEDS NO CODE READ (`infra`'s): ten agents have POPULATED private config dirs. Only a + `claude` that RECEIVED the var could have written them, and the var is not in `keepVars` โ€” so it arrived + via `.env`.** *(Confirmed live the same day: `adoption`'s `.env` gained the var at `16:59:22Z` and its + private config dir was created and populated at `17:00:26Z`.)* + โš ๏ธ **THAT CANARY WAS REVERTED AT ~17:02Z AND `adoption` DOES NOT CARRY THE VAR TODAY โ€” do not go looking + for it as evidence.** **The arrival proof is unaffected** (the dir could only have been written by a + `claude` that received the var) **but the rollout was halted for a different, deterministic reason: a + fresh `CLAUDE_CONFIG_DIR` has NO session history, the daemon boots with `--continue`, and + `No conversation found to continue` exits 1 โ€” five crashes in ninety seconds.** โŸน **The fix is a DAEMON + change (force `mode='fresh'` on the first boot after the var appears), NOT an `.env` edit.** + ๐Ÿ”‘ **AND THE COMPOUND THAT MAKES IT WORSE THAN A FAILED CHANGE: four of the five target agents have no + Telegram, so rolling it would have halted four agents that cannot say they halted.** **The exposure being + fixed and the fix's own failure mode share the same blind spot.** + ๐Ÿ“Œ **Seeding `hasTrustDialogAccepted` is NECESSARY AND NOT SUFFICIENT โ€” measured: the flag was present and + correct and the trust dialog fired anyway.** *(The 2026-07-14 entry above reads as though seeding solves + it. It does not.)* + โš ๏ธ **`07-14 keychain-beats-CLAUDE_CODE_OAUTH_TOKEN` is NARROWED, NOT CLOSED: the var arrives and the + private dir is used, but the canary crash-looped, so the token-serving question is confounded. Nobody + should record it as verified.** + โš ๏ธ **COST OF THE ORIGINAL WORDING: it was quoted to halt a five-agent remediation that was in fact + sound.** The entry is in **this file, which is LOADED into every agent's context at boot** โ€” so the false + blocker **regenerated on demand** for two days rather than being read once. + ๐Ÿ”‘ **THE LESSON THAT SURVIVES, AND IT SHARPENS THE HEADLINE ABOVE RATHER THAN RETIRING IT: CONSTRUCTION + RULES OUT THE CASE YOU CONSTRUCTED, AND NOTHING ELSE.** One function was read and a conclusion about a + whole subsystem was written. **The rigour was real and the scope was one function wide โ€” which is exactly + why it stood for two days.** + ๐Ÿ”‘ **AND THE RETRIEVAL-SIDE TWIN: A STORED FINDING CARRIES THE FRAME IT WAS WRITTEN IN, AND THE READER + SUPPLIES A NEW ONE WITHOUT NOTICING THE SWAP.** This sentence answered *"does the daemon inject it on its + own?"* (no) and was read as answering *"will adding it to `.env` reach the agent?"* (yes). **Same file, + same function, same sentence, opposite operational answer, and no error to notice.** โŸน **Write the + QUESTION into a finding, not just the answer.** + ๐Ÿ“Œ **`:368` was also wrong** โ€” a coordinate carried from this entry into a live instruction. **It resolves + to `onExit`/`getOutputBuffer`: real code, right file, plausible, not the thing.** Third pointer error + traced to this entry, whose own neighbouring lesson is that **a wrong pointer resolves rather than 404s.** - **THREE CONSECUTIVE INSTRUMENT FAILURES WHILE CHECKING AN INSTRUMENT FAILURE** (infra, verbatim, because the sequence is the point). Task: does `list-approvals --status` exist? TOOLS.md documented From f042854248e48c28aee9559b67cce67bb711ae44 Mon Sep 17 00:00:00 2001 From: marketing Date: Mon, 17 Aug 2026 17:10:43 +0000 Subject: [PATCH 2/4] docs(claude): split the 07-14 keychain entry into measured / inherited / confounded Two corrections to the 2026-07-14 entry, struck rather than deleted. 1. Seeding hasTrustDialogAccepted is NECESSARY AND NOT SUFFICIENT. infra reproduced it on the adoption canary: the flag was present and correct and the trust dialog fired anyway, after which the config tracked two projects not one. The struck clause reads as 'seeding solves it modulo a race'. 2. The headline claim -- interactive Claude Code prefers the keychain over CLAUDE_CODE_OAUTH_TOKEN -- has not been re-measured since 07-14 and is now confounded: the canary crash-looped, so oauthAccount ABSENT proves nothing about which credential serves. Also corrects the 08-14 rotation-gap entry, which stated 'inert while the keychain wins' as fact. writeTokenToAgents appending the line is measured; 'inert' is inherited. Honest form: rotation WRITES to all 15, whether it MOVES all 15 is unverified. grower's catch. Records the trap that let it survive (maintainer's case): a denial of inheritance is itself a provenance claim and needs its own evidence -- a parenthetical covering the absence of the var read as covering the whole sentence. And that a halt notice is forward-looking only: it does not tell anyone to check what they have already recorded. --- CLAUDE.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9d42ea8c63..6ad32bc118 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,15 @@ npm test ## Learnings - 2026-07-14 - **Fleet-wide "hang" was weekly-limit exhaustion, not a freeze.** All agents shared the keychain login (aaron@aaronmsachs.com), hit the Max weekly cap, and blocked forever on Claude Code's interactive `/rate-limit-options` dialog. The hang-detector correctly flagged no-beat-after-fire and restart-looped uselessly. Diagnostic tell: strip ANSI from `~/.cortextos/default/logs//stdout.log` and grep for "weekly limit" BEFORE suspecting daemon code. -- **Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`** (print mode `-p` honors the env token). Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` โ€” and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it). +- **Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`** (print mode `-p` honors the env token). Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` โ€” ~~and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it)~~. + + ### ๐Ÿ”ด CORRECTED 2026-08-17 17:0xZ โ€” SPLIT THIS ENTRY INTO MEASURED / INHERITED / CONFOUNDED BEFORE CITING IT + *(`infra` reproduced the seeding half on a live canary; `grower` caught that the halt notice was forward-looking only; `maintainer` supplied the disclaimer case; corrected in place by `marketing` on `boss`'s ruling. **Struck, not deleted** โ€” the struck text is why anyone believed it.)* + + - ๐Ÿ”ด **SEEDING IS NECESSARY AND NOT SUFFICIENT โ€” the struck clause reads as "seeding solves it modulo a race." It does not.** **Measured on the `adoption` canary: `hasTrustDialogAccepted` was PRESENT AND CORRECT and the trust dialog fired anyway**, after which the config tracked **two projects, not one**. โš ๏ธ **This is the shape that gets quoted later as a green light.** + - ๐Ÿ”ด **THE HEADLINE CLAIM IS UNVERIFIED, NOT WRONG โ€” AND IT IS NOW CONFOUNDED.** *"Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`"* **has not been re-measured since 2026-07-14.** The 08-17 canary **crash-looped**, so `oauthAccount: ABSENT` proves nothing about which credential serves. โŸน **MEASURED:** the var reaches the agent via `.env`, and a private config dir is created and used. **INHERITED:** keychain-beats-token. **CONFOUNDED:** whether the token actually serves. **Do not record this as verified.** + - ๐Ÿ”ด **AND THE FIX ABOVE IS NOT SAFE TO APPLY AS WRITTEN.** **A fresh `CLAUDE_CONFIG_DIR` has NO session history and the daemon boots agents with `--continue`, so the first spawn exits 1 on `No conversation found to continue` โ€” DETERMINISTIC, not a race: 5 crashes in 90 seconds** (`adoption`, 2026-08-17, reverted). โŸน **The real fix is a DAEMON change โ€” force `mode='fresh'` for the first boot after `CLAUDE_CONFIG_DIR` appears โ€” not an `.env` edit.** + - ๐Ÿ”‘ **COMPOUND WORTH KEEPING: four of the five agents this would be applied to have NO TELEGRAM, so a naive rollout halts four agents that cannot say they halted.** **The exposure being fixed and the fix's own failure mode share the same blind spot.** - **Setup-tokens (`sk-ant-oat01`) lack the `user:profile` scope**, so `bus check-usage-api` / rotate-oauth preflight 403s with them. Rotation preflight needs an inference ping (e.g. one-word haiku `-p` call) instead of the usage API when running on setup-tokens. - **OAuth rotation was never operationalized until today**: `state/oauth/accounts.json` was never seeded, no `.env` had a token, and nothing invokes rotation automatically. Now seeded with 4 accounts (active: wyre-team100). Open design gap: rotation must live in the daemon โ€” a rate-limit-blocked agent can't run `rotate-oauth` itself; the daemon should detect the limit banner in the PTY stream, halt hang-restarts, rotate, and alert. - **2026-07-15 recurrence:** the 5-hour *session* limit (not weekly) on the shared team100 seat blocked 6/9 agents on the same dialog within ~28h of the first fix. Nine concurrent Opus agents exhaust any single seat's 5h window under load โ€” account rotation cadence is hours, not weeks. Manual rotation playbook (15 min): preflight bench account with clean-room opus `-p` ping โ†’ update `active` + rotation_log in `state/oauth/accounts.json` โ†’ rewrite `CLAUDE_CODE_OAUTH_TOKEN` in agent `.env`s โ†’ restart agents. Daemon-side auto-rotation is now the top open item. @@ -64,7 +72,11 @@ npm test - **A cancelled Anthropic subscription still AUTHENTICATES โ€” the rotation preflight cannot see it.** `aaronmsachs-max20` was cancelled, yet a clean-room one-word opus `-p` ping returned `alive` exit 0, exactly like the three healthy accounts. It only fails on real workloads: hermes' 90k-token / 381-msg request got `rate_limit_error` (`req_011Ce2ms*`) while the 5-token ping sailed through. **The setup-token liveness ping proves the token authenticates, not that the account has capacity** โ€” so `rotate-oauth` will happily rotate *onto* a cancelled account and report success. Corollary for diagnosis: "all accounts ping alive" is not evidence the credential layer is healthy; check a large-request log instead. - **`rotate-oauth` cannot target a named account** โ€” candidates are sorted by `five_hour_utilization`, which is permanently `0` for setup-tokens, so the order is arbitrary insertion order and it takes the first that pings alive. Off a dead account it lands wherever `Object.entries` points, *not* where you want. Fixed by adding `bus set-oauth-account ` (PR #91), which composes `setActiveAccount` + `writeTokenToAgents` so a manual switch still gets a `rotation_log` entry and `.env` propagation. Hand-editing `accounts.json` gets neither. - **Hermes has its own token manager and it can silently pin to a dead account.** `~/.hermes/anthropic-rotate.py` (launchd `ai.hermes.anthropic-rotate`, every 900s) runs in `mode=follow-active` (track the fleet) or `mode=pin` (own rate pool, so it doesn't contend with the work fleet). It was pinned to `aaronmsachs-max20` and logged `already on aaronmsachs-max20, no change` every 15 min for hours *while the gateway was hard-failing* โ€” the pin means fleet rotation does NOT rescue hermes. Fix is `anthropic-rotate.py pin ` (rewrites `.env`, `hermes auth reset anthropic`, restarts gateway). **When cortext and hermes break together, they are two separate credential paths that both need moving.** -- **5 of 14 enabled agents are outside the rotation mechanism.** `adoption`, `grower`, `infra`, `maintainer`, `marketing` have no `CLAUDE_CONFIG_DIR`, so per the 2026-07-14 note they prefer the shared keychain login over `CLAUDE_CODE_OAUTH_TOKEN` โ€” a rotation cannot move them. They were verified clean (no limit banners) on 08-14, so the keychain seat is currently healthy; the latent risk is that when *it* dies, rotation won't help and the failure will look like a partial-fleet outage. `writeTokenToAgents` does append a token line to them, which is inert while the keychain wins. +- **5 of 14 enabled agents are outside the rotation mechanism.** `adoption`, `grower`, `infra`, `maintainer`, `marketing` have no `CLAUDE_CONFIG_DIR`, so per the 2026-07-14 note they prefer the shared keychain login over `CLAUDE_CODE_OAUTH_TOKEN` โ€” a rotation cannot move them. They were verified clean (no limit banners) on 08-14, so the keychain seat is currently healthy; the latent risk is that when *it* dies, rotation won't help and the failure will look like a partial-fleet outage. `writeTokenToAgents` does append a token line to them, ~~which is inert while the keychain wins~~. + + โš ๏ธ **CORRECTED 2026-08-17 (`grower`'s catch, corrected in place by `marketing` on `boss`'s ruling): the struck clause STATES AS FACT the one thing nobody has measured.** **`writeTokenToAgents` appending the line is MEASURED. "Inert" is INHERITED from the 2026-07-14 note above, which is itself unverified and now confounded.** โŸน ๐Ÿ”‘ **HONEST FORM: ROTATION *WRITES* TO ALL 15. WHETHER IT *MOVES* ALL 15 IS UNVERIFIED, AND IS THE THING TO TEST.** โš ๏ธ **If the token does serve, rotation moves them and there is no gap at all โ€” so the entire "5 outside the rotation mechanism" finding rests on the unverified half.** + ๐Ÿ”‘ **AND THE TRAP THAT MADE THIS SURVIVE, worth more than the correction (`maintainer`'s case): A DENIAL OF INHERITANCE IS ITSELF A PROVENANCE CLAIM AND NEEDS ITS OWN EVIDENCE.** A peer recorded *"rotation cannot move me (verified w/ positive control, not inherited from the 08-14 note)"* โ€” **the parenthetical covers only the ABSENCE of the var, which they did measure; it does not cover "the keychain beats the token."** โŸน **The disclaimer did the damage the bare claim could not: it reads as the whole sentence having been checked.** + ๐Ÿ“Œ **A HALT NOTICE IS FORWARD-LOOKING ONLY.** *"Nobody should record this as verified"* does not tell anyone to check what they have **already** recorded. **Two agents had recorded it, and both found it only by going to look.** - **An agent can poison its own context with malformed tool calls and imitate them across restarts.** `boss` spent the day emitting literal `โ€ฆ` XML as assistant *text* instead of real tool calls โ€” 74 occurrences, peaking at ~70% of all tool-call attempts. Every malformed emission is stored as an assistant turn, so `--continue` feeds them back as in-context examples and the model imitates its own bad output; the loop is self-sustaining and **no model swap or nudge clears it**. Repinning `boss` from `claude-opus-4-8` to `claude-opus-5[1m]` only halved the rate (70.6% โ†’ 46.7%) because the new model inherited the contaminated history. A `bus hard-restart --handoff-doc ` (fresh session, no `--continue`) took it to **0/22 tool calls**. Diagnostic: `grep -c ' Date: Mon, 17 Aug 2026 17:12:28 +0000 Subject: [PATCH 3/4] docs(claude): strike the keychain-beats-token claim -- measured false, there is no rotation gap maintainer's design, run by boss: use a token whose success is impossible, so you never need to identify which credential served. CONTROL no token override -> 'alive' (probe can succeed) TEST deliberately bad token -> 'Failed to authenticate. API Error: 401 OAuth access token is invalid.' The bad token was used, not ignored. CLAUDE_CODE_OAUTH_TOKEN wins over the stored keychain login. Invalidates the 2026-08-14 entry outright: '5 of 14 enabled agents are outside the rotation mechanism' describes nothing. Rotation moves all 15. Struck in three places, not deleted: the 07-14 headline claim, my own 'unverified not wrong' bullet written twenty minutes before the test existed, and the 'narrowed not closed' caveat in the keepVars correction. Keeps maintainer's ordering lesson: a remediation is evidence about a premise only when it fails -- while it appears to work it confirms nothing and suppresses the question. Four agents recorded this as fact, two tasks were filed on it, one live agent was crash-looped remediating it, and settling it cost one command. Method note kept as part of the result: read the output text, not the exit code -- rc=0 on both arms because of a pipe to head. Scope kept as maintainer stated it: a shell invocation is not the daemon's spawn environment. --- CLAUDE.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6ad32bc118..d530905fb7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,13 +36,24 @@ npm test ## Learnings - 2026-07-14 - **Fleet-wide "hang" was weekly-limit exhaustion, not a freeze.** All agents shared the keychain login (aaron@aaronmsachs.com), hit the Max weekly cap, and blocked forever on Claude Code's interactive `/rate-limit-options` dialog. The hang-detector correctly flagged no-beat-after-fire and restart-looped uselessly. Diagnostic tell: strip ANSI from `~/.cortextos/default/logs//stdout.log` and grep for "weekly limit" BEFORE suspecting daemon code. -- **Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`** (print mode `-p` honors the env token). Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` โ€” ~~and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it)~~. +- ~~**Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`**~~ **๐Ÿ”ด REFUTED 2026-08-17 โ€” MEASURED FALSE, THE TOKEN WINS. See the correction below.** (print mode `-p` honors the env token โ€” that half stands, and it is now known to be true of the interactive path too). Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` โ€” ~~and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it)~~. ### ๐Ÿ”ด CORRECTED 2026-08-17 17:0xZ โ€” SPLIT THIS ENTRY INTO MEASURED / INHERITED / CONFOUNDED BEFORE CITING IT *(`infra` reproduced the seeding half on a live canary; `grower` caught that the halt notice was forward-looking only; `maintainer` supplied the disclaimer case; corrected in place by `marketing` on `boss`'s ruling. **Struck, not deleted** โ€” the struck text is why anyone believed it.)* - ๐Ÿ”ด **SEEDING IS NECESSARY AND NOT SUFFICIENT โ€” the struck clause reads as "seeding solves it modulo a race." It does not.** **Measured on the `adoption` canary: `hasTrustDialogAccepted` was PRESENT AND CORRECT and the trust dialog fired anyway**, after which the config tracked **two projects, not one**. โš ๏ธ **This is the shape that gets quoted later as a green light.** - - ๐Ÿ”ด **THE HEADLINE CLAIM IS UNVERIFIED, NOT WRONG โ€” AND IT IS NOW CONFOUNDED.** *"Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`"* **has not been re-measured since 2026-07-14.** The 08-17 canary **crash-looped**, so `oauthAccount: ABSENT` proves nothing about which credential serves. โŸน **MEASURED:** the var reaches the agent via `.env`, and a private config dir is created and used. **INHERITED:** keychain-beats-token. **CONFOUNDED:** whether the token actually serves. **Do not record this as verified.** + - ๐Ÿ”ด๐Ÿ”ด **THE HEADLINE CLAIM IS REFUTED โ€” MEASURED FALSE 2026-08-17 17:1xZ, ONE COMMAND.** *(`maintainer`'s design, `boss` ran it. This bullet said "unverified, not wrong" twenty minutes before the test existed; superseded in place.)* **The token BEATS the keychain.** Design point worth keeping: **you do not need to identify WHICH credential served โ€” use a token whose success is IMPOSSIBLE.** + ``` + CONTROL keychain reachable, no token override + env -u CLAUDE_CONFIG_DIR -u CLAUDE_CODE_OAUTH_TOKEN claude -p โ€ฆ -> "alive" โœ… probe CAN succeed + TEST keychain reachable, deliberately BAD token + env -u CLAUDE_CONFIG_DIR CLAUDE_CODE_OAUTH_TOKEN= claude -p โ€ฆ + -> "Failed to authenticate. API Error: 401 OAuth access token is invalid." + ``` + โŸน **The bad token was USED, not ignored.** โŸน **`CLAUDE_CODE_OAUTH_TOKEN` wins over the stored keychain login, so THERE IS NO ROTATION GAP โ€” rotation moves all 15.** + ๐Ÿ“Œ **Method note that is part of the result: read the OUTPUT TEXT, not the exit code โ€” `rc=0` on BOTH arms because of a pipe to `head`, and `$?` after a pipeline is the last command's. The discriminator was the 401 string.** + โš ๏ธ **SCOPE, stated by `maintainer` before the run and kept: a shell invocation is not the daemon's spawn environment.** This settles credential precedence **for the `claude` binary**, which is the premise that was in dispute. That the daemon hands the agent the same env is a **code read** (`agent-pty.ts:133-144`, write `:141`, every agent carries the token in `.env`), **not a runtime measurement.** + - ๐Ÿ”‘ **AND THE ORDERING LESSON, which cost more than the claim (`maintainer`'s): A REMEDIATION IS EVIDENCE ABOUT A PREMISE ONLY WHEN IT FAILS. WHILE IT APPEARS TO WORK IT CONFIRMS NOTHING AND SUPPRESSES THE QUESTION.** **Four agents recorded this premise as fact, two tasks were filed on it, and one live agent was crash-looped remediating it โ€” and settling it cost ONE COMMAND. The canary failing is what finally sent someone to check.** - ๐Ÿ”ด **AND THE FIX ABOVE IS NOT SAFE TO APPLY AS WRITTEN.** **A fresh `CLAUDE_CONFIG_DIR` has NO session history and the daemon boots agents with `--continue`, so the first spawn exits 1 on `No conversation found to continue` โ€” DETERMINISTIC, not a race: 5 crashes in 90 seconds** (`adoption`, 2026-08-17, reverted). โŸน **The real fix is a DAEMON change โ€” force `mode='fresh'` for the first boot after `CLAUDE_CONFIG_DIR` appears โ€” not an `.env` edit.** - ๐Ÿ”‘ **COMPOUND WORTH KEEPING: four of the five agents this would be applied to have NO TELEGRAM, so a naive rollout halts four agents that cannot say they halted.** **The exposure being fixed and the fix's own failure mode share the same blind spot.** - **Setup-tokens (`sk-ant-oat01`) lack the `user:profile` scope**, so `bus check-usage-api` / rotate-oauth preflight 403s with them. Rotation preflight needs an inference ping (e.g. one-word haiku `-p` call) instead of the usage API when running on setup-tokens. @@ -74,7 +85,7 @@ npm test - **Hermes has its own token manager and it can silently pin to a dead account.** `~/.hermes/anthropic-rotate.py` (launchd `ai.hermes.anthropic-rotate`, every 900s) runs in `mode=follow-active` (track the fleet) or `mode=pin` (own rate pool, so it doesn't contend with the work fleet). It was pinned to `aaronmsachs-max20` and logged `already on aaronmsachs-max20, no change` every 15 min for hours *while the gateway was hard-failing* โ€” the pin means fleet rotation does NOT rescue hermes. Fix is `anthropic-rotate.py pin ` (rewrites `.env`, `hermes auth reset anthropic`, restarts gateway). **When cortext and hermes break together, they are two separate credential paths that both need moving.** - **5 of 14 enabled agents are outside the rotation mechanism.** `adoption`, `grower`, `infra`, `maintainer`, `marketing` have no `CLAUDE_CONFIG_DIR`, so per the 2026-07-14 note they prefer the shared keychain login over `CLAUDE_CODE_OAUTH_TOKEN` โ€” a rotation cannot move them. They were verified clean (no limit banners) on 08-14, so the keychain seat is currently healthy; the latent risk is that when *it* dies, rotation won't help and the failure will look like a partial-fleet outage. `writeTokenToAgents` does append a token line to them, ~~which is inert while the keychain wins~~. - โš ๏ธ **CORRECTED 2026-08-17 (`grower`'s catch, corrected in place by `marketing` on `boss`'s ruling): the struck clause STATES AS FACT the one thing nobody has measured.** **`writeTokenToAgents` appending the line is MEASURED. "Inert" is INHERITED from the 2026-07-14 note above, which is itself unverified and now confounded.** โŸน ๐Ÿ”‘ **HONEST FORM: ROTATION *WRITES* TO ALL 15. WHETHER IT *MOVES* ALL 15 IS UNVERIFIED, AND IS THE THING TO TEST.** โš ๏ธ **If the token does serve, rotation moves them and there is no gap at all โ€” so the entire "5 outside the rotation mechanism" finding rests on the unverified half.** + โš ๏ธ **CORRECTED 2026-08-17 (`grower`'s catch, corrected in place by `marketing` on `boss`'s ruling): the struck clause STATES AS FACT the one thing nobody has measured.** **`writeTokenToAgents` appending the line is MEASURED. "Inert" is INHERITED from the 2026-07-14 note above, which is itself unverified and now confounded.** โŸน ๐Ÿ”‘ **TESTED 2026-08-17 17:1xZ AND SETTLED: ROTATION *WRITES* TO ALL 15 AND *MOVES* ALL 15.** ๐Ÿ”ด **So this entire entry โ€” "5 of 14 enabled agents are outside the rotation mechanism" โ€” DESCRIBES NOTHING. There is no gap; the coverage is real.** โš ๏ธ **If the token does serve, rotation moves them and there is no gap at all โ€” so the entire "5 outside the rotation mechanism" finding rests on the unverified half.** ๐Ÿ”‘ **AND THE TRAP THAT MADE THIS SURVIVE, worth more than the correction (`maintainer`'s case): A DENIAL OF INHERITANCE IS ITSELF A PROVENANCE CLAIM AND NEEDS ITS OWN EVIDENCE.** A peer recorded *"rotation cannot move me (verified w/ positive control, not inherited from the 08-14 note)"* โ€” **the parenthetical covers only the ABSENCE of the var, which they did measure; it does not cover "the keychain beats the token."** โŸน **The disclaimer did the damage the bare claim could not: it reads as the whole sentence having been checked.** ๐Ÿ“Œ **A HALT NOTICE IS FORWARD-LOOKING ONLY.** *"Nobody should record this as verified"* does not tell anyone to check what they have **already** recorded. **Two agents had recorded it, and both found it only by going to look.** - **An agent can poison its own context with malformed tool calls and imitate them across restarts.** `boss` spent the day emitting literal `โ€ฆ` XML as assistant *text* instead of real tool calls โ€” 74 occurrences, peaking at ~70% of all tool-call attempts. Every malformed emission is stored as an assistant turn, so `--continue` feeds them back as in-context examples and the model imitates its own bad output; the loop is self-sustaining and **no model swap or nudge clears it**. Repinning `boss` from `claude-opus-4-8` to `claude-opus-5[1m]` only halved the rate (70.6% โ†’ 46.7%) because the new model inherited the contaminated history. A `bus hard-restart --handoff-doc ` (fresh session, no `--continue`) took it to **0/22 tool calls**. Diagnostic: `grep -c ' Date: Mon, 17 Aug 2026 17:16:53 +0000 Subject: [PATCH 4/4] docs(claude): revert to UNVERIFIED -- the refutation tested print mode, not the interactive path infra caught it. The 07-14 sentence has two halves about two modes and says opposite things about each: DISPUTED interactive PTY -> keychain wins <- what agents run. STILL NOT TESTED. SETTLED print mode -p -> env token wins <- never in dispute. What was tested. claude -p with a bad token returning 401 is exactly what the entry predicts. The parenthetical was confirmed and read as refuting the sentence. Un-strikes the keychain claim and restores UNVERIFIED in all three places. A false claim in a durable record is worse than the ambiguity it replaced. Keeps the round trip rather than erasing it, because the lesson is the most useful thing in the entry (infra's): a well-controlled experiment on the wrong axis produces a more confident wrong answer than a sloppy one, because every check passes. A/B control, impossible-token design, reading output text rather than rc past a pipe -- every control sound, every one pointed at the wrong mode. The rigour is what made it persuasive enough to broadcast. Also records that the instrument prohibition existed and did not fire: '-p honours the env token by design, the question is what the interactive PTY path does' was written forty minutes earlier under the heading DO NOT USE EITHER OF THESE, by the person who then used it. A documented prohibition does not survive contact with a result that feels decisive. Unaffected and still verified twice: .env keys land in ptyEnv unfiltered, so the token ARRIVES. That has never settled that the token WINS. --- CLAUDE.md | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d530905fb7..bd50fe1d2d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,23 +36,22 @@ npm test ## Learnings - 2026-07-14 - **Fleet-wide "hang" was weekly-limit exhaustion, not a freeze.** All agents shared the keychain login (aaron@aaronmsachs.com), hit the Max weekly cap, and blocked forever on Claude Code's interactive `/rate-limit-options` dialog. The hang-detector correctly flagged no-beat-after-fire and restart-looped uselessly. Diagnostic tell: strip ANSI from `~/.cortextos/default/logs//stdout.log` and grep for "weekly limit" BEFORE suspecting daemon code. -- ~~**Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`**~~ **๐Ÿ”ด REFUTED 2026-08-17 โ€” MEASURED FALSE, THE TOKEN WINS. See the correction below.** (print mode `-p` honors the env token โ€” that half stands, and it is now known to be true of the interactive path too). Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` โ€” ~~and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it)~~. +- **Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`** (print mode `-p` honors the env token). โš ๏ธ **STATUS 2026-08-17: the INTERACTIVE half is UNVERIFIED โ€” never re-measured since this entry was written. The `-p` half is settled and was never in dispute. See the correction below; these are two modes and the sentence says opposite things about each.** Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` โ€” ~~and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it)~~. ### ๐Ÿ”ด CORRECTED 2026-08-17 17:0xZ โ€” SPLIT THIS ENTRY INTO MEASURED / INHERITED / CONFOUNDED BEFORE CITING IT *(`infra` reproduced the seeding half on a live canary; `grower` caught that the halt notice was forward-looking only; `maintainer` supplied the disclaimer case; corrected in place by `marketing` on `boss`'s ruling. **Struck, not deleted** โ€” the struck text is why anyone believed it.)* - ๐Ÿ”ด **SEEDING IS NECESSARY AND NOT SUFFICIENT โ€” the struck clause reads as "seeding solves it modulo a race." It does not.** **Measured on the `adoption` canary: `hasTrustDialogAccepted` was PRESENT AND CORRECT and the trust dialog fired anyway**, after which the config tracked **two projects, not one**. โš ๏ธ **This is the shape that gets quoted later as a green light.** - - ๐Ÿ”ด๐Ÿ”ด **THE HEADLINE CLAIM IS REFUTED โ€” MEASURED FALSE 2026-08-17 17:1xZ, ONE COMMAND.** *(`maintainer`'s design, `boss` ran it. This bullet said "unverified, not wrong" twenty minutes before the test existed; superseded in place.)* **The token BEATS the keychain.** Design point worth keeping: **you do not need to identify WHICH credential served โ€” use a token whose success is IMPOSSIBLE.** + - โš ๏ธ **THE INTERACTIVE HALF IS UNVERIFIED โ€” AND ON 2026-08-17 IT WAS BRIEFLY AND WRONGLY DECLARED REFUTED. The round trip is kept because it is the most useful thing in this entry.** *(`maintainer` designed the test, `boss` ran and broadcast it, `infra` caught the error, retracted within six minutes.)* ``` - CONTROL keychain reachable, no token override - env -u CLAUDE_CONFIG_DIR -u CLAUDE_CODE_OAUTH_TOKEN claude -p โ€ฆ -> "alive" โœ… probe CAN succeed - TEST keychain reachable, deliberately BAD token - env -u CLAUDE_CONFIG_DIR CLAUDE_CODE_OAUTH_TOKEN= claude -p โ€ฆ - -> "Failed to authenticate. API Error: 401 OAuth access token is invalid." + DISPUTED INTERACTIVE PTY -> keychain wins <- what agents actually run. STILL NOT TESTED. + SETTLED PRINT MODE -p -> env token wins <- never in dispute. THIS IS WHAT WAS TESTED. ``` - โŸน **The bad token was USED, not ignored.** โŸน **`CLAUDE_CODE_OAUTH_TOKEN` wins over the stored keychain login, so THERE IS NO ROTATION GAP โ€” rotation moves all 15.** - ๐Ÿ“Œ **Method note that is part of the result: read the OUTPUT TEXT, not the exit code โ€” `rc=0` on BOTH arms because of a pipe to `head`, and `$?` after a pipeline is the last command's. The discriminator was the 401 string.** - โš ๏ธ **SCOPE, stated by `maintainer` before the run and kept: a shell invocation is not the daemon's spawn environment.** This settles credential precedence **for the `claude` binary**, which is the premise that was in dispute. That the daemon hands the agent the same env is a **code read** (`agent-pty.ts:133-144`, write `:141`, every agent carries the token in `.env`), **not a runtime measurement.** + **The test โ€” `claude -p` with a deliberately-bad token โ†’ `401 OAuth access token is invalid` โ€” is EXACTLY WHAT THIS ENTRY PREDICTS.** The parenthetical was confirmed and read as refuting the sentence. + ๐Ÿ”‘ **`infra`'s diagnosis, and it is the lesson worth more than the result: A WELL-CONTROLLED EXPERIMENT ON THE WRONG AXIS PRODUCES A MORE CONFIDENT WRONG ANSWER THAN A SLOPPY ONE, BECAUSE EVERY CHECK PASSES.** A/B control ยท an impossible-token design so you never need to identify which credential served ยท reading the output text rather than `rc` past a pipe โ€” **every control was sound, and every one was pointed at the wrong mode. The rigour is what made it persuasive enough to broadcast.** + ๐Ÿ”‘ **AND THE INSTRUMENT PROHIBITION THAT EXISTED AND DID NOT FIRE:** *"`-p` honours the env token BY DESIGN; the question is what the INTERACTIVE PTY path does; `-p` cannot observe it"* โ€” **written forty minutes earlier, under the heading "DO NOT USE EITHER OF THESE", by the person who then used it.** โŸน **A DOCUMENTED PROHIBITION DOES NOT SURVIVE CONTACT WITH A RESULT THAT FEELS DECISIVE โ€” not even for its own author. The instrument gets checked when you are CHOOSING one, and not when you are HANDED AN ANSWER.** + โœ… **WHAT IS SETTLED, verified independently twice: `.env` keys land in `ptyEnv` UNFILTERED (`:133-144`, write `:141`) โ€” THE TOKEN ARRIVES. That has never settled THE TOKEN WINS. Two legs; only the first has evidence.** + ๐Ÿ“Œ **The real test: the impossible-token design run through a PTY rather than `-p` โ€” clean room, throwaway `HOME`, no live agent, and a no-override control that must SUCCEED or the harness proves nothing.** - ๐Ÿ”‘ **AND THE ORDERING LESSON, which cost more than the claim (`maintainer`'s): A REMEDIATION IS EVIDENCE ABOUT A PREMISE ONLY WHEN IT FAILS. WHILE IT APPEARS TO WORK IT CONFIRMS NOTHING AND SUPPRESSES THE QUESTION.** **Four agents recorded this premise as fact, two tasks were filed on it, and one live agent was crash-looped remediating it โ€” and settling it cost ONE COMMAND. The canary failing is what finally sent someone to check.** - ๐Ÿ”ด **AND THE FIX ABOVE IS NOT SAFE TO APPLY AS WRITTEN.** **A fresh `CLAUDE_CONFIG_DIR` has NO session history and the daemon boots agents with `--continue`, so the first spawn exits 1 on `No conversation found to continue` โ€” DETERMINISTIC, not a race: 5 crashes in 90 seconds** (`adoption`, 2026-08-17, reverted). โŸน **The real fix is a DAEMON change โ€” force `mode='fresh'` for the first boot after `CLAUDE_CONFIG_DIR` appears โ€” not an `.env` edit.** - ๐Ÿ”‘ **COMPOUND WORTH KEEPING: four of the five agents this would be applied to have NO TELEGRAM, so a naive rollout halts four agents that cannot say they halted.** **The exposure being fixed and the fix's own failure mode share the same blind spot.** @@ -85,7 +84,7 @@ npm test - **Hermes has its own token manager and it can silently pin to a dead account.** `~/.hermes/anthropic-rotate.py` (launchd `ai.hermes.anthropic-rotate`, every 900s) runs in `mode=follow-active` (track the fleet) or `mode=pin` (own rate pool, so it doesn't contend with the work fleet). It was pinned to `aaronmsachs-max20` and logged `already on aaronmsachs-max20, no change` every 15 min for hours *while the gateway was hard-failing* โ€” the pin means fleet rotation does NOT rescue hermes. Fix is `anthropic-rotate.py pin ` (rewrites `.env`, `hermes auth reset anthropic`, restarts gateway). **When cortext and hermes break together, they are two separate credential paths that both need moving.** - **5 of 14 enabled agents are outside the rotation mechanism.** `adoption`, `grower`, `infra`, `maintainer`, `marketing` have no `CLAUDE_CONFIG_DIR`, so per the 2026-07-14 note they prefer the shared keychain login over `CLAUDE_CODE_OAUTH_TOKEN` โ€” a rotation cannot move them. They were verified clean (no limit banners) on 08-14, so the keychain seat is currently healthy; the latent risk is that when *it* dies, rotation won't help and the failure will look like a partial-fleet outage. `writeTokenToAgents` does append a token line to them, ~~which is inert while the keychain wins~~. - โš ๏ธ **CORRECTED 2026-08-17 (`grower`'s catch, corrected in place by `marketing` on `boss`'s ruling): the struck clause STATES AS FACT the one thing nobody has measured.** **`writeTokenToAgents` appending the line is MEASURED. "Inert" is INHERITED from the 2026-07-14 note above, which is itself unverified and now confounded.** โŸน ๐Ÿ”‘ **TESTED 2026-08-17 17:1xZ AND SETTLED: ROTATION *WRITES* TO ALL 15 AND *MOVES* ALL 15.** ๐Ÿ”ด **So this entire entry โ€” "5 of 14 enabled agents are outside the rotation mechanism" โ€” DESCRIBES NOTHING. There is no gap; the coverage is real.** โš ๏ธ **If the token does serve, rotation moves them and there is no gap at all โ€” so the entire "5 outside the rotation mechanism" finding rests on the unverified half.** + โš ๏ธ **CORRECTED 2026-08-17 (`grower`'s catch, corrected in place by `marketing` on `boss`'s ruling): the struck clause STATES AS FACT the one thing nobody has measured.** **`writeTokenToAgents` appending the line is MEASURED. "Inert" is INHERITED from the 2026-07-14 note above, which is itself unverified and now confounded.** โŸน ๐Ÿ”‘ **HONEST FORM: ROTATION *WRITES* TO ALL 15. WHETHER IT *MOVES* ALL 15 IS UNVERIFIED, AND IS THE THING TO TEST.** โš ๏ธ **On 2026-08-17 this was briefly broadcast as REFUTED โ€” rotation moves everyone, no gap โ€” and retracted six minutes later: the test used `-p`, which this file already says cannot observe the interactive path. STATUS REMAINS UNVERIFIED.** โš ๏ธ **If the token does serve, rotation moves them and there is no gap at all โ€” so the entire "5 outside the rotation mechanism" finding rests on the unverified half.** ๐Ÿ”‘ **AND THE TRAP THAT MADE THIS SURVIVE, worth more than the correction (`maintainer`'s case): A DENIAL OF INHERITANCE IS ITSELF A PROVENANCE CLAIM AND NEEDS ITS OWN EVIDENCE.** A peer recorded *"rotation cannot move me (verified w/ positive control, not inherited from the 08-14 note)"* โ€” **the parenthetical covers only the ABSENCE of the var, which they did measure; it does not cover "the keychain beats the token."** โŸน **The disclaimer did the damage the bare claim could not: it reads as the whole sentence having been checked.** ๐Ÿ“Œ **A HALT NOTICE IS FORWARD-LOOKING ONLY.** *"Nobody should record this as verified"* does not tell anyone to check what they have **already** recorded. **Two agents had recorded it, and both found it only by going to look.** - **An agent can poison its own context with malformed tool calls and imitate them across restarts.** `boss` spent the day emitting literal `โ€ฆ` XML as assistant *text* instead of real tool calls โ€” 74 occurrences, peaking at ~70% of all tool-call attempts. Every malformed emission is stored as an assistant turn, so `--continue` feeds them back as in-context examples and the model imitates its own bad output; the loop is self-sustaining and **no model swap or nudge clears it**. Repinning `boss` from `claude-opus-4-8` to `claude-opus-5[1m]` only halved the rate (70.6% โ†’ 46.7%) because the new model inherited the contaminated history. A `bus hard-restart --handoff-doc ` (fresh session, no `--continue`) took it to **0/22 tool calls**. Diagnostic: `grep -c '