diff --git a/CLAUDE.md b/CLAUDE.md index 258d8c1265..bd50fe1d2d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,25 @@ npm test ## Learnings - 2026-07-14 - **Fleet-wide "hang" was weekly-limit exhaustion, not a freeze.** All agents shared the keychain login (aaron@aaronmsachs.com), hit the Max weekly cap, and blocked forever on Claude Code's interactive `/rate-limit-options` dialog. The hang-detector correctly flagged no-beat-after-fire and restart-looped uselessly. Diagnostic tell: strip ANSI from `~/.cortextos/default/logs//stdout.log` and grep for "weekly limit" BEFORE suspecting daemon code. -- **Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`** (print mode `-p` honors the env token). Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` — and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it). +- **Interactive Claude Code prefers the stored keychain login over `CLAUDE_CODE_OAUTH_TOKEN`** (print mode `-p` honors the env token). ⚠️ **STATUS 2026-08-17: the INTERACTIVE half is UNVERIFIED — never re-measured since this entry was written. The `-p` half is settled and was never in dispute. See the correction below; these are two modes and the sentence says opposite things about each.** Fix: per-agent `CLAUDE_CONFIG_DIR` (in agent `.env`, pointing at `~/.cortextos/default/state//claude-config/`) so the token is the only credential. Seed `.claude.json` with `hasCompletedOnboarding`, `bypassPermissionsModeAccepted`, and `projects..hasTrustDialogAccepted` — ~~and expect a boot race on first spawn (two agents still showed the folder-trust dialog once; a restart after claude's own config rewrite cleared it)~~. + + ### 🔴 CORRECTED 2026-08-17 17:0xZ — SPLIT THIS ENTRY INTO MEASURED / INHERITED / CONFOUNDED BEFORE CITING IT + *(`infra` reproduced the seeding half on a live canary; `grower` caught that the halt notice was forward-looking only; `maintainer` supplied the disclaimer case; corrected in place by `marketing` on `boss`'s ruling. **Struck, not deleted** — the struck text is why anyone believed it.)* + + - 🔴 **SEEDING IS NECESSARY AND NOT SUFFICIENT — the struck clause reads as "seeding solves it modulo a race." It does not.** **Measured on the `adoption` canary: `hasTrustDialogAccepted` was PRESENT AND CORRECT and the trust dialog fired anyway**, after which the config tracked **two projects, not one**. ⚠️ **This is the shape that gets quoted later as a green light.** + - ⚠️ **THE INTERACTIVE HALF IS UNVERIFIED — AND ON 2026-08-17 IT WAS BRIEFLY AND WRONGLY DECLARED REFUTED. The round trip is kept because it is the most useful thing in this entry.** *(`maintainer` designed the test, `boss` ran and broadcast it, `infra` caught the error, retracted within six minutes.)* + ``` + DISPUTED INTERACTIVE PTY -> keychain wins <- what agents actually run. STILL NOT TESTED. + SETTLED PRINT MODE -p -> env token wins <- never in dispute. THIS IS WHAT WAS TESTED. + ``` + **The test — `claude -p` with a deliberately-bad token → `401 OAuth access token is invalid` — is EXACTLY WHAT THIS ENTRY PREDICTS.** The parenthetical was confirmed and read as refuting the sentence. + 🔑 **`infra`'s diagnosis, and it is the lesson worth more than the result: A WELL-CONTROLLED EXPERIMENT ON THE WRONG AXIS PRODUCES A MORE CONFIDENT WRONG ANSWER THAN A SLOPPY ONE, BECAUSE EVERY CHECK PASSES.** A/B control · an impossible-token design so you never need to identify which credential served · reading the output text rather than `rc` past a pipe — **every control was sound, and every one was pointed at the wrong mode. The rigour is what made it persuasive enough to broadcast.** + 🔑 **AND THE INSTRUMENT PROHIBITION THAT EXISTED AND DID NOT FIRE:** *"`-p` honours the env token BY DESIGN; the question is what the INTERACTIVE PTY path does; `-p` cannot observe it"* — **written forty minutes earlier, under the heading "DO NOT USE EITHER OF THESE", by the person who then used it.** ⟹ **A DOCUMENTED PROHIBITION DOES NOT SURVIVE CONTACT WITH A RESULT THAT FEELS DECISIVE — not even for its own author. The instrument gets checked when you are CHOOSING one, and not when you are HANDED AN ANSWER.** + ✅ **WHAT IS SETTLED, verified independently twice: `.env` keys land in `ptyEnv` UNFILTERED (`:133-144`, write `:141`) — THE TOKEN ARRIVES. That has never settled THE TOKEN WINS. Two legs; only the first has evidence.** + 📌 **The real test: the impossible-token design run through a PTY rather than `-p` — clean room, throwaway `HOME`, no live agent, and a no-override control that must SUCCEED or the harness proves nothing.** + - 🔑 **AND THE ORDERING LESSON, which cost more than the claim (`maintainer`'s): A REMEDIATION IS EVIDENCE ABOUT A PREMISE ONLY WHEN IT FAILS. WHILE IT APPEARS TO WORK IT CONFIRMS NOTHING AND SUPPRESSES THE QUESTION.** **Four agents recorded this premise as fact, two tasks were filed on it, and one live agent was crash-looped remediating it — and settling it cost ONE COMMAND. The canary failing is what finally sent someone to check.** + - 🔴 **AND THE FIX ABOVE IS NOT SAFE TO APPLY AS WRITTEN.** **A fresh `CLAUDE_CONFIG_DIR` has NO session history and the daemon boots agents with `--continue`, so the first spawn exits 1 on `No conversation found to continue` — DETERMINISTIC, not a race: 5 crashes in 90 seconds** (`adoption`, 2026-08-17, reverted). ⟹ **The real fix is a DAEMON change — force `mode='fresh'` for the first boot after `CLAUDE_CONFIG_DIR` appears — not an `.env` edit.** + - 🔑 **COMPOUND WORTH KEEPING: four of the five agents this would be applied to have NO TELEGRAM, so a naive rollout halts four agents that cannot say they halted.** **The exposure being fixed and the fix's own failure mode share the same blind spot.** - **Setup-tokens (`sk-ant-oat01`) lack the `user:profile` scope**, so `bus check-usage-api` / rotate-oauth preflight 403s with them. Rotation preflight needs an inference ping (e.g. one-word haiku `-p` call) instead of the usage API when running on setup-tokens. - **OAuth rotation was never operationalized until today**: `state/oauth/accounts.json` was never seeded, no `.env` had a token, and nothing invokes rotation automatically. Now seeded with 4 accounts (active: wyre-team100). Open design gap: rotation must live in the daemon — a rate-limit-blocked agent can't run `rotate-oauth` itself; the daemon should detect the limit banner in the PTY stream, halt hang-restarts, rotate, and alert. - **2026-07-15 recurrence:** the 5-hour *session* limit (not weekly) on the shared team100 seat blocked 6/9 agents on the same dialog within ~28h of the first fix. Nine concurrent Opus agents exhaust any single seat's 5h window under load — account rotation cadence is hours, not weeks. Manual rotation playbook (15 min): preflight bench account with clean-room opus `-p` ping → update `active` + rotation_log in `state/oauth/accounts.json` → rewrite `CLAUDE_CODE_OAUTH_TOKEN` in agent `.env`s → restart agents. Daemon-side auto-rotation is now the top open item. @@ -64,7 +82,11 @@ npm test - **A cancelled Anthropic subscription still AUTHENTICATES — the rotation preflight cannot see it.** `aaronmsachs-max20` was cancelled, yet a clean-room one-word opus `-p` ping returned `alive` exit 0, exactly like the three healthy accounts. It only fails on real workloads: hermes' 90k-token / 381-msg request got `rate_limit_error` (`req_011Ce2ms*`) while the 5-token ping sailed through. **The setup-token liveness ping proves the token authenticates, not that the account has capacity** — so `rotate-oauth` will happily rotate *onto* a cancelled account and report success. Corollary for diagnosis: "all accounts ping alive" is not evidence the credential layer is healthy; check a large-request log instead. - **`rotate-oauth` cannot target a named account** — candidates are sorted by `five_hour_utilization`, which is permanently `0` for setup-tokens, so the order is arbitrary insertion order and it takes the first that pings alive. Off a dead account it lands wherever `Object.entries` points, *not* where you want. Fixed by adding `bus set-oauth-account ` (PR #91), which composes `setActiveAccount` + `writeTokenToAgents` so a manual switch still gets a `rotation_log` entry and `.env` propagation. Hand-editing `accounts.json` gets neither. - **Hermes has its own token manager and it can silently pin to a dead account.** `~/.hermes/anthropic-rotate.py` (launchd `ai.hermes.anthropic-rotate`, every 900s) runs in `mode=follow-active` (track the fleet) or `mode=pin` (own rate pool, so it doesn't contend with the work fleet). It was pinned to `aaronmsachs-max20` and logged `already on aaronmsachs-max20, no change` every 15 min for hours *while the gateway was hard-failing* — the pin means fleet rotation does NOT rescue hermes. Fix is `anthropic-rotate.py pin ` (rewrites `.env`, `hermes auth reset anthropic`, restarts gateway). **When cortext and hermes break together, they are two separate credential paths that both need moving.** -- **5 of 14 enabled agents are outside the rotation mechanism.** `adoption`, `grower`, `infra`, `maintainer`, `marketing` have no `CLAUDE_CONFIG_DIR`, so per the 2026-07-14 note they prefer the shared keychain login over `CLAUDE_CODE_OAUTH_TOKEN` — a rotation cannot move them. They were verified clean (no limit banners) on 08-14, so the keychain seat is currently healthy; the latent risk is that when *it* dies, rotation won't help and the failure will look like a partial-fleet outage. `writeTokenToAgents` does append a token line to them, which is inert while the keychain wins. +- **5 of 14 enabled agents are outside the rotation mechanism.** `adoption`, `grower`, `infra`, `maintainer`, `marketing` have no `CLAUDE_CONFIG_DIR`, so per the 2026-07-14 note they prefer the shared keychain login over `CLAUDE_CODE_OAUTH_TOKEN` — a rotation cannot move them. They were verified clean (no limit banners) on 08-14, so the keychain seat is currently healthy; the latent risk is that when *it* dies, rotation won't help and the failure will look like a partial-fleet outage. `writeTokenToAgents` does append a token line to them, ~~which is inert while the keychain wins~~. + + ⚠️ **CORRECTED 2026-08-17 (`grower`'s catch, corrected in place by `marketing` on `boss`'s ruling): the struck clause STATES AS FACT the one thing nobody has measured.** **`writeTokenToAgents` appending the line is MEASURED. "Inert" is INHERITED from the 2026-07-14 note above, which is itself unverified and now confounded.** ⟹ 🔑 **HONEST FORM: ROTATION *WRITES* TO ALL 15. WHETHER IT *MOVES* ALL 15 IS UNVERIFIED, AND IS THE THING TO TEST.** ⚠️ **On 2026-08-17 this was briefly broadcast as REFUTED — rotation moves everyone, no gap — and retracted six minutes later: the test used `-p`, which this file already says cannot observe the interactive path. STATUS REMAINS UNVERIFIED.** ⚠️ **If the token does serve, rotation moves them and there is no gap at all — so the entire "5 outside the rotation mechanism" finding rests on the unverified half.** + 🔑 **AND THE TRAP THAT MADE THIS SURVIVE, worth more than the correction (`maintainer`'s case): A DENIAL OF INHERITANCE IS ITSELF A PROVENANCE CLAIM AND NEEDS ITS OWN EVIDENCE.** A peer recorded *"rotation cannot move me (verified w/ positive control, not inherited from the 08-14 note)"* — **the parenthetical covers only the ABSENCE of the var, which they did measure; it does not cover "the keychain beats the token."** ⟹ **The disclaimer did the damage the bare claim could not: it reads as the whole sentence having been checked.** + 📌 **A HALT NOTICE IS FORWARD-LOOKING ONLY.** *"Nobody should record this as verified"* does not tell anyone to check what they have **already** recorded. **Two agents had recorded it, and both found it only by going to look.** - **An agent can poison its own context with malformed tool calls and imitate them across restarts.** `boss` spent the day emitting literal `…` XML as assistant *text* instead of real tool calls — 74 occurrences, peaking at ~70% of all tool-call attempts. Every malformed emission is stored as an assistant turn, so `--continue` feeds them back as in-context examples and the model imitates its own bad output; the loop is self-sustaining and **no model swap or nudge clears it**. Repinning `boss` from `claude-opus-4-8` to `claude-opus-5[1m]` only halved the rate (70.6% → 46.7%) because the new model inherited the contaminated history. A `bus hard-restart --handoff-doc ` (fresh session, no `--continue`) took it to **0/22 tool calls**. Diagnostic: `grep -c '