Skip to content

Regression after d53a513 (#1664): agent-sdk fallback silently overrides a configured OpenAI-compatible provider; compression retry loop spawns headless claude sessions #1762

Open

Description

@nobyArdor

Regression after d53a513 (#1664): agent-sdk fallback silently overrides a configured OpenAI-compatible provider; compression retry loop spawns headless claude sessions

Summary

Commit d53a513 (PR #1664) replaced provider auto-detection with an explicit
AGENTMEMORY_LLM_PROVIDER opt-in. For users upgrading from ≤0.9.31 with an existing
~/.agentmemory/.env that contains both a working OPENAI_API_KEY (OpenAI-compatible
endpoint, e.g. DeepSeek) and AGENTMEMORY_ALLOW_AGENT_SDK=true, the upgrade silently
switches all LLM compression from their configured provider to the claude-agent-sdk
fallback
— spawning headless claude CLI child sessions that authenticate with the
ambient Claude Code login and burn the user's Claude subscription quota.

In our incident this combined with two further gaps (retry queue ignores
AGENTMEMORY_AUTO_COMPRESS=false; every new observation still gets one ungated
compression dispatch) into a self-sustaining loop: with the Claude quota exhausted,
each retry spawned a new headless claude.exe, every ~40–90 seconds for hours —
11–12 concurrent instances (~2.8 GB RAM), each initializing the full MCP stack from
~/.claude.json (12 serena processes), and enough failed session starts that the
account got flagged by Anthropic.

Environment

  • agentmemory 0.9.33 (daemon via Worker Compose, native Windows 11)
  • ~/.agentmemory/.env (relevant keys):
    AGENTMEMORY_ALLOW_AGENT_SDK=true      # carried over from earlier experiments
    OPENAI_API_KEY=sk-...                 # DeepSeek
    OPENAI_BASE_URL=https://api.deepseek.com
    OPENAI_MODEL=deepseek-chat
    # AGENTMEMORY_LLM_PROVIDER not set (upgraded from 0.9.31 where it was not needed)

Before / after

≤ 0.9.31 (detectProvider) 0.9.33 (after d53a513)
Provider chosen with the env above openai (auto-detected from OPENAI_API_KEY) → DeepSeek noop → agent-sdk fallback → headless claude.exe using the Claude Code login
Compression traffic DeepSeek API Claude subscription quota
If the provider fails DeepSeek error, no side effects Retry loop spawns a new claude session per attempt

Reproduction

  1. Configure the env above on a machine with Claude Code logged in.
  2. Let observations flow (any capture hooks) until a batch triggers compression.
  3. Watch claude.exe (path: node_modules/@anthropic-ai/claude-agent-sdk-win32-x64/)
    spawn per compression call, each starting MCP servers from ~/.claude.json.
  4. Exhaust the Claude quota → every call fails → compression-recovery re-dispatches
    indefinitely (see related bugs) → process pile-up.

Incident chain (verified live, 2026-11-11/12)

  1. Compression dispatched via AgentSDKProvider → headless claude.exe per batch.
  2. Each claude.exe initializes the full Claude Code MCP stack (~/.claude.json),
    including serena — so the symptom surfaced as "serena processes multiplying".
  3. Quota exhausted → compression_failed → observation retained for retry.
  4. compression-recovery re-dispatched regardless of
    AGENTMEMORY_AUTO_COMPRESS=false
    (set early in the incident — did not help).
  5. New observations kept receiving one ungated compression dispatch despite the flag.
  6. Net: ~1 new claude.exe+serena every 40–90 s for ~4 h; 53+ "retained for compression
    retry" warnings in one worker-log window; account/IP flagged by Anthropic.

Worker log, line 2 (the tool's own warning — present but easy to miss in daemon stdout):

stderr  [agentmemory] WARNING: agent-sdk fallback enabled via AGENTMEMORY_ALLOW_AGENT_SDK=true.
This spawns @anthropic-ai/claude-agent-sdk child sessions ...

Related bugs found during the incident (candidates for separate issues)

  1. compression-recovery ignores AGENTMEMORY_AUTO_COMPRESS=false — keeps LLM-retrying
    a feature the user disabled.
  2. New observations get one ungated mem::compress dispatch even with the flag off.
  3. mem::compress fails with compression_failed via the OpenAI-compatible provider
    (DeepSeek) while mem::summarize succeeds on the same provider — compress path bug.
  4. Engine crashes at startup (<100 ms, misleading os error 5) when all engine.log*
    are exactly at the 10 MiB rotation cap in compose-state; workaround: move logs away.
  5. .env template ships AGENTMEMORY_ALLOW_AGENT_SDK=true with a leading space
    (line 64) — fragile against dotenv parsing differences.

Suggested fixes

  1. Provider precedence: never let the agent-sdk fallback win over a configured
    OPENAI_API_KEY/explicit provider; at minimum emit a loud migration warning on
    upgrade when auto-detection previously selected a provider and no
    AGENTMEMORY_LLM_PROVIDER is set.
  2. Gate compression-recovery and the per-observation dispatch behind
    AGENTMEMORY_AUTO_COMPRESS=false (mark backlog synthetic instead of retrying).
  3. Investigate the compress-vs-summarize asymmetry on OpenAI-compatible providers.
  4. Fix startup rotation with capped logs; surface the real engine error instead of the
    secondary os error 5.
  5. Remove the leading space in the .env template.

Workaround (current production config here)

AGENTMEMORY_AUTO_COMPRESS=false
AGENTMEMORY_ALLOW_AGENT_SDK=false
AGENTMEMORY_LLM_PROVIDER=openai    # explicit; routes retries to DeepSeek

Verified: zero claude.exe spawns over 10+ minutes of heavy observation traffic;
summarization works via DeepSeek (qualityScore 100).

Evidence available on request

  • Live process listings during the incident (12× claude.exe with command lines/RAM,
    12× serena with parents, spawner chain iii.exe → node dist/index.mjs → claude.exe)
  • Worker-log excerpts with original line numbers (retained for compression retry ×53)
  • 21 client-side session transcripts (~/.claude/projects/...) created by the spawned
    SDK sessions in the incident window
  • agentmemory-incident-evidence-20261112.txt, agentmemory-log-fragments-20261112.txt

Activity

  1. nobyArdor commented on Nov 11, 2026

    @nobyArdor
    Author

    Evidence 1: live process listings during the incident

    === INCIDENT EVIDENCE: automated claude-agent-sdk spawn loop, 2026-11-11/12 ===
    Machine: single developer workstation (Windows 11), owner: gorbunov
    Root cause: agentmemory v0.9.33 provider-selection regression (commit d53a513, PR #1664)
      - fell back to claude-agent-sdk (ambient Claude Code login) instead of the
        configured DeepSeek API key; with plan quota exhausted, its retry loop
        spawned a headless claude.exe every ~40-90 s for hours.
    All process listings below were captured LIVE via PowerShell Get-CimInstance
    Win32_Process during the incident (timestamps local = UTC+3).
    
    --- TIMELINE (local, UTC+3) ---
    ~21:30  first pile-up of serena/claude processes noticed by user
    23:50   user's legitimate interactive Claude Code session started (PID 6248)
    00:42-00:48  spawn loop observed directly: 11 SDK claude.exe in 6 minutes
    00:50   loop cause identified; SDK processes killed
    01:05   AGENTMEMORY_ALLOW_AGENT_SDK=false applied; loop stopped permanently
    
    --- LISTING 1: serena.exe processes (captured 2026-11-12 00:49 local) ---
    PID    start    parent                     context
    23108  23:50:13 claude.exe 6248            claude-code (user's interactive session - legit)
    26096  00:04:09 DeepSeek Harness.exe       deepseek-harness (legit)
    272    00:41:02 ZCode.exe                  zcode (legit)
    21812  00:42:48 claude.exe 10208 (SDK)     claude-code  <-- spawned by loop
    16796  00:43:37 claude.exe 19384 (SDK)     claude-code  <-- spawned by loop
    12192  00:43:46 claude.exe 24432 (SDK)     claude-code  <-- spawned by loop
    25956  00:44:05 claude.exe 15656 (SDK)     claude-code  <-- spawned by loop
    13280  00:44:09 claude.exe 21892 (SDK)     claude-code  <-- spawned by loop
    16604  00:44:24 claude.exe 15172 (SDK)     claude-code  <-- spawned by loop
    18804  00:46:48 claude.exe 13432 (SDK)     claude-code  <-- spawned by loop
    21096  00:47:18 claude.exe 12872 (SDK)     claude-code  <-- spawned by loop
    12300  00:48:02 claude.exe 23084 (SDK)     claude-code  <-- spawned by loop
    
    --- LISTING 2: claude.exe processes (captured 2026-11-12 00:50 local) ---
    PID    start    RAM     command line
    6248   23:50:07 289MB   WinGet\Links\claude.exe                              <-- legit interactive
    10208  00:42:47 224MB   agentmemory\node_modules\@anthropic-ai\claude-agent-sdk-win32-x64\claude.exe --output-format stream-json --verbose ...
    19384  00:43:35 287MB   (same SDK path, headless)
    24432  00:43:44 222MB   (same SDK path)
    15656  00:44:03 238MB   (same SDK path)
    21892  00:44:05 222MB   (same SDK path)
    15172  00:44:20 219MB   (same SDK path)
    13432  00:46:46 223MB   (same SDK path)
    12872  00:47:16 237MB   (same SDK path)
    23084  00:47:59 239MB   (same SDK path)
    16680  00:48:24 223MB   (same SDK path)
    28868  00:48:51 222MB   (same SDK path)
    TOTAL: 12 claude.exe, ~2846 MB RAM; 11 of them headless SDK children spawned in 6 minutes.
    
    --- SPAWNER PARENT CHAIN (traced live) ---
    claude.exe (SDK headless)
     <- node.exe  "node dist/index.mjs"            (agentmemory worker)
     <- cmd.exe   /C "node.exe dist/index.mjs"
     <- iii.exe   compose --namespace agentmemory-3111   (agentmemory daemon)
    
    --- CLAUDE-SIDE ARTIFACTS (still on disk) ---
    21 session transcripts created by the spawned SDK sessions in the window
    2026-11-12 00:40-01:10 local (2026-11-11 21:40-22:10 UTC), each initializing
    the full MCP+hooks stack before failing:
    112083 bytes  2026-11-12 00:51:03.7630352000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/b4007e57-4249-474f-8168-28aea3504d95.jsonl
    117660 bytes  2026-11-12 01:08:03.1831265000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/81f667d6-fe11-4819-a7d5-db53cf233f07.jsonl
    78041 bytes  2026-11-12 00:50:55.7676982000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/aa1baf62-72ce-4b38-8a2b-946e027d8566.jsonl
    78535 bytes  2026-11-12 00:52:57.2445869000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/d57550d8-5ce5-4ff4-8320-ff576a7637f3.jsonl
    79877 bytes  2026-11-12 00:52:56.4059921000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/b67deb00-5828-43f9-b045-f5bd883fc581.jsonl
    79946 bytes  2026-11-12 00:52:55.0862841000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/5ef9d3f5-1734-47e0-8536-b94a89e98f3c.jsonl
    82221 bytes  2026-11-12 00:50:48.5494437000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/97e5c71a-eea7-4f3b-8b67-7733013c636d.jsonl
    83394 bytes  2026-11-12 00:50:43.2355634000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/4568c596-68e6-48f5-9405-7bec9b363f7a.jsonl
    83455 bytes  2026-11-12 01:06:07.8371207000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/724d928d-f75c-46fc-85fe-118febdee852.jsonl
    83598 bytes  2026-11-12 00:50:56.6615866000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/356810e4-41e7-493b-ac21-961d6dba10e4.jsonl
    83833 bytes  2026-11-12 00:42:59.9223958000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/63632406-a082-4e89-b2f8-26b19e832d87.jsonl
    83842 bytes  2026-11-12 00:49:16.7359816000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/16af86a8-768d-4aec-8068-59c2ff9d0542.jsonl
    84607 bytes  2026-11-12 00:50:32.2627896000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/99029e18-93c2-4b6c-a860-058bbec7bda1.jsonl
    89100 bytes  2026-11-12 00:40:14.7189524000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/5a1fb496-1dc5-4423-b451-dfd45b917875.jsonl
    89216 bytes  2026-11-12 00:50:34.7724223000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/cca429c0-7e1d-4f3c-aa38-cde517c11d77.jsonl
    89573 bytes  2026-11-12 00:50:12.8479027000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/8c68e9d1-c9c2-45a8-bfd3-89e1ffb25bde.jsonl
    91605 bytes  2026-11-12 01:06:22.5707564000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/276aff21-21be-46d0-aa7f-4d10aa95f03c.jsonl
    91907 bytes  2026-11-12 00:50:36.9356269000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/13ddbeb3-8614-4465-8384-65c2a5640076.jsonl
    92901 bytes  2026-11-12 00:50:59.1974762000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/6a9fa3e6-a163-4022-91c1-35dfc8126de4.jsonl
    95398 bytes  2026-11-12 00:49:57.2800532000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/f3432ee8-4b42-42b3-8da3-ffb8b541042e.jsonl
    98544 bytes  2026-11-12 00:50:46.8937562000  C:/Users/gorbunov/.claude/projects/C--Develop-node-js-agentmemory/10f64a61-9bd8-43d0-9427-b24183dfb262.jsonl
    
    --- SERVER-SIDE LOGS (available on request) ---
    agentmemory daemon log: %APPDATA%\agentmemory\compose-state\...\logs\agentmemory.log
      (contains 53+ "Observation retained for compression retry" warnings = the retry loop)
    compose log: ~/.agentmemory/compose-3111.log
    Fix applied: AGENTMEMORY_ALLOW_AGENT_SDK=false + AGENTMEMORY_LLM_PROVIDER=openai
    in ~/.agentmemory/.env; loop verified stopped (zero spawns over 10+ min of traffic).
    Upstream bugs documented; maintainer of agentmemory is the account owner.
    

    Evidence 2: worker-log fragments (original line numbers preserved, skips marked)

    ================================================================================
    EVIDENCE EXCERPTS (FRAGMENTS) — NOT THE FULL FILE
    Source : %APPDATA%\agentmemory\compose-state\compose-a502c8e7\agentmemory-3111\logs\agentmemory.log.txt
             (byte-identical copy of agentmemory.log taken 2026-11-12 01:49 local; 353 lines, 60561 bytes)
    Method : excerpts extracted by ripgrep/sed with ORIGINAL LINE NUMBERS preserved;
             lines between fragments are omitted and marked as [ ... N lines skipped ... ].
    Context: agentmemory daemon worker log. Log lines carry no inline timestamps
             (format: <stream>\t[agentmemory] <level> <message>); chronological order
             follows line order; observation IDs (obs_...) are time-ordered by design.
    ================================================================================
    
    --- FRAGMENT A: lines 1-2 — log identity header + the agent-sdk fallback WARNING
    --- (the tool itself warns that child claude sessions will be spawned)
       1| # iii-compose-worker-log-v1 cb9a516d-2300-4bbe-a754-9cad3afd5a37
       2| stderr	[agentmemory] WARNING: agent-sdk fallback enabled via AGENTMEMORY_ALLOW_AGENT_SDK=true. This spawns @anthropic-ai/claude-agent-sdk child sessions that can trigger the Stop-hook recursion loop. A SDK-child env marker is set to block re-entry, but prefer setting a real API key in ~/.agentmemory/.env instead.
    
    [ ... 8 lines skipped (startup noise: engine config, port bindings) ... ]
    
    --- FRAGMENT B: lines 11-12, 18 — FIRST retry-loop warnings (loop begins)
      11| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwch6weq_9f2c9dc17ab2","sessionId":"f4dab943-37e8-4905-9262-bd74b727e55f"}
      12| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcdo121_3fdeebf0804e","sessionId":"e3d1c4ee-fda7-4cc2-b7d7-33268ce0f403"}
      18| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwccrmro_03bf2b559b4d","sessionId":"caa36244-cc6c-40fc-8de7-75792942e38e"}
    
    [ ... 24 lines skipped ... ]
    
    --- FRAGMENT C: lines 103-106 — BURST: 4 retry warnings back-to-back
    --- (this is the cadence that spawned headless claude.exe instances)
     103| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcjauxp_d3f57d881333","sessionId":"f4dab943-37e8-4905-9262-bd74b727e55f"}
     104| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcdlqwf_e069ec73b8fd","sessionId":"e3d1c4ee-fda7-4cc2-b7d7-33268ce0f403"}
     105| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcdlzmb_799fbc629b4c","sessionId":"e3d1c4ee-fda7-4cc2-b7d7-33268ce0f403"}
     106| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcdm0p6_1ddad7694c01","sessionId":"e3d1c4ee-fda7-4cc2-b7d7-33268ce0f403"}
    
    [ ... 6 lines skipped ... ]
    
    --- FRAGMENT D: lines 199-208 — loop at full speed (6 warnings in 10 lines)
     199| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnhvpj_0fe2953e3c91","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
     201| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnjjl3_1337085bf94a","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
     203| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnipu8_3f29e3266e58","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
     205| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnkh4w_581009af7a3b","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
     207| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnkpe2_061deedad423","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
     208| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnjzt6_85fd28b26196","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
    
    [ ... 6 lines skipped ... ]
    
    --- FRAGMENT E: lines 228-233 — AFTER FIX (ALLOW_AGENT_SDK=false, LLM_PROVIDER=openai):
    --- observations switch to zero-LLM synthetic compression; DeepSeek summarize succeeds
     228| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwcnomid_05ca6bc3d841","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"post_tool_use","compress":"synthetic"}
     229| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwcnpkyb_80cc0e379c79","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"stop","compress":"synthetic"}
     230| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwcntb91_a8ab86057a91","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"prompt_submit","compress":"synthetic"}
     231| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwcnu190_00c114746bcc","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"post_tool_use","compress":"synthetic"}
     232| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwcnuj1s_3b8b39814dd9","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"post_tool_use","compress":"synthetic"}
     233| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwcnvk3f_4e5c9e6126ed","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"post_tool_use","compress":"synthetic"}
    
    --- FRAGMENT F: lines 253-254 — LAST retry warning (residual backlog) and the
    --- provider-hint line proving explicit DeepSeek provider was then chosen
     253| stdout	[agentmemory] warn Observation retained for compression retry {"observationId":"obs_mwcnange_dbe19f340230","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529"}
     254| stdout	[agentmemory] info Observation captured {"obsId":"obs_mwco7htt_99251ca9da3f","sessionId":"sess_7bd70ea7-8d61-4bda-b0eb-ff303c0d7529","hook":"post_tool_use","compress":"synthetic"}
    
    ================================================================================
    SUMMARY: 53 occurrences of "Observation retained for compression retry"
    at lines: 11 12 18 43 103-106 112-123 130-144 154-158 164-217 235 253 (grouped).
    Each retry dispatched an LLM compression via the claude-agent-sdk fallback,
    spawning a headless claude.exe child session (see companion process evidence:
    agentmemory-incident-evidence-20261112.txt). Loop stopped after the config fix.
    ================================================================================
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions