Activity

1. adboio

   added

   bugSomething isn't working rightSomething isn't working right

   on Apr 30, 2026
2. adboio commented on Apr 30, 2026

   

   adboio

   on Apr 30, 2026

   Last edited by adboio

   ContributorAuthor

   More actions

   maybe slop, maybe not... plan from posthog code investigation:

   ---

   Investigating flaky skill loading in slash commands

   Context

   Users report that skills sometimes don't show up / aren't recognized when typing slash commands — often at session start, sometimes mid-task. The behaviour is flaky and inconsistent, which points to race conditions and missing reactive refresh paths rather than a single deterministic bug.

   After tracing the full pipeline from skill discovery on disk → SDK plugin loading → ACP available_commands_update → renderer command suggestions, I found multiple independent failure modes that all manifest as "skill missing from slash menu" or "agent doesn't know about skill". Below is a triage of the most likely root causes, ranked by how often each is plausibly hit.

   This plan is a diagnosis + remediation proposal, not yet a code change. The user asked us to "try to identify why this might be happening" — so the primary deliverable is the failure-mode analysis, with concrete fix options listed for each so we can pick what to ship.

   How the pipeline works (quick reference)

   There are two parallel sources of truth for "what skills are available", and the UI merges them:

   1. Renderer-side filesystem scan — apps/code/src/renderer/features/task-detail/components/TaskInput.tsx:479-492 calls trpcClient.skills.list.query() once on mount; result stored in useDraftStore.commands[sessionId].
   2. Agent-side SDK report — packages/agent/src/adapters/claude/claude-agent.ts:1509-1517 calls query.supportedCommands() and emits ACP available_commands_update; renderer extracts via extractAvailableCommandsFromEvents (apps/code/src/renderer/utils/session.ts:159-178).

   getCommandSuggestions (apps/code/src/renderer/features/message-editor/suggestions/getSuggestions.ts:156-175) merges [CODE_COMMANDS, agentCommands or filesystemCommands]. agentCommands takes precedence when a taskId is present, otherwise the filesystem fallback is used.

   The agent's own awareness of skills comes from the SDK reading SKILL.md files inside the plugins array passed at session creation (apps/code/src/main/services/agent/service.ts:683-694 → buildClaudeCodeOptions). This is a one-shot snapshot at session start.

   Failure modes (ranked by likelihood)

   1. available_commands_update arrives after the user opens the slash menu

   Where: packages/agent/src/adapters/claude/claude-agent.ts:1567-1581

   private deferBackgroundFetches(q: Query): void {
     Promise.all([
       new Promise<void>((resolve) => setTimeout(resolve, 10)).then(() =>
         this.sendAvailableCommandsUpdate(),
       ),
       fetchMcpToolMetadata(...),
     ])...
   }

   sendAvailableCommandsUpdate is fire-and-forget with a 10ms setTimeout. On a busy event loop (especially on app cold start when many things initialise simultaneously), 10ms easily becomes hundreds of ms. The user sees the editor become interactive and types / before the update arrives. They get only CODE_COMMANDS + whatever the filesystem fallback returned (which races the same way — see PostHog/code#3).

   Worse for resumed sessions (claude-agent.ts:267-291): loadSession passes skipBackgroundFetches: true, replays history (which can take seconds for long sessions), and only then calls deferBackgroundFetches. For a multi-thousand-message session, the slash menu is stale for the entire replay.

   Fix options:

   • await sendAvailableCommandsUpdate() synchronously inside createSession before returning (drop the 10ms setTimeout). Trade-off: small latency on session create.
   • Add a loading flag to the renderer command suggestions so the dropdown shows "Loading skills…" instead of an incomplete list.
   • Re-emit available_commands_update on a skillsUpdated event from PosthogPluginService (currently a dead signal — see feat: repo selection, edit task title and description code#5).

   2. Concurrent buildSyntheticPlugin clobbers symlinks

   Where: apps/code/src/main/services/agent/discover-plugins.ts:128-186

   async function buildSyntheticPlugin(...) {
     // ...
     await fs.promises.mkdir(syntheticSkillsDir, { recursive: true });
     // delete every symlink
     const existing = await fs.promises.readdir(syntheticSkillsDir);
     await Promise.all(existing.map((e) => fs.promises.rm(...)));
     // recreate symlinks
     await Promise.all(skillDirs.map(async (skillName) => {
       await fs.promises.symlink(realSrc, dest);
     }));
   }

   Each session start wipes and recreates the symlinks under <userData>/plugins/user-skills/skills/, <userData>/plugins/repo-skills-<hash>/skills/. There is no mutex.

   When two sessions start concurrently — e.g.

   • App launch with N reconnecting sessions
   • Two windows open at once
   • A user creates a new task while another is initialising

   …session B's rm can run between session A's rm and symlink. The Claude SDK loads plugin skills lazily by readdir-ing syntheticSkillsDir; if it reads during the gap, it sees an empty or partial directory. The agent's system reminder lists fewer skills than exist on disk.

   Fix options:

   • Use a per-pluginDir mutex (e.g. a shared Map<string, Promise<void>>) so only one rebuild runs at a time.
   • Write to a temp dir then atomically rename (rename is atomic on POSIX for same-filesystem dirs).
   • Skip the rebuild when the symlink set hasn't changed (compare skillDirs to existing entries first).

   3. PosthogPluginService.initialize() races with first session create

   Where: apps/code/src/main/services/posthog-plugin/service.ts:63-95

   @postConstruct()
   init(): void {
     this.initialize().catch(...);  // fire-and-forget
   }
   
   private async initialize() {
     if (!existsSync(join(this.runtimePluginDir, "plugin.json"))) {
       await this.copyBundledPlugin();
     }
     await overlayDownloadedSkills(...);
     await syncCodexSkills(...);
     this.intervalId = setInterval(...);
     await this.updateSkills();  // network download
   }

   @postConstruct is async-fire-and-forget. If a tRPC agent.createSession call comes in before initialize() has finished:

   • getPluginPath() may return a runtime dir whose copyBundledPlugin is half-done (partial files).
   • overlayDownloadedSkills may be mid-write when the SDK reads.
   • updateSkills() (the periodic one too) atomically swaps in new skills, but the swap happens directory-by-directory — a session that starts during that window sees a torn state.

   This is especially bad on cold start: the splash → first session window is exactly when initialize() is doing the most work.

   Fix options:

   • Have getPluginPath() await an initializationPromise cached on the service, so callers block until first init is done.
   • Move copyBundledPlugin to a synchronous path on app start before any agent service is instantiated.
   • For the periodic updateSkills swap, use a write-to-temp + atomic-rename pattern at the plugin directory level, not per-skill.

   4. Renderer's filesystem skill list is fetched once on mount, never refreshed

   Where: apps/code/src/renderer/features/task-detail/components/TaskInput.tsx:479-492

   useEffect(() => {
     let cancelled = false;
     trpcClient.skills.list.query().then((skills) => {
       if (cancelled) return;
       useDraftStore.getState().actions.setCommands(...);
     });
     return () => { cancelled = true; ... };
   }, [promptSessionId]);

   This effect runs once when the task input mounts. If the user:

   • adds a SKILL.md to ~/.claude/skills/ mid-task,
   • installs a marketplace plugin,
   • has the periodic updateSkills download new skills,

   …the slash menu doesn't update until they navigate away and back. This matches the user-reported pattern "sometimes mid-task as well".

   Even at mount time it races with PostHog/code#3 — if the skills list is fetched before PosthogPluginService.initialize() populates the runtime plugin dir, it returns only bundled skills (or empty).

   Fix options:

   • Subscribe to a tRPC subscription that emits when PosthogPluginService fires skillsUpdated (currently emitted but unused — see feat: repo selection, edit task title and description code#5) and refetch.
   • Use TanStack Query with staleTime and an invalidation key tied to skillsUpdated.
   • Watch ~/.claude/skills/ and the runtime plugin dir for changes, push refresh notifications.

   5. skillsUpdated event has no consumers

   Where: apps/code/src/main/services/posthog-plugin/service.ts:147 emits skillsUpdated. Grep across the codebase finds zero subscribers outside tests.

   Consequence:

   • After the periodic 30-min updateSkills succeeds, the running agent session's plugins are not refreshed (the SDK was given a snapshot at session start).
   • The renderer's filesystem command list is not refreshed.
   • Existing slash menus continue showing stale data until session/task remount.

   Fix options:

   • Wire it through tRPC subscription to renderer so TaskInput refetches.
   • Wire it through agent service so it can re-emit available_commands_update on running sessions.

   6. The agent's system-reminder skill list is frozen at session creation

   This is the deepest issue. The system reminder you see at the start of a turn (The following skills are available for use with the Skill tool: …) is generated by the Claude Agent SDK from the plugins array given at session creation. There is no mechanism to update this list mid-session. Even if available_commands_update is re-sent to the UI, the model's context still has the old list.

   This means:

   • A skill added after session start is invisible to the agent even if the UI shows it.
   • The user types /new-skill, the UI accepts it, the prompt is sent, the agent has no idea what /new-skill is and treats it as plain text.

   This may explain reports where the slash menu shows the skill but the agent then fails to apply it.

   Fix options:

   • Inject the up-to-date skill list as a system-reminder content block on each prompt (similar to how available-skills is already shown). Requires changes in packages/agent.
   • Detect when a /<skill-name> prompt references a skill the SDK didn't know about, and force a session restart.
   • Document this as a known limitation and surface a "restart session to load new skills" UX hint when skillsUpdated fires.

   7. SKILLS_ZIP_URL env var unset → only bundled skills

   Where: apps/code/src/main/services/posthog-plugin/service.ts:21-24

   const SKILLS_ZIP_URL = process.env.SKILLS_ZIP_URL ?? "";
   if (!SKILLS_ZIP_URL) log.warn("SKILLS_ZIP_URL environment variable is not set");

   If a build ships without SKILLS_ZIP_URL set, updateSkills does nothing useful and only bundled skills are available. Worth confirming with the user whether their failures are correlated with builds where this might be missing — would explain "missing skills" but not "flakiness".

   Recommended priority

   User confirmed the symptom is "skill missing from slash menu" (UI staleness), not "agent ignored a selected skill". That narrows scope: failure mode PostHog/code#6 (agent's frozen in-context skill list) is real but out of scope for this fix — defer it. The UI dropdown is fed by two sources (renderer filesystem fetch + agent's available_commands_update), and both have race / staleness bugs.

   1. Highest impact: Fix feat: generated api client for tasks code#1 (await sendAvailableCommandsUpdate in createSession, including the resume path) and feat: task creation and task list improvements code#4 (refresh renderer skill list reactively, not just on mount). These together fix the "open slash menu, skill missing, close + reopen, skill there" symptom — the most-reported one.
   2. Required for correctness: Fix chore: set up biome and ci code#3 (getPluginPath awaits init) and chore: config housekeeping code#2 (mutex around buildSyntheticPlugin). chore: set up biome and ci code#3 prevents the renderer's filesystem fetch from returning empty for bundled skills during cold start. chore: config housekeeping code#2 prevents the agent-side commands list from being torn when multiple sessions start concurrently.
   3. Closes the mid-task gap: Fix feat: repo selection, edit task title and description code#5 (consume skillsUpdated) — wire it to both the renderer (refetch skills.list) and to agent sessions (re-emit available_commands_update). Without this, slash menus stay stale until task remount even though the data is right there.
   4. Deferred: feat: workflow CRUD code#6 (mid-session agent skill awareness). Not the reported symptom — track separately.
   5. To confirm, not fix: feat: link up to new agent code#7 (env var) — verify with the user whether their build has SKILLS_ZIP_URL set; if not, that's a deployment fix, not a code fix.

   Critical files

   • packages/agent/src/adapters/claude/claude-agent.ts — deferBackgroundFetches, sendAvailableCommandsUpdate, loadSession
   • apps/code/src/main/services/agent/discover-plugins.ts — buildSyntheticPlugin
   • apps/code/src/main/services/posthog-plugin/service.ts — initialize, updateSkills, skillsUpdated
   • apps/code/src/main/services/agent/service.ts — calls discoverExternalPlugins (line 671-694)
   • apps/code/src/main/trpc/routers/skills.ts — skills.list tRPC procedure
   • apps/code/src/renderer/features/task-detail/components/TaskInput.tsx — mount-time fetch
   • apps/code/src/renderer/features/message-editor/suggestions/getSuggestions.ts — merge logic
   • apps/code/src/renderer/utils/session.ts — extractAvailableCommandsFromEvents

   Verification

   For each fix landed, the right end-to-end check is:

   1. Cold start race (Reports that users will need #1, chore: set up biome and ci code#3): Add a 200ms artificial delay in PosthogPluginService.initialize and 100ms in the network download. Cold-start the app and immediately open a task; type / within 50ms. Slash menu should still show full skill list.
   2. Concurrent symlink (Mapping xpaths to our product #2): Open three windows simultaneously pointing at different repos. Each session should see the full set of user + repo skills, no missing entries in their system-reminder.
   3. Mid-task refresh (/Actions/Configuration #4, feat: repo selection, edit task title and description code#5): Start a task, drop a new SKILL.md into ~/.claude/skills/, type /. New skill should appear without remounting the task.
   4. Mid-session agent awareness (references need removing #6): Same as chore: set up biome and ci code#3, then send /<new-skill> as a prompt. Agent should recognise and invoke the skill.
   5. Unit tests: add tests around discover-plugins.ts for concurrent calls and around claude-agent.ts ensuring available_commands_update is sent before createSession resolves.

   Confirmed scope

   • Symptom: skill missing from slash menu, intermittent, both at session start and mid-task.
   • Resume vs fresh-task: unknown — both paths must be fixed.

   Still worth checking before implementation

   • Is SKILLS_ZIP_URL set in the affected builds? If not, downloaded skills never arrive at all and feat: task creation and task list improvements code#4//Users #5 won't help — that's a deployment issue (made the layout nicer #7).
   • Are the missing skills more often bundled (PostHog-shipped), user (~/.claude/skills), repo (.claude/skills), or marketplace (~/.claude/plugins)? Each comes from a different code path; the answer would tell us whether chore: config housekeeping code#2 (synthetic plugin clobber) or chore: set up biome and ci code#3 (PosthogPluginService init race) is the dominant cause.
3. charlesvien commented on May 2, 2026

   

   charlesvien

   on May 2, 2026

   Member

   More actions

   Reported here as well: https://posthog.slack.com/archives/C09G8Q32R6F/p1777385929691309
4. charlesvien

   changed the title [-]skills availability bugs[/-] [+]Skills not reliably available in chat despite being configured[/+] on May 2, 2026
5. charlesvien

   added

   bugSomething isn't working rightSomething isn't working right

   feature/desktopFeature Tag: DesktopFeature Tag: Desktop

   on Aug 1, 2026
6. charlesvien

   transferred this issue fromPostHog/codeon Aug 1, 2026