fix: Slack bridge self-heals channels resolved after startup (#677)#142
Draft
drewdrewthis wants to merge 3 commits into
Draft
fix: Slack bridge self-heals channels resolved after startup (#677)#142drewdrewthis wants to merge 3 commits into
drewdrewthis wants to merge 3 commits into
Conversation
The bridge resolved each agent's Slack channel once at startup and skipped (never retried) any that didn't resolve. A channel created — or the bot invited — AFTER the bridge last restarted was therefore never mirrored until a manual `systemctl restart agents-slack-bridge`. config-sync restarts the bridge on a roster add, but that restart fires before the channel exists (the human creates the channel + invites the bot afterward), so every newly-added agent hit this. Fix: keep unresolved agents in a `pending` set and retry them against a fresh conversations.list snapshot on a self-rescheduling timer (not setInterval — overlapping ticks could double-append a tail), gating promotion on bot MEMBERSHIP (is_member): a public channel is listed before the bot is invited, so resolving on mere presence would mirror a channel the bot can't post to (not_in_channel) — the exact silent-agent symptom. New tails/mapping are appended live (read by reference by the poll loop + inbound router), so a late resolve needs no restart. The picker loop now reads `tails` live so a late-resolved Claude agent gets pickers too. - src/slack/reresolve.ts: pure matchPendingChannels (membership-gated, promote-once) - src/slack/client.ts: listChannels() one-shot snapshot (id/name/is_member) - src/slack/bridge.ts: pending set + self-rescheduling resolver + picker heal - src/slack-reresolve.test.ts: 9 tests incl. the is_member falsifiability case Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ifiability) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(#677) A throw in resolvePendingOnce (e.g. statSync racing a just-deleted transcript) would reject the awaited pass and skip the re-schedule, silently killing the self-heal retry chain. Wrap in .catch().finally(schedule) so a bad pass logs and the chain always re-arms while agents remain pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolveChannelId(resolve on presence) to a singleconversations.listsnapshot gated onis_member. Ifis_memberwere wrong for a currently-healthy agent, that agent would drop topendingand go silent on the next bridge restart. Mitigated: existing agents' bots are members (they work today) →is_member=true→ they resolve at startup exactly as before; the full suite (286/286) stays green; the change only adds a retry path for the case that was previously dropped on the floor.cli/src/slack/reresolve.ts(matchPendingChannels, the membership gate) and the two edited blocks incli/src/slack/bridge.ts: the startup resolve (partition into mirrored vspending) and the self-rescheduling resolver just beforesocket.start().Why
Part of #677 (this is the bridge fix; the closing PR is the companion langwatch-saas deploy change that lands it on the box — see Anything surprising). The Slack bridge resolved each agent's channel once at startup and skipped (never retried) any that didn't resolve.
config-syncrestarts the bridge when the roster changes, but the human creates the agent's Slack channel and invites the bot after that restart — so the channel isn't resolvable when the bridge reads it, the agent is dropped, and it stays silent until a manualsystemctl restart agents-slack-bridge. This recurred for every newly-added agent.What changed
is_member), not channel resolution → alternative: resolve-on-presence (today) → consequence:conversations.listreturns a public channel the moment it exists, before the bot is invited; mirroring it then posts into a channel the bot can't write to (not_in_channel) and receives no events from — a silent half-mirrored agent. Membership makes "mirrored" mean "can actually converse."setTimeoutretry over apendingset → alternative:setInterval→ consequence: overlapping ticks under Slack rate-limiting could both see an agent pending and double-append its tail (every turn posts twice). The chain re-arms only while agents remain pending, so it costs nothing once all are mirrored.conversations.listsnapshot per pass, matched against all pending agents → alternative: a paginated lookup per agent → consequence: N-agents × pages every interval on the same token bucket as posts/pills would 429-starve the healthy agents.tailslive → alternative: the existing once-builttails.filter(...)snapshot → consequence: a late-resolved Claude agent would get text mirroring but no interactive pickers, and zero Claude agents at boot would never schedule the loop at all.How it works
Human verification (for if you really don't trust me)
git checkout fix/677-bridge-channel-reresolve && cd cli && pnpm installpnpm run build→ tsc exits 0.pnpm test→ 286 pass.src/slack/reresolve.tschangeif (hit && hit.isMember)→if (hit); runpnpm exec tsx --test src/slack-reresolve.test.ts→ the "not a member" and "mixed roster" tests FAIL. Revert → 9/9 pass.How I can prove I was successful
[proven] tsc typecheck clean; full suite 286/286; and the
is_membergate is load-bearing — removing it fails exactly the two membership tests (not a member,mixed roster) and restoring it returns 9/9. Terminal evidence:[claimed — not exercisable from the dev environment] the live end-to-end behavior (a running bridge mirroring a real Slack channel created after startup, with no restart) requires the agents-box and the Slack app tokens, which live only in the box's
secrets.env;kanban slack bridgeexits at startup withoutSLACK_BOT_TOKEN/SLACK_APP_TOKEN, so there is no user-observable runtime surface reachable from here. The wiring around the tested core is typechecked and readstails/mappingby reference by construction. The box E2E is the deploy-time gate — runbook attached to #677: add a test agent → apply → create its channel + invite the bot → send a message → get a reply with no manual restart; confirmjournalctl -u agents-slack-bridgeshowsnow mirroring <slug>andExecMainStartTimestampunchanged across create→reply.Anything surprising?
/opt/kanban-codeis built only at provision time on the agents box; nothing updates it post-boot. A companion langwatch-saas change (pinkanban_code_refin tfvars + a config-sync converge — "Track B3" in the plan) is required to land this on the running box without re-provisioning (the box hasprevent_destroy). Until that ships, this reaches the box only on its next provision.