Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,14 @@ whose seams had diverged enough that several ports needed a different fix, and t

### Fixed

- **Diagnose a lost multiplexer session on the crash path (#489).** A dead window and a session
destroyed under the run (a reaper, this tool's own prune/stop, an operator `kill-session`, a
server crash) both scored `crashed` and read as an agent fault. Probe `has_session` on a crash
verdict and carry the answer in the failure reason, on every role's `session-end` journal entry
(`session_vanished`), and as a `session-vanished` lifecycle breadcrumb, composed with an
environment-fault pause. Diagnosis only: routing is unchanged and a retry re-creates the
session.

- **A native-Windows install driven from a WSL shell now says so (#332).** WSL appends the Windows
`PATH` to its own, so a bash prompt can reach a Windows-installed `bmad-loop`: that interpreter
reports `win32`, takes the psmux platform default, and never sees the distro's tmux — while
Expand Down
3 changes: 2 additions & 1 deletion docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ See [README.md](../README.md) for the narrative overview and [setup-guide.md](se
- Dispatched sessions are told the sprint board is orchestrator-owned (#437) — the sibling of the park contract above, injected into the prompt the same way. The board advances as soon as dev verifies, but the story's single commit lands only after the review loop, so a session dispatched in between opens on an uncommitted, unattributed change to `sprint-status.yaml` with nothing in the repo naming its author (one read it as a spec violation, reverted it, and tripped the sign-off-regression gate on a story both sessions agreed was finished). Story dev prompts and the review prompts of sprint and sweep runs carry the same prohibition: never write the board, never revert it, and a row at `done` or `awaiting-operator` is the orchestrator's own bookkeeping — not a defect to fix, and not proof that the work is verified, deliberately, since the row is written _before_ the deterministic dev verification runs and a repair session opens on a red tree under a `done` row. Only the **review** prompt adds where to go instead: a story that cannot be finished without a human decision is finalized to `status: blocked` with a reason — the one hand-back that both withholds the commit and reaches a human, where any other non-terminal status just burns the review budget onto a defer that rolls the work back. A dev prompt gets no such invitation, because `blocked` halts the whole run — the exact failure park exists to avoid — and a dev session that cannot finish already has park. A deferred-work bundle's dev prompt carries nothing (a bundle has no board row) while a bundle's _review_ prompt does, since a sweep runs inside a project whose board exists and is just as revertible; every injected plugin-workflow session carries the prohibition too — `post_dev_phase`, `post_review_result` and `pre_commit_gate` all fire inside that same window — as its own `## Sprint board` section appended _after_ the session-gate hooks, so a plugin prompt rewrite cannot strip it, and without the `blocked` redirect for the same reason a dev prompt has none; stories mode carries none of it, having no board at all.
- Typed escalations: `CRITICAL` pauses the run + notifies (desktop + `ATTENTION` file); `PREFERENCE` is journaled and continues.
- Environment faults pause without burning budget (#194): a session whose coding CLI never reached the API — a verify command whose _environment_ is broken (`sh` reports rc `126`/`127`; on Windows a missing tool is caught by its `is not recognized` message or by resolving the command's leading token, and a command naming a file `cmd` cannot execute — a `.sh`, or any extension outside `PATHEXT`, which cmd hands to the file association and which exits `0` without running anything — is a fault rather than a silent rc `0` pass, #302) **or** a session whose log matches the profile's `env_fault_patterns` (an `API Error … Connection refused`-class transport failure, or a provider quota/usage-limit refusal, that idled out the session clock) — pauses the run with the matched evidence instead of charging the attempt and deferring the story as if its code were broken. Re-arm restores the budget. Patterns are per-profile: `claude` seeds the connection signature, `opencode` a provider quota/rate-limit and connection pair (#323), and the other four profiles ship none. Each adapter matches them against the log named by its `ENV_FAULT_LOG_SUFFIX` — the tmux pane capture `logs/<task-id>.log`, or `<task-id>.server.out` (the `opencode serve` process's own stdout) for `opencode-http`, never that adapter's model-written transcript. A pattern is only sound against a log the model cannot write to, which is why the pane-capture profiles anchor an error token plus a cause on the same line and seed no bare quota patterns. Extend or disable them in a project profile overlay.
- A session the multiplexer lost says so (#489). Sessions complete on a hook `Stop` or on window death, and a window is gone whether the CLI exited or something destroyed the whole mux session out from under the run — an external reaper, a concurrent prune or `bmad-loop stop`, an operator `kill-session`, a server crash, the host sleeping. Both are `crashed`, so the retry/defer reason an operator reads said only `dev session crashed` — pointing at the agent when the host was at fault. The crash verdict now asks whether the _session_ still exists and, when it does not, says so in the reason (`… session crashed: the multiplexer no longer reports the session, so the window's disappearance is not evidence the CLI exited`), as `session_vanished` on every role's `session-end` journal entry (and on `dev-decision`, beside the routing it fed), and as a `session-vanished` breadcrumb in `session-lifecycle.jsonl`. The wording states what the evidence _withdraws_, not what it proves: `has_session` maps every nonzero backend result to False, so a negative lookup is "the backend did not confirm it" rather than proof the session is gone — enough to stop an operator reading window death as a CLI exit, not enough to name a destroyer. It composes with an environment-fault pause instead of being swallowed by it. A session reaped _after_ flushing its result still scores `completed` and is not diagnosed — it produced something. Diagnosis only — the routing is unchanged, and a retry re-creates the session.
- CRITICAL resolution: `bmad-loop resolve <run-id>` opens an interactive resolve agent seeded with the escalation + frozen spec; you disambiguate, it re-arms the story (`escalated → pending`, spec reset to `ready-for-dev`) and resumes. `--no-interactive` skips to re-arm if you fixed the spec yourself.
- Intent-gap patch-restore (BMAD-METHOD#2564): when review halts on an `intent gap`, the dev primitive saves the attempted change as a patch file (referenced from the halt output) before reverting the tree. If that reading turns out to be correct, the resolve agent adds `"restore_patch": "<path>"` to its `resolution.json`; the orchestrator re-arms the spec to `in-review` (not `ready-for-dev`) and re-applies the patch after every reset, so the re-driven session resumes _review_ on the restored diff instead of re-implementing. `bmad-loop resolve --no-interactive --restore-patch <path>` does the same by hand. A patch that fails to apply escalates rather than dispatching onto a half-restored tree. Sweep bundles get the same recovery.

Expand Down Expand Up @@ -101,7 +102,7 @@ See [README.md](../README.md) for the narrative overview and [setup-guide.md](se

- Every run is a resumable on-disk state machine: `bmad-loop resume <run-id>` continues from a gate, escalation, or interruption.
- A graceful stop (`stop --graceful` / TUI `S`) is resumable too: unlike a hard stop killed mid-item, it lets the in-flight item finish through commit and finalizes cleanly, ending as a `stopped` run that `resume` picks up at the next item.
- All run state in `.bmad-loop/runs/<run-id>/` (gitignored): `state.json`; `journal.jsonl` (every decision, including the `session-synthesized-from-frontmatter` catch and its `spec-marker-repaired` repair, #276); `events/` (hook signals); `tasks/<id>/` (per-session prompt + `result.json` + breadcrumbs — `session-lifecycle.jsonl` records timeout fires, budget-guard trips (`budget-tripped` / `over-budget-fired`), transport-failure classification (`env-fault-classified`, #194) and the #276 forensics (`spec-status-transition-observed`, `frontmatter-unmodified-refused`, `contract-nudge-sent`); `heartbeat.json` is the wait loop's proof-of-life; `resultless-stops.jsonl` records give-up Stops with a verdict — `no-artifact`, `ambiguous-frontmatter`, `unmodified-since-launch`, `terminal-frontmatter-pending`); `logs/`; `deferred/`; `resolve/`; `ATTENTION`.
- All run state in `.bmad-loop/runs/<run-id>/` (gitignored): `state.json`; `journal.jsonl` (every decision, including the `session-synthesized-from-frontmatter` catch and its `spec-marker-repaired` repair, #276); `events/` (hook signals); `tasks/<id>/` (per-session prompt + `result.json` + breadcrumbs — `session-lifecycle.jsonl` records timeout fires, budget-guard trips (`budget-tripped` / `over-budget-fired`), transport-failure classification (`env-fault-classified`, #194), a mux session lost under the run (`session-vanished`, #489) and the #276 forensics (`spec-status-transition-observed`, `frontmatter-unmodified-refused`, `contract-nudge-sent`); `heartbeat.json` is the wait loop's proof-of-life; `resultless-stops.jsonl` records give-up Stops with a verdict — `no-artifact`, `ambiguous-frontmatter`, `unmodified-since-launch`, `terminal-frontmatter-pending`); `logs/`; `deferred/`; `resolve/`; `ATTENTION`.
- `journal.jsonl` records `session-end` for every session unconditionally — even a teardown that throws still lands one (status `aborted` when the outcome is unknowable). A timed-out session's entry carries `fired_at` (wall time the deadline was declared), `teardown_s` (wall seconds from that fire to this entry — the teardown gap), and `expired_clock` (`monotonic` / `wall` / `both` — `wall` alone fingerprints a host suspend that froze the monotonic clock). Every entry whose usage was read carries `tokens` (raw) and `tokens_weighted` (cache reads at `limits.cache_read_weight`), keeping per-session spend reconstructible; both are `null` when the usage read failed, and both are absent on an `aborted` end. `tokens_weighted` is the end-of-session total — distinct from a tripped session's `budget_weighted`, the guard's mid-session sample at trip time.

### Hook-based transport (no pane-scraping)
Expand Down
3 changes: 2 additions & 1 deletion docs/tui-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,8 @@ One row per story (or sweep bundle/triage task) in the selected run:
(the guard's mid-session sample at trip time). The
matching `tasks/<id>/` dir holds the forensic breadcrumbs the adapter wrote
while the session ran: `session-lifecycle.jsonl` (timeout-fire,
budget-guard `budget-tripped` / `over-budget-fired`, kill-escalation, and the
budget-guard `budget-tripped` / `over-budget-fired`, kill-escalation,
`session-vanished` (the mux no longer reported the session during the run, #489), and the
#276 missing-marker forensics `spec-status-transition-observed` /
`frontmatter-unmodified-refused` / `contract-nudge-sent`),
`heartbeat.json` (the wait loop's proof-of-life —
Expand Down
8 changes: 8 additions & 0 deletions src/bmad_loop/adapters/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,14 @@ class SessionResult:
# and both fire on a CLI that launched and wedged without doing anything. Stop
# is the only canonical event that means a turn actually ended.
stop_seen: bool = False
# Set on a `crashed` verdict when the mux no longer reports the SESSION, not
# just its window (#489) — see `GenericAdapter._session_vanished` for why the
# two are otherwise indistinguishable. Diagnostic label only: it changes the
# reason text, never the routing. Deliberately NOT carried by
# `_post_kill_reconcile`'s hand-built result — that path gates on
# stalled/timeout/over_budget, which this flag can never accompany; add it
# there if `crashed` ever joins that rescue set.
session_vanished: bool = False


class CodingCLIAdapter(ABC):
Expand Down
64 changes: 62 additions & 2 deletions src/bmad_loop/adapters/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,19 @@ def _log_evidence(self, handle: SessionHandle) -> bool | None:
tees a pane log."""
return None

def _session_vanished(self) -> bool:
"""Whether the whole multiplexer session is gone, asked only once a
crash verdict has already been reached (#489). Base: False — an adapter
with no session to lose (opencode-http) never vanishes. Overridden by
`GenericAdapter`.

Same failure convention as `_window_alive`: `MultiplexerError` is the
seam's declared "couldn't ask" and the override swallows it to False.
Anything else propagates, exactly as it does from the liveness probe —
this is a label on a verdict already made, so it degrades rather than
second-guessing the verdict, but it does not swallow unknown faults."""
return False

def _final(
self,
handle: SessionHandle,
Expand Down Expand Up @@ -274,13 +287,37 @@ def _final(
)
result_json = None
status = "completed" if result_json is not None else fallback
# Diagnose the crash verdict only (#489) — see `_session_vanished`. A
# read-back upgrade to `completed` is deliberately not diagnosed: a
# session reaped AFTER flushing its result did produce something, and the
# verdict it earned is the honest one. `crashed` also covers the
# `SessionEnd` arm of `GenericAdapter.run()`, where the CLI announced
# its own exit rather than the window dying — the label stays truthful
# there because it reports what the mux answered, not how the window
# ended.
vanished = status == "crashed" and self._session_vanished()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict the vanished-session probe to window-death verdicts

When the crashed fallback comes from the SessionEnd arm in wait_for_completion, the CLI has explicitly announced its exit; if the mux session is also absent by the time this probe runs, this stamps session_vanished=True and produces a reason claiming that the window disappearance is not evidence the CLI exited. That misdiagnoses exactly the exited-CLI case this change is meant to distinguish. Pass the crash origin into _final, or skip this probe for the SessionEnd path, so only a crash inferred from a failed window-liveness check receives this label.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against the code at 264d84a — declining this one. The conjunction it describes (SessionEnd received AND session absent at probe time) can only arise when a host-level actor destroyed the session, and that is precisely when the suffix must fire:

  • A normal CLI exit can never make _session_vanished() true. Window 0 is a parked shell ("Window 0 is a plain shell so the session survives task windows closing", tmux_base.py), and run()'s teardown kills only the window, never the session (comment in _session_vanished). So reaching this corner requires an independent destroyer — mux server crash, external reaper, operator/concurrent kill-session, host sleep — the exact candidates enumerated in generic.py's probe comment.
  • In every one of those cases, "the multiplexer no longer reports the session" is a true and operator-relevant fact that an announced CLI exit cannot explain. The likeliest route into the corner is causal, not coincidental: the destroyer HUPs the CLI, whose hook flushes SessionEnd on the way down. Skipping the probe on the SessionEnd arm would make exactly that case read as a plain CLI crash — reintroducing the psmux can destroy a live session out from under a run (psmux#546) — decide the exposure and whether the reconcile can tell #489 misdiagnosis this PR fixes.
  • The suffix withdraws an inference; it does not assert the CLI failed to exit. That wording is deliberate on both sides of the seam: session_failure_reason's docstring ("states what the evidence withdraws, not what it proves") and _final's comment, which names the SessionEnd arm explicitly as considered. Routing is unchanged either way, and the SessionEnd receipt remains on disk in the run's events/ directory (SignalWatcher never unlinks consumed event files), so no evidence is lost to the operator.

Threading the crash origin into _final to vary one explanation string in a double-fault corner would trade real plumbing for a marginal wording refinement; if anything, the improvement would be phrasing that acknowledges both facts, not narrowing the probe's scope.

if vanished:
# Evidence rides along like every neighbouring crumb: which session
# went missing (several runs share a host) and what verdict it lands.
# getattr because the mixin does not declare `session_name` (opencode-
# http has none) and only a mux-backed adapter can reach this branch
# (the base `_session_vanished` is a constant False). No default — an
# override on an adapter without a session name must fail loud here,
# not write evidence-free crumbs.
self._note_lifecycle(
handle.task_id,
"session-vanished",
session=getattr(self, "session_name"),
status=status,
)
return SessionResult(
status=status,
result_json=result_json,
session_id=session_id,
transcript_path=transcript,
budget_weighted=budget_weighted,
stop_seen=stop_seen,
session_vanished=vanished,
)

def _result_path(self, task_id: str) -> Path:
Expand Down Expand Up @@ -914,6 +951,29 @@ def _log_activity_key(self, task_id: str) -> tuple[int, int] | None:
def _window_alive(self, handle: SessionHandle) -> bool:
return handle.native_id in self.mux.list_window_ids(self.session_name)

def _session_vanished(self) -> bool:
# The disambiguating probe (#489): `list_window_ids` returns [] for a
# dead window AND for a session that no longer exists, so a plain
# window-death verdict cannot tell an exited CLI from a session destroyed
# under the run. Only `has_session` separates them.
#
# The destroyer is NOT necessarily foreign. Candidates: an external
# reaper (psmux/psmux#546), a concurrent
# `runs.kill_session` from this tool's own prune/stop/crash paths or the
# TUI, an operator `kill-session`, a mux server crash, the host sleeping.
# The reason text stays neutral about which, because this probe cannot
# tell them apart — it reports that the mux no longer answers for the
# session, nothing more.
#
# Safe to ask this late: `run()`'s teardown kills the WINDOW, never the
# session, so our own kill cannot fake a vanishing, and a session once
# gone stays gone.
try:
return not self.mux.has_session(self.session_name)
except MultiplexerError:
# Unknown is not vanished — the same rule the liveness probe follows.
return False

def send_text(self, handle: SessionHandle, text: str) -> None:
self.mux.send_text(handle.native_id, text)

Expand Down Expand Up @@ -1771,8 +1831,8 @@ def _post_kill_reconcile(
# this rescue exists for a session that finished but lost its Stop, not for
# one that never ran. A session that ended no turn and whose pane log never
# grew produced nothing, so a qualifying artifact is not its output — keep
# the stall/timeout verdict. This is the call path the issue's second
# occurrence (story i-11) took.
# the stall/timeout verdict. This is the call path the incident's second
# occurrence took.
if not self._produced_work(handle, result.stop_seen):
self._note_lifecycle(
handle.task_id,
Expand Down
11 changes: 10 additions & 1 deletion src/bmad_loop/adapters/multiplexer.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,16 @@ def target(self, session: str, window: str | None = None) -> str:

@abstractmethod
def has_session(self, name: str) -> bool:
"""True iff a session named exactly ``name`` exists."""
"""True iff a session named exactly ``name`` exists.

Weak False (#489): a False means the backend did not *confirm* the
session, not that it provably no longer exists — implementations map
any failed lookup ("no such session", "no server running", a target
the grammar could not parse) to False alike. A transport failure
(the backend could not be asked at all) raises ``MultiplexerError``
rather than returning False. Callers that surface a False as
evidence must word it as what the negative withdraws, not what it
proves — see ``escalation.session_failure_reason``."""

@abstractmethod
def new_session(
Expand Down
10 changes: 10 additions & 0 deletions src/bmad_loop/adapters/tmux_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,16 @@ def has_session(self, name: str) -> bool:
# error), so this can't use check=True. But a timeout or a missing binary
# is a real backend failure: raise the seam type so callers catch it via
# MultiplexerError instead of a raw subprocess error escaping.
#
# Strength of a False: EVERY nonzero exit maps to it — "no such session",
# "no server running", and a target the grammar could not parse alike. That
# is exactly right for the create-if-missing callers this predicate was
# written for, where a wrong False self-corrects on the next line. It is
# weaker than it looks for a caller that reports the answer as evidence
# (#489), which is why that one words its output as what the negative
# withdraws rather than what it proves. Deliberately NOT tightened here:
# `list_window_ids` raises on transport failure because it backs a liveness
# probe, and this predicate has no such duty to its existing callers.
try:
probe = self._run(["has-session", "-t", f"={name}"], check=False)
except (subprocess.TimeoutExpired, OSError) as exc:
Expand Down
Loading