fix(brainstorming): pin brain.py console streams to UTF-8

[aranellaeth]

What

Pins brain.py's sys.stdout / sys.stderr to UTF-8 — preserving each stream's own errors= handler — plus four regression tests.

Rescoped 2026-08-07. This PR opened on 2026-07-12 covering four files. Three are now resolved elsewhere:

file	status
resolve_party.py	landed as #2687
resolve_personas.py	landed as #2688
resolve_config.py	deferring to @armelhbobdad's #2693 — it is further along (tests + two review rounds addressed) and there is no reason to have two PRs on one file
brain.py	still unfixed on main; this PR is now only this

Rebased onto current main and narrowed, so this and #2693 no longer overlap and neither blocks the other.

Why

brain.py prints two kinds of arbitrary user text:

• stdout — --extra overlay techniques (customize.toml's additional_techniques), a documented first-class feature; their category names and descriptions go straight out through fmt_list / fmt_show.
• stderr — the technique name echoed back by show NAME when nothing matches: print(f"# not found: {m}", file=sys.stderr), where m is argv.

Either can carry a character the platform default cannot encode, and print() then raises UnicodeEncodeError.

Scoping this honestly: it is not a shipped-catalog crash. brain-methods.csv's only non-ASCII is U+2014 (34 occurrences), which cp1252 encodes fine, and the --json path is ensure_ascii=True. The exposure is the overlay and argv paths — brain.py show 日本語, or an overlay technique named Fikir Fırtınası 🌪.

The errors= detail

reconfigure(encoding=...) on its own resets the error handler to strict:

>>> s = io.TextIOWrapper(io.BytesIO(), encoding="ascii", errors="backslashreplace")
>>> s.reconfigure(encoding="utf-8")
>>> s.encoding, s.errors
('utf-8', 'strict')             # backslashreplace lost

>>> t = io.TextIOWrapper(io.BytesIO(), encoding="ascii", errors="backslashreplace")
>>> t.reconfigure(encoding="utf-8", errors=t.errors)
>>> t.encoding, t.errors
('utf-8', 'backslashreplace')   # preserved

stderr defaults to backslashreplace on POSIX, so the naive form would turn a clean diagnostic about a surrogate-escaped path into a traceback — a regression on the exact platform this patch is not even about. pin_utf8() passes errors= through.

Credit to @armelhbobdad for spotting this on #2693: the version of this patch that was up before today had precisely that bug.

Testing

uv run --python 3.11 --with pytest pytest src/core-skills/bmad-brainstorming/scripts/tests/test_brain.py -q
33 passed

29 existing + 4 new. All four new tests are red against the unpatched script — verified by checking out main's brain.py and re-running (4 failed, 29 passed), so they are real tripwires and not assertions that were already true.

One note for maintainers: test_brain.py is wired into no npm script — CI's Python coverage is test:renderer, test:retrospective and test:sprint-planning. Unlike test_context.py, this file is healthy (29/29 green on a clean main), so adding it to test:renderer would be a cheap win with no pre-existing failures to import. Happy to do that here if you want it.

Follow-up worth its own PR

#2687 and #2688 took the encoding="utf-8" half of the consumer fix but not the errors="replace" and out.stdout or "" half this PR originally carried. On a strict-decode failure the exception is raised on subprocess's reader thread, so subprocess.run returns returncode=0 with stdout=None — which walks straight past if out.returncode != 0 and dies in .strip(), uncaught by the surrounding except json.JSONDecodeError. Reproduced on Windows 11 / Python 3.13 against a child emitting invalid UTF-8:

#2687 merged form  (encoding="utf-8", strict):   returncode=0  stdout=None
    out.stdout.strip() -> AttributeError: 'NoneType' object has no attribute 'strip'

#2578 original form (encoding="utf-8", errors="replace"):
                                                 returncode=0  stdout='{"agents": ...}'

Same class as the finding @armelhbobdad fixed in _installed_resolver_config on #2693, one exception type off because these two call .strip() before json.loads. Say the word and I'll open it.

🤖 Generated with Claude Code

[coderabbitai]

📝 Walkthrough

Walkthrough

The changes make CLI and resolver output handling explicitly UTF-8-aware. Console streams are reconfigured when supported, JSON output preserves non-ASCII characters, and subprocess stdout decoding uses UTF-8 with replacement errors.

Changes

UTF-8 Output Handling

Layer / File(s)	Summary
Console output encoding src/core-skills/bmad-brainstorming/scripts/brain.py, src/scripts/resolve_config.py	CLI streams are configured for UTF-8 when supported, and merged JSON is emitted with non-ASCII characters preserved.
Resolver subprocess decoding src/core-skills/bmad-forge-idea/scripts/resolve_personas.py, src/core-skills/bmad-party-mode/scripts/resolve_party.py	Captured subprocess output is decoded as UTF-8 with replacement handling before existing validation and JSON parsing.og

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• bmad-code-org/BMAD-METHOD#2414: Adds a similar UTF-8 stdout helper for JSON output.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description clearly explains the UTF-8 stream fix in brain.py, its scope, rationale, and test results.
Title check	✅ Passed	The title clearly and concisely identifies the brain.py console stream UTF-8 fix, which matches the rescoped changeset.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

src/core-skills/bmad-forge-idea/scripts/resolve_personas.py (1)

49-65: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting the duplicated _run_json helper.

_run_json is byte-for-byte identical in resolve_personas.py (lines 48-67) and resolve_party.py (lines 46-65). If these scripts share a common package or utility module, extracting this function would prevent the implementations from diverging over time. If they're intentionally standalone per-skill scripts, the duplication is acceptable.

The UTF-8 encoding change itself is correct — encoding="utf-8", errors="replace" with text=True properly decodes child process output, and the 60s timeout and failure checks are preserved.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/core-skills/bmad-forge-idea/scripts/resolve_personas.py` around lines 49
- 65, Extract the identical _run_json helper shared by resolve_personas.py and
resolve_party.py into their common utility or package module, then update both
scripts to import and reuse it. Preserve the existing UTF-8 decoding, timeout,
failure handling, and JSON parsing behavior; if no shared module is appropriate,
leave the standalone implementations unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/core-skills/bmad-forge-idea/scripts/resolve_personas.py`:
- Around line 49-65: Extract the identical _run_json helper shared by
resolve_personas.py and resolve_party.py into their common utility or package
module, then update both scripts to import and reuse it. Preserve the existing
UTF-8 decoding, timeout, failure handling, and JSON parsing behavior; if no
shared module is appropriate, leave the standalone implementations unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a7664864-bac5-4a60-949b-9915a6acacfa

📥 Commits

Reviewing files that changed from the base of the PR and between 49069b8 and 5f9ef65.

📒 Files selected for processing (4)

• src/core-skills/bmad-brainstorming/scripts/brain.py
• src/core-skills/bmad-forge-idea/scripts/resolve_personas.py
• src/core-skills/bmad-party-mode/scripts/resolve_party.py
• src/scripts/resolve_config.py

[aranellaeth]

Thanks!

_run_json is intentionally duplicated—these are standalone, stdlib-only per-skill scripts with no shared package to import from, and the duplication predates this PR (I only applied the encoding fix to the existing copies). Extracting a shared module would be a larger packaging change, out of scope for this targeted fix.

Happy to follow up separately if a shared script utility is ever introduced.

[sanmaxdev]

Checked the UTF-8 stream changes against the existing resolver pattern. The focused suite passes with 59 tests, and a cp1252 reproduction now exits cleanly with the emoji preserved. The base branch fails with UnicodeEncodeError. The diff is focused and looks correct.

[armelhbobdad]

This PR still has the only fix for the producer-side crash, and it has been waiting on review since 2026-07-12 — flagging it because its hunks are being merged piecemeal in a way that risks losing the important one.

Of the four files here, the three consumer hunks have now landed separately: resolve_party.py via #2687 and resolve_personas.py via #2688. What has not landed is src/scripts/resolve_config.py — the producer, and the one that actually fixes #2682. Adding encoding="utf-8" to the callers only governs how the parent decodes the child's bytes; the child still raises UnicodeEncodeError on a cp1252 stdout before it emits anything, so _run_json keeps seeing a non-zero exit. #2682 was auto-closed by #2687 even though the crash it reports is still reproducible on main (verified on Windows 11 / Python 3.11 — exit 1, same U+1F4CA).

Since this PR has been idle a while, I opened #2693 with just the resolve_config.py guard plus the stdout regression test the suite was missing. It is deliberately scoped to that one piece so it can land independently — happy for it to be closed in favour of merging this PR whole, which is the better outcome; the credit for the resolve_config.py fix is yours either way, @aranellaeth.

One review note while this is open: reconfigure(encoding="utf-8") without errors= also resets the stream's error handler to strict. That is harmless for stdout, but this PR also reconfigures stderr in brain.py, where it silently downgrades POSIX's default backslashreplace — a surrogateescaped path interpolated into an error message would turn a clean diagnostic into a traceback. errors=stream.errors preserves it.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

The PR prevents Windows console encoding failures in the brainstorming CLI while preserving each stream’s existing error-handling behavior.

• Adds a helper that reconfigures stdout and stderr to UTF-8 when supported.
• Calls the helper when the CLI starts.
• Adds coverage for cp1252 stdout and stderr, preserved error handlers, and streams without reconfigure.

Confidence Score: 5/5

The PR appears safe to merge, with focused tests covering the changed console-encoding behavior.

The normal CLI path uses standard Python process streams that support the guarded UTF-8 reconfiguration, and the new tests exercise both affected streams and the unsupported-stream fallback.

Important Files Changed

Filename	Overview
src/core-skills/bmad-brainstorming/scripts/brain.py	Adds guarded UTF-8 configuration for both console streams while preserving their current error handlers.
src/core-skills/bmad-brainstorming/scripts/tests/test_brain.py	Verifies non-cp1252 output, stderr diagnostics, error-handler preservation, and compatibility with streams lacking reconfiguration support.

_{Reviews (1): Last reviewed commit: "fix(brainstorming): pin brain.py console..." | Re-trigger Greptile}

[aranellaeth]

Rescoped and rebased onto current main.

@armelhbobdad — thank you for flagging this. You were right that the piecemeal merges were leaving the important hunk behind, and right about the errors= downgrade.

• resolve_party.py → merged as fix(party-mode): decode subprocess JSON as UTF-8 on Windows #2687
• resolve_personas.py → merged as fix(forge-idea): decode persona resolver JSON as UTF-8 #2688
• resolve_config.py → deferring to fix: pin UTF-8 stdout in resolve_config so Windows cp1252 doesn't crash #2693. It is further along (tests + two review rounds addressed) and there is no reason to have two PRs open on one file.
• brain.py → still unfixed on main; that is now all this PR is.

Force-pushed 5f9ef65 → 3021e86: two files, +69 lines, four new tests — all four red against the unpatched script. The stderr backslashreplace downgrade you called out is fixed here too. Details and the repro in the updated description.