Skip to content

ops/demo.py: subprocess.run(text=True) without encoding fails on cp1252 Windows; real error swallowed into a misleading TypeError #108

Description

@RRDANAI

Summary

ops/demo.py --snapshots fails on a Windows box whose locale encoding is cp1252.
subprocess.run(..., text=True) is called without encoding, so the child's
UTF-8 output is decoded as cp1252. The resulting UnicodeDecodeError is raised
on subprocess's reader thread, where it cannot propagate — stdout comes
back as None, and the run dies much later with a TypeError that points at an
unrelated line.

Reproduction

py ops/demo.py --snapshots out

on Windows with locale.getpreferredencoding() == cp1252. Exit code 1.

Traceback

The real error, on the reader thread:

  File ".../Lib/subprocess.py", line 1615, in _readerthread
    buffer.append(fh.read())
  File ".../Lib/encodings/cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2672:
character maps to <undefined>

The error that actually stops the process:

  File ".../ops/demo.py", line 1619, in _render_isolated
    render_scenario(env, fixture, tmpdir, session_id, cfg, out_dir, theme=theme)
  File ".../ops/demo.py", line 1589, in render_scenario
    dest.write_text('\n\n'+out+'\n\n')
TypeError: can only concatenate str (not "NoneType") to str

Nothing in the second traceback points at an encoding problem, which is what
makes this expensive to diagnose.

Cause

ops/demo.py:743-748:

result = subprocess.run(
    [sys.executable, str(STATUSLINE_SCRIPT)],
    input=payload,
    text=True,
    env=env,
    capture_output=True,

text=True with no encoding uses the locale default. Byte 0x90 is an
undefined slot in cp1252, so the decode raises rather than producing mojibake.

ops/demo.py:1589 has the same latent defect on the write side:

dest.write_text('\n\n'+out+'\n\n')

Path.write_text with no encoding also uses the locale default, so it would
fail to encode the statusline's glyphs on cp1252 even once out is a valid
string.

Suggested fix

Pass encoding='utf-8' at both sites:

result = subprocess.run(..., text=True, encoding='utf-8', ...)
dest.write_text('\n\n'+out+'\n\n', encoding='utf-8')

The statusline already forces UTF-8 on its own stdout (app.py:76-77 calls
sys.stdout.reconfigure(encoding='utf-8')), so the producer side is
unambiguous — only the consumer is guessing.

Workaround

PYTHONUTF8=1 fixes it. With the flag set the same command exits 0 and writes
all 43 snapshot files.

Environment

Key Value
OS Windows 11 Home 10.0.26200
Python 3.13.14
Locale cp1252
Terminal WezTerm
Commit 74f2add

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions