Skip to content

fix(pixel-teams): stream UTF-8 chunks incrementally without splitting multi-byte sequences - #5698

Open
vaibhavsrv wants to merge 1 commit into
Osmantic:public-betafrom
vaibhavsrv:fix/pixel-teams-stream-multibyte-utf8
Open

vaibhavsrv wants to merge 1 commit into
Osmantic:public-betafrom
vaibhavsrv:fix/pixel-teams-stream-multibyte-utf8

Conversation

@vaibhavsrv

Copy link
Copy Markdown
Contributor

Why this matters

In ods/extensions/services/dashboard-api/routers/pixel_teams.py, the _run(owner, agent) task consumes SSE events from _retained_chat_stream(...) via async for chunk in response.body_iterator:. For incoming byte chunks, it directly executed buffered += chunk.decode("utf-8") if isinstance(chunk, bytes) else chunk.

If a multi-byte UTF-8 character (such as emoji, symbols, or localized characters) is split across network TCP or chunk boundaries by the upstream inference stream, calling chunk.decode("utf-8") on the first chunk containing an incomplete sequence raises an unhandled UnicodeDecodeError ('utf-8' codec can't decode bytes in position ...: unexpected end of data). This abruptly crashes the agent team worker task and marks the worker as failed.

This surgical fix introduces codecs.getincrementaldecoder("utf-8")() to buffer incomplete multi-byte code units across chunk boundaries before appending decoded text to the SSE line buffer.

Validation

  • Baseline reproduction: Simulated a split multi-byte character sequence (b'\xf0\x9f' in chunk 1 followed by b'\x9a\x80' in chunk 2) on unpatched pixel_teams.py; baseline immediately failed with UnicodeDecodeError.
  • Post-fix behavior: Incremental decoding buffers partial code units across chunk boundaries; _run decodes the full character (🚀) without errors and emits the delta choice event.
  • Telemetry: pixel-teams suites: 29 passed. New-test syntax and diff checks pass; new regressions wired into test suite.

Overlap check

Risk / AI disclosure

AI-assisted investigation, implementation and CLI regressions. This uses Python's standard incremental UTF-8 decoder during SSE chunk buffering and does not alter downstream SSE event parsing or contract schemas. Independent human review and platform/runtime qualification remain gates. No running configuration, deployment or upstream merge changed.

Follow-up integration evidence

Composed with #5697 at 8b9f0e2e3 without conflicts. Production and test diffs passed together; team coordinator and streaming checks remain intact.
Backlog composition was local-only (production/test diffs, excluding workflow/Makefile wiring); it is not an upstream merge or independent human approval. Declared live-review gates remain open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant