Skip to content

fix(gateway): sniff image media type when channel reports octet-stream - #1643

Open
jayfeng7 wants to merge 17 commits into
TencentCloud:developfrom
jayfeng7:fix/attachment-mime-sniff-fallback
Open

jayfeng7 wants to merge 17 commits into
TencentCloud:developfrom
jayfeng7:fix/attachment-mime-sniff-fallback

Conversation

@jayfeng7

@jayfeng7 jayfeng7 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1640

Why

Image attachments reported as application/octet-stream or with missing MIME can reach the model with an unsupported or incorrect data URL type. Stored workspace image references repeat this failure when they are rematerialized on later requests.

What changed

  • Use the existing magic-byte sniffer for inline and downloaded images with generic or missing MIME, before applying the previous fallback.
  • Retain missing MIME on workspace-only references until bytes are available. Correct generic MIME in the shared request-time finalizer, covering both async/sync expansion and existing stored workspace references without rewriting checkpoints.
  • Preserve explicitly declared types. If bytes cannot be recognized, retain the prior generic type or extension/PNG fallback.
  • Add coverage for inline, URL, path-backed and historical references, unknown bytes and explicit-type controls; add an Unreleased changelog entry.
  • Synchronize this existing branch with develop (cc52ef2e).

Surface area

  • Backend API — attachment materialization and model-request image blocks
  • Frontend UI
  • Agents / LangGraph
  • Sandbox
  • Dependencies
  • Default behavior change — sniff missing/generic MIME when image bytes are available
  • Docs / tests / CI only

Bug fix verification

  • Before the follow-up, the new regression matrix had 10 failures, 14 passes: missing MIME, workspace-only expansion and historical generic references exposed the gaps.
  • After the fix, the full attachment file has 41 passing tests.
  • Attachment, chat-image pipeline and workspace-image middleware tests together: 46 passed. The tests compare exact outbound data URLs and verify that historical checkpoint blocks remain unchanged.

Earlier validation (before the 2026-10-09 sync)

  • Python 3.12.14 with locked dependencies; full Ruff lint/format checks (1,157 files) and mypy (562 source files) passed.
  • Actual pre-commit hook and dashboard production build passed. The hook's testmon scope was the three image-related test files; the earlier gate ran all 46 cases, and the commit hook reused its cache. This is not a passing full suite.
  • Broader gateway tests: 476 passed, 2 skipped, 1 failed at an unchanged Windows browser-profile path assertion. The same case also fails with the unchanged browser-profile module from develop; it is outside this image patch.
  • git diff --check passed. Upstream CI still needs maintainer authorization and execution.

This fixes rematerializable workspace references. It does not migrate arbitrary legacy inline data URLs or redesign channel-wide MIME normalization.

AI assistance

Tool(s) used: AI coding agent

How you used it: read the issue's request-time reproduction, traced inline/download/workspace paths, repaired MIME fallback ordering, added red/green and preservation tests, and ran the checks above.

  • I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

Latest synchronization and validation (2026-11-18)

Head 348b6c032c20399ca17c9a1b17d326084235359a merges develop 3d85f07e76122c7323448527809422f42a9f3666. Relative to that baseline, the PR contains only its implementation, regression tests and Unreleased fix entry.

  • Full Linux Python run on this candidate: 5,953 passed, 70 skipped.
  • Windows attachment regressions: 45 passed.
  • Full backend Ruff lint/format checks and mypy (692 source files) passed.
  • The repository's native pre-commit hook passed on Linux, including its testmon check and dashboard production build. The signed commit used a hook runner that verified the Linux tree matched the staged Windows tree before and after the hook.
  • The frontend build uses NODE_OPTIONS=--max-old-space-size=4096, matching the repository build configuration.
  • GitHub verifies the commit signature. Current-head CI results are available in Checks.

Channels may report image attachments with content_type
application/octet-stream (QQ C2C does this, sometimes with a .png
filename over JPEG bytes). materialize_image_part trusted that value,
so the data URL sent to the model carried an unsupported media type
and the provider returned 400.

Apply the existing sniff_image_media_type() magic-byte detection as a
fallback when the reported mime is application/octet-stream or empty,
at all three materialization sites: inline data, downloaded URL
content, and path-backed workspace refs.

Fixes TencentCloud#1640
@jayfeng7
jayfeng7 force-pushed the fix/attachment-mime-sniff-fallback branch from dda63ce to f6130e9 Compare October 5, 2026 10:27
…e-sniff-fallback

# Conflicts:
#	CHANGELOG.md
@jayfeng7

jayfeng7 commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Friendly follow-up — the MIME fallback fix covers inline/downloaded images and request-time expansion of historical workspace references, while preserving explicit image types. The 46 focused image-pipeline tests passed locally; broader-suite limitations are disclosed in the PR. The current fork CI run is awaiting maintainer approval, with no merge conflict reported. Could a maintainer approve CI and review the patch when convenient? Happy to address feedback.

Jay Feng and others added 7 commits October 9, 2026 16:54
Resolve CHANGELOG.md conflict: adopt the 1.0.2b7 release structure and
re-add the image MIME sniffing entry under [Unreleased] until this PR lands.
…e-sniff-fallback

# Conflicts:
#	CHANGELOG.md
…e-sniff-fallback

# Conflicts:
#	CHANGELOG.md
…e-sniff-fallback

# Conflicts:
#	CHANGELOG.md
…e-sniff-fallback

# Conflicts:
#	CHANGELOG.md
@jayfeng7

Copy link
Copy Markdown
Contributor Author

This is ready to squash-merge; there is no reviewer assigned, so flagging it in case it fell off the queue.

  • Conflicts with develop are resolved as of 373eca43c2 (the PR had been dirty). Relative to current develop the diff is the same three files plus one CHANGELOG.md line, placed under ### 修复 next to the OCTOP_HOME entry.
  • CI on this head: Python 3.12 and Windows / Python 3.12 both green; Live tests (real credentials) skipped, which is expected without credentials on a fork head.
  • Local: tests/unit/gateway/test_attachment_hints.py — 41 passed.

Scope is unchanged from the original description: application/octet-stream (or empty) image MIME is sniffed from magic bytes for inline attachments, downloaded URLs, and request-time expansion of workspace refs, while an explicit image/* type is still trusted. Historical checkpoints are expanded at request time, so no rewrite is needed.

Happy to adjust anything, or to split it if you would rather take the sniffing helper without the request-time expansion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant