Skip to content

✨ Mail-in: text bodies as notes, video, 35 MiB SIZE, streamed and chunked relay - #18577

Merged
kubestellar-prow[bot] merged 2 commits into
v5from
feat/18564-mailin-body-video-size
Nov 17, 2026
Merged

kubestellar-prow[bot] merged 2 commits into
v5from
feat/18564-mailin-body-video-size

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #18564

What changes

hive-mx (src/pkg/mailin, src/cmd/hive-mx)

  • A message's text body becomes a Markdown note. text/plain is kept as sent; an HTML-only body is reduced to text, headings, lists and link targets (note.go), with scripts, styles, images, frames and other remote content dropped. A body-only mail becomes one message.md attachment (old spokes read it as Markdown); a mail with attachments carries its body in the new files.note.
  • Video: .mp4, .mov, .mkv, sniffed by container (ISO brand, QuickTime atoms, EBML DocType) and matched to the extension.
  • SIZE is 35 MiB (36700160), 25 MB per file, 30 MB per message, as named constants with env overrides (HIVE_MX_MAX_*).
  • DATA is streamed to a temp file in HIVE_MX_SPOOL_DIR, and audio, video and image attachments are streamed from it to their own spool files (ParseReader). DKIM reads from the spool too. HIVE_MX_MAX_CONCURRENT_DATA bounds the work in flight (451 past it), and a spool failure answers 452.
  • Per-hive daily byte budget (HIVE_MX_HIVE_DAILY_BYTES, 512 MiB, 452 past it). Failed deliveries are refunded. The per-IP and per-domain limits are unchanged.
  • The hub client sends JSON when the attachments total at most 512 KiB (every spoke reads that), and multipart/form-data streamed from the spool otherwise. Routing headers (X-Hive-Mail-Delivery, -Kind, -Recipient) go on every delivery.

Hub (src/pkg/hub/mail_relay.go)

  • /api/mail/inbound spools the body to disk, routes on the headers (falling back to the JSON body for an older hive-mx), and raises its own body cap to 48 MiB for routed deliveries on this route only. It also extends the read and write deadlines for this route only.
  • A files delivery that is multipart or larger than 768 KiB goes to the spoke's new /api/inbox/email/chunk in order. Each piece is signed with the relay delivery id <id>.<index>.<count>, which keeps every piece under the 1 MiB request limit nginx applies by default on hosted spoke ingresses (they set no proxy-body-size). The chunk route is on the hosted auth-check public list like the other relay targets.

Spoke (src/pkg/inbox, src/pkg/dashboard)

  • /api/inbox/email/chunk verifies each piece's relay signature and replay. It stages the piece under .incoming/<id> (pruned after an hour, with spool quota and HIVE_INBOX_EMAIL_MAX_BYTES caps). On the last piece it decodes the multipart body, streaming files to disk and checking size and SHA-256, then stores it like /api/inbox/email.
  • The email source moves staged files into items, keeps the note per item (wrapped as untrusted data in the item text), and stores video. MP4 and QuickTime video is pending transcription through the existing audio path (as audio/mp4, the same as dropped videos). MKV is stored only, and the item says so. The spool quota goes to 1 GiB.

Docs: meetings.md, inbox-voice-sources.md, env-vars.md, and the API reference (hub and knowledge).

Deploy manifests (memory/ephemeral-storage limits, spool emptyDir) are in hivecommons/quorum deploy/hive-oke/mail/.

🤖 Generated with Claude Code

…nked relay

hive-mx keeps a message's text body as a Markdown note (a body-only mail
becomes message.md; HTML is sanitized with no remote content), accepts
.mp4/.mov/.mkv video, and raises SIZE to 35 MiB with 25 MB per file. Mail
and its binary attachments are spooled to disk, never held in memory
whole, with a concurrency cap and a per-hive daily byte budget.

Large deliveries go to the hub as multipart with routing headers; the hub
spools them and relays files deliveries to the spoke in signed pieces
under the 1 MiB ingress default (/api/inbox/email/chunk). The spoke stages
the pieces, verifies each file's size and SHA-256 and stores video,
transcribing MP4 and QuickTime audio.

Fixes #18564

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18593 ✨ Betty answers questions about the hive from its live state same files src/docs/meetings.md
#18584 ✨ Show why a calendar-email proof failed on its Settings row CONFLICT dashboard/openapi.json
src/docs/api-reference/knowledge.md (conflict)
src/docs/meetings.md
src/pkg/dashboard/api_meetings_email.go
src/pkg/dashboard/api_meetings_email_test.go
#18592 🐛 Betty closes the loop on deep dives; research agents survive config reloads CONFLICT src/docs/api-reference/knowledge.md (conflict)
src/docs/meetings.md
#18583 ✨ Governor: opt-in advisory report in every repo (default off) same files dashboard/openapi.json
src/docs/env-vars.md
#18595 🐛 Meetings panel says plainly when notes and clarifying questions are off same files src/docs/meetings.md
#18536 ✨ feature: choose remote repository write identity same files dashboard/openapi.json
#18576 ✨ Owner-only Send to on model-owned stage-tracker rows same files dashboard/openapi.json

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

⏱️ The 900s mutation budget ran out, so 3 package(s) were not fully mutated and the results below do not cover them:

  • pkg/hub: cut short (partial, no report)
  • pkg/inbox: skipped (not started)
  • pkg/mailin: skipped (not started)

1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/pkg/dashboard/api_inbox_email.go 222 CONDITIONALS_NEGATION

Verdict: uncovered (#17349).

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 17, 2026
@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Mail-in: accept plain-text and HTML bodies as notes, accept video, and raise the 2 MB size limit

1 participant