Repository navigation
✨ Mail-in: text bodies as notes, video, 35 MiB SIZE, streamed and chunked relay - #18577
Conversation
…nked relay hive-mx keeps a message's text body as a Markdown note (a body-only mail becomes message.md; HTML is sanitized with no remote content), accepts .mp4/.mov/.mkv video, and raises SIZE to 35 MiB with 25 MB per file. Mail and its binary attachments are spooled to disk, never held in memory whole, with a concurrency cap and a per-hive daily byte budget. Large deliveries go to the hub as multipart with routing headers; the hub spools them and relays files deliveries to the spoke in signed pieces under the 1 MiB ingress default (/api/inbox/email/chunk). The spoke stages the pieces, verifies each file's size and SHA-256 and stores video, transcribing MP4 and QuickTime audio. Fixes #18564 Signed-off-by: Andy Anderson <andy@clubanderson.com>
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
Signed-off-by: Andy Anderson <andy@clubanderson.com>
Mutation testing⏱️ The 900s mutation budget ran out, so 3 package(s) were not fully mutated and the results below do not cover them:
1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the
Verdict: |
|
[APPROVALNOTIFIER] This PR is APPROVED Approval requirements bypassed by manually added approval. This pull-request has been approved by: The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Fixes #18564
What changes
hive-mx (
src/pkg/mailin,src/cmd/hive-mx)text/plainis kept as sent; an HTML-only body is reduced to text, headings, lists and link targets (note.go), with scripts, styles, images, frames and other remote content dropped. A body-only mail becomes onemessage.mdattachment (old spokes read it as Markdown); a mail with attachments carries its body in the newfiles.note..mp4,.mov,.mkv, sniffed by container (ISO brand, QuickTime atoms, EBML DocType) and matched to the extension.36700160), 25 MB per file, 30 MB per message, as named constants with env overrides (HIVE_MX_MAX_*).HIVE_MX_SPOOL_DIR, and audio, video and image attachments are streamed from it to their own spool files (ParseReader). DKIM reads from the spool too.HIVE_MX_MAX_CONCURRENT_DATAbounds the work in flight (451 past it), and a spool failure answers 452.HIVE_MX_HIVE_DAILY_BYTES, 512 MiB, 452 past it). Failed deliveries are refunded. The per-IP and per-domain limits are unchanged.multipart/form-datastreamed from the spool otherwise. Routing headers (X-Hive-Mail-Delivery,-Kind,-Recipient) go on every delivery.Hub (
src/pkg/hub/mail_relay.go)/api/mail/inboundspools the body to disk, routes on the headers (falling back to the JSON body for an older hive-mx), and raises its own body cap to 48 MiB for routed deliveries on this route only. It also extends the read and write deadlines for this route only./api/inbox/email/chunkin order. Each piece is signed with the relay delivery id<id>.<index>.<count>, which keeps every piece under the 1 MiB request limit nginx applies by default on hosted spoke ingresses (they set noproxy-body-size). The chunk route is on the hosted auth-check public list like the other relay targets.Spoke (
src/pkg/inbox,src/pkg/dashboard)/api/inbox/email/chunkverifies each piece's relay signature and replay. It stages the piece under.incoming/<id>(pruned after an hour, with spool quota andHIVE_INBOX_EMAIL_MAX_BYTEScaps). On the last piece it decodes the multipart body, streaming files to disk and checking size and SHA-256, then stores it like/api/inbox/email.audio/mp4, the same as dropped videos). MKV is stored only, and the item says so. The spool quota goes to 1 GiB.Docs:
meetings.md,inbox-voice-sources.md,env-vars.md, and the API reference (hub and knowledge).Deploy manifests (memory/ephemeral-storage limits, spool emptyDir) are in hivecommons/quorum
deploy/hive-oke/mail/.🤖 Generated with Claude Code