From 1e2796fb62032f29336e783905e173bdace09c6a Mon Sep 17 00:00:00 2001 From: Andy Anderson Date: Mon, 16 Nov 2026 20:16:26 -0500 Subject: [PATCH 1/2] =?UTF-8?q?=E2=9C=A8=20mail-in:=20text=20bodies=20as?= =?UTF-8?q?=20notes,=20video,=2035=20MiB=20SIZE,=20streamed=20and=20chunke?= =?UTF-8?q?d=20relay?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hive-mx keeps a message's text body as a Markdown note (a body-only mail becomes message.md; HTML is sanitized with no remote content), accepts .mp4/.mov/.mkv video, and raises SIZE to 35 MiB with 25 MB per file. Mail and its binary attachments are spooled to disk, never held in memory whole, with a concurrency cap and a per-hive daily byte budget. Large deliveries go to the hub as multipart with routing headers; the hub spools them and relays files deliveries to the spoke in signed pieces under the 1 MiB ingress default (/api/inbox/email/chunk). The spoke stages the pieces, verifies each file's size and SHA-256 and stores video, transcribing MP4 and QuickTime audio. Fixes #18564 Signed-off-by: Andy Anderson --- .../added-18564-mailin-body-video-size.md | 1 + src/cmd/hive-mx/main.go | 7 +- src/docs/api-reference/hub-server.md | 2 +- src/docs/api-reference/knowledge.md | 19 +- src/docs/env-vars.md | 19 +- src/docs/inbox-voice-sources.md | 45 +-- src/docs/meetings.md | 9 +- src/pkg/dashboard/api_inbox_email.go | 126 +++++++- .../dashboard/api_inbox_email_chunk_test.go | 201 ++++++++++++ src/pkg/dashboard/api_meetings_email.go | 4 +- src/pkg/dashboard/api_meetings_email_test.go | 2 +- src/pkg/hub/mail_relay.go | 213 ++++++++++-- src/pkg/hub/mail_relay_test.go | 182 +++++++++++ src/pkg/hub/saas_auth.go | 2 + src/pkg/inbox/email_source.go | 134 ++++++-- src/pkg/inbox/email_source_test.go | 170 ++++++++++ src/pkg/inbox/email_stage.go | 186 +++++++++++ src/pkg/mailin/auth.go | 10 +- src/pkg/mailin/auth_test.go | 2 +- src/pkg/mailin/config.go | 61 ++-- src/pkg/mailin/config_test.go | 11 + src/pkg/mailin/files.go | 172 +++++++--- src/pkg/mailin/files_test.go | 37 ++- src/pkg/mailin/hub.go | 90 +++++- src/pkg/mailin/hub_test.go | 80 +++++ src/pkg/mailin/message.go | 306 +++++++++++++++--- src/pkg/mailin/message_test.go | 157 ++++++++- src/pkg/mailin/note.go | 172 ++++++++++ src/pkg/mailin/note_test.go | 62 ++++ src/pkg/mailin/ratelimit.go | 71 ++++ src/pkg/mailin/ratelimit_test.go | 44 +++ src/pkg/mailin/server.go | 123 +++++-- src/pkg/mailin/server_test.go | 113 ++++++- src/pkg/mailin/wire/multipart.go | 169 ++++++++++ src/pkg/mailin/wire/multipart_test.go | 155 +++++++++ src/pkg/mailin/wire/wire.go | 61 +++- 36 files changed, 2977 insertions(+), 241 deletions(-) create mode 100644 changelog.d/added-18564-mailin-body-video-size.md create mode 100644 src/pkg/dashboard/api_inbox_email_chunk_test.go create mode 100644 src/pkg/inbox/email_stage.go create mode 100644 src/pkg/mailin/note.go create mode 100644 src/pkg/mailin/note_test.go create mode 100644 src/pkg/mailin/wire/multipart.go create mode 100644 src/pkg/mailin/wire/multipart_test.go diff --git a/changelog.d/added-18564-mailin-body-video-size.md b/changelog.d/added-18564-mailin-body-video-size.md new file mode 100644 index 0000000000..9281eed732 --- /dev/null +++ b/changelog.d/added-18564-mailin-body-video-size.md @@ -0,0 +1 @@ +- Mail-in to `hive@` now accepts a plain-text or HTML body as a Markdown note (a body-only mail becomes a `message.md` Inbox item), accepts `.mp4`, `.mov` and `.mkv` video (MP4 and QuickTime audio is transcribed), and raises the SMTP SIZE limit to 35 MiB with 25 MB per file; hive-mx spools mail to disk, the hub relays large deliveries to the spoke in signed pieces under the 1 MiB ingress limit, and each hive gets a daily byte budget (#18564). diff --git a/src/cmd/hive-mx/main.go b/src/cmd/hive-mx/main.go index b0f9460545..97f908fc82 100644 --- a/src/cmd/hive-mx/main.go +++ b/src/cmd/hive-mx/main.go @@ -1,7 +1,8 @@ // Command hive-mx is hive's receive-only mail receiver: it accepts mail for -// hive@, keeps only calendar invites, mail-in files and proof -// codes, and posts them to the hub that routes them to each hive. It never -// sends mail. Configuration is environment only; see pkg/mailin and +// hive@, keeps only calendar invites, mail-in files and text +// bodies, and proof codes, and posts them to the hub that routes them to +// each hive. Messages are spooled to disk (HIVE_MX_SPOOL_DIR), never held +// in memory whole. It never sends mail. Configuration is environment only; see pkg/mailin and // src/docs/meetings.md ("Inviting Betty by email"). package main diff --git a/src/docs/api-reference/hub-server.md b/src/docs/api-reference/hub-server.md index 35b5e2603f..a6aae1454a 100644 --- a/src/docs/api-reference/hub-server.md +++ b/src/docs/api-reference/hub-server.md @@ -65,7 +65,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `GET` | `/api/contribute/ws` | Hub handler-specific | Contribute WSProxy | `pkg/hub/server.go:1793` | | `POST` | `/api/github/webhook` | Hub handler-specific | GitHub Webhook | `pkg/hub/server.go:1794` | | `GET` | `/api/mail/hosts` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | Hosts the mail receiver accepts `hive@` for ([#18395](https://github.com/hivecommons/hive/issues/18395)): `{hosts[]}`, sorted, lower-case: every hosted hive's subdomain host, vanity URL host and reported dashboard URL host. A host claimed by two hives is left out. `401` bad token, `404` while `HIVE_HUB_MAIL_TOKEN` is unset. `Cache-Control: no-store` | `pkg/hub/mail_relay.go` | -| `POST` | `/api/mail/inbound` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | One parsed mail delivery from hive-mx (`pkg/mailin/wire.Delivery`, at most 8 MiB). The hub matches the recipient host to a hosted hive, signs the body byte for byte with that hive's relay key (`X-Hive-Relay-Timestamp`, `X-Hive-Relay-Signature-256` bound to `X-Hive-Relay-Delivery`) and forwards it to the spoke's `/api/meetings/invites`, `/api/inbox/email` or `/api/meetings/calendar-emails/proof` by `kind`, behind the webhook relay's private-address guard and without following redirects. Stores nothing; logs the delivery id only. `202` delivered; `400` invalid delivery; `404` no such hive; `413` too large; `422` the spoke refused it (permanent); `502`/`503` hive not reachable yet or unavailable (hive-mx asks the sender to retry) | `pkg/hub/mail_relay.go` | +| `POST` | `/api/mail/inbound` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | One parsed mail delivery from hive-mx (`pkg/mailin/wire.Delivery`): JSON, or `multipart/form-data` for a large files delivery, routed by the `X-Hive-Mail-Delivery`, `X-Hive-Mail-Kind` and `X-Hive-Mail-Recipient` headers (at most 48 MiB, `HIVE_HUB_MAIL_MAX_BODY_BYTES`, this route only; without the headers, JSON of at most 8 MiB). The body is spooled to disk, never held whole in memory. The hub matches the recipient host to a hosted hive, signs the body byte for byte with that hive's relay key (`X-Hive-Relay-Timestamp`, `X-Hive-Relay-Signature-256` bound to `X-Hive-Relay-Delivery`) and forwards it to the spoke's `/api/meetings/invites`, `/api/inbox/email` or `/api/meetings/calendar-emails/proof` by `kind`, behind the webhook relay's private-address guard and without following redirects. A files delivery that is multipart or larger than one piece (`HIVE_HUB_MAIL_CHUNK_BYTES`, 768 KiB) goes to `/api/inbox/email/chunk` in signed pieces instead ([#18564](https://github.com/hivecommons/hive/issues/18564)). Stores nothing once relayed; logs the delivery id only. `202` delivered; `400` invalid delivery; `404` no such hive; `413` too large; `422` the spoke refused it (permanent); `502`/`503` hive not reachable yet or unavailable (hive-mx asks the sender to retry) | `pkg/hub/mail_relay.go` | | `GET` | `/gh-setup` | Hub handler-specific | GitHub App Setup Router | `pkg/hub/server.go:1797` | | `GET` | `/learn` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1798` | | `GET` | `/get-started` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1799` | diff --git a/src/docs/api-reference/knowledge.md b/src/docs/api-reference/knowledge.md index 33257dccbe..dee53ced6e 100644 --- a/src/docs/api-reference/knowledge.md +++ b/src/docs/api-reference/knowledge.md @@ -68,10 +68,11 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `POST` | `/api/knowledge/connectors/{name}/auth/complete` | Owner only | Finish a sign-in from pasted material on a hosted hive: `{redirect_url}` (Plaud: the `localhost:8199/auth/callback?…` URL the browser landed on, forwarded in-pod while the login is pending; Zoom, Google Meet and Webex: the registered redirect URL with `code` and `state`, exchanged for tokens kept in the connector token store) or `{tokens}` (Plaud only: contents of `~/.plaud/tokens-mcp.json`); see [inbox-voice-sources.md](../inbox-voice-sources.md) | `pkg/dashboard/api_inbox.go:32` | | `GET` | `/api/inbox` | Dashboard auth/session | Inbox items (voice recordings, dictations and meeting recordings keyed by `(source, id)`) with per-destination route state, source connection status, candidate repos and auto-routing settings; see [inbox-voice-sources.md](../inbox-voice-sources.md) | `pkg/dashboard/api_inbox.go:24` | | `POST` | `/api/inbox/route` | Owner only | Route one item: `{source, item_id, destination: spek_rfc\|dibs\|dismiss, repo?}` (202 queued; 400 unconfigured destination, 404 unknown item, 409 already queued/running) | `pkg/dashboard/api_inbox.go:25` | -| `POST` | `/api/inbox/email` | Hub relay signature | Mail-in files hive-mx received for `hive@` ([#18395](https://github.com/hivecommons/hive/issues/18395)): Markdown, audio and image attachments become `email` Inbox items under `connector-state/email/items`. A sender who passed DMARC alignment and is a proven calendar email of a user with access gets normal items; anyone else's items carry `awaiting_approval` and cannot be shared, routed or transcribed until approved. `200 {items, awaiting_approval}`; `409` off (`meetings.email.files_enabled`); `507` spool full (256 MiB); `401` bad relay signature | `pkg/dashboard/api_inbox_email.go:21` | -| `POST` | `/api/inbox/email/{id}/approve` | Owner only | Approve a mailed-in item from outside the allowlist. Audited as `inbox-email-approve` | `pkg/dashboard/api_inbox_email.go:22` | -| `POST` | `/api/inbox/email/{id}/decline` | Owner only | Delete a mailed-in item and its file. Unapproved items are also deleted after 14 days. Audited as `inbox-email-decline` | `pkg/dashboard/api_inbox_email.go:23` | -| `GET` | `/api/inbox/email/{id}/file` | Owner or read-write | The stored file with its sniffed type (Markdown as `text/plain`), inline for images, `nosniff` and a sandbox CSP | `pkg/dashboard/api_inbox_email.go:24` | +| `POST` | `/api/inbox/email` | Hub relay signature | Mail-in files hive-mx received for `hive@` ([#18395](https://github.com/hivecommons/hive/issues/18395), [#18564](https://github.com/hivecommons/hive/issues/18564)): Markdown, audio, video and image attachments, and a body-only mail as `message.md`, become `email` Inbox items under `connector-state/email/items`; the mail's text body is kept as each item's note (`files.note`). A sender who passed DMARC alignment and is a proven calendar email of a user with access gets normal items; anyone else's items carry `awaiting_approval` and cannot be shared, routed or transcribed until approved. `200 {items, awaiting_approval}`; `409` off (`meetings.email.files_enabled`); `507` spool full (1 GiB); `401` bad relay signature | `pkg/dashboard/api_inbox_email.go:29` | +| `POST` | `/api/inbox/email/chunk` | Hub relay signature, per piece | One piece of a large mail-in delivery the hub relays in order (`X-Hive-Relay-Delivery: ..`, at most 8 MiB per piece, `X-Hive-Mail-Body-Type` the whole body's type: `multipart/form-data` from hive-mx or JSON). Pieces are staged under `connector-state/email/items/.incoming/` (dropped after an hour if incomplete); the last one decodes the body, checks each file's size and SHA-256, and answers like `/api/inbox/email`. `200 {piece, pieces}` for an earlier piece; `400` invalid piece or body; `413` over `HIVE_INBOX_EMAIL_MAX_BYTES` (48 MiB); `507` spool full | `pkg/dashboard/api_inbox_email.go:30` | +| `POST` | `/api/inbox/email/{id}/approve` | Owner only | Approve a mailed-in item from outside the allowlist. Audited as `inbox-email-approve` | `pkg/dashboard/api_inbox_email.go:31` | +| `POST` | `/api/inbox/email/{id}/decline` | Owner only | Delete a mailed-in item and its file. Unapproved items are also deleted after 14 days. Audited as `inbox-email-decline` | `pkg/dashboard/api_inbox_email.go:32` | +| `GET` | `/api/inbox/email/{id}/file` | Owner or read-write | The stored file with its sniffed type (Markdown as `text/plain`), inline for images (audio and video download), `nosniff` and a sandbox CSP | `pkg/dashboard/api_inbox_email.go:33` | | `POST` | `/api/inbox/dismiss` | Owner only | Bulk "Knowledge only": `{items: [{source, item_id}], undo?}` | `pkg/dashboard/api_inbox.go:26` | | `GET` | `/api/inbox/settings` | Dashboard auth/session | Auto-routing rules, default RFC repo, dibs URL, destination readiness | `pkg/dashboard/api_inbox.go:27` | | `POST` | `/api/inbox/settings` | Owner only | Update `inbox.rules` / `inbox.spek.repo` / `inbox.dibs.url` in the runtime config (omitted fields unchanged) | `pkg/dashboard/api_inbox.go:28` | @@ -117,8 +118,8 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `PUT` | `/api/meetings/persona` | Owner or read-write | Save `meetings.persona` to `hive.yaml`: `{enabled?, preset?, clear_overrides?, name?, display_name?, aliases?, pronouns?, language?, voice?, voice_type?, accent?, intro_template?, leave_phrases?, humor?, unprompted?: {per_10m?, min_gap_s?, per_meeting?, answer_wait_s?, min_confidence?, queue_depth?, stale_s?}, voice_on_address?}`. An absent field is unchanged, an empty one (or 0 for a cap) clears the override, a value equal to the preset's is stored as no override, and `clear_overrides` drops every field override first. Validated as if the persona were on; returns the GET body. 400 invalid. Audited as `meetings_persona`. An optional `email: {calendar_invites?, mail_in_files?}` sets `meetings.email.invites_enabled` / `files_enabled` (owner only, else 403; [#18395](https://github.com/hivecommons/hive/issues/18395)) | `pkg/dashboard/api_meetings_persona.go:165` | | `POST` | `/api/meetings/persona/sample` | Owner or read-write | Speak a short line through the meetings TTS service (Kokoro `POST /v1/audio/speech`) and return the audio (`audio/mpeg`, at most 2 MiB, `Cache-Control: no-store`). Optional `{voice?, language?, display_name?, text?}` (unsaved form values; `text` at most 200 characters, default the persona's sample line in its language). 503 "TTS not configured" until `meetings.tts_url` is set, 400 invalid, 502 the TTS service failed | `pkg/dashboard/api_meetings_persona.go:166` | | `GET` | `/api/meetings/persona/sample` | Owner or read-write | The same sample with `voice`, `language`, `display_name` and `text` as query parameters, for the dashboard's `