diff --git a/changelog.d/added-18564-mailin-body-video-size.md b/changelog.d/added-18564-mailin-body-video-size.md new file mode 100644 index 0000000000..9281eed732 --- /dev/null +++ b/changelog.d/added-18564-mailin-body-video-size.md @@ -0,0 +1 @@ +- Mail-in to `hive@` now accepts a plain-text or HTML body as a Markdown note (a body-only mail becomes a `message.md` Inbox item), accepts `.mp4`, `.mov` and `.mkv` video (MP4 and QuickTime audio is transcribed), and raises the SMTP SIZE limit to 35 MiB with 25 MB per file; hive-mx spools mail to disk, the hub relays large deliveries to the spoke in signed pieces under the 1 MiB ingress limit, and each hive gets a daily byte budget (#18564). diff --git a/dashboard/openapi.json b/dashboard/openapi.json index 5ba6f9db19..daee0eb1cd 100644 --- a/dashboard/openapi.json +++ b/dashboard/openapi.json @@ -39671,14 +39671,14 @@ "Inbox" ], "summary": "Receive mail-in files (hub relay)", - "description": "Markdown, audio and image attachments hive-mx received for hive@. A sender whose From address passed DMARC alignment and is a proven calendar email of a user with access becomes normal Inbox items; anyone else's items await the owner's approval. Relay signature: X-Hive-Relay-Timestamp, X-Hive-Relay-Signature-256 (HMAC-SHA256 with this hive's relay key over timestamp.delivery.body) and X-Hive-Relay-Delivery; no session.", + "description": "Markdown, audio, video and image attachments and text bodies hive-mx received for hive@ (#18395, #18564); a body-only mail arrives as message.md and a mail's body is kept as each item's note. A sender whose From address passed DMARC alignment and is a proven calendar email of a user with access becomes normal Inbox items; anyone else's items await the owner's approval. Relay signature: X-Hive-Relay-Timestamp, X-Hive-Relay-Signature-256 (HMAC-SHA256 with this hive's relay key over timestamp.delivery.body) and X-Hive-Relay-Delivery; no session.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "type": "object", - "description": "hive-mx delivery (pkg/mailin/wire.Delivery) of kind files: id, kind, recipient, message_id, received_at, auth, files {title, attachments[] {name, kind, content_type, size, sha256, data}}" + "description": "hive-mx delivery (pkg/mailin/wire.Delivery) of kind files: id, kind, recipient, message_id, received_at, auth, files {title, note, attachments[] {name, kind, content_type, size, sha256, data}}" } } } @@ -39726,6 +39726,74 @@ } } }, + "/api/inbox/email/chunk": { + "post": { + "tags": [ + "Inbox" + ], + "summary": "Receive one piece of a large mail-in delivery (hub relay)", + "description": "The hub relays a files delivery larger than one request in order, each piece signed with this hive's relay key over the relay delivery id .. (X-Hive-Relay-Delivery), so the order and count are covered and each piece passes the replay check. X-Hive-Mail-Body-Type is the whole body's type (multipart/form-data from hive-mx, or application/json). Pieces are staged on disk; the last one decodes the body, checks each file's size and SHA-256 and stores it like POST /api/inbox/email. No session.", + "requestBody": { + "required": true, + "content": { + "application/octet-stream": { + "schema": { + "type": "string", + "format": "binary", + "description": "One piece of the delivery body, at most 8 MiB" + } + } + } + }, + "responses": { + "200": { + "description": "Piece staged ({piece, pieces}), or, for the last piece, stored ({items, awaiting_approval})", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "piece": { + "type": "integer" + }, + "pieces": { + "type": "integer" + }, + "items": { + "type": "array", + "items": { + "type": "string" + } + }, + "awaiting_approval": { + "type": "boolean" + } + } + } + } + } + }, + "400": { + "description": "Invalid piece, missing piece or undecodable body" + }, + "401": { + "description": "Relay signature rejected or piece replayed" + }, + "409": { + "description": "Mail-in files are off or unavailable" + }, + "413": { + "description": "Piece or delivery too large" + }, + "503": { + "description": "This hive holds no relay key" + }, + "507": { + "description": "The email spool is full" + } + } + } + }, "/api/inbox/email/{id}/approve": { "post": { "tags": [ diff --git a/src/cmd/hive-mx/main.go b/src/cmd/hive-mx/main.go index b0f9460545..97f908fc82 100644 --- a/src/cmd/hive-mx/main.go +++ b/src/cmd/hive-mx/main.go @@ -1,7 +1,8 @@ // Command hive-mx is hive's receive-only mail receiver: it accepts mail for -// hive@, keeps only calendar invites, mail-in files and proof -// codes, and posts them to the hub that routes them to each hive. It never -// sends mail. Configuration is environment only; see pkg/mailin and +// hive@, keeps only calendar invites, mail-in files and text +// bodies, and proof codes, and posts them to the hub that routes them to +// each hive. Messages are spooled to disk (HIVE_MX_SPOOL_DIR), never held +// in memory whole. It never sends mail. Configuration is environment only; see pkg/mailin and // src/docs/meetings.md ("Inviting Betty by email"). package main diff --git a/src/docs/api-reference/hub-server.md b/src/docs/api-reference/hub-server.md index 35b5e2603f..a6aae1454a 100644 --- a/src/docs/api-reference/hub-server.md +++ b/src/docs/api-reference/hub-server.md @@ -65,7 +65,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `GET` | `/api/contribute/ws` | Hub handler-specific | Contribute WSProxy | `pkg/hub/server.go:1793` | | `POST` | `/api/github/webhook` | Hub handler-specific | GitHub Webhook | `pkg/hub/server.go:1794` | | `GET` | `/api/mail/hosts` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | Hosts the mail receiver accepts `hive@` for ([#18395](https://github.com/hivecommons/hive/issues/18395)): `{hosts[]}`, sorted, lower-case: every hosted hive's subdomain host, vanity URL host and reported dashboard URL host. A host claimed by two hives is left out. `401` bad token, `404` while `HIVE_HUB_MAIL_TOKEN` is unset. `Cache-Control: no-store` | `pkg/hub/mail_relay.go` | -| `POST` | `/api/mail/inbound` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | One parsed mail delivery from hive-mx (`pkg/mailin/wire.Delivery`, at most 8 MiB). The hub matches the recipient host to a hosted hive, signs the body byte for byte with that hive's relay key (`X-Hive-Relay-Timestamp`, `X-Hive-Relay-Signature-256` bound to `X-Hive-Relay-Delivery`) and forwards it to the spoke's `/api/meetings/invites`, `/api/inbox/email` or `/api/meetings/calendar-emails/proof` by `kind`, behind the webhook relay's private-address guard and without following redirects. Stores nothing; logs the delivery id only. `202` delivered; `400` invalid delivery; `404` no such hive; `413` too large; `422` the spoke refused it (permanent); `502`/`503` hive not reachable yet or unavailable (hive-mx asks the sender to retry) | `pkg/hub/mail_relay.go` | +| `POST` | `/api/mail/inbound` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | One parsed mail delivery from hive-mx (`pkg/mailin/wire.Delivery`): JSON, or `multipart/form-data` for a large files delivery, routed by the `X-Hive-Mail-Delivery`, `X-Hive-Mail-Kind` and `X-Hive-Mail-Recipient` headers (at most 48 MiB, `HIVE_HUB_MAIL_MAX_BODY_BYTES`, this route only; without the headers, JSON of at most 8 MiB). The body is spooled to disk, never held whole in memory. The hub matches the recipient host to a hosted hive, signs the body byte for byte with that hive's relay key (`X-Hive-Relay-Timestamp`, `X-Hive-Relay-Signature-256` bound to `X-Hive-Relay-Delivery`) and forwards it to the spoke's `/api/meetings/invites`, `/api/inbox/email` or `/api/meetings/calendar-emails/proof` by `kind`, behind the webhook relay's private-address guard and without following redirects. A files delivery that is multipart or larger than one piece (`HIVE_HUB_MAIL_CHUNK_BYTES`, 768 KiB) goes to `/api/inbox/email/chunk` in signed pieces instead ([#18564](https://github.com/hivecommons/hive/issues/18564)). Stores nothing once relayed; logs the delivery id only. `202` delivered; `400` invalid delivery; `404` no such hive; `413` too large; `422` the spoke refused it (permanent); `502`/`503` hive not reachable yet or unavailable (hive-mx asks the sender to retry) | `pkg/hub/mail_relay.go` | | `GET` | `/gh-setup` | Hub handler-specific | GitHub App Setup Router | `pkg/hub/server.go:1797` | | `GET` | `/learn` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1798` | | `GET` | `/get-started` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1799` | diff --git a/src/docs/api-reference/knowledge.md b/src/docs/api-reference/knowledge.md index 33257dccbe..dee53ced6e 100644 --- a/src/docs/api-reference/knowledge.md +++ b/src/docs/api-reference/knowledge.md @@ -68,10 +68,11 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `POST` | `/api/knowledge/connectors/{name}/auth/complete` | Owner only | Finish a sign-in from pasted material on a hosted hive: `{redirect_url}` (Plaud: the `localhost:8199/auth/callback?…` URL the browser landed on, forwarded in-pod while the login is pending; Zoom, Google Meet and Webex: the registered redirect URL with `code` and `state`, exchanged for tokens kept in the connector token store) or `{tokens}` (Plaud only: contents of `~/.plaud/tokens-mcp.json`); see [inbox-voice-sources.md](../inbox-voice-sources.md) | `pkg/dashboard/api_inbox.go:32` | | `GET` | `/api/inbox` | Dashboard auth/session | Inbox items (voice recordings, dictations and meeting recordings keyed by `(source, id)`) with per-destination route state, source connection status, candidate repos and auto-routing settings; see [inbox-voice-sources.md](../inbox-voice-sources.md) | `pkg/dashboard/api_inbox.go:24` | | `POST` | `/api/inbox/route` | Owner only | Route one item: `{source, item_id, destination: spek_rfc\|dibs\|dismiss, repo?}` (202 queued; 400 unconfigured destination, 404 unknown item, 409 already queued/running) | `pkg/dashboard/api_inbox.go:25` | -| `POST` | `/api/inbox/email` | Hub relay signature | Mail-in files hive-mx received for `hive@` ([#18395](https://github.com/hivecommons/hive/issues/18395)): Markdown, audio and image attachments become `email` Inbox items under `connector-state/email/items`. A sender who passed DMARC alignment and is a proven calendar email of a user with access gets normal items; anyone else's items carry `awaiting_approval` and cannot be shared, routed or transcribed until approved. `200 {items, awaiting_approval}`; `409` off (`meetings.email.files_enabled`); `507` spool full (256 MiB); `401` bad relay signature | `pkg/dashboard/api_inbox_email.go:21` | -| `POST` | `/api/inbox/email/{id}/approve` | Owner only | Approve a mailed-in item from outside the allowlist. Audited as `inbox-email-approve` | `pkg/dashboard/api_inbox_email.go:22` | -| `POST` | `/api/inbox/email/{id}/decline` | Owner only | Delete a mailed-in item and its file. Unapproved items are also deleted after 14 days. Audited as `inbox-email-decline` | `pkg/dashboard/api_inbox_email.go:23` | -| `GET` | `/api/inbox/email/{id}/file` | Owner or read-write | The stored file with its sniffed type (Markdown as `text/plain`), inline for images, `nosniff` and a sandbox CSP | `pkg/dashboard/api_inbox_email.go:24` | +| `POST` | `/api/inbox/email` | Hub relay signature | Mail-in files hive-mx received for `hive@` ([#18395](https://github.com/hivecommons/hive/issues/18395), [#18564](https://github.com/hivecommons/hive/issues/18564)): Markdown, audio, video and image attachments, and a body-only mail as `message.md`, become `email` Inbox items under `connector-state/email/items`; the mail's text body is kept as each item's note (`files.note`). A sender who passed DMARC alignment and is a proven calendar email of a user with access gets normal items; anyone else's items carry `awaiting_approval` and cannot be shared, routed or transcribed until approved. `200 {items, awaiting_approval}`; `409` off (`meetings.email.files_enabled`); `507` spool full (1 GiB); `401` bad relay signature | `pkg/dashboard/api_inbox_email.go:29` | +| `POST` | `/api/inbox/email/chunk` | Hub relay signature, per piece | One piece of a large mail-in delivery the hub relays in order (`X-Hive-Relay-Delivery: ..`, at most 8 MiB per piece, `X-Hive-Mail-Body-Type` the whole body's type: `multipart/form-data` from hive-mx or JSON). Pieces are staged under `connector-state/email/items/.incoming/` (dropped after an hour if incomplete); the last one decodes the body, checks each file's size and SHA-256, and answers like `/api/inbox/email`. `200 {piece, pieces}` for an earlier piece; `400` invalid piece or body; `413` over `HIVE_INBOX_EMAIL_MAX_BYTES` (48 MiB); `507` spool full | `pkg/dashboard/api_inbox_email.go:30` | +| `POST` | `/api/inbox/email/{id}/approve` | Owner only | Approve a mailed-in item from outside the allowlist. Audited as `inbox-email-approve` | `pkg/dashboard/api_inbox_email.go:31` | +| `POST` | `/api/inbox/email/{id}/decline` | Owner only | Delete a mailed-in item and its file. Unapproved items are also deleted after 14 days. Audited as `inbox-email-decline` | `pkg/dashboard/api_inbox_email.go:32` | +| `GET` | `/api/inbox/email/{id}/file` | Owner or read-write | The stored file with its sniffed type (Markdown as `text/plain`), inline for images (audio and video download), `nosniff` and a sandbox CSP | `pkg/dashboard/api_inbox_email.go:33` | | `POST` | `/api/inbox/dismiss` | Owner only | Bulk "Knowledge only": `{items: [{source, item_id}], undo?}` | `pkg/dashboard/api_inbox.go:26` | | `GET` | `/api/inbox/settings` | Dashboard auth/session | Auto-routing rules, default RFC repo, dibs URL, destination readiness | `pkg/dashboard/api_inbox.go:27` | | `POST` | `/api/inbox/settings` | Owner only | Update `inbox.rules` / `inbox.spek.repo` / `inbox.dibs.url` in the runtime config (omitted fields unchanged) | `pkg/dashboard/api_inbox.go:28` | @@ -117,8 +118,8 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `PUT` | `/api/meetings/persona` | Owner or read-write | Save `meetings.persona` to `hive.yaml`: `{enabled?, preset?, clear_overrides?, name?, display_name?, aliases?, pronouns?, language?, voice?, voice_type?, accent?, intro_template?, leave_phrases?, humor?, unprompted?: {per_10m?, min_gap_s?, per_meeting?, answer_wait_s?, min_confidence?, queue_depth?, stale_s?}, voice_on_address?}`. An absent field is unchanged, an empty one (or 0 for a cap) clears the override, a value equal to the preset's is stored as no override, and `clear_overrides` drops every field override first. Validated as if the persona were on; returns the GET body. 400 invalid. Audited as `meetings_persona`. An optional `email: {calendar_invites?, mail_in_files?}` sets `meetings.email.invites_enabled` / `files_enabled` (owner only, else 403; [#18395](https://github.com/hivecommons/hive/issues/18395)) | `pkg/dashboard/api_meetings_persona.go:165` | | `POST` | `/api/meetings/persona/sample` | Owner or read-write | Speak a short line through the meetings TTS service (Kokoro `POST /v1/audio/speech`) and return the audio (`audio/mpeg`, at most 2 MiB, `Cache-Control: no-store`). Optional `{voice?, language?, display_name?, text?}` (unsaved form values; `text` at most 200 characters, default the persona's sample line in its language). 503 "TTS not configured" until `meetings.tts_url` is set, 400 invalid, 502 the TTS service failed | `pkg/dashboard/api_meetings_persona.go:166` | | `GET` | `/api/meetings/persona/sample` | Owner or read-write | The same sample with `voice`, `language`, `display_name` and `text` as query parameters, for the dashboard's `