From c8eecd740d874e862272dfe8fdfdaa7fec6fee10 Mon Sep 17 00:00:00 2001 From: Andrew Anderson Date: Mon, 16 Nov 2026 11:11:41 -0500 Subject: [PATCH 1/2] feat(deploy): hive-mx mail receiver on hive-oke in namespace mail hive-mx (hivecommons/hive#18395) accepts mail only for hive@ and posts calendar invites, mail-in files and proof codes to the hub. It runs as 2 replicas in a new namespace mail (Pod Security restricted, non-root 65532, read-only root, no capabilities), behind an OCI Network Load Balancer on port 25 with externalTrafficPolicy Local so SPF sees the sender's IP. NetworkPolicy: SMTP in from anywhere, egress only to CoreDNS and the hub Service. install.sh generates the hub bearer token once, never prints it, and with --hub-secret copies it into hive-hub/hive-hub-secrets (mail-token); the README lists the hub env change and the DNS records the operator adds. Signed-off-by: Andrew Anderson --- deploy/hive-oke/README.md | 25 ++--- deploy/hive-oke/mail/README.md | 102 +++++++++++++++++++ deploy/hive-oke/mail/hive-mx.yaml | 145 ++++++++++++++++++++++++++++ deploy/hive-oke/mail/install.sh | 45 +++++++++ deploy/hive-oke/mail/namespace.yaml | 12 +++ deploy/hive-oke/mail/policy.yaml | 47 +++++++++ 6 files changed, 364 insertions(+), 12 deletions(-) create mode 100644 deploy/hive-oke/mail/README.md create mode 100644 deploy/hive-oke/mail/hive-mx.yaml create mode 100755 deploy/hive-oke/mail/install.sh create mode 100644 deploy/hive-oke/mail/namespace.yaml create mode 100644 deploy/hive-oke/mail/policy.yaml diff --git a/deploy/hive-oke/README.md b/deploy/hive-oke/README.md index 9451d96..25ec31c 100644 --- a/deploy/hive-oke/README.md +++ b/deploy/hive-oke/README.md @@ -4,18 +4,19 @@ Quorum (RFC #1) uses one shared Vexa control plane per cluster. On hive-oke it r `meetings`, and nothing outside that namespace is touched. The only cluster-scoped object is the `meetings` Namespace. -| File | What it is | -| -------------------- | ------------------------------------------------------------------------------------------- | -| `namespace.yaml` | Namespace `meetings` (Pod Security baseline in warn/audit mode) | -| `values.yaml` | Helm values for the Vexa chart, plus the `quorum` block read by `install.sh` | -| `versitygw.yaml` | In-namespace S3 endpoint (versitygw, POSIX backend on a 50Gi `oci-bv` PVC), bucket Job | -| `transcription.yaml` | CPU Whisper (faster-whisper `small`, int8) Deployment and Service | -| `tts.yaml` | CPU text-to-speech (Kokoro via Kokoro-FastAPI) Deployment, Service, voice-map ConfigMap | -| `searxng.yaml` | SearXNG web search for hive Betty: Deployment, Service `searxng:8080`, settings ConfigMap | -| `policy.yaml` | LimitRange and gateway NetworkPolicy | -| `install.sh` | Idempotent install/upgrade: Secrets, policy, ResourceQuota, storage, STT, TTS, search, Helm | -| `smoke.sh` | Smoke test with the tenant key from a throwaway Pod (includes a direct TTS call) | -| `smoke_api.py` | The API half of the smoke test (health, `/ws`, chat, speak), run inside that Pod | +| File | What it is | +| -------------------- | -------------------------------------------------------------------------------------------- | +| `namespace.yaml` | Namespace `meetings` (Pod Security baseline in warn/audit mode) | +| `values.yaml` | Helm values for the Vexa chart, plus the `quorum` block read by `install.sh` | +| `versitygw.yaml` | In-namespace S3 endpoint (versitygw, POSIX backend on a 50Gi `oci-bv` PVC), bucket Job | +| `transcription.yaml` | CPU Whisper (faster-whisper `small`, int8) Deployment and Service | +| `tts.yaml` | CPU text-to-speech (Kokoro via Kokoro-FastAPI) Deployment, Service, voice-map ConfigMap | +| `searxng.yaml` | SearXNG web search for hive Betty: Deployment, Service `searxng:8080`, settings ConfigMap | +| `policy.yaml` | LimitRange and gateway NetworkPolicy | +| `install.sh` | Idempotent install/upgrade: Secrets, policy, ResourceQuota, storage, STT, TTS, search, Helm | +| `smoke.sh` | Smoke test with the tenant key from a throwaway Pod (includes a direct TTS call) | +| `smoke_api.py` | The API half of the smoke test (health, `/ws`, chat, speak), run inside that Pod | +| `mail/` | hive-mx, hive's receive-only mail receiver, in its own namespace `mail` (see mail/README.md) | ## Versions (all pinned by tag and digest) diff --git a/deploy/hive-oke/mail/README.md b/deploy/hive-oke/mail/README.md new file mode 100644 index 0000000..cc71069 --- /dev/null +++ b/deploy/hive-oke/mail/README.md @@ -0,0 +1,102 @@ +# hive-mx on hive-oke: mail in for hive + +hive-mx is hive's receive-only mail receiver (hivecommons/hive#18395, plan #18391). People invite +Betty by adding `hive@` (for example `hive@acme.hive.hivecommons.dev`) to a +calendar invite, prove their calendar emails, and mail Markdown, audio and image files into a +hive's Inbox. One receiver serves every hive on the cluster through a wildcard MX record; no spoke +opens port 25. + +hive-mx runs in namespace `mail` and touches nothing else, except one key it can add to the hub's +Secret (see "Hub" below). + +| File | What it is | +| ---------------- | ---------------------------------------------------------------------------------- | +| `namespace.yaml` | Namespace `mail` (Pod Security restricted, enforced) | +| `policy.yaml` | NetworkPolicy: SMTP in from anywhere; egress only to CoreDNS and the hub Service | +| `hive-mx.yaml` | Deployment (2 replicas), PodDisruptionBudget, Service type LoadBalancer on port 25 | +| `install.sh` | Idempotent install/upgrade; generates the hub bearer token once, never prints it | + +## What hive-mx does + +- Accepts `RCPT TO` only for `hive@` where the host is a hive the hub knows (subdomain, vanity + or dashboard host). The list comes from `GET /api/mail/hosts` on the hub every 60 seconds; until + the first list loads, RCPT gets a temporary 451. +- At most 5 recipients and 2 MB per message, per-IP and per-sender-domain rate limits, at most 100 + connections, no AUTH, no relay. It never sends mail. +- After DATA it records SPF, DKIM and DMARC alignment, keeps only calendar invites (scheduling + fields and join details), Markdown/audio/image attachments, or a proof code, and posts the result + to `POST /api/mail/inbound` on the hub. The hub signs it with the target hive's relay key and + forwards it; it stores nothing. A hub or spoke that cannot take the mail makes hive-mx answer a + temporary 451, so the sender's server retries. + +## Versions (pinned by tag and digest) + +| Component | Image | +| --------- | ------------------------------------------------------------------------------------------------------------------------------- | +| hive-mx | `ghcr.io/hivecommons/hive-mx:aa286e8@sha256:bbed51d5f92fe208f566ca19b2dab54e14a5985481dcb6ae0c1893ef9d6c764f` (manifest list, linux/amd64 and linux/arm64; distroless static, uid 65532) | + +Built from `src/Dockerfile.mx` in hivecommons/hive by `docker.yml` (`build-mx`, `merge-mx`). Every +hive-oke node is arm64. Update by changing both the tag and the digest in `hive-mx.yaml`. + +## Install + +```sh +KUBE_CONTEXT=hive-oke deploy/hive-oke/mail/install.sh --hub-secret +``` + +`install.sh` applies the namespace, creates Secret `mail/hive-mx-hub-token` with a random token if it +does not exist, and with `--hub-secret` copies the same value into key `mail-token` of Secret +`hive-hub/hive-hub-secrets`. Then it applies the policy and the Deployment and prints the Service. + +## Hub + +The hub's mail endpoints answer 404 until it has `HIVE_HUB_MAIL_TOKEN`. That needs one env entry on +the `hub` container of Deployment `hive-hub/hive-hub`, which rolls the hub, so it is an operator +step: + +```sh +kubectl --context hive-oke -n hive-hub patch deployment hive-hub --type json -p '[ + {"op":"add","path":"/spec/template/spec/containers/1/env/-","value": + {"name":"HIVE_HUB_MAIL_TOKEN","valueFrom":{"secretKeyRef":{"name":"hive-hub-secrets","key":"mail-token"}}}} +]' +``` + +(container index 1 is `hub`; check with +`kubectl -n hive-hub get deploy hive-hub -o jsonpath='{.spec.template.spec.containers[*].name}'`). +The hub must also run a build that has `/api/mail/*` (hivecommons/hive#18395 part 3). + +## DNS + +hivecommons.dev DNS is on DNSimple; these two records are added by the operator: + +| Type | Name | Value | Priority | +| ---- | ------------------------- | ------------------------- | -------- | +| A | `mx.hive.hivecommons.dev` | the Service's external IP | n/a | +| MX | `*.hive.hivecommons.dev` | `mx.hive.hivecommons.dev` | 10 | + +The root `hivecommons.dev` MX (a legacy mailbox) is left alone. + +## Load balancer + +The Service is an OCI Network Load Balancer (`oci.oraclecloud.com/load-balancer-type: nlb`) in the +same subnet as ingress-nginx, with `externalTrafficPolicy: Local`, so hive-mx sees the sender's IP +(SPF and the per-IP rate limit need it). The node security list or NSG must allow TCP 25 to the +Service's node port from the internet. + +## STARTTLS + +hive-mx offers STARTTLS only when a certificate is mounted (`HIVE_MX_TLS_CERT`, `HIVE_MX_TLS_KEY`, +re-read when the file changes). None is mounted here yet: mail is accepted over plain SMTP, which +every sender falls back to. To add it, issue a cert-manager Certificate for `mx.hive.hivecommons.dev` +in namespace `mail` (DNS-01; HTTP-01 does not reach port 25), mount its Secret and set the two +variables. + +## Check + +```sh +kubectl --context hive-oke -n mail get pods,svc +kubectl --context hive-oke -n mail logs deploy/hive-mx | tail +``` + +`/readyz` is 503 until the first host list loaded from the hub; the logs carry only delivery ids, +kinds and DMARC results, never message content. diff --git a/deploy/hive-oke/mail/hive-mx.yaml b/deploy/hive-oke/mail/hive-mx.yaml new file mode 100644 index 0000000..0108d71 --- /dev/null +++ b/deploy/hive-oke/mail/hive-mx.yaml @@ -0,0 +1,145 @@ +# hive-mx (src/cmd/hive-mx in hivecommons/hive, #18395): accepts mail only for +# hive@, keeps calendar invites, mail-in files and proof codes, and posts +# them to the hub, which routes each one to its hive. It never sends mail, offers no AUTH and +# relays nothing. The bearer token for the hub comes from Secret `hive-mx-hub-token` +# (created by install.sh; the hub holds the same value as HIVE_HUB_MAIL_TOKEN). +# +# Image: ghcr.io/hivecommons/hive-mx, multi-arch (linux/amd64, linux/arm64), distroless static, +# uid 65532, pinned by tag and digest. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hive-mx + namespace: mail + labels: + app.kubernetes.io/name: hive-mx + app.kubernetes.io/part-of: hive-mail +spec: + replicas: 2 + selector: + matchLabels: + app.kubernetes.io/name: hive-mx + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + template: + metadata: + labels: + app.kubernetes.io/name: hive-mx + app.kubernetes.io/part-of: hive-mail + spec: + automountServiceAccountToken: false + enableServiceLinks: false + securityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + # Secret volume files are group-owned by fsGroup, so the token is + # readable by the non-root user without being world-readable. + fsGroup: 65532 + seccompProfile: + type: RuntimeDefault + # One replica per node, so a node drain never takes both. + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: hive-mx + terminationGracePeriodSeconds: 30 + containers: + - name: hive-mx + image: ghcr.io/hivecommons/hive-mx:aa286e8@sha256:bbed51d5f92fe208f566ca19b2dab54e14a5985481dcb6ae0c1893ef9d6c764f + imagePullPolicy: IfNotPresent + env: + - name: HIVE_MX_HUB_URL + value: http://hive-hub.hive-hub.svc.cluster.local + - name: HIVE_MX_HUB_TOKEN_FILE + value: /var/run/secrets/hive-mx/token + - name: HIVE_MX_DOMAIN + value: mx.hive.hivecommons.dev + - name: HIVE_MX_LISTEN + value: ":2525" + - name: HIVE_MX_HEALTH_LISTEN + value: ":8080" + ports: + - name: smtp + containerPort: 2525 + protocol: TCP + - name: health + containerPort: 8080 + protocol: TCP + readinessProbe: + httpGet: + path: /readyz + port: health + periodSeconds: 10 + livenessProbe: + httpGet: + path: /healthz + port: health + periodSeconds: 20 + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - name: hub-token + mountPath: /var/run/secrets/hive-mx + readOnly: true + volumes: + - name: hub-token + secret: + secretName: hive-mx-hub-token + defaultMode: 0440 +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: hive-mx + namespace: mail + labels: + app.kubernetes.io/name: hive-mx + app.kubernetes.io/part-of: hive-mail +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: hive-mx +--- +# Port 25 from the internet. An OCI Network Load Balancer (not the proxying classic LB) with +# externalTrafficPolicy Local keeps the sender's IP, which SPF and the per-IP rate limit need. +apiVersion: v1 +kind: Service +metadata: + name: hive-mx + namespace: mail + labels: + app.kubernetes.io/name: hive-mx + app.kubernetes.io/part-of: hive-mail + annotations: + oci.oraclecloud.com/load-balancer-type: "nlb" + oci-network-load-balancer.oraclecloud.com/subnet: "ocid1.subnet.oc1.iad.aaaaaaaaom27jaylwwnovy5gnfm75rceuvmpcqhe3ttgp6te3zlkz73i4pva" +spec: + type: LoadBalancer + externalTrafficPolicy: Local + ipFamilies: [IPv4] + ipFamilyPolicy: SingleStack + selector: + app.kubernetes.io/name: hive-mx + ports: + - name: smtp + port: 25 + targetPort: smtp + protocol: TCP diff --git a/deploy/hive-oke/mail/install.sh b/deploy/hive-oke/mail/install.sh new file mode 100755 index 0000000..dafabd3 --- /dev/null +++ b/deploy/hive-oke/mail/install.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Idempotent install/upgrade of hive-mx in namespace `mail` on hive-oke (hivecommons/hive#18395). +# +# KUBE_CONTEXT=hive-oke deploy/hive-oke/mail/install.sh # mail namespace only +# KUBE_CONTEXT=hive-oke deploy/hive-oke/mail/install.sh --hub-secret +# +# The bearer token hive-mx shows the hub is generated once into Secret mail/hive-mx-hub-token +# and never printed. With --hub-secret the same value is also written as key `mail-token` of +# Secret hive-hub/hive-hub-secrets (the only change this script makes outside `mail`); the hub +# reads it as HIVE_HUB_MAIL_TOKEN once its Deployment references that key and has rolled out, +# which is an operator step (README.md, "Hub"). +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ctx="${KUBE_CONTEXT:-hive-oke}" +k() { kubectl --context "$ctx" "$@"; } + +hub_secret=false +for arg in "$@"; do + case "$arg" in + --hub-secret) hub_secret=true ;; + *) echo "unknown argument: $arg" >&2; exit 2 ;; + esac +done + +k apply -f "$here/namespace.yaml" + +if ! k -n mail get secret hive-mx-hub-token >/dev/null 2>&1; then + token="$(openssl rand -hex 32)" + k -n mail create secret generic hive-mx-hub-token --from-literal=token="$token" >/dev/null + unset token + echo "created Secret mail/hive-mx-hub-token" +fi + +if [ "$hub_secret" = true ]; then + b64="$(k -n mail get secret hive-mx-hub-token -o jsonpath='{.data.token}')" + k -n hive-hub patch secret hive-hub-secrets --type merge -p "{\"data\":{\"mail-token\":\"$b64\"}}" >/dev/null + unset b64 + echo "set key mail-token in Secret hive-hub/hive-hub-secrets" +fi + +k apply -f "$here/policy.yaml" +k apply -f "$here/hive-mx.yaml" +k -n mail rollout status deploy/hive-mx --timeout=180s +k -n mail get svc hive-mx -o wide diff --git a/deploy/hive-oke/mail/namespace.yaml b/deploy/hive-oke/mail/namespace.yaml new file mode 100644 index 0000000..765216b --- /dev/null +++ b/deploy/hive-oke/mail/namespace.yaml @@ -0,0 +1,12 @@ +# hive-mx, the cluster's receive-only mail receiver for hive (hivecommons/hive#18395). +# Everything mail runs lives in this namespace; nothing else on hive-oke is touched. +apiVersion: v1 +kind: Namespace +metadata: + name: mail + labels: + app.kubernetes.io/part-of: hive-mail + # hive-mx runs as non-root with no capabilities, so restricted is enforced. + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/warn: restricted + pod-security.kubernetes.io/audit: restricted diff --git a/deploy/hive-oke/mail/policy.yaml b/deploy/hive-oke/mail/policy.yaml new file mode 100644 index 0000000..db3484e --- /dev/null +++ b/deploy/hive-oke/mail/policy.yaml @@ -0,0 +1,47 @@ +# Network policy for namespace `mail`. +# +# Ingress: SMTP (2525, the Service maps port 25 to it) from anywhere, since mail comes from +# the internet; health (8080) only from inside the cluster (kubelet probes are not subject +# to NetworkPolicy). +# Egress: DNS (CoreDNS in kube-system resolves SPF, DKIM and DMARC records upstream) and the +# hub Service in namespace hive-hub on port 80. Nothing else: hive-mx never sends mail. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: hive-mx + namespace: mail + labels: + app.kubernetes.io/name: hive-mx + app.kubernetes.io/part-of: hive-mail +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: hive-mx + policyTypes: [Ingress, Egress] + ingress: + - ports: + - port: 2525 + protocol: TCP + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hive-hub + podSelector: + matchLabels: + app: hive-hub + ports: + - port: 80 + protocol: TCP From fa2fd3ecc2ac34510bd2b97e21f13aa9c40c0d04 Mon Sep 17 00:00:00 2001 From: Andrew Anderson Date: Mon, 16 Nov 2026 12:45:24 -0500 Subject: [PATCH 2/2] style(deploy): prettier on the mail README Signed-off-by: Andrew Anderson --- deploy/hive-oke/mail/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deploy/hive-oke/mail/README.md b/deploy/hive-oke/mail/README.md index cc71069..97f2a4c 100644 --- a/deploy/hive-oke/mail/README.md +++ b/deploy/hive-oke/mail/README.md @@ -31,8 +31,8 @@ Secret (see "Hub" below). ## Versions (pinned by tag and digest) -| Component | Image | -| --------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Component | Image | +| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive-mx | `ghcr.io/hivecommons/hive-mx:aa286e8@sha256:bbed51d5f92fe208f566ca19b2dab54e14a5985481dcb6ae0c1893ef9d6c764f` (manifest list, linux/amd64 and linux/arm64; distroless static, uid 65532) | Built from `src/Dockerfile.mx` in hivecommons/hive by `docker.yml` (`build-mx`, `merge-mx`). Every