Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions deploy/hive-oke/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,19 @@ Quorum (RFC #1) uses one shared Vexa control plane per cluster. On hive-oke it r
`meetings`, and nothing outside that namespace is touched. The only cluster-scoped object is the
`meetings` Namespace.

| File | What it is |
| -------------------- | ------------------------------------------------------------------------------------------- |
| `namespace.yaml` | Namespace `meetings` (Pod Security baseline in warn/audit mode) |
| `values.yaml` | Helm values for the Vexa chart, plus the `quorum` block read by `install.sh` |
| `versitygw.yaml` | In-namespace S3 endpoint (versitygw, POSIX backend on a 50Gi `oci-bv` PVC), bucket Job |
| `transcription.yaml` | CPU Whisper (faster-whisper `small`, int8) Deployment and Service |
| `tts.yaml` | CPU text-to-speech (Kokoro via Kokoro-FastAPI) Deployment, Service, voice-map ConfigMap |
| `searxng.yaml` | SearXNG web search for hive Betty: Deployment, Service `searxng:8080`, settings ConfigMap |
| `policy.yaml` | LimitRange and gateway NetworkPolicy |
| `install.sh` | Idempotent install/upgrade: Secrets, policy, ResourceQuota, storage, STT, TTS, search, Helm |
| `smoke.sh` | Smoke test with the tenant key from a throwaway Pod (includes a direct TTS call) |
| `smoke_api.py` | The API half of the smoke test (health, `/ws`, chat, speak), run inside that Pod |
| File | What it is |
| -------------------- | -------------------------------------------------------------------------------------------- |
| `namespace.yaml` | Namespace `meetings` (Pod Security baseline in warn/audit mode) |
| `values.yaml` | Helm values for the Vexa chart, plus the `quorum` block read by `install.sh` |
| `versitygw.yaml` | In-namespace S3 endpoint (versitygw, POSIX backend on a 50Gi `oci-bv` PVC), bucket Job |
| `transcription.yaml` | CPU Whisper (faster-whisper `small`, int8) Deployment and Service |
| `tts.yaml` | CPU text-to-speech (Kokoro via Kokoro-FastAPI) Deployment, Service, voice-map ConfigMap |
| `searxng.yaml` | SearXNG web search for hive Betty: Deployment, Service `searxng:8080`, settings ConfigMap |
| `policy.yaml` | LimitRange and gateway NetworkPolicy |
| `install.sh` | Idempotent install/upgrade: Secrets, policy, ResourceQuota, storage, STT, TTS, search, Helm |
| `smoke.sh` | Smoke test with the tenant key from a throwaway Pod (includes a direct TTS call) |
| `smoke_api.py` | The API half of the smoke test (health, `/ws`, chat, speak), run inside that Pod |
| `mail/` | hive-mx, hive's receive-only mail receiver, in its own namespace `mail` (see mail/README.md) |

## Versions (all pinned by tag and digest)

Expand Down
102 changes: 102 additions & 0 deletions deploy/hive-oke/mail/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# hive-mx on hive-oke: mail in for hive

hive-mx is hive's receive-only mail receiver (hivecommons/hive#18395, plan #18391). People invite
Betty by adding `hive@<hive dashboard host>` (for example `hive@acme.hive.hivecommons.dev`) to a
calendar invite, prove their calendar emails, and mail Markdown, audio and image files into a
hive's Inbox. One receiver serves every hive on the cluster through a wildcard MX record; no spoke
opens port 25.

hive-mx runs in namespace `mail` and touches nothing else, except one key it can add to the hub's
Secret (see "Hub" below).

| File | What it is |
| ---------------- | ---------------------------------------------------------------------------------- |
| `namespace.yaml` | Namespace `mail` (Pod Security restricted, enforced) |
| `policy.yaml` | NetworkPolicy: SMTP in from anywhere; egress only to CoreDNS and the hub Service |
| `hive-mx.yaml` | Deployment (2 replicas), PodDisruptionBudget, Service type LoadBalancer on port 25 |
| `install.sh` | Idempotent install/upgrade; generates the hub bearer token once, never prints it |

## What hive-mx does

- Accepts `RCPT TO` only for `hive@<host>` where the host is a hive the hub knows (subdomain, vanity
or dashboard host). The list comes from `GET /api/mail/hosts` on the hub every 60 seconds; until
the first list loads, RCPT gets a temporary 451.
- At most 5 recipients and 2 MB per message, per-IP and per-sender-domain rate limits, at most 100
connections, no AUTH, no relay. It never sends mail.
- After DATA it records SPF, DKIM and DMARC alignment, keeps only calendar invites (scheduling
fields and join details), Markdown/audio/image attachments, or a proof code, and posts the result
to `POST /api/mail/inbound` on the hub. The hub signs it with the target hive's relay key and
forwards it; it stores nothing. A hub or spoke that cannot take the mail makes hive-mx answer a
temporary 451, so the sender's server retries.

## Versions (pinned by tag and digest)

| Component | Image |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive-mx | `ghcr.io/hivecommons/hive-mx:aa286e8@sha256:bbed51d5f92fe208f566ca19b2dab54e14a5985481dcb6ae0c1893ef9d6c764f` (manifest list, linux/amd64 and linux/arm64; distroless static, uid 65532) |

Built from `src/Dockerfile.mx` in hivecommons/hive by `docker.yml` (`build-mx`, `merge-mx`). Every
hive-oke node is arm64. Update by changing both the tag and the digest in `hive-mx.yaml`.

## Install

```sh
KUBE_CONTEXT=hive-oke deploy/hive-oke/mail/install.sh --hub-secret
```

`install.sh` applies the namespace, creates Secret `mail/hive-mx-hub-token` with a random token if it
does not exist, and with `--hub-secret` copies the same value into key `mail-token` of Secret
`hive-hub/hive-hub-secrets`. Then it applies the policy and the Deployment and prints the Service.

## Hub

The hub's mail endpoints answer 404 until it has `HIVE_HUB_MAIL_TOKEN`. That needs one env entry on
the `hub` container of Deployment `hive-hub/hive-hub`, which rolls the hub, so it is an operator
step:

```sh
kubectl --context hive-oke -n hive-hub patch deployment hive-hub --type json -p '[
{"op":"add","path":"/spec/template/spec/containers/1/env/-","value":
{"name":"HIVE_HUB_MAIL_TOKEN","valueFrom":{"secretKeyRef":{"name":"hive-hub-secrets","key":"mail-token"}}}}
]'
```

(container index 1 is `hub`; check with
`kubectl -n hive-hub get deploy hive-hub -o jsonpath='{.spec.template.spec.containers[*].name}'`).
The hub must also run a build that has `/api/mail/*` (hivecommons/hive#18395 part 3).

## DNS

hivecommons.dev DNS is on DNSimple; these two records are added by the operator:

| Type | Name | Value | Priority |
| ---- | ------------------------- | ------------------------- | -------- |
| A | `mx.hive.hivecommons.dev` | the Service's external IP | n/a |
| MX | `*.hive.hivecommons.dev` | `mx.hive.hivecommons.dev` | 10 |

The root `hivecommons.dev` MX (a legacy mailbox) is left alone.

## Load balancer

The Service is an OCI Network Load Balancer (`oci.oraclecloud.com/load-balancer-type: nlb`) in the
same subnet as ingress-nginx, with `externalTrafficPolicy: Local`, so hive-mx sees the sender's IP
(SPF and the per-IP rate limit need it). The node security list or NSG must allow TCP 25 to the
Service's node port from the internet.

## STARTTLS

hive-mx offers STARTTLS only when a certificate is mounted (`HIVE_MX_TLS_CERT`, `HIVE_MX_TLS_KEY`,
re-read when the file changes). None is mounted here yet: mail is accepted over plain SMTP, which
every sender falls back to. To add it, issue a cert-manager Certificate for `mx.hive.hivecommons.dev`
in namespace `mail` (DNS-01; HTTP-01 does not reach port 25), mount its Secret and set the two
variables.

## Check

```sh
kubectl --context hive-oke -n mail get pods,svc
kubectl --context hive-oke -n mail logs deploy/hive-mx | tail
```

`/readyz` is 503 until the first host list loaded from the hub; the logs carry only delivery ids,
kinds and DMARC results, never message content.
145 changes: 145 additions & 0 deletions deploy/hive-oke/mail/hive-mx.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# hive-mx (src/cmd/hive-mx in hivecommons/hive, #18395): accepts mail only for
# hive@<known hive host>, keeps calendar invites, mail-in files and proof codes, and posts
# them to the hub, which routes each one to its hive. It never sends mail, offers no AUTH and
# relays nothing. The bearer token for the hub comes from Secret `hive-mx-hub-token`
# (created by install.sh; the hub holds the same value as HIVE_HUB_MAIL_TOKEN).
#
# Image: ghcr.io/hivecommons/hive-mx, multi-arch (linux/amd64, linux/arm64), distroless static,
# uid 65532, pinned by tag and digest.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hive-mx
namespace: mail
labels:
app.kubernetes.io/name: hive-mx
app.kubernetes.io/part-of: hive-mail
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: hive-mx
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app.kubernetes.io/name: hive-mx
app.kubernetes.io/part-of: hive-mail
spec:
automountServiceAccountToken: false
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
# Secret volume files are group-owned by fsGroup, so the token is
# readable by the non-root user without being world-readable.
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
# One replica per node, so a node drain never takes both.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: hive-mx
terminationGracePeriodSeconds: 30
containers:
- name: hive-mx
image: ghcr.io/hivecommons/hive-mx:aa286e8@sha256:bbed51d5f92fe208f566ca19b2dab54e14a5985481dcb6ae0c1893ef9d6c764f
imagePullPolicy: IfNotPresent
env:
- name: HIVE_MX_HUB_URL
value: http://hive-hub.hive-hub.svc.cluster.local
- name: HIVE_MX_HUB_TOKEN_FILE
value: /var/run/secrets/hive-mx/token
- name: HIVE_MX_DOMAIN
value: mx.hive.hivecommons.dev
- name: HIVE_MX_LISTEN
value: ":2525"
- name: HIVE_MX_HEALTH_LISTEN
value: ":8080"
ports:
- name: smtp
containerPort: 2525
protocol: TCP
- name: health
containerPort: 8080
protocol: TCP
readinessProbe:
httpGet:
path: /readyz
port: health
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: health
periodSeconds: 20
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
volumeMounts:
- name: hub-token
mountPath: /var/run/secrets/hive-mx
readOnly: true
volumes:
- name: hub-token
secret:
secretName: hive-mx-hub-token
defaultMode: 0440
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: hive-mx
namespace: mail
labels:
app.kubernetes.io/name: hive-mx
app.kubernetes.io/part-of: hive-mail
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: hive-mx
---
# Port 25 from the internet. An OCI Network Load Balancer (not the proxying classic LB) with
# externalTrafficPolicy Local keeps the sender's IP, which SPF and the per-IP rate limit need.
apiVersion: v1
kind: Service
metadata:
name: hive-mx
namespace: mail
labels:
app.kubernetes.io/name: hive-mx
app.kubernetes.io/part-of: hive-mail
annotations:
oci.oraclecloud.com/load-balancer-type: "nlb"
oci-network-load-balancer.oraclecloud.com/subnet: "ocid1.subnet.oc1.iad.aaaaaaaaom27jaylwwnovy5gnfm75rceuvmpcqhe3ttgp6te3zlkz73i4pva"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ipFamilies: [IPv4]
ipFamilyPolicy: SingleStack
selector:
app.kubernetes.io/name: hive-mx
ports:
- name: smtp
port: 25
targetPort: smtp
protocol: TCP
45 changes: 45 additions & 0 deletions deploy/hive-oke/mail/install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
# Idempotent install/upgrade of hive-mx in namespace `mail` on hive-oke (hivecommons/hive#18395).
#
# KUBE_CONTEXT=hive-oke deploy/hive-oke/mail/install.sh # mail namespace only
# KUBE_CONTEXT=hive-oke deploy/hive-oke/mail/install.sh --hub-secret
#
# The bearer token hive-mx shows the hub is generated once into Secret mail/hive-mx-hub-token
# and never printed. With --hub-secret the same value is also written as key `mail-token` of
# Secret hive-hub/hive-hub-secrets (the only change this script makes outside `mail`); the hub
# reads it as HIVE_HUB_MAIL_TOKEN once its Deployment references that key and has rolled out,
# which is an operator step (README.md, "Hub").
set -euo pipefail

here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ctx="${KUBE_CONTEXT:-hive-oke}"
k() { kubectl --context "$ctx" "$@"; }

hub_secret=false
for arg in "$@"; do
case "$arg" in
--hub-secret) hub_secret=true ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
done

k apply -f "$here/namespace.yaml"

if ! k -n mail get secret hive-mx-hub-token >/dev/null 2>&1; then
token="$(openssl rand -hex 32)"
k -n mail create secret generic hive-mx-hub-token --from-literal=token="$token" >/dev/null
unset token
echo "created Secret mail/hive-mx-hub-token"
fi

if [ "$hub_secret" = true ]; then
b64="$(k -n mail get secret hive-mx-hub-token -o jsonpath='{.data.token}')"
k -n hive-hub patch secret hive-hub-secrets --type merge -p "{\"data\":{\"mail-token\":\"$b64\"}}" >/dev/null
unset b64
echo "set key mail-token in Secret hive-hub/hive-hub-secrets"
fi

k apply -f "$here/policy.yaml"
k apply -f "$here/hive-mx.yaml"
k -n mail rollout status deploy/hive-mx --timeout=180s
k -n mail get svc hive-mx -o wide
12 changes: 12 additions & 0 deletions deploy/hive-oke/mail/namespace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# hive-mx, the cluster's receive-only mail receiver for hive (hivecommons/hive#18395).
# Everything mail runs lives in this namespace; nothing else on hive-oke is touched.
apiVersion: v1
kind: Namespace
metadata:
name: mail
labels:
app.kubernetes.io/part-of: hive-mail
# hive-mx runs as non-root with no capabilities, so restricted is enforced.
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/audit: restricted
47 changes: 47 additions & 0 deletions deploy/hive-oke/mail/policy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Network policy for namespace `mail`.
#
# Ingress: SMTP (2525, the Service maps port 25 to it) from anywhere, since mail comes from
# the internet; health (8080) only from inside the cluster (kubelet probes are not subject
# to NetworkPolicy).
# Egress: DNS (CoreDNS in kube-system resolves SPF, DKIM and DMARC records upstream) and the
# hub Service in namespace hive-hub on port 80. Nothing else: hive-mx never sends mail.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hive-mx
namespace: mail
labels:
app.kubernetes.io/name: hive-mx
app.kubernetes.io/part-of: hive-mail
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hive-mx
policyTypes: [Ingress, Egress]
ingress:
- ports:
- port: 2525
protocol: TCP
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hive-hub
podSelector:
matchLabels:
app: hive-hub
ports:
- port: 80
protocol: TCP
Loading