Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-inbox-upload-pdf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- PDFs can be dropped onto the Inbox or linked: each page with text becomes a `## Page N` section and the document title names the item, while a scanned PDF without a text layer is refused with "no text layer (scanned PDF); OCR not supported yet". PDFs are never parsed in the hive process: the new `hive-pdftext` sandbox (pure Go, `github.com/ledongthuc/pdf`, BSD-3-Clause, already a dependency) runs as the `hive-transcribe` uid through su-exec with an empty environment, under prlimit and nice, in an empty network namespace where available, from a private 0700 work dir, capped at 200 pages and 1 MiB of text and killed with its process group after 60 seconds (`HIVE_PDF_SANDBOX`, `HIVE_PDF_RUN_AS`, `HIVE_PDF_MEMORY_MB`).
26 changes: 17 additions & 9 deletions dashboard/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -38633,7 +38633,7 @@
"Knowledge"
],
"summary": "Add dropped files to the Inbox",
"description": "Owner or read-write. Each multipart `file` field is typed by its content (magic bytes, UTF-8 validation and a parse check), never by its name or client MIME type alone. Markdown, text, JSON, JSON Lines, YAML, CSV, TSV, HTML and Word (.docx, read with zip-bomb caps and no DTDs or external entities) become `upload` Inbox items with extracted markdown, stored under `connector-state/uploads/<id>/` (0700 directory, 0600 files) and written as a Knowledge fact with the text inside an `<untrusted-inbox-data>` envelope. Limits come from `inbox.uploads` (`max_file_bytes`, default 25 MiB; `max_files_per_drop`, default 20; `quota_bytes`, default 2 GiB). Requests marked `Sec-Fetch-Site: cross-site` are refused. Every file is audited as `inbox_upload` with its outcome.",
"description": "Owner or read-write. Each multipart `file` field is typed by its content (magic bytes, UTF-8 validation and a parse check), never by its name or client MIME type alone. Markdown, text, JSON, JSON Lines, YAML, CSV, TSV, HTML, Word (.docx, read with zip-bomb caps and no DTDs or external entities) and PDF (parsed out of process in the hive-pdftext sandbox; a PDF without a text layer is refused) become `upload` Inbox items with extracted markdown, stored under `connector-state/uploads/<id>/` (0700 directory, 0600 files) and written as a Knowledge fact with the text inside an `<untrusted-inbox-data>` envelope. Limits come from `inbox.uploads` (`max_file_bytes`, default 25 MiB; `max_files_per_drop`, default 20; `quota_bytes`, default 2 GiB). Requests marked `Sec-Fetch-Site: cross-site` are refused. Every file is audited as `inbox_upload` with its outcome.",
"requestBody": {
"required": true,
"content": {
Expand Down Expand Up @@ -38695,7 +38695,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -38762,7 +38763,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -38829,7 +38831,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -38902,7 +38905,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -38969,7 +38973,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -39036,7 +39041,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -39103,7 +39109,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down Expand Up @@ -39873,7 +39880,8 @@
"csv",
"tsv",
"html",
"docx"
"docx",
"pdf"
]
},
"code": {
Expand Down
6 changes: 5 additions & 1 deletion src/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,8 @@ RUN --mount=type=cache,id=hive-go-build,target=/tmp/gobuild \
GOCACHE=/tmp/gobuild GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 go build -o /hive-backup ./cmd/hive-backup && \
GOCACHE=/tmp/gobuild GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 go build -o /hivectl ./cmd/hivectl && \
GOCACHE=/tmp/gobuild GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 go build -o /crux ./cmd/crux && \
GOCACHE=/tmp/gobuild GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 go build -o /hive-whisper ./cmd/hive-whisper
GOCACHE=/tmp/gobuild GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 go build -o /hive-whisper ./cmd/hive-whisper && \
GOCACHE=/tmp/gobuild GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 go build -o /hive-pdftext ./cmd/hive-pdftext


# --- Go security scanners for the sec-check lane (#18192) ---
Expand Down Expand Up @@ -738,6 +739,9 @@ COPY --from=builder /crux /usr/local/bin/crux
# whisper.cpp + LGPL FFmpeg engine from the whisper-builder stage. They run as
# the dedicated hive-transcribe uid (created below), never as dev.
COPY --from=builder /hive-whisper /usr/local/bin/hive-whisper
# Inbox PDF uploads are parsed out of process by the hive-pdftext sandbox
# (pure Go, github.com/ledongthuc/pdf), as the same hive-transcribe uid.
COPY --from=builder /hive-pdftext /usr/local/bin/hive-pdftext
COPY --from=whisper-builder /opt/hive-whisper /opt/hive-whisper

# Go security scanners for the sec-check lane (#18192); built in the
Expand Down
59 changes: 59 additions & 0 deletions src/cmd/hive-pdftext/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Command hive-pdftext is the sandbox the Inbox reads one uploaded PDF in.
// The hive starts it as the hive-transcribe uid through su-exec, with an
// empty environment, under prlimit and nice and, where the kernel allows it,
// in an empty network namespace; the PDF arrives on stdin and the text
// leaves as JSON on stdout. See pkg/inbox/pdfbox and
// src/docs/inbox-uploads.md.
package main

import (
"context"
"errors"
"flag"
"fmt"
"io"
"os"
"os/signal"
"syscall"

"github.com/hivecommons/hive/pkg/inbox/pdfbox"
)

// version is the sandbox protocol version printed by -version.
const version = "hive-pdftext 1"

func main() { os.Exit(runMain(os.Args[1:])) }

func runMain(args []string) int {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
defer stop()
return run(ctx, args, os.Stdin, os.Stdout, os.Stderr)
}

// run parses args and reads one PDF.
func run(ctx context.Context, args []string, stdin io.Reader, stdout, stderr io.Writer) int {
var o pdfbox.Options
fs := flag.NewFlagSet("hive-pdftext", flag.ContinueOnError)
fs.SetOutput(stderr)
showVersion := fs.Bool("version", false, "print the version and exit")
fs.Int64Var(&o.MaxBytes, "max-bytes", 0, "largest PDF accepted on stdin (0: no cap)")
fs.IntVar(&o.MaxPages, "max-pages", pdfbox.DefaultMaxPages, "pages read at most")
fs.IntVar(&o.MaxChars, "max-chars", pdfbox.DefaultMaxChars, "characters returned at most")
fs.StringVar(&o.WorkRoot, "work-root", "", "where the private work dir is created (default: $TMPDIR or /tmp)")
if err := fs.Parse(args); err != nil {
return pdfbox.ExitUsage
}
if *showVersion {
fmt.Fprintln(stdout, version)
return pdfbox.ExitOK
}
if err := pdfbox.Run(ctx, o, stdin, stdout); err != nil {
fmt.Fprintln(stderr, "hive-pdftext: "+err.Error())
var pe *pdfbox.Error
if errors.As(err, &pe) {
return pe.Code
}
return pdfbox.ExitSetup
}
return pdfbox.ExitOK
}
48 changes: 48 additions & 0 deletions src/cmd/hive-pdftext/main_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package main

import (
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"

"github.com/hivecommons/hive/pkg/inbox/pdfbox"
)

func TestRunVersionUsageAndExtract(t *testing.T) {
var out, errb bytes.Buffer
if code := run(context.Background(), []string{"-version"}, nil, &out, &errb); code != pdfbox.ExitOK || !strings.HasPrefix(out.String(), "hive-pdftext ") {
t.Fatalf("-version = %d %q", code, out.String())
}
if code := run(context.Background(), []string{"-nope"}, nil, &out, &errb); code != pdfbox.ExitUsage {
t.Fatalf("bad flag = %d", code)
}
data, err := os.ReadFile(filepath.Join("..", "..", "pkg", "inbox", "pdfbox", "testdata", "text.pdf"))
if err != nil {
t.Fatal(err)
}
out.Reset()
if code := run(context.Background(), []string{"-work-root", t.TempDir()}, bytes.NewReader(data), &out, &errb); code != pdfbox.ExitOK {
t.Fatalf("extract = %d %s", code, errb.String())
}
var res pdfbox.Result
if err := json.Unmarshal(out.Bytes(), &res); err != nil || res.Pages != 2 || !res.HasText() {
t.Fatalf("result = %+v, %v", res, err)
}
errb.Reset()
if code := run(context.Background(), []string{"-max-bytes", "10", "-work-root", t.TempDir()}, bytes.NewReader(data), &out, &errb); code != pdfbox.ExitTooLarge || !strings.Contains(errb.String(), "larger") {
t.Fatalf("too large = %d %q", code, errb.String())
}
if code := run(context.Background(), []string{"-work-root", t.TempDir()}, strings.NewReader("%PDF-1.4 garbage"), &out, &errb); code != pdfbox.ExitParse {
t.Fatalf("garbage = %d", code)
}
}

func TestRunMainVersion(t *testing.T) {
if code := runMain([]string{"-version"}); code != pdfbox.ExitOK {
t.Fatalf("runMain -version = %d", code)
}
}
2 changes: 1 addition & 1 deletion src/docs/api-reference/knowledge.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f
| `GET` | `/api/inbox/transcriptions` | Dashboard auth | Live progress of queued and running transcriptions in queue order (`{active: [{source, id, state}]}` with `phase`, `progress`, `estimated`, `started_at`, `audio_seconds`, `processed_seconds`, `eta_seconds`, `queue_position`); reads no source, so the Inbox polls it every 3 s while a job runs; `Cache-Control: no-store`; see [inbox-transcription.md](../inbox-transcription.md) | `pkg/dashboard/api_inbox.go:34` |
| `GET` | `/api/inbox/{source}/{id}/audio` | Owner or read-write | Mint a fresh short-lived audio/video link for one item (`{url, expires_at}`; Plaud: presigned, 24h; Zoom: audio-only or video `download_url` with a one-hour download token; Webex: the recording's temporary direct audio (else MP4) download link, valid 3 hours; Teams and Google Meet: 404, no browser-playable link; Slack huddles: 404, Slack keeps no huddle audio; Otter: the conversation's MP3 download link (`GET /conversations/{id}/audio`), minted per click; Granola: 404, Granola keeps no meeting audio; Limitless: 404, Pendant audio is served only behind the API key; Fathom: the signed url of a recording download (`POST /recordings/{id}/download`, audio-only recordings complete at once, video is polled), valid about 24 hours; Fireflies: the transcript's hashed `audio_url` (else `video_url`), valid 24 hours, Pro plan or higher); never stored by the hive, replied with `Cache-Control: no-store`, audited as `inbox_audio` (403 role, 404 unknown item or no audio, 502 source unreachable) | `pkg/dashboard/api_inbox.go:30` |
| `GET` | `/api/inbox/{source}/{id}/audio/stream` | Owner or read-write | Stream one item's audio from the dashboard origin, for the Inbox player (the CSP refuses cross-origin media). The hive mints the link as above, fetches it server-side and copies the bytes with the media `Content-Type` (from storage, else the link's extension); a single `Range: bytes=…` is passed through (206 + `Content-Range`; 200 whole file when storage ignores it; 416 passed through). The link stays in memory for one playback's Range requests (10 min, never past expiry), is never logged, returned or stored, and is re-minted once when storage refuses it; nothing is written to disk. `Cache-Control: no-store`; audited as `inbox_audio_stream` once per minted link (403 role, 404 unknown item or no audio, 413 over the 2 GiB cap, 502 source or storage unreachable, or link refused/expired) | `pkg/dashboard/api_inbox.go:31` |
| `POST` | `/api/inbox/uploads` | Owner or read-write | Add dropped files to the Inbox: `multipart/form-data` with one or more `file` fields (at most `inbox.uploads.max_files_per_drop`, default 20, each up to `inbox.uploads.max_file_bytes`, default 25 MiB). Each file is typed by content (magic bytes, UTF-8, a parse check), never by name or client MIME alone; md, txt, json, jsonl, yaml, csv, tsv, html and docx become `upload` items with extracted markdown and a Knowledge fact. One file: 201 `{results: [{filename, ok, source, item_id, title, kind}]}` or its refusal (400 `empty`/`too_many_files`, 413 `too_large`, 415 `unsupported_type`, 422 `parse_failed`, 507 `quota_exceeded`, each with `code` and `error`); several files: 200 with per-file results unless all failed. Cross-site requests are refused; every file is audited as `inbox_upload` (403 role, 404 no upload store); see [inbox-uploads.md](../inbox-uploads.md) | `pkg/dashboard/api_inbox_uploads.go:27` |
| `POST` | `/api/inbox/uploads` | Owner or read-write | Add dropped files to the Inbox: `multipart/form-data` with one or more `file` fields (at most `inbox.uploads.max_files_per_drop`, default 20, each up to `inbox.uploads.max_file_bytes`, default 25 MiB). Each file is typed by content (magic bytes, UTF-8, a parse check), never by name or client MIME alone; md, txt, json, jsonl, yaml, csv, tsv, html, docx and pdf (parsed in the `hive-pdftext` sandbox) become `upload` items with extracted markdown and a Knowledge fact. One file: 201 `{results: [{filename, ok, source, item_id, title, kind}]}` or its refusal (400 `empty`/`too_many_files`, 413 `too_large`, 415 `unsupported_type`, 422 `parse_failed`, 507 `quota_exceeded`, each with `code` and `error`); several files: 200 with per-file results unless all failed. Cross-site requests are refused; every file is audited as `inbox_upload` (403 role, 404 no upload store); see [inbox-uploads.md](../inbox-uploads.md) | `pkg/dashboard/api_inbox_uploads.go:27` |
| `POST` | `/api/inbox/uploads/url` | Owner or read-write | Add a link: `{url}`. The hive fetches it under the SSRF guard (http/https only, no credentials, its own DNS lookup with the connection pinned to the checked address, private/loopback/link-local/CGNAT/multicast/ULA/metadata ranges refused, every redirect re-checked, at most 3, body capped at `inbox.uploads.max_file_bytes`, 60 s) and runs the body through the upload type pipeline (an HTML page becomes readable markdown titled by its `<title>`). 201 `{results: [{filename: host, ok, source, item_id, title, kind}]}`; 400 `link_refused`, 413, 415, 422, 502 `fetch_failed`, 507; audited as `inbox_upload` with `origin=link` and the host (403 role, 404 no upload store) | `pkg/dashboard/api_inbox_uploads.go:28` |
| `POST` | `/api/inbox/recordings` | Owner or read-write | Start a dashboard recording (● Record): `{title, mime: audio/webm\|audio/mp4, consent_version, includes_tab_audio}`. `consent_version` must name the current consent notice; it is stored with the user in the recording's metadata. One recording per user at a time. 201 `{id, max_chunk_bytes, max_seconds, retention_days, transcribes}`; audited as `inbox_recording_start` (400 consent missing or stale, 403 role, 409 `{active_id}` already recording, 412 `inbox.recording.enabled` off, 415 other media types, 507 `inbox.recording.quota_bytes` used up); see [inbox-recording.md](../inbox-recording.md) | `pkg/dashboard/api_inbox_recordings.go:20` |
| `POST` | `/api/inbox/recordings/{id}/chunks` | The recorder (owner or read-write) | Store one MediaRecorder chunk `?seq=N`, the raw bytes as the body (at most `max_chunk_bytes`, 512 KiB). In order only; each part is fsynced before the 200 `{id, seq, next_seq, bytes, duplicate?}`. The first chunk must start with a WebM or MP4 signature; the hive never decodes audio. A re-send with the same bytes is acknowledged again (`duplicate: true`) (400 bad seq, 403 someone else's, 409 `{expected_seq}` gap, other bytes for a stored seq, or already ended, 413 chunk over 512 KiB or past `max_bytes` / `max_seconds`, 415 not WebM/MP4, 507 quota) | `pkg/dashboard/api_inbox_recordings.go:21` |
Expand Down
3 changes: 3 additions & 0 deletions src/docs/env-vars.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,9 @@ behavior. [Maintenance limitations](#keeping-this-reference-current) still apply
| `HIVE_INBOX_UPLOADS_MAX_FILES_PER_DROP` | No | `20` | Overrides `inbox.uploads.max_files_per_drop`: files in one drop or one upload request. |
| `HIVE_INBOX_UPLOADS_DOCUMENT_RETENTION_DAYS` | No | `0` | Overrides `inbox.uploads.document_retention_days`: days after which a text document's original is deleted (its extracted text and item stay); unset or `0` keeps originals. |
| `HIVE_INBOX_UPLOADS_MEDIA_RETENTION_DAYS` | No | `5` | Overrides `inbox.uploads.media_retention_days`: days after which an audio or video upload's original is deleted. |
| `HIVE_PDF_SANDBOX` | No | `/usr/local/bin/hive-pdftext` | The sandbox that reads PDFs dropped onto the Inbox out of process ([inbox-uploads.md](inbox-uploads.md)). |
| `HIVE_PDF_RUN_AS` | No | `hive-transcribe` | The uid the PDF sandbox runs as through su-exec; `self` runs it as the hive's own uid (development only). |
| `HIVE_PDF_MEMORY_MB` | No | `1024` | Address-space limit (prlimit `--as`) of the PDF sandbox, in MiB. |
| `HIVE_INBOX_TRANSCRIPTION_DAILY_MAX_ITEMS` | No | `20` | Overrides `inbox.transcription.daily_max_items`: transcription runs per UTC day. |
| `GEMINI_API_KEY` / `GOOGLE_API_KEY` / `GOOGLE_GENAI_API_KEY` | No | none | The hive's own Gemini API key, read in this order. Gemini model discovery already uses it; Inbox transcription with `inbox.transcription.provider: gemini` (or `auto` when whisper is unavailable) sends recordings to the Gemini API with it, in the `x-goog-api-key` header. Never stored in `hive.yaml`, never shown ([inbox-transcription.md](inbox-transcription.md)). |
| `HIVE_WHISPER_MODEL_DIR` | No | `/data/models/whisper` | Where the built-in whisper provider keeps its ggml model (`ggml-<model>.bin`, mode 0644). Air-gapped hives pre-seed the file here; it is checked against its pinned size and SHA-256 before use ([inbox-transcription.md](inbox-transcription.md)). |
Expand Down
Loading
Loading