Repository navigation
✨ Inbox: read PDFs in a sandboxed hive-pdftext process - #18426
Conversation
PDF text is extracted out of process by the new hive-pdftext command (pure Go, github.com/ledongthuc/pdf, BSD-3-Clause, already in go.mod): as the hive-transcribe uid via su-exec, empty environment, prlimit + nice, empty network namespace where available, private 0700 work dir, page and character caps, and a wall-clock timeout that kills the process group. Scanned PDFs without a text layer are refused with a clear message. Signed-off-by: Andy Anderson <andy@clubanderson.com>
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
Mutation testinggremlins found 14 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.
9 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the
Verdict: |
Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. @clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
605f7d7
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
There was a problem hiding this comment.
Decision needed: @clubanderson should confirm that a silently-disabled network namespace (fail-open) is acceptable for the PDF sandbox, since the PR also touches the Dockerfile and the security model.
Read upload_pdf.go, upload_source.go, upload_types.go and the Dockerfile hunk in the diff. The design (su-exec to hive-transcribe, empty env, prlimit, process-group kill, capped stdout) looks sound to me. One verified concern:
Network isolation can be disabled for the life of the process by one cancelled request (low–medium) — src/pkg/inbox/upload_pdf.go, isolation() (the isoOnce.Do block). The probe runs exec.CommandContext(ctx, …) with the first caller's request context, and r.iso = probe.Run() == nil. If that context is cancelled or times out (client disconnect, or the 60 s Run timeout) before the probe finishes, Run() errors, iso is stored as false, and sync.Once never retries. Every later PDF then runs without unshare --user --net, with no log line saying so. The sandbox still has uid/prlimit/env limits, but the "empty network namespace" the PR body and security-model.md promise would silently not apply. Suggest probing with context.Background() plus a short timeout, and logging when isolation is unavailable.
Minor, not a defect: Run reports any ctx.Err() != nil as "took longer than %s", so a cancelled client shows the timeout message.
Because this adds a privileged-uid exec path and edits src/Dockerfile and security-model.md, it needs a maintainer's sign-off on the fail-open behaviour. I did not review the pdfbox parser internals or the test files in depth. Several CI jobs were still in progress when I read it.
— hive: agent=reviewer-2 backend=claude model=claude-sonnet-5-5 claude=2.1.325
|
[APPROVALNOTIFIER] This PR is APPROVED Approval requirements bypassed by manually added approval. This pull-request has been approved by: The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Fourth PR of drag-and-drop ingest: PDFs, parsed out of process.
What it does
%PDF-and named.pdf(or with no extension), or a link servingapplication/pdf, becomes apdfitem: one## Page Nsection per page with text, a page-count summary, and the Info title. A PDF with no text layer is refused with "no text layer (scanned PDF); OCR not supported yet".github.com/ledongthuc/pdf(BSD-3-Clause, pure Go). It is already ingo.modand NOTICE (the knowledge document parser uses it), so no new dependency. Popplerpdftotext(GPL) was avoided.hive-pdftextcommand (built and shipped in the hive image next tohive-whisper) is the only place an uploaded PDF is parsed. The hive runs it:hive-transcribeuid viasu-exec, with an environment of onlyPATHandLC_ALL;prlimit(1 GiB address space, CPU = timeout + 10 s, file size = the PDF's size, 32 open files, no core dumps) andnice -n 10;unshare --user --net) when the kernel allows it (probed once);HIVE_PDF_SANDBOX,HIVE_PDF_RUN_AS,HIVE_PDF_MEMORY_MBoverride path, uid and memory. Without the sandbox the upload is refused with a clear message.Tests
pkg/inbox/pdfboxwith hand-built fixtures: a two-page text PDF with an Info title, a scanned-like PDF without text, page/char/byte caps, empty and malformed input, and the private work dir (0700, removed after the run).cmd/hive-pdftext: flags, version, exit codes.🤖 Generated with Claude Code