Skip to content

fix: sandbox silent-failure hardening (returncode check, boot-time docker check, token redaction) - #44

Merged
NameetP merged 1 commit into
mainfrom
fix/sandbox-silent-failure-hardening
Oct 8, 2026
Merged

NameetP merged 1 commit into
mainfrom
fix/sandbox-silent-failure-hardening

Conversation

@NameetP

@NameetP NameetP commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Summary

Adopts the two findings from the 2026-10-07 MWF sandbox-outage receipt (products/pdfmux/decisions/2026-10-07-mwf-tick-receipt.md, Drumworks business-agents repo) that were deliberately left unshipped pending review — the receipt's own words: "shipping them unreviewed would repeat the failure mode that caused today's outage." Dispatch packet: shared/dispatch-packets/2026-10-08/01-pdfmux-sandbox-silent-failure-hardening-cto-agent.md.

That outage is already fixed in production (image built, live call verified). This PR closes the silent-failure gap that let it run unnoticed for days — it's hardening, not an active incident fix.

Fix 1 — run_parse() never checked proc.returncode (src/pdfmux/remote/sandbox.py:97-112)

When docker run failed for an infra reason (image missing, daemon down, registry denied, OOM), proc.stdout was empty, so json.loads(proc.stdout.decode() or "{}") silently returned {} — no "error" key — and the caller got SandboxResult(data={}) as if the parse succeeded. This is exactly why the outage ran unnoticed for days: pm2's error log stayed at 0 bytes.

Now: non-zero exit → log a server-side diagnostic (stderr tail + exit code, sandbox.py:106-111) and raise SandboxError("This PDF couldn't be read.") — reusing the existing neutral string from the JSON-decode-failure branch, never inventing new user-facing copy.

Also added assert_sandbox_ready() (sandbox.py:124-151), called from create_app() (server.py:498): when PDFMUX_SANDBOX=docker, runs docker image inspect <DOCKER_IMAGE> at boot and raises loudly if it's missing/unreachable. This is the change that turns a multi-day silent outage into an immediate boot-time crash.

Fix 2 — plugin token leaked into uvicorn's access log (src/pdfmux/remote/server.py)

The token is the secret path segment in POST /mcp/<token>; uvicorn's access logger records the full request path, so every call wrote a live credential in plaintext to root-only pm2 logs. Severity is low (logs are root-only; the token itself is unchanged) but it violates this codebase's "tokens never land in plaintext logs" discipline.

Added RedactTokenFilter (server.py:523-534), wired onto uvicorn's access handler via a custom log_config dict (_log_config(), server.py:537-552) passed to uvicorn.run() (server.py:555-564). A filter attached directly to the logger object beforehand would be wiped — uvicorn's own Config.configure_logging() calls logging.config.dictConfig() at server start, which rebuilds any logger named in the config (here, uvicorn.access) from scratch. Declaring the filter inside the dict survives that. /mcp/<token> now logs as /mcp/<redacted>.

Test plan

  • 9 new tests in tests/test_remote_server.py: non-zero-returncode path (safe message vs. logged diagnostic), assert_sandbox_ready() (noop outside docker mode / passes when image present / raises when image missing / raises when docker binary missing), create_app() wiring, RedactTokenFilter unit behavior, and a dictConfig-survival integration test proving the filter actually reaches a real emitted access-log line.
  • Full suite with CI's exact extras ([dev,serve,remote]): 856 passed, 7 skipped, 0 failed.
  • ruff check / ruff format --check / pyright src/pdfmux all clean.
  • endpoint-audit.mjs (per business-agents CLAUDE.md's mandatory deploy checklist): 0 BLOCKING findings. Caveat for the record: that script is a Next.js app/api-route scanner; this repo has no app//pages/ directories, so it scanned 0 files — the 0-BLOCKING result reflects "nothing in its scan scope," not an endpoint-specific pass on this Starlette server.

Scope note

Code + tests only, merging to main. Not deployed — this session has no VPS/Tailscale access to mc.pdfmux.com (confirmed, out of scope per the dispatch packet). The currently-running image keeps serving the unpatched (but today correctly-working) code until the next session with VPS access redeploys.

🤖 Generated with Claude Code

…ocker check, token redaction

Adopts the two findings from the 2026-10-07 MWF sandbox-outage receipt
(products/pdfmux/decisions/2026-10-07-mwf-tick-receipt.md) that were
deliberately left unshipped pending review.

1. run_parse() never checked proc.returncode. When `docker run` failed for
   an infra reason (image missing, daemon down, OOM), stdout was empty, so
   json.loads(... or "{}") silently returned {} — no "error" key — and the
   caller got SandboxResult(data={}) as if the parse succeeded. This is why
   the outage ran unnoticed for days: pm2's error log stayed at 0 bytes.
   Now a non-zero exit logs a server-side diagnostic (stderr tail + exit
   code, never user-facing) and raises SandboxError("This PDF couldn't be
   read.") — reusing the existing neutral string from the JSON-decode path.

2. Added assert_sandbox_ready(), called from create_app(), which runs
   `docker image inspect` on DOCKER_IMAGE at boot when PDFMUX_SANDBOX=docker.
   A missing/broken image now crashes at startup instead of failing silently
   on every request.

3. The plugin token is the secret path segment in `POST /mcp/<token>`, and
   uvicorn's access logger recorded the full path in plaintext — a live
   credential landing in root-only pm2 logs. Added RedactTokenFilter, wired
   onto uvicorn's "access" log handler via a custom log_config (a filter
   attached to the logger object directly would be wiped when uvicorn's own
   dictConfig() call rebuilds it), so `/mcp/<token>` is logged as
   `/mcp/<redacted>`.

Tests: 9 new cases covering the returncode-failure path (safe message vs.
logged diagnostic), the boot-time assertion (noop/pass/image-missing/
docker-missing), create_app() wiring, and the redaction filter (unit +
dictConfig-survival integration). Full suite: 856 passed, 7 skipped
(matching CI's `[dev,serve,remote]` extras exactly).

Scope: code + tests only, merged to main. NOT deployed to mc.pdfmux.com —
no VPS access this session; the currently-running image keeps serving the
unpatched code until the next session with VPS access redeploys.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@NameetP
NameetP merged commit 04e002d into main Oct 8, 2026
7 checks passed
@NameetP
NameetP deleted the fix/sandbox-silent-failure-hardening branch October 8, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant