Repository navigation
fix: sandbox silent-failure hardening (returncode check, boot-time docker check, token redaction) - #44
Merged
Conversation
…ocker check, token redaction
Adopts the two findings from the 2026-10-07 MWF sandbox-outage receipt
(products/pdfmux/decisions/2026-10-07-mwf-tick-receipt.md) that were
deliberately left unshipped pending review.
1. run_parse() never checked proc.returncode. When `docker run` failed for
an infra reason (image missing, daemon down, OOM), stdout was empty, so
json.loads(... or "{}") silently returned {} — no "error" key — and the
caller got SandboxResult(data={}) as if the parse succeeded. This is why
the outage ran unnoticed for days: pm2's error log stayed at 0 bytes.
Now a non-zero exit logs a server-side diagnostic (stderr tail + exit
code, never user-facing) and raises SandboxError("This PDF couldn't be
read.") — reusing the existing neutral string from the JSON-decode path.
2. Added assert_sandbox_ready(), called from create_app(), which runs
`docker image inspect` on DOCKER_IMAGE at boot when PDFMUX_SANDBOX=docker.
A missing/broken image now crashes at startup instead of failing silently
on every request.
3. The plugin token is the secret path segment in `POST /mcp/<token>`, and
uvicorn's access logger recorded the full path in plaintext — a live
credential landing in root-only pm2 logs. Added RedactTokenFilter, wired
onto uvicorn's "access" log handler via a custom log_config (a filter
attached to the logger object directly would be wiped when uvicorn's own
dictConfig() call rebuilds it), so `/mcp/<token>` is logged as
`/mcp/<redacted>`.
Tests: 9 new cases covering the returncode-failure path (safe message vs.
logged diagnostic), the boot-time assertion (noop/pass/image-missing/
docker-missing), create_app() wiring, and the redaction filter (unit +
dictConfig-survival integration). Full suite: 856 passed, 7 skipped
(matching CI's `[dev,serve,remote]` extras exactly).
Scope: code + tests only, merged to main. NOT deployed to mc.pdfmux.com —
no VPS access this session; the currently-running image keeps serving the
unpatched code until the next session with VPS access redeploys.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adopts the two findings from the 2026-10-07 MWF sandbox-outage receipt (
products/pdfmux/decisions/2026-10-07-mwf-tick-receipt.md, Drumworks business-agents repo) that were deliberately left unshipped pending review — the receipt's own words: "shipping them unreviewed would repeat the failure mode that caused today's outage." Dispatch packet:shared/dispatch-packets/2026-10-08/01-pdfmux-sandbox-silent-failure-hardening-cto-agent.md.That outage is already fixed in production (image built, live call verified). This PR closes the silent-failure gap that let it run unnoticed for days — it's hardening, not an active incident fix.
Fix 1 —
run_parse()never checkedproc.returncode(src/pdfmux/remote/sandbox.py:97-112)When
docker runfailed for an infra reason (image missing, daemon down, registry denied, OOM),proc.stdoutwas empty, sojson.loads(proc.stdout.decode() or "{}")silently returned{}— no"error"key — and the caller gotSandboxResult(data={})as if the parse succeeded. This is exactly why the outage ran unnoticed for days: pm2's error log stayed at 0 bytes.Now: non-zero exit → log a server-side diagnostic (stderr tail + exit code,
sandbox.py:106-111) and raiseSandboxError("This PDF couldn't be read.")— reusing the existing neutral string from the JSON-decode-failure branch, never inventing new user-facing copy.Also added
assert_sandbox_ready()(sandbox.py:124-151), called fromcreate_app()(server.py:498): whenPDFMUX_SANDBOX=docker, runsdocker image inspect <DOCKER_IMAGE>at boot and raises loudly if it's missing/unreachable. This is the change that turns a multi-day silent outage into an immediate boot-time crash.Fix 2 — plugin token leaked into uvicorn's access log (
src/pdfmux/remote/server.py)The token is the secret path segment in
POST /mcp/<token>; uvicorn's access logger records the full request path, so every call wrote a live credential in plaintext to root-only pm2 logs. Severity is low (logs are root-only; the token itself is unchanged) but it violates this codebase's "tokens never land in plaintext logs" discipline.Added
RedactTokenFilter(server.py:523-534), wired onto uvicorn'saccesshandler via a customlog_configdict (_log_config(),server.py:537-552) passed touvicorn.run()(server.py:555-564). A filter attached directly to the logger object beforehand would be wiped — uvicorn's ownConfig.configure_logging()callslogging.config.dictConfig()at server start, which rebuilds any logger named in the config (here,uvicorn.access) from scratch. Declaring the filter inside the dict survives that./mcp/<token>now logs as/mcp/<redacted>.Test plan
tests/test_remote_server.py: non-zero-returncode path (safe message vs. logged diagnostic),assert_sandbox_ready()(noop outside docker mode / passes when image present / raises when image missing / raises when docker binary missing),create_app()wiring,RedactTokenFilterunit behavior, and adictConfig-survival integration test proving the filter actually reaches a real emitted access-log line.[dev,serve,remote]): 856 passed, 7 skipped, 0 failed.ruff check/ruff format --check/pyright src/pdfmuxall clean.endpoint-audit.mjs(per business-agents CLAUDE.md's mandatory deploy checklist): 0 BLOCKING findings. Caveat for the record: that script is a Next.jsapp/api-route scanner; this repo has noapp//pages/directories, so it scanned 0 files — the 0-BLOCKING result reflects "nothing in its scan scope," not an endpoint-specific pass on this Starlette server.Scope note
Code + tests only, merging to
main. Not deployed — this session has no VPS/Tailscale access tomc.pdfmux.com(confirmed, out of scope per the dispatch packet). The currently-running image keeps serving the unpatched (but today correctly-working) code until the next session with VPS access redeploys.🤖 Generated with Claude Code