Skip to content

fix(platform): harden lib reliability, authorization, and dead-end doors - #3183

Draft
larryro wants to merge 9 commits into
mainfrom
fix/lib-reliability-hardening
Draft

fix(platform): harden lib reliability, authorization, and dead-end doors#3183
larryro wants to merge 9 commits into
mainfrom
fix/lib-reliability-hardening

Conversation

@larryro

@larryro larryro commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Batch of verified medium-severity reliability / authorization / dead-end defects in the platform libs and their doors (source: the backend deep-review triage, lib / tts / node_only / onedrive rows). Base: origin/main @ 899fcc0. Every still-live finding carries a regression test that is red on the base tree and green here.

Per-finding outcome

# Finding Outcome Where
1 SOPS decrypt runs execFileSync on the request path fixed — both verbs spawn sops async (execFile), 10s timeout kills a wedged child with a legible error, concurrent cold reads of one file share one child; encryptJsonWithSops is now async (4 callers awaited) core/lib/sops.ts
2 Image uploads end as rag_status 'failed' — Indexing skipped (empty) fixed — an image (vision seam retired, no extractor can yield text) is stamped unsupported up front (terminal, no retry affordance, no blob fetch); docs (en/de/fr) name images under Unsupported domains/knowledge/service.ts, extraction/router.ts (isImageFile)
3 readJsonFile labels every stat() failure not_found fixed — only ENOENT/ENOTDIR are not_found; EACCES/EPERM/EIO surface as inaccessible (callers already handle it; readGovernancePolicy now warns instead of silently defaulting) core/lib/file_io.ts
4 TTS synthesis never checks messageId ∈ thread; audio serve org-gated while chunk listing owner-gated fixedsynthesizeChunk refuses (403 + tts.synthesize_denied audit, reason message_not_in_thread) a message that is not a row of the owned thread, before any reservation; getChunkForServe and /usage go through the same loadOwnedThread gate as the listing door domains/tts/service.ts, domains/tts/routes.ts
5 Legacy resolveOrgObjectStore swallows default-tree errors and falls back to the dead Convex backend fixed — one fail-closed resolver + one cache: a broken default tree surfaces as its own error, "neither configured" throws ObjectStoreUnconfiguredError; the Convex store variant and blob_access's dead ctx.storage.* branches are retired; lib/object-store.ts delegates (a config write now invalidates every cached resolution) core/lib/storage/object_store.ts, blob_access.ts, lib/object-store.ts
6 Render batch output budget counts UTF-16 chars post-hoc; pages.json can exceed the 20MB read cap fixed — the worker budgets the serialized file in UTF-8 bytes BEFORE admitting each page (per-page bound in bytes too); a page that does not fit is handed back not_attempted for the next batch core/node_only/sandbox/render_fetch.ts
7 Exec drain retries only ever attach; a lost first POST burns the whole budget fixed — an attach the spawner answers with exec <id> not found while nothing was consumed re-POSTs the exec (ExecNotFoundError); never after progress (no double-run), never on a resume core/node_only/sandbox/helpers/session_client.ts
8 30-min claim staleness lets a live long sync run twice fixed — a 5-min heartbeat renews updated_at_ms while the run is alive (cleared in finally), so only a dead worker's claim ever reads stale domains/onedrive/service.ts (shared by the Google Drive adapter)
9 Provider credential endpointUrl update skips the https validation create enforces already fixed on base — updateCredential rejects non-https (provider_credentials/service.ts:461-467)

Nothing deferred; no migration needed.

Tests (all red on base, green here)

  • core/lib/sops.test.ts (new, 11): timer fires while decrypt is in flight (base: loop blocked 637ms); hung sops killed at the timeout (base: 10s spawnSync ETIMEDOUT); 5 concurrent cold reads → 1 child (base: 5); encrypt off-loop; cache/invalidation, plaintext, no-key refusal; real sops+age round trip when installed.
  • core/lib/file_io.test.ts (+4): ENOENT and ENOTDIR stay not_found; EACCES → inaccessible (base: not_found).
  • core/lib/storage/object_store.test.ts (+6): unconfigured → ObjectStoreUnconfiguredError (base: resolved { backend: 'convex' }); corrupt default tree → its own error (base: swallowed → convex); default/own precedence; missing credentials; cache clear.
  • domains/tts/service.access-gate.test.ts (new, 4): foreign message → 403 + audit, no reservation (base: proceeds); owned message passes; audio serve owner → chunk, stranger → null (base: served).
  • domains/knowledge/service.image-status.test.ts (new, 2) + extraction/router.test.ts (+2): image → single unsupported write, no blob fetch (base: failed); text still extracts.
  • core/node_only/sandbox/render_fetch.test.ts (+2): the staged worker run under node against a fake playwright-core: pages.json ≤ cap, third page handed back (base: 14151 B over a 10000 B cap); per-page bound in bytes (base: 1700 'é' accepted under a 3000-byte bound).
  • core/node_only/sandbox/helpers/session_client.test.ts (+2): lost POST → attach(not found) → POST again completes (base: exec exec-3 not found after the budget); after progress a not-found attach fails, never re-POSTs.
  • domains/onedrive/service.claim-heartbeat.test.ts (new, 2): 3 renewals over 3 periods, none after the run (base: 0); no heartbeat without a claim.
  • Integration harness (backend/integration-check.ts): TTS section now synthesizes a REAL app.messages row and probes foreign-message 403 + audit + no squatting rows + stranger audio 404 / chunks []; knowledge section uploads a PNG → unsupported; onedrive section runs a long sync with heartbeatMs: 100, ages the claim 31 min, and proves the next job no-ops (listCalls stays 1, one document).

Verification

  • bunx tsc --noEmit (platform): exit 0.
  • bunx oxlint --type-aware (platform): exit 0.
  • Backend unit suite (vitest --project server backend): 239 files / 2539 tests green.
  • backend:integration on fresh throwaway tale-db + MinIO (SANDBOX_LLM_GATEWAY_ADMIN_PASSWORD set): branch 381/381 checks passed (exit 0; the three new probes: foreignMessage=403 (audited=1, squatRows=0) strangerAudio=404 strangerChunks=0, image status=unsupported, claim stampAge=0s listCalls=1) · base @ 899fcc0 with the same harness edits: 378/381 checks passed (exit 1) — the only failures are the three new probes, each showing the pre-fix defect: TTS foreignMessage=200 (audited=0, squatRows=1) strangerAudio=200; knowledge image status=failed, error=Indexing skipped (empty).; onedrive stampAge=1860s listCalls=2.

Cross-class discoveries (not fixed here)

  • Fully scanned PDFs still index only --- Page N --- markers as completed (sibling of update dashboard ui (#508) #2; needs the vision lane or a "no indexable text" outcome).
  • A non-image extraction that yields no text (empty .txt, image-only .docx) still lands as failed — Indexing skipped (empty); there is no "nothing to index" status in the rag_status CHECK.
  • generateReplacementBlobUpload / ReplacementBlobUploadHandoff have no consumers in 0.5 (dead export, kept).
  • docs/en/platform/knowledge/documents.md:18 says the legacy Office trio shows Not indexed while line 82 says Unsupported (pre-existing inconsistency).

decryptSecretsFile shelled out with execFileSync on the request path (every per-org object-store resolve on cache expiry, every credential save), stalling the single-threaded event loop for the whole decrypt — up to the 10s timeout when sops hung. Both verbs now spawn sops with execFile, kill a wedged child at the timeout with a legible error, and concurrent cold reads of one file share a single child. encryptJsonWithSops is async; its four callers await it.
readJsonFile mapped every stat() failure to not_found, so a mis-permissioned config volume read as an absent file and callers (readGovernancePolicy among them) silently fell back to defaults. Only ENOENT/ENOTDIR are not_found now; EACCES/EPERM/EIO surface as inaccessible, which the callers already handle by warning.
The reused resolveOrgObjectStore swallowed a broken default tree (.catch(() => null)) and fell back to the retired Convex store, whose ctx.storage.* verbs no longer exist — a real misconfiguration surfaced as a TypeError deep in the video, harvest, and transcription lanes. One fail-closed resolver and one cache now serve both the 0.5 lane and blob_access: a broken default tree throws its own error, neither-configured throws ObjectStoreUnconfiguredError, the Convex variant and its dead branches are gone, and a config write invalidates every cached resolution.
Images route to the vision extractor and the vision seam is retired, so every image upload extracted nothing and landed as rag_status 'failed — Indexing skipped (empty)' with a retry affordance that could only fail again. An image is now stamped 'unsupported' up front — the terminal, honest state the badge already renders — before any bytes are fetched. Docs (en/de/fr) name images under Unsupported.
The render worker summed html.length after storing each page and only broke before the next one, so pages.json could exceed the 20MB sessionReadFile cap (more so with multibyte text); the host then saw no output and the crawl retried the same batch forever. The worker now budgets the serialized file in UTF-8 bytes before admitting each page, applies the per-page bound in bytes, and hands a page that does not fit back as not attempted for the next batch.
Every drain retry attached, so when the initial exec POST never reached the spawner the exec did not exist and each attach answered 'exec <id> not found' until the whole budget burned against a healthy session. An attach answered not-found while nothing was consumed now re-POSTs the exec (ExecNotFoundError); never after progress, so a turn is never run twice, and never on a resume.
runSyncConfigJobWith treated a 'running' stamp older than 30 minutes as a crashed worker but nothing refreshed the stamp during a run, so a folder sync that merely outlived the window was claimed again by the next cron tick and ran twice concurrently, racing createDocument into duplicates. A heartbeat now renews updated_at_ms every 5 minutes while the run is alive and stops with it; only a dead worker's claim reads stale.
synthesizeChunk verified thread ownership but not that messageId belonged to the thread, so any member could mint the global (message_id, chunk_index) reservation for another user's message under their own thread and lock the real owner out with a 403 blaming them. The audio door checked org membership only while the chunk listing required ownership. Synthesis now refuses (403 + security audit) a message that is not a row of the owned thread before any reservation, and the audio and usage doors go through the same loadOwnedThread gate as the listing.
TTS synthesizes a real app.messages row and probes the foreign-message refusal (403, audit row, no squatting chunk) and the stranger's audio/chunk doors; knowledge uploads a PNG and expects 'unsupported'; onedrive runs a long sync with a fast heartbeat, ages its claim past the stale window, and proves the next job no-ops.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant