fix(platform): harden lib reliability, authorization, and dead-end doors - #3183
Draft
larryro wants to merge 9 commits into
Draft
fix(platform): harden lib reliability, authorization, and dead-end doors#3183larryro wants to merge 9 commits into
larryro wants to merge 9 commits into
Conversation
decryptSecretsFile shelled out with execFileSync on the request path (every per-org object-store resolve on cache expiry, every credential save), stalling the single-threaded event loop for the whole decrypt — up to the 10s timeout when sops hung. Both verbs now spawn sops with execFile, kill a wedged child at the timeout with a legible error, and concurrent cold reads of one file share a single child. encryptJsonWithSops is async; its four callers await it.
readJsonFile mapped every stat() failure to not_found, so a mis-permissioned config volume read as an absent file and callers (readGovernancePolicy among them) silently fell back to defaults. Only ENOENT/ENOTDIR are not_found now; EACCES/EPERM/EIO surface as inaccessible, which the callers already handle by warning.
The reused resolveOrgObjectStore swallowed a broken default tree (.catch(() => null)) and fell back to the retired Convex store, whose ctx.storage.* verbs no longer exist — a real misconfiguration surfaced as a TypeError deep in the video, harvest, and transcription lanes. One fail-closed resolver and one cache now serve both the 0.5 lane and blob_access: a broken default tree throws its own error, neither-configured throws ObjectStoreUnconfiguredError, the Convex variant and its dead branches are gone, and a config write invalidates every cached resolution.
Images route to the vision extractor and the vision seam is retired, so every image upload extracted nothing and landed as rag_status 'failed — Indexing skipped (empty)' with a retry affordance that could only fail again. An image is now stamped 'unsupported' up front — the terminal, honest state the badge already renders — before any bytes are fetched. Docs (en/de/fr) name images under Unsupported.
The render worker summed html.length after storing each page and only broke before the next one, so pages.json could exceed the 20MB sessionReadFile cap (more so with multibyte text); the host then saw no output and the crawl retried the same batch forever. The worker now budgets the serialized file in UTF-8 bytes before admitting each page, applies the per-page bound in bytes, and hands a page that does not fit back as not attempted for the next batch.
Every drain retry attached, so when the initial exec POST never reached the spawner the exec did not exist and each attach answered 'exec <id> not found' until the whole budget burned against a healthy session. An attach answered not-found while nothing was consumed now re-POSTs the exec (ExecNotFoundError); never after progress, so a turn is never run twice, and never on a resume.
runSyncConfigJobWith treated a 'running' stamp older than 30 minutes as a crashed worker but nothing refreshed the stamp during a run, so a folder sync that merely outlived the window was claimed again by the next cron tick and ran twice concurrently, racing createDocument into duplicates. A heartbeat now renews updated_at_ms every 5 minutes while the run is alive and stops with it; only a dead worker's claim reads stale.
synthesizeChunk verified thread ownership but not that messageId belonged to the thread, so any member could mint the global (message_id, chunk_index) reservation for another user's message under their own thread and lock the real owner out with a 403 blaming them. The audio door checked org membership only while the chunk listing required ownership. Synthesis now refuses (403 + security audit) a message that is not a row of the owned thread before any reservation, and the audio and usage doors go through the same loadOwnedThread gate as the listing.
TTS synthesizes a real app.messages row and probes the foreign-message refusal (403, audit row, no squatting chunk) and the stranger's audio/chunk doors; knowledge uploads a PNG and expects 'unsupported'; onedrive runs a long sync with a fast heartbeat, ages its claim past the stale window, and proves the next job no-ops.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch of verified medium-severity reliability / authorization / dead-end defects in the platform libs and their doors (source: the backend deep-review triage,
lib/tts/node_only/onedriverows). Base:origin/main@ 899fcc0. Every still-live finding carries a regression test that is red on the base tree and green here.Per-finding outcome
execFileSyncon the request pathsopsasync (execFile), 10s timeout kills a wedged child with a legible error, concurrent cold reads of one file share one child;encryptJsonWithSopsis now async (4 callers awaited)core/lib/sops.tsrag_status 'failed' — Indexing skipped (empty)unsupportedup front (terminal, no retry affordance, no blob fetch); docs (en/de/fr) name images under Unsupporteddomains/knowledge/service.ts,extraction/router.ts(isImageFile)readJsonFilelabels everystat()failurenot_foundENOENT/ENOTDIRarenot_found;EACCES/EPERM/EIOsurface asinaccessible(callers already handle it;readGovernancePolicynow warns instead of silently defaulting)core/lib/file_io.tsmessageId∈ thread; audio serve org-gated while chunk listing owner-gatedsynthesizeChunkrefuses (403 +tts.synthesize_deniedaudit, reasonmessage_not_in_thread) a message that is not a row of the owned thread, before any reservation;getChunkForServeand/usagego through the sameloadOwnedThreadgate as the listing doordomains/tts/service.ts,domains/tts/routes.tsresolveOrgObjectStoreswallows default-tree errors and falls back to the dead Convex backendObjectStoreUnconfiguredError; the Convex store variant andblob_access's deadctx.storage.*branches are retired;lib/object-store.tsdelegates (a config write now invalidates every cached resolution)core/lib/storage/object_store.ts,blob_access.ts,lib/object-store.tspages.jsoncan exceed the 20MB read capnot_attemptedfor the next batchcore/node_only/sandbox/render_fetch.tsexec <id> not foundwhile nothing was consumed re-POSTs the exec (ExecNotFoundError); never after progress (no double-run), never on a resumecore/node_only/sandbox/helpers/session_client.tsupdated_at_mswhile the run is alive (cleared infinally), so only a dead worker's claim ever reads staledomains/onedrive/service.ts(shared by the Google Drive adapter)endpointUrlupdate skips the https validation create enforcesupdateCredentialrejects non-https (provider_credentials/service.ts:461-467)Nothing deferred; no migration needed.
Tests (all red on base, green here)
core/lib/sops.test.ts(new, 11): timer fires while decrypt is in flight (base: loop blocked 637ms); hung sops killed at the timeout (base: 10sspawnSync ETIMEDOUT); 5 concurrent cold reads → 1 child (base: 5); encrypt off-loop; cache/invalidation, plaintext, no-key refusal; realsops+age round trip when installed.core/lib/file_io.test.ts(+4): ENOENT and ENOTDIR staynot_found; EACCES →inaccessible(base:not_found).core/lib/storage/object_store.test.ts(+6): unconfigured →ObjectStoreUnconfiguredError(base: resolved{ backend: 'convex' }); corrupt default tree → its own error (base: swallowed → convex); default/own precedence; missing credentials; cache clear.domains/tts/service.access-gate.test.ts(new, 4): foreign message → 403 + audit, no reservation (base: proceeds); owned message passes; audio serve owner → chunk, stranger → null (base: served).domains/knowledge/service.image-status.test.ts(new, 2) +extraction/router.test.ts(+2): image → singleunsupportedwrite, no blob fetch (base:failed); text still extracts.core/node_only/sandbox/render_fetch.test.ts(+2): the staged worker run under node against a fakeplaywright-core:pages.json≤ cap, third page handed back (base: 14151 B over a 10000 B cap); per-page bound in bytes (base: 1700 'é' accepted under a 3000-byte bound).core/node_only/sandbox/helpers/session_client.test.ts(+2): lost POST → attach(not found) → POST again completes (base:exec exec-3 not foundafter the budget); after progress a not-found attach fails, never re-POSTs.domains/onedrive/service.claim-heartbeat.test.ts(new, 2): 3 renewals over 3 periods, none after the run (base: 0); no heartbeat without a claim.backend/integration-check.ts): TTS section now synthesizes a REALapp.messagesrow and probes foreign-message 403 + audit + no squatting rows + stranger audio 404 / chunks []; knowledge section uploads a PNG →unsupported; onedrive section runs a long sync withheartbeatMs: 100, ages the claim 31 min, and proves the next job no-ops (listCallsstays 1, one document).Verification
bunx tsc --noEmit(platform): exit 0.bunx oxlint --type-aware(platform): exit 0.vitest --project server backend): 239 files / 2539 tests green.backend:integrationon fresh throwaway tale-db + MinIO (SANDBOX_LLM_GATEWAY_ADMIN_PASSWORDset): branch 381/381 checks passed (exit 0; the three new probes:foreignMessage=403 (audited=1, squatRows=0) strangerAudio=404 strangerChunks=0, imagestatus=unsupported, claimstampAge=0s listCalls=1) · base @ 899fcc0 with the same harness edits: 378/381 checks passed (exit 1) — the only failures are the three new probes, each showing the pre-fix defect: TTSforeignMessage=200 (audited=0, squatRows=1) strangerAudio=200; knowledge imagestatus=failed, error=Indexing skipped (empty).; onedrivestampAge=1860s listCalls=2.Cross-class discoveries (not fixed here)
--- Page N ---markers ascompleted(sibling of update dashboard ui (#508) #2; needs the vision lane or a "no indexable text" outcome)..txt, image-only.docx) still lands asfailed — Indexing skipped (empty); there is no "nothing to index" status in therag_statusCHECK.generateReplacementBlobUpload/ReplacementBlobUploadHandoffhave no consumers in 0.5 (dead export, kept).docs/en/platform/knowledge/documents.md:18says the legacy Office trio shows Not indexed while line 82 says Unsupported (pre-existing inconsistency).