docs: recapture the knowledge screenshots - #3214
Merged
Merged
Conversation
Regenerate project-knowledge-files and documents-list through `bun run docs:screenshots` against the hermetic docs-demo stack, with the knowledge-db back on a healthy BM25 index (#3212): every seeded file reaches Indexed through the product's own path — the org's embedding model under Settings > Data residency, then each row's Retry indexing button. The Documents tab now also lists the markdown documents behind the knowledge entries (0.5 no longer files them into the "Knowledge entries" folder), so the documents-list captions name them in all three locales.
Knowledge indexing refuses every upload until the organization names an embedding model, so the docs-demo seeder left all its documents Failed. Wire the mock provider and the embedding model (Settings > Data residency, pointed at the mock gateway's /v1/embeddings at the width the knowledge-db stores) before the first upload, re-queue rows an earlier run left Failed through their own Retry indexing button, and give every uploaded or re-queued row its indexing window before the shots are taken. The documents-list shot now gates on the last file's Indexed badge instead of its name, and the README states what TALE_DEV_SKIP_DOCKER actually skips.
larryro
marked this pull request as ready for review
September 4, 2026 09:34
The RAG reconcile sweep settled a dead ingest chain with a rag_error that told the person to re-upload the file. The blob is still stored (every candidate row has a storage_ref) and the failed badge's Retry indexing button re-runs the pipeline on it — the ingest upserts the corpus row ON CONFLICT (org_slug, file_id), so a half-written document is reset and re-chunked; a re-upload only made them recreate a document the retry recovers. The text now names that recovery. Export the constant so the real-Postgres probe asserts the exact text instead of sniffing for the word "interrupted", and add a unit lock for the settle writes: the guidance names Retry indexing and never a re-upload, the corpus's own error always wins over the generic text, and an already-failed row is never overwritten with it.
KNOWLEDGE_ENTRIES_FOLDER ("Knowledge entries") has had no importer since
the 0.5 rewrite (#3107): knowledge-entry backing documents are listed at
the Documents root, not filed into a reserved folder. knip never flagged
it because every backend/**/*.ts file is a knip entry for the platform
workspace, so backend exports are never reported unused. No message key,
helper, test, or docs sentence existed only for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Regenerates the two knowledge screenshots through the real capture pipeline (
bun run docs:screenshots) on the hermetic docs-demo stack, as the end-to-end proof that knowledge ingestion works again after the BM25 self-heal (#3212). Two commits:docs: recapture the knowledge screenshots—project-knowledge-files.webp,documents-list.webp, and thedocuments-listcaptions (en/de/fr × 3 pages).manifest.jsonis byte-identical (it stores route/viewport/dimensions only).chore(platform): seed the embedding model before the docs-shot uploads— the rig gap that had left every seeded documentFailed(details below).Why the documents were failed
The seeded files were not PANIC victims. All five
.txtfiles and two of the three knowledge-entry documents had failed with "No embedding model is configured — set knowledge/embedding.json for this organization." — the demo org never had an embedding model, because the seeder never configured one (and its previous config dir lived inside a worktree that no longer exists). OnlyOnboarding kickoff window.mdwas the PANIC victim: knowledge-db row stuck atprocessing, file marked "Indexing was interrupted and did not finish. Re-upload the file to try again."Recovery through the product path (no DB repair)
e2e-mock, modeltext-embedding-3-small, width 1536 — exactly what the failure dialog tells an admin to do. The seeder now does this step through the same UI before the first upload (ensureEmbeddingModel).Failedthe same way and waits forIndexed.Onboarding kickoff window.md) also recovered through Retry indexing — the ingest upsertsON CONFLICT (org_slug, file_id), so the stuck knowledge-db row was reset and re-chunked. Itsrag_errortext tells the user to re-upload; the retry button already suffices (product nit).No dead-end: every failed document had a retry affordance. Nothing was touched in either database by hand.
Heal evidence (knowledge-db
tale_knowledge, localhost:5433)select count(*) from private_knowledge.chunksBGB.pdf, orgtest).txt, +3 knowledge-entry.md; the deleted+re-uploaded brand-guidelines chunk nets zero)pdb.verify_index('private_knowledge.idx_pk_chunks_bm25')schema_valid t · index_readable t · checksums_valid t · segment_metadata_valid t (2 segments)PANIClines in newest log (postgresql-2026-09-04_071700.log)ERRORlines are this morning'sassertion left == rightat 07:17/07:33 UTC, before the repair)private_knowledge.documents(orgnorthlight-labs)processingcompletedincl.Onboarding kickoff window.mdat 09:26:24 UTCapp.file_metadata.rag_status(org)failedcompleted/v1/embeddingscallsBoot-time verification of the same stack (the hermetic
bun scripts/dev.tsswallows backend stdout, so this is an observation-only backend booted with the identical env on :3105):app.audit_logsholds twoknowledge_index_repaired(success) events onprivate_knowledge.idx_pk_chunks_bm25from 07:33 UTC today — the merged self-heal's live repair — and none since.Rig changes (
chore(platform))ensureEmbeddingModelseeds the org's embedding model through Settings → Data residency (mock provider, 1536-dim as the knowledge-db stores;EMBEDDING_DIMENSIONSis now exported from the mock so the width has one source). The mock-provider step moves ahead of the uploads to make the provider selectable.retryFailedIndexing+awaitIndexed: rows leftFailedare re-queued through their own Retry indexing button; every uploaded/re-queued row gets a bounded, non-fatal indexing window (a stack without the knowledge-db still seeds the non-knowledge shots and only warns).ensureKnowledgeEntryDocumentsIndexed: the knowledge entries' markdown documents share thedocuments-listframe, so theirFailedrows are retried too.documents-listgate waits for the last file'sIndexedbadge instead of its filename.TALE_DEV_SKIP_DOCKER=1only skipsdocker compose up; the backend still dials the knowledge-db on :5433, so with the containers already running indexing works.Product findings (1 and 2 fixed below; 3 not fixed here)
KNOWLEDGE_ENTRIES_FOLDER("Knowledge entries",backend/core/knowledge_entries/constants.ts) has no usage since fd5c009 (feat(platform): move the platform off Convex onto Postgres #3107); the 0.4 rig shot showed them filed in that folder. The newdocuments-listshot and captions describe the 0.5 behaviour; regenerate once the folder is restored. → fixed in this PR (see Also fixes): the constant is dead code and is dropped; the root listing is the 0.5 behaviour.rag_errorfor an interrupted index says "Re-upload the file to try again", but Retry indexing recovers it — the message could name the retry. → fixed in this PR (see Also fixes).bunx convex run agents/installations:upsertInstallationfor the researcher agent and skips on failure (pre-existing, logged every run).Rig state handled through the UI
relaunch-content-inventory.txt(seeder race from an earlier run, since fixed) was detached from the project and deleted from Knowledge.2026-brand-guidelines.txtcarried av1 · Draftrecord badge from the earlier run'scontrolled-document-replace-fileshot (records cannot be un-controlled); it was deleted and re-uploaded by the seeder, which also exercised the plain upload → index path.Gates
bun run --filter @tale/docs test— 194/194 (30 files, incl. the de/fr docs i18n checks);bun run --filter @tale/docs lint— clean.oxlinton the changed platform files — clean;tsc --noEmit -p services/platform/tsconfig.json— rc 0;oxfmt --check— clean.Also fixes
Two follow-up commits close product findings 1 and 2 above:
fix(platform): point interrupted-indexing errors at Retry indexing— the RAG reconcile sweep'srag_errorfor a dead ingest chain (backend/domains/file_metadata/watchdogs.ts, both settle branches) now reads "Indexing was interrupted before it finished. The stored file is intact — use Retry indexing to index it again." instead of asking for a re-upload.rag_errorprose is an English backend literal rendered verbatim in the failed dialog (like its siblings indomains/knowledge/service.ts), so there is no locale work, and norag_error_codeis added because Retry indexing already sits beside every failed badge. The constant is exported: the real-Postgres probe (integration-check.ts) asserts the exact text instead of sniffing for "interrupted", and a new unit lock (domains/file_metadata/watchdogs.test.ts, vitestserver) pins that the guidance names Retry indexing and never a re-upload, that the corpus's own error always wins over the generic text, and that an already-failed row is never overwritten with it.chore(platform): drop the unused knowledge entries folder constant—KNOWLEDGE_ENTRIES_FOLDERremoved frombackend/core/knowledge_entries/constants.ts. No importer since feat(platform): move the platform off Convex onto Postgres #3107; knip never flagged it because everybackend/**/*.tsis a knip entry for the platform workspace.rgoverservices/platformanddocs/{en,de,fr}finds no message key, helper, test, or docs sentence that existed only for it (the seeder's comment describing the root listing stays — it is the 0.5 behaviour).Gates on the final tree: vitest
server—file_metadata/watchdogs.test.ts,lib/i18n/messages.test.ts(orphan-key guard),core/knowledge/index_health.test.ts,core/knowledge_entries/helpers.test.ts(4 files, 69 tests);tsc --noEmitrc 0;oxlint --type-awareandoxfmt --checkclean on the four touched files;kniprc 0. Docs are untouched by these two commits. The backend integration probe was type-checked, not re-run.Draft — not for merge yet.