Repository navigation
feat: pre-chunked JSON ingestion with per-chunk metadata and summaries - #34
Merged
Merged
Conversation
Add POST /v1/tables/{name}/ingest/chunks endpoint that accepts
pre-split text chunks with free-form metadata, bypassing the extraction
and chunking stages entirely.
Key changes:
- NodeMetadata gains start_line / end_line fields (+ with_lines builder)
so each node can record its source file position
- TextChunk gains start_line / end_line / attributes fields
- TreeBuilder threads all three through BuildNode → PageNode.metadata;
unknown metadata keys land in NodeMetadata.attributes
- IngestPipeline.ingest_chunks() / ingest_chunks_and_store() feed
pre-chunked content straight to the tree builder + summarizer
- ChunkInput accepts a free-form HashMap<String, Value>; well-known keys
(page_number, start_line, end_line, section_type) are extracted to
typed fields, the rest go into NodeMetadata.attributes
- JobRequest::Chunks variant wires the new endpoint into the job queue
…ies for chunk ingestion - Add `summary: Option<String>` to `ChunkBodyItem`, `ChunkInput`, `TextChunk`, and `BuildNode` - Thread pre-supplied summaries through the pipeline into `PageNode.summary` at tree-build time - `BatchSummarizer` now skips nodes that already carry a non-empty summary - Remove `generate_summaries` from `IngestChunksBody` / `IngestChunksRequest` — chunk ingestion always generates summaries (hardcoded `true` in job processor) - Update `docs/guides/ingestion.mdx` with a full "Ingest Pre-chunked Content" section - Update `README.md` ingestion diagram and pipeline description to reflect pre-chunked path
Replace old logos with new hexagonal green logo across all locations: - Regenerate all Tauri app icons (desktop, iOS, Android, Windows Store) - Update src/assets logo.svg and logo.png - Update assets/logo-black.png with black variant of new logo - Replace logo.svg in all 6 tutorial public directories
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /v1/tables/{table_name}/ingest/chunksendpoint that accepts an array of pre-split text chunks, each with optional metadata and an optional pre-computed summarypage_number,start_line,end_line,section_type, plus any custom keys) is threaded all the way through toNodeMetadataon eachPageNodeBatchSummarizernow skips nodes that already carry a non-empty summary (useful when callers provide pre-computed summaries)generate_summariesis alwaystruefor chunk ingestion (removed as a caller-facing option) — pre-chunked content almost never includes summaries so LLM generation is the right defaultdocs/guides/ingestion.mdx, README diagram and pipeline description updatedChanges
New API
Request body:
{ "title": "My Doc", "chunks": [ { "text": "The mitochondria is the powerhouse of the cell...", "heading": "Biology Basics", "metadata": { "page_number": 1, "start_line": 1, "end_line": 12 } }, { "text": "Photosynthesis converts sunlight into glucose...", "heading": "Plant Biology", "summary": "Plants make food from sunlight via photosynthesis.", "metadata": { "page_number": 2, "start_line": 13, "end_line": 28 } } ], "tags": ["biology"], "metadata": { "author": "Alice" } }Crates modified
reasondb-ingestTextChunk,ChunkInput,BuildNodeall gainsummary: Option<String>;BatchSummarizerskips nodes with existing summariesreasondb-serverChunkBodyItemgainssummaryfield;generate_summariesremoved fromIngestChunksBody/IngestChunksRequest; job processor hardcodestruefor chunksTest plan
docker compose build)summary→ summary used as-is, LLM skipped for that nodepage_number,start_line,end_linestored correctly onNodeMetadata