Skip to content

feat: pre-chunked JSON ingestion with per-chunk metadata and summaries - #34

Merged
ajainvivek merged 4 commits into
mainfrom
feat/pre-chunked-json-ingestion
Mar 13, 2026
Merged

ajainvivek merged 4 commits into
mainfrom
feat/pre-chunked-json-ingestion

Conversation

@ajainvivek

Copy link
Copy Markdown
Contributor

Summary

  • Adds a new POST /v1/tables/{table_name}/ingest/chunks endpoint that accepts an array of pre-split text chunks, each with optional metadata and an optional pre-computed summary
  • Chunk ingestion bypasses the extraction and chunking pipeline stages — content goes straight into the tree builder and LLM summarizer
  • Per-chunk metadata (page_number, start_line, end_line, section_type, plus any custom keys) is threaded all the way through to NodeMetadata on each PageNode
  • BatchSummarizer now skips nodes that already carry a non-empty summary (useful when callers provide pre-computed summaries)
  • generate_summaries is always true for chunk ingestion (removed as a caller-facing option) — pre-chunked content almost never includes summaries so LLM generation is the right default
  • Docs updated: new "Ingest Pre-chunked Content" section in docs/guides/ingestion.mdx, README diagram and pipeline description updated

Changes

New API

POST /v1/tables/{table_name}/ingest/chunks

Request body:

{
  "title": "My Doc",
  "chunks": [
    {
      "text": "The mitochondria is the powerhouse of the cell...",
      "heading": "Biology Basics",
      "metadata": { "page_number": 1, "start_line": 1, "end_line": 12 }
    },
    {
      "text": "Photosynthesis converts sunlight into glucose...",
      "heading": "Plant Biology",
      "summary": "Plants make food from sunlight via photosynthesis.",
      "metadata": { "page_number": 2, "start_line": 13, "end_line": 28 }
    }
  ],
  "tags": ["biology"],
  "metadata": { "author": "Alice" }
}

Crates modified

Crate Change
reasondb-ingest TextChunk, ChunkInput, BuildNode all gain summary: Option<String>; BatchSummarizer skips nodes with existing summaries
reasondb-server ChunkBodyItem gains summary field; generate_summaries removed from IngestChunksBody/IngestChunksRequest; job processor hardcodes true for chunks

Test plan

  • Build passes (docker compose build)
  • Container starts healthy
  • Chunk without pre-supplied summary → LLM generates summary automatically
  • Chunk with pre-supplied summary → summary used as-is, LLM skipped for that node
  • Root/parent nodes always get LLM-generated rollup summaries
  • page_number, start_line, end_line stored correctly on NodeMetadata

Add POST /v1/tables/{name}/ingest/chunks endpoint that accepts
pre-split text chunks with free-form metadata, bypassing the extraction
and chunking stages entirely.

Key changes:
- NodeMetadata gains start_line / end_line fields (+ with_lines builder)
  so each node can record its source file position
- TextChunk gains start_line / end_line / attributes fields
- TreeBuilder threads all three through BuildNode → PageNode.metadata;
  unknown metadata keys land in NodeMetadata.attributes
- IngestPipeline.ingest_chunks() / ingest_chunks_and_store() feed
  pre-chunked content straight to the tree builder + summarizer
- ChunkInput accepts a free-form HashMap<String, Value>; well-known keys
  (page_number, start_line, end_line, section_type) are extracted to
  typed fields, the rest go into NodeMetadata.attributes
- JobRequest::Chunks variant wires the new endpoint into the job queue
…ies for chunk ingestion

- Add `summary: Option<String>` to `ChunkBodyItem`, `ChunkInput`, `TextChunk`, and `BuildNode`
- Thread pre-supplied summaries through the pipeline into `PageNode.summary` at tree-build time
- `BatchSummarizer` now skips nodes that already carry a non-empty summary
- Remove `generate_summaries` from `IngestChunksBody` / `IngestChunksRequest` — chunk ingestion
  always generates summaries (hardcoded `true` in job processor)
- Update `docs/guides/ingestion.mdx` with a full "Ingest Pre-chunked Content" section
- Update `README.md` ingestion diagram and pipeline description to reflect pre-chunked path
Replace old logos with new hexagonal green logo across all locations:
- Regenerate all Tauri app icons (desktop, iOS, Android, Windows Store)
- Update src/assets logo.svg and logo.png
- Update assets/logo-black.png with black variant of new logo
- Replace logo.svg in all 6 tutorial public directories
@ajainvivek
ajainvivek merged commit 5954775 into main Mar 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant