Skip to content

fix: add fallback for heading-less documents in structural chunker - #1032

Open
JairVilleda wants to merge 8 commits into
ascherj:mainfrom
JairVilleda:fix/149-structural-chunker-fallback
Open

fix: add fallback for heading-less documents in structural chunker#1032
JairVilleda wants to merge 8 commits into
ascherj:mainfrom
JairVilleda:fix/149-structural-chunker-fallback

Conversation

@JairVilleda

Copy link
Copy Markdown

Summary

StructuralChunker.chunk() returned an empty list for documents that contained no markdown headings, causing those documents to be silently dropped from the ingestion pipeline. This PR updates the structural chunking behavior so heading-less content is captured and can be processed instead of being discarded.

Issue

Closes #149

Changes

Root cause: StructuralChunker._extract_sections() only collected content when a markdown heading had been detected. For a document with no headings, heading_stack remained empty, so the document's content was never added to a section and chunk() ultimately returned an empty list.

  • ingestion/chunking/structural_chunker.py — updated section extraction so content from documents without headings is captured instead of being silently dropped
  • Preserved the existing structural chunking behavior for documents that do contain headings
  • Added/updated test coverage for the heading-less document case so the regression is detected

Testing

  • Unit tests pass
  • New/updated tests cover the changes

Reproduce the original bug:

  1. Run the structural chunker test for a document with no markdown headings.
  2. Before the fix, StructuralChunker.chunk() returned an empty list.
  3. The test failed because it expected at least one chunk but received zero.

Verify the fix:

  1. Activate the project virtual environment.
  2. Run the structural chunker unit test:
    python -m pytest tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings -v
  3. The test should now pass because the heading-less document produces a chunk instead of an empty list.
  4. A minimal reproduction can also be tested with a plain-text document containing no markdown headings.

Screenshots / Demo

N/A — this is a backend ingestion/chunking change with no visible UI changes. The observable behavior is verified through the structural chunker unit test.

Notes for Reviewers

The implementation was verified with the project's testing/check commands after the fix. No unrelated changes are included in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Structural chunker silently drops documents that contain no headings

1 participant