From 7a00784918927013793ce2e2c8e37aefce960fea Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Tue, 28 Jul 2026 23:47:21 -0400 Subject: [PATCH 1/8] docs: add week 7 journal --- JOURNAL.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 JOURNAL.md diff --git a/JOURNAL.md b/JOURNAL.md new file mode 100644 index 000000000..7184a7a26 --- /dev/null +++ b/JOURNAL.md @@ -0,0 +1,19 @@ +## Week 7 — Issue selection + +**Issue link:** [https://github.com/ascherj/pathreview/issues/149] + +**Issue title:** [Structural chunker silently drops documents that contain no headings] + +**Tier:** [x] Tier 1 [ ] Tier 2 [ ] Tier 3 + +**Problem summary:** +[The StructuralChunker used by the ingestion pipeline currently fails to create chunks when a document does not contain markdown headings. In ingestion/chunking/structural_chunker.py, the _extract_sections() helper only captures content after detecting a heading, causing heading-free documents to return no chunks even when they contain valid text. This affects README ingestion because StructuralChunker is selected for README documents through the ingestion pipeline. A successful fix would ensure that any non-empty document can still produce at least one chunk while preserving the current behavior for empty documents.] + +**Branch name:** [fix/149-structural-chunker-fallback] + +**Setup confirmation:** [x] App runs locally at localhost:5173 + +**Cohort ledger:** [x] Issue added to cohort ledger + +**Issue selection notes ("Is this right for me?" checklist):** +[I chose this issue because it is my first open source contribution and I wanted to start with a manageable Tier 1 bug. I was able to understand the problem, locate the affected ingestion and chunking files, and identify what the expected behavior should be after the fix. The issue has clear reproduction steps and a related test, which makes it a good fit for learning how to contribute to a larger codebase while keeping the scope realistic for the project timeline.] \ No newline at end of file From 8b851147b6b2d73ee5a711a93f915ca95782afea Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Sun, 2 Aug 2026 22:23:05 -0400 Subject: [PATCH 2/8] docs(ingestion): add reproduction step for issue #149 --- JOURNAL.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/JOURNAL.md b/JOURNAL.md index 7184a7a26..f6fe2687b 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -16,4 +16,24 @@ **Cohort ledger:** [x] Issue added to cohort ledger **Issue selection notes ("Is this right for me?" checklist):** -[I chose this issue because it is my first open source contribution and I wanted to start with a manageable Tier 1 bug. I was able to understand the problem, locate the affected ingestion and chunking files, and identify what the expected behavior should be after the fix. The issue has clear reproduction steps and a related test, which makes it a good fit for learning how to contribute to a larger codebase while keeping the scope realistic for the project timeline.] \ No newline at end of file +[I chose this issue because it is my first open source contribution and I wanted to start with a manageable Tier 1 bug. I was able to understand the problem, locate the affected ingestion and chunking files, and identify what the expected behavior should be after the fix. The issue has clear reproduction steps and a related test, which makes it a good fit for learning how to contribute to a larger codebase while keeping the scope realistic for the project timeline.] + + +## Week 8 — Reproduction & solution planning + +**Reproduction steps:** +Option A: +```bash +python -m pytest tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings -v +``` + +Option B: +```python +from ingestion.chunking.structural_chunker import StructuralChunker +c = StructuralChunker() +text = "This is plain text without any markdown headings. " * 20 +print(c.chunk(text, {"source": "test"})) +``` + +- The test fails because StructuralChunker.chunk() returns an empty list for a document without markdown headings +- The issue originates in ingestion/chunking/structural_chunker.py, where _extract_sections() only collects content after encountering a markdown heading From 8532f5faa3532f0829f7359cfe9aa5975e53357e Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Mon, 3 Aug 2026 02:06:30 -0400 Subject: [PATCH 3/8] docs(ingestion): add solution plan for issue #149 --- PLAN.md | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 PLAN.md diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 000000000..4994639db --- /dev/null +++ b/PLAN.md @@ -0,0 +1,68 @@ +## Solution plan + +**Issue:** [Structural chunker silently drops documents that contain no headings #149 +https://github.com/ascherj/pathreview/issues/149] + +### Understand + +StructuralChunker.chunk() returns an empty list when processing a document without markdown headings. This causes the document to be skipped during ingestion because no chunks are created for embedding or indexing. + +The root cause is in ingestion/chunking/structural_chunker.py inside _extract_sections(). The function only collects content after a markdown heading has been detected. For heading-less documents: +- heading_stack remains empty. +- current_section_lines never receives content. +- The final section is never saved because the save condition requires a heading. + +Expected behavior: Heading-less documents should still produce chunks and be included in retrieval. + +The fix should happen in _extract_sections() by allowing heading-less content to become a valid section. This allows the existing chunk() logic to continue handling token limits and semantic fallback behavior through SemanticChunker. + +### Map +Files involved: + +- ingestion/chunking/structural_chunker.py + - _extract_sections() — currently drops content without headings + - chunk() — contains existing semantic fallback behavior for large sections + +- ingestion/chunking/semantic_chunker.py + - SemanticChunker.chunk() — existing chunking logic that can be reused for heading-less documents + +- tests/unit/test_structural_chunker.py + - test_document_with_no_headings — existing regression test that should be strengthened. + - New test for text appearing before the first markdown heading + +### Plan +1. Update _extract_sections() so documents without markdown headings are returned as valid sections instead of being dropped. +2. Ensure the existing chunk() flow handles these sections through the normal size checks and SemanticChunker fallback. +3. Preserve metadata for heading-less sections, including an empty heading_path and appropriate heading level. +4. Strengthen test_document_with_no_headings to verify: + - At least one chunk is returned. + - Original content is preserved. + - Source metadata is maintained. + - Heading metadata uses the expected fallback values. +5. Add a test for pre-heading text to verify that content before the first markdown heading is not dropped. +6. Run the relevant unit tests to confirm existing heading-based chunking behavior remains unchanged. + +### Inputs & outputs +Input: +- A document string with or without markdown headings. +- Metadata containing source information + +Current output: +- Documents without headings return an empty list + +Expected output: +- Heading-less documents are passed through semantic chunking and return one or more valid chunks + +### Risks & unknowns +- Adding support for heading-less sections changes current behavior because previously this content was silently ignored. +- Need to confirm the expected metadata values for fallback sections (heading_path, heading_level). +- Need to verify that documents with existing markdown headings continue producing the same chunk structure. +- Need to ensure long heading-less documents still use semantic chunking instead of creating oversized chunks. + +### Edge cases +- Empty documents +- Documents containing only whitespace +- Short documents without headings +- Long documents without headings +- Documents containing text before the first markdown heading +- Documents with valid markdown heading sections \ No newline at end of file From bfad5ad840fd1b348077af0174d5f156d66f9ff2 Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Mon, 3 Aug 2026 02:12:50 -0400 Subject: [PATCH 4/8] docs: update week 8 journal --- JOURNAL.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/JOURNAL.md b/JOURNAL.md index f6fe2687b..c08033b5c 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -18,6 +18,7 @@ **Issue selection notes ("Is this right for me?" checklist):** [I chose this issue because it is my first open source contribution and I wanted to start with a manageable Tier 1 bug. I was able to understand the problem, locate the affected ingestion and chunking files, and identify what the expected behavior should be after the fix. The issue has clear reproduction steps and a related test, which makes it a good fit for learning how to contribute to a larger codebase while keeping the scope realistic for the project timeline.] +--- ## Week 8 — Reproduction & solution planning @@ -37,3 +38,17 @@ print(c.chunk(text, {"source": "test"})) - The test fails because StructuralChunker.chunk() returns an empty list for a document without markdown headings - The issue originates in ingestion/chunking/structural_chunker.py, where _extract_sections() only collects content after encountering a markdown heading + +**Reproduction commit link:** +[(https://github.com/JairVilleda/pathreview/commit/8b851147b6b2d73ee5a711a93f915ca95782afea)] + +**Reproduction summary:** +I reproduced the issue by running the existing unit test for StructuralChunker and by testing it with a document containing no markdown headings. In both cases, StructuralChunker.chunk() returned an empty list instead of producing at least one chunk. + +**PLAN.md link:** +[(https://github.com/JairVilleda/pathreview/blob/fix/149-structural-chunker-fallback/PLAN.md)] + +**Blockers or open questions:** +[None at this time.] + +--- \ No newline at end of file From 47686de18fac39c13e6a8c87df1db886dba3e4ce Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Wed, 12 Aug 2026 00:59:57 -0400 Subject: [PATCH 5/8] fix(ingestion): preserve documents without headings --- ingestion/chunking/structural_chunker.py | 63 +++++++++++++----------- 1 file changed, 34 insertions(+), 29 deletions(-) diff --git a/ingestion/chunking/structural_chunker.py b/ingestion/chunking/structural_chunker.py index d5bcf0530..7eea91380 100644 --- a/ingestion/chunking/structural_chunker.py +++ b/ingestion/chunking/structural_chunker.py @@ -49,22 +49,26 @@ def chunk(self, text: str, metadata: dict) -> list[Chunk]: if section_tokens > self.SECTION_TOKEN_LIMIT: # Sub-chunk using semantic chunker section_metadata = metadata.copy() - section_metadata.update({ - "heading_path": heading_path, - "heading_level": section["level"], - }) + section_metadata.update( + { + "heading_path": heading_path, + "heading_level": section["level"], + } + ) sub_chunks = self.semantic_chunker.chunk(section_text, section_metadata) chunks.extend(sub_chunks) else: # Single chunk for this section section_metadata = metadata.copy() - section_metadata.update({ - "heading_path": heading_path, - "heading_level": section["level"], - "chunk_index": len(chunks), - "char_start": 0, - "char_end": len(section_text), - }) + section_metadata.update( + { + "heading_path": heading_path, + "heading_level": section["level"], + "chunk_index": len(chunks), + "char_start": 0, + "char_end": len(section_text), + } + ) chunks.append(Chunk(text=section_text, metadata=section_metadata)) return chunks @@ -79,7 +83,6 @@ def _extract_sections(self, text: str) -> list[dict]: sections = [] heading_stack = [] # Stack of (level, heading_text) current_section_lines = [] - current_level = 0 for line in lines: heading_match = re.match(r"^(#{1,6})\s+(.+)$", line) @@ -88,11 +91,13 @@ def _extract_sections(self, text: str) -> list[dict]: # Save previous section if exists if current_section_lines: if heading_stack: - sections.append({ - "content": "\n".join(current_section_lines).strip(), - "path": [h[1] for h in heading_stack], - "level": heading_stack[-1][0] if heading_stack else 0, - }) + sections.append( + { + "content": "\n".join(current_section_lines).strip(), + "path": [h[1] for h in heading_stack], + "level": heading_stack[-1][0] if heading_stack else 0, + } + ) current_section_lines = [] # Process new heading @@ -104,19 +109,19 @@ def _extract_sections(self, text: str) -> list[dict]: heading_stack.pop() heading_stack.append((heading_level, heading_text)) - current_level = heading_level else: - # Regular content line - if heading_stack or current_section_lines: # Only collect if we have a heading - current_section_lines.append(line) - - # Save final section - if current_section_lines and heading_stack: - sections.append({ - "content": "\n".join(current_section_lines).strip(), - "path": [h[1] for h in heading_stack], - "level": heading_stack[-1][0] if heading_stack else 0, - }) + # Regular content line (collected even before the first heading) + current_section_lines.append(line) + + # Save final section (may have no heading at all) + if current_section_lines: + sections.append( + { + "content": "\n".join(current_section_lines).strip(), + "path": [h[1] for h in heading_stack], + "level": heading_stack[-1][0] if heading_stack else 0, + } + ) return sections From e740a58c44bc9b960c3426247139d6fa41c0db26 Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Wed, 12 Aug 2026 01:14:22 -0400 Subject: [PATCH 6/8] test(ingestion): strengthen headingless document regression --- tests/unit/test_structural_chunker.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/unit/test_structural_chunker.py b/tests/unit/test_structural_chunker.py index a0b5d0d5d..42f717db9 100644 --- a/tests/unit/test_structural_chunker.py +++ b/tests/unit/test_structural_chunker.py @@ -34,6 +34,16 @@ def test_document_with_no_headings(self, chunker): assert isinstance(result[0], Chunk) assert all(isinstance(c, Chunk) for c in result) + # Content is preserved, not silently dropped + assert result[0].text == text.strip() + + # Source metadata is carried through + assert result[0].metadata["source"] == "test" + + # Heading-less sections fall back to an empty path at level 0 + assert result[0].metadata["heading_path"] == "" + assert result[0].metadata["heading_level"] == 0 + def test_document_with_nested_headings(self, chunker): """Test document with nested headings preserves heading_path.""" text = """# Main Title From af6085467e6b54bd625d05bef75b7f2d5f1406c6 Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Wed, 12 Aug 2026 01:43:40 -0400 Subject: [PATCH 7/8] fix(ingestion): preserve pre-heading content --- ingestion/chunking/structural_chunker.py | 17 ++++++++--------- tests/unit/test_structural_chunker.py | 23 +++++++++++++++++++++++ 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/ingestion/chunking/structural_chunker.py b/ingestion/chunking/structural_chunker.py index 7eea91380..2f38477c3 100644 --- a/ingestion/chunking/structural_chunker.py +++ b/ingestion/chunking/structural_chunker.py @@ -88,16 +88,15 @@ def _extract_sections(self, text: str) -> list[dict]: heading_match = re.match(r"^(#{1,6})\s+(.+)$", line) if heading_match: - # Save previous section if exists + # Save previous section if exists (may precede the first heading) if current_section_lines: - if heading_stack: - sections.append( - { - "content": "\n".join(current_section_lines).strip(), - "path": [h[1] for h in heading_stack], - "level": heading_stack[-1][0] if heading_stack else 0, - } - ) + sections.append( + { + "content": "\n".join(current_section_lines).strip(), + "path": [h[1] for h in heading_stack], + "level": heading_stack[-1][0] if heading_stack else 0, + } + ) current_section_lines = [] # Process new heading diff --git a/tests/unit/test_structural_chunker.py b/tests/unit/test_structural_chunker.py index 42f717db9..f2557b3d1 100644 --- a/tests/unit/test_structural_chunker.py +++ b/tests/unit/test_structural_chunker.py @@ -44,6 +44,29 @@ def test_document_with_no_headings(self, chunker): assert result[0].metadata["heading_path"] == "" assert result[0].metadata["heading_level"] == 0 + def test_text_before_first_heading_is_preserved(self, chunker): + """Test content before the first heading becomes its own section.""" + text = """Introductory text before the heading. + +# Main Heading + +Body text under the heading. +""" + result = chunker.chunk(text, {"source": "test"}) + + assert len(result) == 2 + + # Pre-heading content is not dropped + assert "Introductory text before the heading." in result[0].text + assert result[0].metadata["heading_path"] == "" + assert result[0].metadata["heading_level"] == 0 + assert result[0].metadata["source"] == "test" + + # Normal heading-based chunking still works + assert "Body text under the heading." in result[1].text + assert result[1].metadata["heading_path"] == "Main Heading" + assert result[1].metadata["heading_level"] == 1 + def test_document_with_nested_headings(self, chunker): """Test document with nested headings preserves heading_path.""" text = """# Main Title From 6b513440903579b291812518bf7eef1209952c6d Mon Sep 17 00:00:00 2001 From: JairVilleda Date: Wed, 12 Aug 2026 02:34:51 -0400 Subject: [PATCH 8/8] docs: add week 9 journal --- JOURNAL.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/JOURNAL.md b/JOURNAL.md index c08033b5c..e235b7661 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -51,4 +51,35 @@ I reproduced the issue by running the existing unit test for StructuralChunker a **Blockers or open questions:** [None at this time.] ---- \ No newline at end of file +--- + +## Week 9 — Solution building & PR submission + +### Check-in 1 (mid-week) + +**Current progress:** +Implemented the fix for issue #149. Updated the structural chunker so documents without markdown headings are no longer silently dropped and can produce a chunk. The reproduction test for heading-less documents now passes. + +**Next steps:** +Finish the final testing/checks, review the changes, update the PR, and complete the PR submission. + +**Blockers:** +None. + +--- + +### Check-in 2 (end of week) + +**PR link:** [(https://github.com/ascherj/pathreview/pull/1032)] + +**Branch:** `fix/149-structural-chunker-fallback` + +**What you built:** +Fixed issue #149 by updating the structural chunker so heading-less documents are captured instead of causing `StructuralChunker.chunk()` to return an empty list. This prevents documents without markdown headings from being silently dropped during ingestion. + +**Tests added or updated:** +Updated the structural chunker test for documents without headings in `tests/unit/test_structural_chunker.py`. The test verifies that a document without markdown headings produces at least one chunk instead of an empty result. + +**Self-review confirmation:** [x] make check passes [x] make test-unit passes + +**Draft PR feedback received from:** none \ No newline at end of file