Fix/149 structural chunker silently drops documents

[JohnPhm]

Summary

This pull request proposes changes to the functionality of the structural chunker. It fixes the issue where the structural chunker simply drops documents silently if certain conditions are not met. By fixing this issue, it prevents data loss during ingestion (notably for README documents, the main path routed through this chunker), keeping the search index complete.

Issue

Closes #149 structural chunker silently drops documents

Changes

Root Cause: flawed assumption in _extract_sections(): the code assumes every document is organized by headings, so it only keeps content once a heading exists. When no heading exists, content is never collected and no section is ever saved so the whole document silently disappears. This means that a heading-less document produced zero chunks and was excluded from the RAG index with no error or warning.
+added JOURNAL.md, detailing what was built/added and it also discusses what tests were used to confirm that the fix is working, the actual fix itself, and includes information on how to find this PR itself and the branch that contains all of the changes and commits
+edited structural_chunker.py, detailing the fix with the content guard
+edited test_structural_chunker.py, adding an assert statement to both test_heading_path_format and test_heading_path_breadcrumb functions to prevent the tests from passing if the chunk() returned nothing.

##Reproduction

• Use test document with no headings within test_structural_chunker.py to test. This can be done using the command 'python -m pytest tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings -v'. The result of using the command should be assert 0>=1, meaning that the structural chunker is not working as intended. A successful fix will yield 'tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings PASSED'.

Testing

• Unit tests pass (make test-unit), running 'make test-unit' introduces no new failures. Furthermore, using the command 'python -m pytest tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings -v' allows us to use the testing function for the heading-less document and it passes with the fix when it failed before. We could also use the entire testing file (test_structural_chunker.py) with the command 'python -m pytest tests/unit/test_structural_chunker.py -v'. By using the testing file (more specifically, using the heading-less document test allows us to confirm that the fix works as intended) The results show that all 15 tests passes when before all tests did not pass.
• Integration tests pass (make test-integration)
• Linter passes (make lint), running 'make lint' introduces no new errors. The same amount of errors that occur in other files still remain the same after implementing the fix, meaning that the new additions do not contribute to more errors.
• Type checker passes (make typecheck)
• New/updated tests cover the changes

Screenshots / Demo

Notes for Reviewers

For structural_chunker.py, I changed the function test_heading_path_format and test_heading_path_breadcrumb by adding an assertion at the end of each function.