feat: add DirectoryLoader for folder ingestion

[Guru2907]

Summary

Implements DirectoryLoader for ingesting documents from a directory with per-extension loader dispatch.

Changes

• Added DirectoryLoader to ragframework/document/loaders.py
• Added default loaders for .txt, .md, .markdown, and .pdf
• Added configurable extension-to-loader dispatch
• Added recursive directory traversal with deterministic sorted paths
• Added relative_path metadata to loaded documents
• Added on_error="skip" and on_error="raise" handling
• Added directory validation with LoaderError
• Exported DirectoryLoader from ragframework.document
• Added changelog entry
• Added tests covering mixed extensions, nested directories, ordering, loader errors, invalid configuration, defaults, and unreadable files

Testing

Focused document loader tests:

• 15 passed

Full test suite:

• 258 passed
• 12 failed

The 12 failures are in the ChromaDB tests because the optional chromadb dependency is not installed in the current environment. These failures are unrelated to DirectoryLoader.

Related issue

Closes #34

[adaumsilva]

@Guru2907 Thanks for adding DirectoryLoader! Please address these before merging:

• Fix recursive=False: the default **/* pattern still loads nested files.
• Register the default PDF loader only when pypdf is available, as required by #34.
• Add regression tests for both scenarios.
• Count skipped unsupported files as requested in the issue.
• Move the changelog entry from 0.1.0 to Unreleased.

All CI checks passed, but I reproduced both functional issues locally.

[Guru2907]

Thanks for the review, @adaumsilva! All points are addressed:

• recursive=False: nested files are now skipped when recursive=False, even with the default **/* pattern. Added a regression test.
• PDF default: the .pdf loader is now registered only when pypdf is available. Added a regression test.
• Skipped files: files with no matching loader are counted in DirectoryLoader.skipped_count (reset on each load() call), with a test.
• Changelog: moved the entry from 0.1.0 to Unreleased.
• Rebased onto the latest main and resolved the conflicts in document/__init__.py and test_loaders.py.

The document tests pass locally (165 passed). Could you approve the workflow run and take another look when you have a moment? Thanks!

[adaumsilva]

Thanks for addressing the previous feedback @Guru2907 ! The non-recursive traversal and conditional PDF registration now work, and the changelog entry is correctly under Unreleased. All four CI checks pass, and I reproduced 165 passing document tests locally.

One issue remains in DirectoryLoader.load(): skipped_count is initialized only in __init__, so it accumulates when the loader is reused, despite the update stating it resets on each call.

With one unsupported file, I reproduced:

• First load: skipped_count == 1
• Second load: skipped_count == 2
• After removing the unsupported file: still 2

Please reset self.skipped_count = 0 at the start of load() and add a regression test that reuses the same loader, verifying the counts are 1, 1, then 0.

The remaining reviewed changes look good.