DirectoryLoader: ingest a whole folder with per-extension loader dispatch

[adaumsilva]

Description

Add DirectoryLoader that walks a directory (optionally recursive, with glob filtering) and delegates each file to the right loader by extension, returning the concatenated list[Document].

loader = DirectoryLoader(
    loaders={".txt": TextFileLoader(), ".md": MarkdownLoader(), ".pdf": PDFLoader()},
    glob="**/*",
    recursive=True,
    on_error="skip",   # or "raise"
)
pipeline = RAGPipeline(loader=loader, ...)
pipeline.ingest("./knowledge_base")

Motivation

Today RAGPipeline.ingest() takes a single file path, so indexing a corpus means writing a loop by hand and choosing loaders manually. This is the single most requested convenience in every RAG library.

Acceptance criteria

• Class DirectoryLoader(DocumentLoader) in ragframework/document/loaders.py
• Sensible default loaders mapping (.txt, .md, .markdown, .pdf when pypdf is importable) built lazily so importing the module never requires optional deps
• Files with no matching loader are skipped (and counted); on_error="raise" re-raises as LoaderError with the offending path
• source must be an existing directory, otherwise LoaderError
• Deterministic ordering (sorted paths) so chunk ids are stable across runs
• Each Document.metadata includes "relative_path" from the root directory
• Tests using tmp_path with mixed extensions, nested folders, an unreadable file with both on_error modes
• Exported from ragframework/document/__init__.py; CHANGELOG.md updated

Files to touch

• ragframework/document/loaders.py
• ragframework/document/__init__.py
• tests/test_document/test_loaders.py

Resources

• Path.rglob

Estimated effort: Medium (half day)

[Guru2907]

Hi @adaumsilva, I would like to work on this issue. I will implement DirectoryLoader with recursive or nonrecursive traversal, glob filtering, extension based loader dispatch, deterministic ordering, relative-path metadata and both on_error modes, along with the requested tests. Thanks.