feat: add OpenAI LLM generator

.github/dependabot.yml

@@ -0,0 +1,22 @@
+version: 2
+updates:
+  - package-ecosystem: pip
+    directory: /
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 10
+    groups:
+      pip-minor-patch:
+        update-types:
+          - minor
+          - patch
+
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 5
+    groups:
+      github-actions:
+        patterns:
+          - "*"

.github/issues/008-llm-generator.md

@@ -1,5 +1,5 @@
 ---
-title: "OpenAI / Anthropic LLM Generator"
+title: "OpenAI LLM Generator"
 labels: ["good first issue", "enhancement"]
 ---
 

.github/issues/011-inmemory-retriever-top-k-zero.md

@@ -0,0 +1,40 @@
+---
+title: "InMemoryRetriever returns every chunk when top_k <= 0"
+labels: ["bug", "good first issue"]
+---
+
+## Description
+
+`InMemoryRetriever.retrieve()` mishandles non-positive `top_k`. With 5 indexed chunks:
+
+```python
+r.retrieve(q, top_k=0)   # returns 5 chunks
+r.retrieve(q, top_k=-1)  # returns 4 chunks
+```
+
+Cause: `k = min(top_k, len(self._chunks))` gives `k = 0`, and `np.argpartition(scores, -0)[-0:]` is `[0:]` — the whole array. `FAISSRetriever` and `ChromaRetriever` both return `[]` for `top_k <= 0`, so the three built-in retrievers disagree.
+
+## Motivation
+
+Retrievers must behave identically for the same inputs so users can swap them freely. Returning the entire corpus for `top_k=0` can also silently blow up a prompt sent to an LLM generator.
+
+## Acceptance criteria
+
+- [ ] `InMemoryRetriever.retrieve()` returns `[]` when `top_k <= 0`
+- [ ] Non-integer / bool `top_k` raises `RetrieverError` (mirror `FAISSRetriever.retrieve`)
+- [ ] Regression tests in `tests/test_retriever/test_in_memory.py` for `top_k` in `(0, -1, True)`
+- [ ] `Retriever.retrieve` docstring in `ragframework/base.py` states the contract: non-positive `top_k` → empty list
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed`
+
+## Files to touch
+
+- `ragframework/retriever/in_memory.py` — `retrieve()` (lines 38–48)
+- `ragframework/base.py` — `Retriever.retrieve` docstring
+- `tests/test_retriever/test_in_memory.py`
+
+## Resources
+
+- `FAISSRetriever.retrieve` in `ragframework/retriever/faiss.py` for the reference behaviour
+- `test_non_positive_top_k_returns_empty` in `tests/test_retriever/test_chroma.py`
+
+**Estimated effort:** Small (< 2 hours)

.github/issues/012-loader-unicode-decode-error.md

@@ -0,0 +1,39 @@
+---
+title: "TextFileLoader / MarkdownLoader leak UnicodeDecodeError instead of LoaderError"
+labels: ["bug", "good first issue"]
+---
+
+## Description
+
+Both file loaders only catch `OSError` around `path.read_text(...)`. A file that is not valid in the configured encoding raises a raw `UnicodeDecodeError` (a `ValueError`), and an unknown encoding name raises `LookupError`:
+
+```python
+TextFileLoader().load("latin1.txt")            # UnicodeDecodeError
+TextFileLoader(encoding="nope").load("a.txt")  # LookupError
+```
+
+The documented contract (`ragframework/exceptions.py`, `CONTRIBUTING.md`) is that loaders raise `LoaderError`.
+
+## Motivation
+
+Callers that catch `LoaderError` (or `RAGFrameworkError`) miss these failures. The pipeline still wraps them in `PipelineError`, but direct loader users do not get the promised exception type.
+
+## Acceptance criteria
+
+- [ ] `UnicodeDecodeError` and `LookupError` are converted to `LoaderError` with the file path and encoding in the message
+- [ ] The duplicated exists / is_file / read_text block in `TextFileLoader` and `MarkdownLoader` is extracted into a private helper (e.g. `_read_text_file(path, encoding)`) so the fix lives in one place
+- [ ] Optional: `errors: str = "strict"` constructor argument passed through to `read_text` so users can opt into `"replace"`
+- [ ] Tests for both failure modes in `tests/test_document/test_loaders.py`
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed`
+
+## Files to touch
+
+- `ragframework/document/loaders.py` — `TextFileLoader.load`, `MarkdownLoader.load`
+- `tests/test_document/test_loaders.py`
+
+## Resources
+
+- [`Path.read_text`](https://docs.python.org/3/library/pathlib.html#pathlib.Path.read_text)
+- [Error handlers](https://docs.python.org/3/library/codecs.html#error-handlers)
+
+**Estimated effort:** Small (< 2 hours)

.github/issues/013-fixed-size-chunker-validation.md

@@ -0,0 +1,42 @@
+---
+title: "FixedSizeChunker accepts chunk_size <= 0 and negative chunk_overlap"
+labels: ["bug", "good first issue"]
+---
+
+## Description
+
+`FixedSizeChunker.__init__` only checks `chunk_overlap >= chunk_size`. Invalid values slip through and produce garbage:
+
+```python
+FixedSizeChunker(chunk_size=0, chunk_overlap=-1).chunk(Document("d", "hello"))
+# -> 5 empty chunks
+
+FixedSizeChunker(chunk_size=10, chunk_overlap=-5).chunk(Document("d", "0123456789ABCDEFGHIJKLMNOPQRS"))
+# -> ['0123456789', 'FGHIJKLMNO']   characters A–E and P–S silently dropped
+```
+
+`RecursiveChunker` and `RAGConfig` already validate all three conditions; `FixedSizeChunker` and `SentenceChunker` do not.
+
+## Motivation
+
+Negative overlap silently loses text from the index — the worst kind of bug for a retrieval system, because nothing fails and recall just degrades.
+
+## Acceptance criteria
+
+- [ ] `FixedSizeChunker` raises `ValueError` for `chunk_size <= 0` or `chunk_overlap < 0` (same messages as `RecursiveChunker`)
+- [ ] `SentenceChunker` raises `ValueError` for `max_sentences <= 0` or `overlap_sentences < 0`
+- [ ] `RecursiveChunker.from_config(config)` added, matching the classmethod `FixedSizeChunker` already has
+- [ ] Parametrised tests in `tests/test_document/test_chunkers.py` covering each invalid value
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed`
+
+## Files to touch
+
+- `ragframework/document/chunkers.py` — `FixedSizeChunker.__init__` (line 21), `SentenceChunker.__init__` (line 65)
+- `tests/test_document/test_chunkers.py`
+
+## Resources
+
+- `RecursiveChunker.__init__` (lines 116–121) for the validation to copy
+- `RAGConfig.__post_init__` in `ragframework/config.py`
+
+**Estimated effort:** Small (< 2 hours)

.github/issues/014-inmemory-retriever-dimension-check.md

@@ -0,0 +1,43 @@
+---
+title: "InMemoryRetriever raises raw numpy ValueError on embedding dimension mismatch"
+labels: ["bug", "good first issue"]
+---
+
+## Description
+
+`InMemoryRetriever` performs no validation of vector shape or content:
+
+```python
+r.add([Chunk(id="a", content="a", embedding=[1.0, 0.0, 0.0])])
+r.add([Chunk(id="b", content="b", embedding=[1.0, 0.0])])   # ValueError from np.array (ragged)
+r.retrieve([1.0, 0.0])                                        # ValueError from matrix @ q
+```
+
+`FAISSRetriever` already has `_validated_vector()` that checks 1-D, non-empty, finite, non-zero, and enforces a consistent dimension across `add()` and `retrieve()`, raising `RetrieverError` with a clear message. `InMemoryRetriever` should behave the same way.
+
+## Motivation
+
+Dimension mismatches are the most common mistake when swapping embedders (e.g. 384-dim MiniLM vs 1536-dim OpenAI). Users deserve `RetrieverError: Query embedding has dimension 384; expected 1536.` rather than a numpy shape error.
+
+## Acceptance criteria
+
+- [ ] Move `FAISSRetriever._validated_vector` into a shared helper, e.g. `ragframework/utils/vectors.py::validate_vector(values, label) -> np.ndarray`
+- [ ] `InMemoryRetriever.add()` validates every chunk vector before mutating state and tracks `self._dimension`
+- [ ] `InMemoryRetriever.retrieve()` validates the query vector and its dimension, raising `RetrieverError`
+- [ ] `FAISSRetriever` uses the shared helper (behaviour unchanged, its tests still pass)
+- [ ] Tests in `tests/test_retriever/test_in_memory.py` for: wrong dimension on add, wrong dimension on query, NaN, zero vector, empty vector
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed`
+
+## Files to touch
+
+- `ragframework/utils/vectors.py` — new
+- `ragframework/utils/__init__.py` — export
+- `ragframework/retriever/in_memory.py`
+- `ragframework/retriever/faiss.py` — replace private static method with the helper
+- `tests/test_retriever/test_in_memory.py`
+
+## Resources
+
+- `FAISSRetriever._validated_vector` (`ragframework/retriever/faiss.py` lines 131–144)
+
+**Estimated effort:** Small (< 2 hours)

.github/issues/015-inmemory-retriever-incremental-add.md

@@ -0,0 +1,37 @@
+---
+title: "InMemoryRetriever.add() re-normalises the entire matrix on every call (O(N²))"
+labels: ["bug", "good first issue"]
+---
+
+## Description
+
+Every call to `InMemoryRetriever.add()` rebuilds `self._matrix` from **all** stored chunks:
+
+```python
+self._chunks.extend(chunks)
+vectors = np.array([c.embedding for c in self._chunks], dtype=np.float32)  # all N, every time
+```
+
+Ingesting N documents one at a time is therefore O(N²). Measured locally: 2 000 single-chunk `add()` calls with 64-dim vectors take ~2 s; 20 000 would take minutes.
+
+## Motivation
+
+`RAGPipeline.ingest()` is called once per source, so per-file ingestion of a folder hits exactly this path. Incremental ingestion should be linear.
+
+## Acceptance criteria
+
+- [ ] `add()` normalises only the **new** vectors and appends them (`np.vstack` onto the existing matrix, or keep a list of row blocks and concatenate lazily on first `retrieve()` after a change)
+- [ ] Retrieval results are identical before/after the change (existing tests pass)
+- [ ] A test that adds chunks in several batches and asserts `len(r)` and the top result match a single-batch add
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed`
+
+## Files to touch
+
+- `ragframework/retriever/in_memory.py` — `add()` (lines 25–36)
+- `tests/test_retriever/test_in_memory.py`
+
+## Resources
+
+- [`numpy.vstack`](https://numpy.org/doc/stable/reference/generated/numpy.vstack.html)
+
+**Estimated effort:** Small (< 2 hours)

.github/issues/016-idempotent-retriever-add.md

@@ -0,0 +1,42 @@
+---
+title: "Re-ingesting the same source duplicates chunks in InMemoryRetriever and FAISSRetriever"
+labels: ["bug", "help wanted"]
+---
+
+## Description
+
+`RAGPipeline.ingest(path)` called twice on the same file doubles the index:
+
+```python
+pipe.ingest("doc.txt")   # 12 chunks
+pipe.ingest("doc.txt")   # 12 chunks
+len(pipe.retriever)      # 24  — same 12 chunk ids stored twice
+```
+
+`ChromaRetriever.add()` uses `upsert`, so it is idempotent on chunk id. `InMemoryRetriever` and `FAISSRetriever` blindly append. The `Retriever.add` docstring in `base.py` does not say which behaviour is correct.
+
+## Motivation
+
+Re-running an ingestion script (the normal workflow while iterating) silently pollutes results: the same chunk appears multiple times in `top_k`, crowding out genuinely different context. Chunk ids are already deterministic (`<doc_id>:<index>`, where `doc_id` is a hash of the source path), so idempotency is achievable.
+
+## Acceptance criteria
+
+- [ ] `Retriever.add` docstring in `ragframework/base.py` defines the contract: adding a chunk whose `id` already exists **replaces** it (upsert semantics)
+- [ ] `InMemoryRetriever.add()` replaces existing rows by id (maintain an `id -> row index` dict)
+- [ ] `FAISSRetriever.add()` handles duplicates. `IndexHNSWFlat` does not support removal, so pick and document one approach: (a) keep an `id -> label` map and tombstone superseded labels (filtered out in `retrieve`, over-fetching to compensate), or (b) wrap the index in `faiss.IndexIDMap2` if it supports the chosen index type. Explain the trade-off in the docstring.
+- [ ] Tests for all three retrievers: add the same chunk twice → `len()` unchanged, retrieval returns it once, and the **new** content/embedding wins
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed`
+
+## Files to touch
+
+- `ragframework/base.py` — `Retriever.add` docstring
+- `ragframework/retriever/in_memory.py`
+- `ragframework/retriever/faiss.py`
+- `tests/test_retriever/test_in_memory.py`, `tests/test_retriever/test_faiss.py`, `tests/test_retriever/test_chroma.py`
+
+## Resources
+
+- [FAISS: removing vectors](https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes#removing-elements-from-an-index)
+- `ChromaRetriever.add` for the upsert reference behaviour
+
+**Estimated effort:** Medium (half day)

.github/issues/017-ragconfig-not-applied.md

@@ -0,0 +1,48 @@
+---
+title: "RAGConfig.chunk_size / chunk_overlap / embedding_dim are silently ignored by RAGPipeline"
+labels: ["bug", "help wanted"]
+---
+
+## Description
+
+`RAGPipeline` reads exactly one field from its config — `top_k`. The other three are dead:
+
+```python
+pipe = RAGPipeline(
+    chunker=FixedSizeChunker(),            # chunk_size=512
+    embedder=RandomEmbedder(dim=16),
+    config=RAGConfig(chunk_size=20, chunk_overlap=5, embedding_dim=8),
+    ...
+)
+pipe.chunker.chunk_size   # 512 — config ignored, no warning
+pipe.embedder.dim         # 16  — embedding_dim=8 never checked
+```
+
+The `RAGConfig` docstring says `embedding_dim` "must match the Embedder in use", but nothing verifies it. Only `FixedSizeChunker.from_config()` consumes the chunk fields, and it must be called by hand.
+
+## Motivation
+
+Users reading the README's `config=RAGConfig(top_k=5)` example reasonably assume the other fields work the same way. A config object that looks authoritative but isn't is a trap.
+
+## Proposed resolution
+
+1. **Enforce `embedding_dim`.** In `ingest()` and `query()`, after the first `embed()` call, compare `len(vector)` to `config.embedding_dim` and raise `PipelineError("Embedder produced 16-dim vectors but RAGConfig.embedding_dim is 8")`. Change the default to `embedding_dim: int | None = None` so the check is opt-in and existing users are not broken.
+2. **Make the chunk fields usable.** Add `RAGPipeline.from_config(config, *, loader, embedder, retriever, generator, chunker_cls=FixedSizeChunker)` classmethod that builds the chunker via `chunker_cls.from_config(config)` (depends on `RecursiveChunker.from_config` from #25).
+3. **Document** in the `RAGConfig` docstring which fields the pipeline enforces vs. which are consumed only by `from_config` helpers.
+
+## Acceptance criteria
+
+- [ ] `embedding_dim` mismatch raises `PipelineError` with both numbers in the message; `None` disables the check
+- [ ] `RAGPipeline.from_config(...)` exists and is covered by a test
+- [ ] `RAGConfig` docstring updated; README Quick Start mentions `from_config`
+- [ ] Existing tests pass unchanged (default `RAGConfig()` must still work with `RandomEmbedder(dim=16)` in `tests/`) — hence the `None` default
+- [ ] `CHANGELOG.md` updated under `[Unreleased]` → `### Fixed` / `### Changed`
+
+## Files to touch
+
+- `ragframework/config.py`
+- `ragframework/pipeline/rag.py`
+- `tests/test_pipeline/test_rag.py`, `tests/test_config.py`
+- `README.md`
+
+**Estimated effort:** Medium (half day)

.github/issues/018-getting-started-markdown-broken.md

@@ -0,0 +1,39 @@
+---
+title: "docs/getting-started.md renders as plain text (escaped headings, HTML entities, 322 blank lines)"
+labels: ["bug", "documentation", "good first issue"]
+---
+
+## Description
+
+`docs/getting-started.md` appears to have been exported from a rich-text editor. On GitHub it renders as one unformatted blob:
+
+- Every heading is escaped: `\# Getting Started`, `\## Installation`, `\### Requirements` (29 occurrences)
+- Indentation inside code blocks is written as ` ` HTML entities
+- Underscores are escaped: `chunk\_size`, `top\_k`
+- 322 of the 480 lines are blank (every paragraph is separated by 3 empty lines)
+- File uses CRLF line endings (see the `.gitattributes` issue)
+
+## Motivation
+
+This is the first document a new contributor or user opens after the README. Right now it is unreadable, which undermines the whole onboarding story.
+
+## Acceptance criteria
+
+- [ ] All `\#` headings converted to real Markdown headings with a sensible hierarchy (`#` → `##` → `###`)
+- [ ] ` ` and `\_` escapes removed; code samples are inside fenced ` ```python ` blocks and actually run against the current API
+- [ ] Blank-line bloat collapsed to standard single blank lines between paragraphs
+- [ ] Content checked against the current codebase: mentions `PDFLoader`, `RecursiveChunker`, `HuggingFaceEmbedder`, `FAISSRetriever`, `ChromaRetriever` where relevant
+- [ ] File saved with LF line endings
+- [ ] Linked from `README.md` (it currently is not) and from `CONTRIBUTING.md`
+
+## Files to touch
+
+- `docs/getting-started.md`
+- `README.md` — add a link under Quick Start
+- `CONTRIBUTING.md` — add a link
+
+## Resources
+
+- `README.md` Quick Start and `examples/basic_rag.py` as the source of truth for working code
+
+**Estimated effort:** Small (< 2 hours)