Move the docs stacklet to Paperless-ngx 3.0.4

AGENTS.md

@@ -86,6 +86,7 @@ Apply to every role, every session.
 8. **Destructive ops need confirmation.** `stack destroy`, `stack uninstall`, `rm -rf`, `git reset --hard`, force-push.
 9. **Announce actions before running them.** No silent long running or integration test runs, scripts, or background commands.
 10. **No em dashes in user-facing prose.** Use hyphens or sentence breaks.
+11. **Put data in through the front door.** Seed and exercise a stacklet the way a family does, never by writing to the service behind it. Documents go through `tools/family-docs/ingest.py` (Matrix room -> archivist -> OCR -> classify -> mirror), never a direct `POST /api/documents/post_document/`. The back door skips the pipeline, so what lands is not what users get: no tags, no correspondent, no document type, no rewritten title, no summary note, no vault entry. Any conclusion drawn from that data is about a system famstack does not ship.
 
 ## Deeper docs (load on demand)
 

docs/admin-guide.md

@@ -740,6 +740,25 @@ If `git pull` shows changes to `lib/stack/` or to a stacklet you are running, re
 
 Pre-1.0 caveat: occasionally a release changes a config schema or env var. Release notes will say so. Read them before pulling.
 
+### Paperless-ngx 3.x (docs stacklet)
+
+This release moves the `docs` stacklet from Paperless-ngx 2.20.15 to 3.0.4.
+
+**Back up `~/famstack-data/docs/` before you restart the stacklet.** Paperless migrates its database the first time 3.x starts, and 2.x refuses to boot on a migrated database. There is no downgrade. If the upgrade goes wrong, restoring that backup is the only way back.
+
+```bash
+./stack down docs
+tar -czf ~/docs-pre-3x-backup.tar.gz -C ~/famstack-data docs
+git pull
+./stack up docs
+```
+
+The backup stacklet does not cover this. It archives files, not the Postgres database, so a `stack backup sync` alone will not get you back to 2.x.
+
+**If Watchtower already moved you to 3.x.** Older famstack releases tracked the `:latest` Paperless tag, and Watchtower's nightly pull rolled some instances from 2.x straight to 3.0 without asking. If that happened to you, your database is already migrated and pinning the image back to 2.20.15 will not start. You need a backup taken before the 3.x boot. Without one, staying on 3.x is the only option, which this release makes the supported path. Paperless is now pinned to an exact tag, so Watchtower can still deliver 3.0.x patches but can no longer jump a major version on its own.
+
+**One behaviour change worth knowing.** 3.0 stopped rejecting a re-uploaded identical file by default and would have filed a second copy instead. famstack sets `PAPERLESS_CONSUMER_DELETE_DUPLICATES` to keep the old behaviour: send the same receipt twice and the archivist replies "already filed", with a link to the document you sent the first time.
+
 ---
 
 ## Uninstall

docs/agent/dev.md

@@ -184,6 +184,7 @@ Testing rules:
 - **Behavioural TDD: RED then GREEN.** Write the failing test that captures the behaviour you want; make it pass with the smallest change; then refactor.
 - **Blackbox at the module boundary.** Test what a module promises through its public surface. Mock only external interfaces (network endpoints, the LLM), and only when truly required.
 - **Prefer a real model over a stubbed one.** `tests/integration/stacktests ai local` points the rig at a self-hosted endpoint: real answers, no cost per call, only slower. A green run against a stub proves the wiring, not the behaviour.
+- **Seed through the front door.** See [AGENTS.md non-negotiable 11](../../AGENTS.md). To populate an instance with documents, run `tools/family-docs/ingest.py`: it posts each rendered demo document into the Matrix `documents` room as the family member who would plausibly have sent it, so the archivist picks it up and the whole pipeline runs. Uploading to `POST /api/documents/post_document/` instead is faster and produces the wrong corpus - bare OCR text with a filename for a title, no tags, correspondent, document type, summary note, or vault entry. Search, classification, and mirror behaviour all read differently against that, so measurements taken on it are worthless. The same rule holds for every stacklet: drive the user-facing surface, not the service behind it.
 - **Docker integration tests when warranted.** If a change crosses a container boundary or depends on a real service's behaviour, add a test under `tests/integration/`.
 - **Tests run against real stacklets and real hooks.** No parallel test-only compose files.
 - **Use real Synapse via the `messages` stacklet.** No handwritten Matrix mocks.

stacklets/docs/bot/pipeline.py

@@ -117,34 +117,43 @@ class EnrichResult:
     llm_error: tuple[str, str] | None = None
 
 
-# ── Whoosh query translation ─────────────────────────────────────────────
-
-# Whoosh special-character set (https://whoosh.readthedocs.io/). When any
-# of these appears in a token we assume the caller is writing Whoosh
-# syntax on purpose (wildcards, fielded search, fuzzy match, grouping,
-# quoted phrases) and leave the token alone.
-_WHOOSH_SPECIAL = set("*?:()\"~")
-
-
-def _to_whoosh_query(q: str) -> str:
-    """Translate a chat search input into a Whoosh query string.
-
-    The visible problem this solves: a user types `pangasius` expecting
-    to find the doc titled "Pangasiusfilet". Whoosh tokenises titles
-    on whitespace and lowercases them, so the index has the token
-    `pangasiusfilet` -- a bare-term query for `pangasius` is not a
-    full token and German compound-splitting only kicks in for words
-    in the analyzer's dictionary (it splits "fisch" out of
-    "fischrezept" but not "pangasius" out of "pangasiusfilet"). The
-    fix is to wildcard each user-supplied token so prefix matching
-    catches "pangasius*" against "pangasiusfilet".
-
-    Each whitespace-separated token gets a `*` appended unless it
-    already contains a Whoosh special character (the caller is using
-    intentional syntax) or is itself a Whoosh operator (AND/OR/NOT).
-    Tokens are joined back with spaces, which Whoosh treats as AND
-    under Paperless's default operator -- so multi-word queries narrow
-    the set rather than blowing it up.
+# ── Search query translation ─────────────────────────────────────────────
+
+# Characters that mean the caller is driving the query parser on purpose
+# (wildcards, fielded search, fuzzy match, grouping, quoted phrases).
+# A token containing any of them is passed through untouched.
+_QUERY_SPECIAL = set("*?:()\"~")
+
+_QUERY_OPERATORS = {"AND", "OR", "NOT"}
+
+
+def _to_search_query(q: str) -> str:
+    """Translate a chat search input into a Paperless search query.
+
+    Two jobs, and they pull in opposite directions.
+
+    **Reach.** A user types `pangasius` expecting the doc titled
+    "Pangasiusfilet". The index holds the whole token, so a bare term
+    does not match; appending `*` makes it a prefix query that does.
+
+    **Precision.** Paperless 3.0 replaced Whoosh with tantivy, whose
+    parser joins bare terms with OR, not AND. That inverts what this
+    function used to claim. Measured against the 25-document demo
+    archive on 3.0.4, `homer* car* insurance*` returned 18 of 24
+    documents, because a document needed only one of the three words.
+    Joining with an explicit AND returns the documents that carry all
+    of them: `marge* AND recipe*` gives 1 (the recipe card) where the
+    OR form gave 11.
+
+    So: wildcard every bare token, then join with AND.
+
+    The AND is only injected when the query is plainly a bag of words.
+    If the caller already wrote an operator or a quoted phrase, their
+    structure is preserved and tokens are rejoined with spaces, because
+    injecting AND into either one breaks it: `"birth certificate"` would
+    become `"birth AND certificate"` and match nothing, and
+    `insurance OR recipe` would become `insurance* AND OR AND recipe*`,
+    which the parser rejects outright.
 
     Empty input passes through unchanged so a defensive caller doesn't
     accidentally search for `*` (which would match every doc).
@@ -153,14 +162,14 @@ def _to_whoosh_query(q: str) -> str:
         return q
     tokens: list[str] = []
     for tok in q.split():
-        if tok.upper() in {"AND", "OR", "NOT"}:
-            tokens.append(tok)
-            continue
-        if any(c in _WHOOSH_SPECIAL for c in tok):
+        if tok.upper() in _QUERY_OPERATORS or any(c in _QUERY_SPECIAL for c in tok):
             tokens.append(tok)
             continue
         tokens.append(f"{tok}*")
-    return " ".join(tokens)
+    caller_wrote_structure = (
+        '"' in q or any(t.upper() in _QUERY_OPERATORS for t in tokens)
+    )
+    return (" " if caller_wrote_structure else " AND ").join(tokens)
 
 
 # ── Paperless HTTP wrapper ───────────────────────────────────────────────
@@ -182,6 +191,32 @@ def _task_document_id(task: dict) -> int | None:
     return int(doc_id) if doc_id else None
 
 
+def _task_duplicate_id(task: dict) -> int | None:
+    """The already-filed twin's id when the upload was rejected as a duplicate.
+
+    Paperless 3.0 replaced the free-text rejection with a structured one.
+    A duplicate task is still reported as ``failure``, but the message is
+    gone: what identifies the twin is ``result_data == {"duplicate_of":
+    <id>, "duplicate_in_trash": <bool>}``. 2.x instead named it in a
+    free-text ``result``, which ``_DUPLICATE_RE`` scrapes.
+
+    Worth knowing when changing the caller: 3.x also sets
+    ``related_document_ids`` to the *twin* on a rejected upload. Nothing
+    but ``result_data`` distinguishes that payload from a successful
+    filing, so a doc-id read on a failed task would hand back a real,
+    existing document as though this upload had produced it. Upstream
+    forces such a task to ``failure`` (``signals/handlers.py``), which is
+    what keeps the success path clear of it.
+
+    Returning None means "this task is not a duplicate rejection", not
+    "no id available".
+    """
+    result_data = task.get("result_data")
+    if isinstance(result_data, dict) and result_data.get("duplicate_of"):
+        return int(result_data["duplicate_of"])
+    return None
+
+
 class PaperlessAPI:
     """Async client for every Paperless endpoint the docs stacklet touches.
 
@@ -242,11 +277,23 @@ async def get_doc(self, doc_id: int) -> dict | None:
         body, _ = await self._req("GET", f"/api/documents/{doc_id}/")
         return body if isinstance(body, dict) else None
 
+    async def _doc_title(self, doc_id: int) -> str | None:
+        """A document's title, or None if it can't be read.
+
+        Only used to name the twin in a duplicate rejection: 3.x reports
+        the duplicate as an id alone, where 2.x embedded the title in the
+        failure text. The chat reply degrades to "(no title)" without it,
+        so an unreachable lookup must not turn a duplicate into an error.
+        """
+        doc = await self.get_doc(doc_id)
+        title = doc.get("title") if doc else None
+        return title if isinstance(title, str) and title else None
+
     async def search(self, query: str, limit: int = 5) -> list[dict]:
         body, _ = await self._req(
             "GET", "/api/documents/",
             params={
-                "query": _to_whoosh_query(query),
+                "query": _to_search_query(query),
                 "page_size": limit,
                 "ordering": "-created",
             },
@@ -341,6 +388,15 @@ async def wait_task(self, task_id: str, timeout: int = 120) -> int | None:
                             if status == "SUCCESS":
                                 return _task_document_id(task)
                             if status == "FAILURE":
+                                # A duplicate is a failed task in both
+                                # generations, but only 2.x explains itself
+                                # in words. Ask the structured field first;
+                                # 3.x leaves `result` empty.
+                                dup_id = _task_duplicate_id(task)
+                                if dup_id is not None:
+                                    raise PaperlessDuplicateError(
+                                        dup_id, await self._doc_title(dup_id),
+                                    )
                                 result = task.get("result") or ""
                                 logger.error("[pipeline] Task failed: {}", result)
                                 match = _DUPLICATE_RE.search(result)

stacklets/docs/bot/recall.py

@@ -50,7 +50,7 @@ async def resolve_search_query(
         to `re.search`. Always non-empty.
       * `paperless_query` is the Whoosh-compatible string the bot
         passes to `PaperlessAPI.search`. The API client wildcards
-        bare tokens itself (`_to_whoosh_query`), so we only have to
+        bare tokens itself (`_to_search_query`), so we only have to
         get the *operators* right here — bare tokens joined with ` OR `
         for question mode, the raw input for literal mode.
       * `keywords_used` is the list the LLM produced (empty when
@@ -107,7 +107,7 @@ async def resolve_search_query(
     # compile step.
     memory_regex = "|".join(re.escape(k) for k in keywords)
     # Paperless side: Whoosh OR alternation, bare tokens. The
-    # PaperlessAPI wraps this in `_to_whoosh_query` which adds the
+    # PaperlessAPI wraps this in `_to_search_query` which adds the
     # prefix wildcard on each bare term -- so "fish OR price" becomes
     # "fish* OR price*" downstream.
     paperless_query = " OR ".join(keywords)

stacklets/docs/docker-compose.yml

@@ -15,11 +15,12 @@ name: stack-docs
 services:
   stack-docs-paperless:
     container_name: stack-docs-paperless
-    # Pinned to the last 2.x release. `:latest` silently rolled to 3.0 (a
-    # major with a redesigned tasks API) and broke document filing; the 3.x
-    # upgrade is a deliberate, separately-branched migration, not a surprise
-    # watchtower pull. Bump within 2.20.x for patches; jump majors on purpose.
-    image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15
+    # Pinned to an exact release, never `:latest`. A watchtower pull of
+    # `:latest` once rolled 2.x straight to 3.0 and broke document filing.
+    # Paperless migrates its database on the first 3.x start and 2.x will
+    # not boot afterwards, so a major bump is one-way: do it deliberately,
+    # with a backup taken first. Bump within 3.0.x for patches.
+    image: ghcr.io/paperless-ngx/paperless-ngx:3.0.4
     labels:
       - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_ENABLE:-true}"
     networks:
@@ -55,6 +56,11 @@ services:
       PAPERLESS_ADMIN_PASSWORD: ${ADMIN_PASSWORD}
       PAPERLESS_URL: ${PAPERLESS_URL}
       PAPERLESS_OCR_MODE: skip
+      # 2.x always rejected a re-uploaded identical file. 3.0 made that
+      # conditional on this flag: with it unset the duplicate is consumed
+      # anyway and the family ends up with two copies of the same letter.
+      # Set it to keep "already filed" the answer, not a silent second copy.
+      PAPERLESS_CONSUMER_DELETE_DUPLICATES: 1
       PAPERLESS_OCR_OUTPUT_TYPE: pdfa
       PAPERLESS_FILENAME_FORMAT: "{created_year}/{created_month}/{created_year}-{created_month}-{created_day} {correspondent} - {title}"
     ports:

tests/integration/stacktests

@@ -95,7 +95,7 @@ case "${1:-}" in
       done
     else
       shift
-      exec stack_cli up "$@"
+      stack_cli up "$@"
     fi
     ;;
 
@@ -111,7 +111,7 @@ case "${1:-}" in
         stack_cli down "$sid" || true
       done
     else
-      exec stack_cli down "$@"
+      stack_cli down "$@"
     fi
     ;;
 
@@ -373,6 +373,6 @@ EOF
 
   *)
     # Anything else — forward untouched to stack.
-    exec stack_cli "$@"
+    stack_cli "$@"
     ;;
 esac