83 regular fixing

.env.example

@@ -92,15 +92,17 @@ ONTOLOGY_CONTEXT_SCOPE=unit              # unit | document
 # of one serialized call. Only worth setting together with
 # ONTOLOGY_CONTEXT_SCOPE=document, where the calls actually share a prefix.
 FANOUT_WARMUP_UNITS=0
-# The catalog ontology for ONTOLOGY_CONTEXT_MODE=fixed_single_ontology in
-# `ontocast process`; it does not change the mode itself. The server does not
-# apply it to requests: pass ontology_context_fixed_ontology_id per request.
+# The catalog ontology for ONTOLOGY_CONTEXT_MODE=fixed_single_ontology, in
+# `ontocast process` and for server requests that name none; it does not change
+# the mode itself.
 #ONTOLOGY_CONTEXT_FIXED_ONTOLOGY_ID=my_ontology_id   # IRI, ontology_id, or author prefix
 # Triple budget for the ontology context serialized into each prompt, in every
 # ontology_context_mode. Over budget, the least load-bearing triples go first
 # (header/list noise, then redundant structure, then comments and definitions);
 # labels, types, hierarchy and domain/range are never dropped, so a graph that
 # cannot fit is passed through with a warning rather than silently gutted.
+# In vector mode with unit scope, VECTOR_STORE_INDUCED_SUBGRAPH_MAX_TOTAL_TRIPLES
+# caps the context first; raise that one to widen it.
 ONTOLOGY_CONTEXT_MAX_TRIPLES=4000
 
 # Stop the run when a FACTS unit's ontology context resolves to zero triples,
@@ -245,11 +247,16 @@ CHUNK_SECTION_TAG_MIN_CHARS=80
 # single-sided ligature gaps that cannot read as two words ("a ffected").
 # Joins the converter cache key when on, so enabling it re-converts.
 # CONVERTER_REPAIR_NUMERIC_ARTIFACTS=false
+# Suffixes converted with Docling (JSON list). Narrow it to refuse formats;
+# .txt/.json/.jsonl are read without Docling. Not part of the cache key.
+# CONVERTER_SUPPORTED_EXTENSIONS=[".pdf", ".docx", ".pptx", ".xlsx", ".html", ".htm", ".md", ".csv", ".adoc", ".png", ".jpg", ".jpeg", ".tif", ".tiff"]
 
 # Triple Store Configuration
-# Leave FUSEKI_URI and FUSEKI_AUTH unset for the default in-memory pyoxigraph
-# backend (process lifetime). Fuseki is used only when BOTH are set.
-# When FUSEKI_DATASET is unset, names derive from default tenant/project (ontocast/test).
+# Leave FUSEKI_URI unset for the default in-memory pyoxigraph backend (process
+# lifetime). Setting it selects Fuseki; FUSEKI_AUTH only if credentials are needed.
+# Dataset names derive from the tenant/project (default ontocast/test). `ontocast
+# serve` and `ontocast process` replace an explicit FUSEKI_*_DATASET with the
+# derived name and log a warning; only an embedded ToolBox reads it.
 
 # Fuseki Configuration (optional — production)
 # Server root only — SPARQL is at {FUSEKI_URI}/{dataset}/sparql (not the UI link "#/dataset/…").
@@ -282,10 +289,9 @@ ONTOCAST_CACHE_DIR=/path/to/cache/directory
 # checkpoint written two ways loads twice. Align all three to hold one resident
 # model instead of two -- see https://growgraph.github.io/ontocast/guides/performance/
 AGG_EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
-# Fallback threshold of the cross-graph entity aligner when the caller names
-# none (the ontocast_align_entities agent tool); POST /match/entities and
-# match-graphs take their own similarity_threshold. The in-pipeline aggregator
-# does NOT read it -- it clusters and gates at AGG_CANDIDATE_SIMILARITY_THRESHOLD.
+# Threshold of the cross-graph entity aligner when the caller names none
+# (POST /match/entities, match-graphs, the ontocast_align_entities agent tool).
+# The in-pipeline aggregator does NOT read it -- it clusters and gates at AGG_CANDIDATE_SIMILARITY_THRESHOLD.
 AGG_SIMILARITY_THRESHOLD=0.80
 # Merge-guard tunables (thresholds for the lexical tiers and corpus inferences)
 # AGG_CANDIDATE_SIMILARITY_THRESHOLD=0.70
@@ -809,7 +815,8 @@ LOGGING_LEVEL=info
 # VECTOR_STORE_DEDUP_INCLUDE_HASH=true
 # VECTOR_STORE_DEDUP_QUERY_HITS_BY_IRI=true
 
-# --- Collection / table names (derived from tenant+project when unset) -----
+# --- Collection / table names (derived from tenant+project; serve and process
+# replace an explicit value, see FUSEKI_DATASET) ----------------------------
 # VECTOR_STORE_ONTOLOGY_TABLE=
 # VECTOR_STORE_FACTS_TABLE=
 # QDRANT_ONTOLOGY_COLLECTION=

.env.example.minimal

@@ -66,9 +66,8 @@ RENDER_MODE=ontology_and_facts
 # fixed_single_ontology           -> pin one ontology for the whole run.
 ONTOLOGY_CONTEXT_MODE=selected_single_ontology
 
-# Only for fixed_single_ontology in `ontocast process`; it does not change the
-# mode itself. The server ignores it: pass ontology_context_fixed_ontology_id
-# with each request instead.
+# Only for fixed_single_ontology (CLI, and server requests that name none);
+# it does not change the mode itself.
 #ONTOLOGY_CONTEXT_FIXED_ONTOLOGY_ID=            # IRI, ontology_id, or author prefix
 
 # Encoding the LLM writes graphs in. `turtle` is worth trying if a provider's
@@ -212,8 +211,8 @@ FACTS_SHACL_AUTOFIX=prune            # prune | rewrite | off
 
 # -- 11. Storage and serving ---------------------------------------------
 # Leave these unset for the in-memory store -- right for trying OntoCast out
-# and for single-shot batch runs; nothing persists. Fuseki is used only when
-# BOTH FUSEKI_URI and FUSEKI_AUTH are set.
+# and for single-shot batch runs; nothing persists. Setting FUSEKI_URI selects
+# Fuseki; FUSEKI_AUTH is needed only if the server requires credentials.
 #FUSEKI_URI=http://localhost:3030
 #FUSEKI_AUTH=admin/admin
 

.gitignore

@@ -70,9 +70,6 @@ instance/
 # Scrapy stuff:
 .scrapy
 
-# Sphinx documentation
-docs/_build/
-
 # PyBuilder
 .pybuilder/
 target/

CHANGELOG.md

@@ -13,13 +13,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   pull request; the `cla` status check tracks acceptances (write access and bots are exempt).
 - **`plot-graph --format`** selects any of `svg`, `png`, `pdf` (default `svg,png`).
 - **`ontocast.config.env_names`:** environment-variable names of every settings field.
+- **More input formats:** Word, Excel, HTML, Markdown, CSV, AsciiDoc and images
+  convert through Docling alongside PDF and PowerPoint;
+  `CONVERTER_SUPPORTED_EXTENSIONS` narrows the set, and `ontocast process` warns
+  about each file it skips by suffix.
+- **`GET /ontologies`** lists the current version of each catalog ontology in the
+  request's tenant and project.
+- **Startup warnings:** `serve` and `process` name any dataset, collection or table
+  setting replaced by the tenant/project-derived name; a raised
+  `ONTOLOGY_CONTEXT_MAX_TRIPLES` that the induced-subgraph cap overrides is reported.
+- **`llm/ontology_selection`** in `budget.counters` counts the per-unit ontology
+  selection calls of `selected_single_ontology` mode.
+- **`ToolBox.ensure_vector_store`** prepares the vector store for a mode on demand.
 
 ### Changed
 
 - **`plot` extra** requires `pygraphviz>=2.0,<3`, whose wheels bundle Graphviz.
 - **`docs` extra:** `properdocs` replaces `mkdocs`; `mkdocs-jupyter` and the
   `griffe<2` pin are dropped.
 - **`plot-graph`** no longer writes `graph.preview.png` (a remote mermaid.ink call).
+- **`strip_provenance`:** an unrecognised value returns `400` (it was read as `false`).
+- **`FUSEKI_URI` alone selects Fuseki;** `FUSEKI_AUTH` is optional. Previously the
+  URI without auth fell back to the in-memory store. A `401`/`403` while creating a
+  dataset now stops startup with `FusekiAccessError` (it was logged and ignored).
+- **`ToolBox.initialize()`** without `ontology_context_mode` prepares what
+  `ONTOLOGY_CONTEXT_MODE` asks for, including the vector index.
+- **`Config.in_memory()`** leaves the vector backend on `auto`, so enabling LanceDB
+  on the result takes effect.
+- **Graph evaluation** (`POST /match/evaluate`, `match-graphs`): a precision,
+  recall or F1 with a zero denominator is `null` (was `0.0`), and entity metrics
+  count subjects and objects that are not vocabulary (predicates were counted).
+- **Section labels from content density:** acknowledgements need two gratitude or
+  funding cues, and an excluded density label no longer spreads to the following
+  unlabelled segments. Fewer methods chunks are dropped as acknowledgements.
+- **LLM cache key** includes `json_mode` when `LLM_JSON_MODE` is on; entries
+  written with it off keep their keys.
+- **Fixed ontology mode over HTTP:** a request that names no
+  `ontology_context_fixed_ontology_id` uses `ONTOLOGY_CONTEXT_FIXED_ONTOLOGY_ID`
+  (every such request returned `400`).
 - **`plot-graph`:** the left-to-right pipeline figure uses larger labels and
   tighter spacing, so it stays legible at page width.
 
@@ -32,19 +63,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **Section-label matching** compiles each heading pattern and keyword once. The
   catalog outgrows `re`'s cache, so schema detection recompiled it for every heading.
-- **`GET /info`** `input_types` lists what `/process` accepts: `txt`, `json`,
-  `pdf`, `pptx` (it advertised `text` and `markdown`).
+- **`GET /info`** `input_types` lists what `/process` accepts on this install (it
+  advertised `text`, `markdown` and `jsonl`, and PDF without the `doc-processing`
+  extra; JSONL is read only by `ontocast process`).
 - **LangChain tools:** `ontocast_delete_ontology` no longer claims to delete the
   seed file (it keeps it; the ontology reloads on the next start), and
-  `ontocast_convert_document` names the formats it converts: PDF and PowerPoint.
+  `ontocast_convert_document` names the formats it converts.
 - **Vector-search context error** names both ways to enable a vector store,
   `QDRANT_URI` or `LANCEDB_ENABLED=true`.
 - **CLI help:** `--section-schema-id` lists every schema (`patent`, `standard`,
   `news` were missing); `--summarize-sections` says summaries are written per
   content unit.
-- **`AGG_SIMILARITY_THRESHOLD`** description and warning say what reads it: the
-  aligner's fallback when a caller names no threshold. `POST /match/entities` and
-  `match-graphs` use their own `similarity_threshold`.
+- **Ontology version bumps** see class and property additions and deletions; they
+  were never detected, so no update produced a major bump.
+- **Section exclusion log** names each dropped label with its size and labelling tier.
+- **Converter availability:** without the `doc-processing` extra no converted format
+  is accepted or advertised (PDF and PPTX were, then failed on import).
+- **`plot-graph`** runs without LLM credentials or storage services.
+- **`VECTOR_STORE_BACKEND=lancedb`** without `LANCEDB_ENABLED` gets the BM25 tool
+  (it failed on first search).
+- **A vector-mode request on the startup scope** prepares the vector index instead
+  of returning `409` when the server started in another mode.
+- **LangChain `ontocast_extract`** is offered only when `docling-core` is installed.
+- **Per-tenant requests** reuse their scope's compiled graph; it was recompiled on
+  every request with `?tenant=` or `?project=`.
+- **A blank `tenant` or `project`** returns `400` on every route; outside `/flush`
+  it returned `500`.
+- **`POST /match/entities` and `match-graphs`** default `similarity_threshold` and
+  `embedding_model` to `AGG_SIMILARITY_THRESHOLD` / `AGG_EMBEDDING_MODEL` (they
+  hard-coded `0.80` and an unprefixed model name that loaded a second copy).
 
 ### Documentation
 
@@ -55,8 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   environment variable; setting descriptions rewritten to state behaviour only.
 - Pipeline and per-unit loop diagrams redrawn in the new palette and served as SVG;
   the loop diagrams state the real convergence rule and the facts loop's exits.
-- Docs no longer claim OntoCast reads a `.env` file (it reads the environment), or
-  that `FUSEKI_URI` alone enables Fuseki (`FUSEKI_AUTH` is also required).
+- Docs no longer claim OntoCast reads a `.env` file (it reads the environment).
 - `demo/README.md` sample paths fixed; public callables fully annotated.
 
 ## [0.6.4] - 2026-09-16

Dockerfile

@@ -12,8 +12,8 @@ WORKDIR /app
 COPY pyproject.toml uv.lock ./
 RUN touch README.md
 
-# Everything except `plot` (pygraphviz builds against system graphviz headers),
-# `dev`, and `docs`. `server` + the LLM providers are what make the image a
+# Everything except `plot` (`plot-graph` is a development tool; its wheel would
+# also need X11/glib libraries and a font in this slim image), `dev`, and `docs`. `server` + the LLM providers are what make the image a
 # runnable API server; a base install has no CLI and no provider.
 RUN uv sync --all-extras --no-extra plot --no-extra dev --no-extra docs
 

README.md

@@ -47,7 +47,7 @@ pip install "ontocast[server,openai,doc-processing]"
 
 `server` provides the `ontocast` command and HTTP API, `openai` the model
 provider (`anthropic`, `google` and `ollama` also exist), and `doc-processing`
-the PDF and PowerPoint converter. All extras:
+the document converter (PDF, Office, HTML, Markdown, images). All extras:
 [Installation](https://growgraph.github.io/ontocast/getting_started/installation/).
 
 ## Quick start

docker/README.md

@@ -7,15 +7,15 @@ This directory contains Docker Compose stacks for **optional** infrastructure On
 | **Apache Fuseki** | `fuseki/` | Persistent RDF triple store (production) |
 | **Qdrant** | `qdrant/` | Vector store for ontology patch retrieval |
 
-OntoCast does **not** require Docker to run. When `FUSEKI_URI` / `FUSEKI_AUTH` are unset, the server uses an **in-memory pyoxigraph** triple store automatically (data is not persisted across restarts).
+OntoCast does **not** require Docker to run. When `FUSEKI_URI` is unset, the server uses an **in-memory pyoxigraph** triple store automatically (data is not persisted across restarts).
 
 ---
 
 ## Triple store backend selection
 
 OntoCast picks the triple store in `ToolBox` at startup:
 
-1. **Fuseki** — when both `FUSEKI_URI` and `FUSEKI_AUTH` are set
+1. **Fuseki** — when `FUSEKI_URI` is set. `FUSEKI_AUTH` is required by this bundled setup (its `shiro.ini` asks for credentials on every path); a Fuseki that allows anonymous access to the admin API and datasets needs none
 2. **In-memory** — otherwise (zero git config, process-local only)
 
 There is no filesystem or Neo4j fallback anymore. Use Fuseki (below) when you need durable RDF storage.
@@ -185,7 +185,7 @@ curl http://localhost:6333/healthz
 ### Common problems
 
 - **Connection refused** — container not running or wrong port in OntoCast `.env`
-- **Authentication failed** — `FUSEKI_AUTH` must be `user/password` (slash-separated)
+- **Authentication failed** (`FusekiAccessError ... 401` at startup) — `FUSEKI_AUTH` is missing or wrong; it is `user/password` and the user needs admin rights to create datasets
 - **Wrong URI** — use the Fuseki HTTP root (`http://host:port`), not `/#/dataset/...` UI links
 - **Data gone after restart** — expected with in-memory backend; run Fuseki for persistence