Provisional SHACL claim validation + alignment-gap analysis

api/shacl_validation.py

@@ -0,0 +1,69 @@
+"""SHACL validation for claims (OFF by default).
+
+Validates a claim's RDF representation against the FWG LinkML-derived SHACL
+shape (``schema/shacl/claim.shacl.ttl``, generated from regen-data-standards
+``Claim.yaml``).
+
+Two hard design constraints, both learned the hard way:
+
+1. **Never block the event loop.** The personal koi-processor runs as a
+   single-worker service; a synchronous ``pyshacl.validate()`` on the hot
+   ``POST /claims/`` path can starve the asyncpg pool and take the whole
+   service down. All validation runs in ``asyncio.to_thread()``.
+
+2. **Fail closed, not silent.** If validation is enabled but the shape file is
+   missing, raise — do not silently accept unvalidated claims (a silent skip
+   gives false confidence).
+
+STATUS (2026-06-05): this module is **not yet wired into create_claim**. The
+koi-processor claim JSON-LD (flat keys: ``claimant_uri``/``claim_type``/
+``statement``) does not match the LinkML ``rfs:Claim`` slot vocabulary
+(``hasClaimant``/``hasSubject``/``hasPrimaryImpact``/...). Enabling validation
+requires aligning the claim representation first. See
+``docs/claims/shacl-alignment-gap.md``.
+"""
+from __future__ import annotations
+
+import asyncio
+import os
+from pathlib import Path
+
+# schema/shacl/claim.shacl.ttl relative to repo root (this file is api/shacl_validation.py)
+_DEFAULT_SHAPE = Path(__file__).resolve().parent.parent / "schema" / "shacl" / "claim.shacl.ttl"
+
+
+def shacl_enabled() -> bool:
+    """True only when VALIDATE_CLAIMS_SHACL is explicitly 'true'. Default off."""
+    return os.getenv("VALIDATE_CLAIMS_SHACL", "false").strip().lower() == "true"
+
+
+def _validate_sync(data_ttl: str, shape_path: str) -> tuple[bool, str]:
+    """Blocking validation — only ever called via asyncio.to_thread()."""
+    import rdflib  # imported lazily so the module loads even if deps are absent
+    from pyshacl import validate
+
+    data_graph = rdflib.Graph().parse(data=data_ttl, format="turtle")
+    shape_graph = rdflib.Graph().parse(shape_path, format="turtle")
+    conforms, _results_graph, results_text = validate(
+        data_graph,
+        shacl_graph=shape_graph,
+        inference="rdfs",
+        advanced=True,
+        meta_shacl=False,
+    )
+    return bool(conforms), results_text
+
+
+async def validate_claim_ttl(data_ttl: str, shape_path: str | os.PathLike | None = None) -> tuple[bool, str]:
+    """Validate a claim RDF graph (Turtle) against the claim SHACL shape.
+
+    Returns (conforms, human_readable_report). Runs pyshacl off the event loop.
+    Raises RuntimeError if the shape file is missing (fail-closed).
+    """
+    shape = Path(shape_path) if shape_path else _DEFAULT_SHAPE
+    if not shape.exists():
+        raise RuntimeError(
+            f"SHACL shape file not found: {shape} — refusing to skip validation (fail-closed). "
+            "Check the schema/shacl/ directory is present in the deployed checkout."
+        )
+    return await asyncio.to_thread(_validate_sync, data_ttl, str(shape))

docs/claims/shacl-alignment-gap.md

@@ -0,0 +1,59 @@
+# Claims SHACL validation — alignment gap (provisional, 2026-06-05)
+
+**Status:** infrastructure landed; **validation NOT yet wired into `create_claim`** because the koi-processor claim representation is not aligned with the canonical LinkML `rfs:Claim` shape. This doc records the gap with an executable proof and the two ways to close it.
+
+## What landed on this branch (`darren/claims-shacl-validation`)
+
+- `schema/shacl/claim.shacl.ttl` — SHACL shape generated from regen-data-standards **PR #53** `Claim.yaml` (`linkml` `ShaclGenerator`, 419 triples). *Provisional:* PR #53 is unmerged (`upstream/pr-53-head`); regenerate when it merges.
+- `api/shacl_validation.py` — validation helper. Runs `pyshacl` inside `asyncio.to_thread()` (never block the single-worker event loop), **fail-closed** if the shape file is missing, gated by `VALIDATE_CLAIMS_SHACL` (default **false**).
+- `requirements.txt` — adds `pyshacl`, `rdflib`.
+- `tests/test_shacl_claim_shape.py` — codifies the proof below.
+
+## The gap
+
+koi-processor already emits an `rfs:Claim` JSON-LD (used for content hashing — `claims_router.py` `_canonical_json` ~line 392 and the proof-pack ~line 2492):
+
+```json
+{ "@context": "https://framework.regen.network/schema/", "@type": "rfs:Claim",
+  "claimant_uri": "...", "claim_type": "ecological", "statement": "...", "about_uri": "..." }
+```
+
+The PR #53 LinkML `rfs:Claim` shape is **`sh:closed`** and requires structured slots:
+`schema:name`, `rfs:hasClaimType`, `rfs:verificationStatus`, `rfs:hasClaimant` (Entity), `rfs:hasSubject` (Entity), `rfs:hasPrimaryImpact` (Impact). Nested `Entity`/`Impact` have their own required slots + controlled-vocabulary enums.
+
+Same `@type` IRI, **completely different property vocabulary**. So today's claims fail validation 100% (closed-shape violations on every flat key + every required slot missing).
+
+## Executable proof (throwaway venv, pyshacl 0.31 / rdflib 7.6)
+
+```
+shape: 419 triples loaded from claim.shacl.ttl
+(A) LinkML-slot-aligned claim: conforms=True
+(B) today's koi flat-key claim: conforms=False
+   Message: Node <urn:claim:koi> is closed. It cannot have value: Literal("Soil carbon increased 2 tC/ha/yr")
+   Message: Node <urn:claim:koi> is closed. It cannot have value: Literal("ecological")
+   Message: Node <urn:claim:koi> is closed. It cannot have value: Literal("orn:koi-net.entity:demo")
+```
+
+(A) passes only when shaped to the LinkML slots with valid enum values (Entity `rfs:type` ∈ {Individual, Organization, Community}; Impact `rfs:hasImpactType` from the ImpactType vocab; `rfs:verificationStatus` ∈ {SelfReported, PeerReviewed, Verified, LedgerAnchored, Withdrawn}). The shape is correct and strict — the toolchain works.
+
+## Two ways to close the gap (pick upstream, with FWG / Marie)
+
+1. **Author a JSON-LD `@context`** mapping koi's flat keys → LinkML slot URIs, and synthesize the required structured nodes (claimant/subject as `Entity`, a primary `Impact`, a `verificationStatus`) at validation time. Lets the existing storage model stand; the mapping is the work. **Problem:** `hasSubject` and `hasPrimaryImpact` are *required* but have **no source field** in `ClaimCreateRequest` — they can't be mapped, only invented. So a pure context mapping is insufficient without model changes.
+2. **Extend the claim model** (`ClaimCreateRequest` + storage) to carry the LinkML semantics (typed claimant/subject entities, primary impact, verification status) and emit slot-keyed JSON-LD. The faithful fix; larger; should follow PR #53 merging and coordinate with the data-standards alignment work.
+
+Either way, **SHACL can't be meaningfully enabled until the claim representation aligns.** Until then this stays off-by-default and unwired.
+
+## When ready to wire (exact spot)
+
+In `api/routers/claims_router.py::create_claim`, after the `about_uri` validation block (~line 652, before RID generation ~line 655):
+
+```python
+from api.shacl_validation import shacl_enabled, validate_claim_ttl
+...
+if shacl_enabled():
+    conforms, report = await validate_claim_ttl(claim_to_aligned_ttl(body))  # claim_to_aligned_ttl: the gap above
+    if not conforms:
+        raise HTTPException(status_code=422, detail=f"Claim failed SHACL validation:\n{report}")
+```
+
+Do **not** enable `VALIDATE_CLAIMS_SHACL=true` on the live single-worker service until (a) alignment is done and (b) latency is profiled — synchronous validation on the hot path is the documented outage risk (`asyncio.to_thread` mitigates event-loop blocking but pool pressure still applies under load).

requirements.txt

@@ -102,3 +102,10 @@ scrapling>=0.4
 tldextract>=5.0
 tiktoken>=0.7
 feedparser>=6.0.10
+
+# =============================================================================
+# SHACL claim validation (OFF by default — VALIDATE_CLAIMS_SHACL; see
+# docs/claims/shacl-alignment-gap.md). pyshacl pulls rdflib.
+# =============================================================================
+pyshacl>=0.27
+rdflib>=7.0