diff --git a/README.md b/README.md index c8478e0..8c3d940 100644 --- a/README.md +++ b/README.md @@ -300,6 +300,8 @@ Soft-start feedback reinforcementは、`soft_start_feedback_reinforcement=True` local database上の比較は、[Soft-start snapshot evaluation](docs/soft-start-snapshot-evaluation.md)でtransaction-consistent private snapshot、固定4 arm、result-free manifest、one-time development、conditional holdout、privacy / mutation / rank hard gateとして分離しています。評価はsnapshotのfresh cloneだけを変更し、source database、live config、snapshot本体、private本文をpublic artifactへ含めません。支持結果も固定local snapshot上のcutover候補に限定し、このissue内でconfigやdefaultを変更しません。 +v1 developmentはsnapshot既存evidenceによりq3/s1がfresh `used_2`で発火し、空baseline由来の`used_3`を固定したboundary gateだけが不合格となりました。この不支持resultを変更せず、[Baseline-aware soft-start snapshot evaluation](docs/baseline-aware-soft-start-snapshot-evaluation.md)はfresh snapshotと別caseを使い、initial evidence stateからq3 first mutationを導出します。baseline不一致やcapacity不足ではregistered resultを作りません。v2はregistered output不在のfreeze-only PRとして固定し、squash merge後の別Issueでdevelopmentを一度だけ実行して、全gate通過時だけholdoutを一度開きます。 + 後続の feedback policy comparison に使う public source は、[feedback-policy-comparison-v1](corpora/feedback-policy-comparison-v1/README.md) に corpus-only で固定しています。development / holdout 各二 cluster の明示 link topology、source-only manifest、raw SHA-256、LF 改行規則、provenance、既存 fixture との identity contamination audit だけを含み、評価の選択入力と観測物は含めません。この source merge commit を後続の別 issue が唯一の入力として扱います。 この corpus を使う `used + q3 + sibling normalization 1.0` と `confirmed + decay 0.5 + sibling normalization 1.0` の比較は、[Feedback policy comparison evaluation](docs/feedback-policy-comparison-evaluation.md) に result-free protocol、one-time development、conditional holdout、exclusive observed result、再計算可能な hard gate を固定しています。controlled result は default adoption や production quality を意味しません。 diff --git a/docs/Decision-Structure.md b/docs/Decision-Structure.md index e48d170..67f9902 100644 --- a/docs/Decision-Structure.md +++ b/docs/Decision-Structure.md @@ -18,7 +18,7 @@ | [longitudinal-feedback-adaptation](https://github.com/Liplus-Project/neuron-graph-rag/wiki/longitudinal-feedback-adaptation) | active | longitudinal feedback-adaptation は、repository-native controlled corpus v3 の相互に独立した cluster と、その source 文書に明記した 0、1、3、10 credit ceiling を用いる。corpus phase は query、gold、schedule、runner、gate、manifest、result、既定値を定義・変更しない。 | | [single-corpus-real-feedback-validation](https://github.com/Liplus-Project/neuron-graph-rag/wiki/single-corpus-real-feedback-validation) | superseded | [repository-native-controlled-corpus](https://github.com/Liplus-Project/neuron-graph-rag/wiki/repository-native-controlled-corpus) がこの node を supersede する。以後の evaluation は、NGR repository に公開する固定 SHA の controlled corpus を source とし、D1 single-corpus experiment は capacity が増えるまで waiting とする。 | | [repository-native-controlled-corpus](https://github.com/Liplus-Project/neuron-graph-rag/wiki/repository-native-controlled-corpus) | active | repository-native controlled corpus v2 は、固定 SHA の公開 documentation と本文中の明示的な同一 directory 相対 link だけから、node、doc path、source URL、credited edge identity が相互に分離した development / holdout の各 3-edge path を導出する。v1 は provenance として保持する。これは controlled benchmark であり、外部 corpus への一般化、評価 query、gold、result、既定値変更を含まない。 | -| [soft-start-feedback-reinforcement](https://github.com/Liplus-Project/neuron-graph-rag/wiki/soft-start-feedback-reinforcement) | active | 最初の credited `used` に通常 bounded update の小さな provisional fractionを適用し、最初の独立 `confirmed` が remainder、後続 confirmation が geometric decayを適用する。transaction-consistent private snapshotの固定4 arm評価はdevelopment全hard gate通過時だけholdoutを一度開き、結果をlocal cutover候補に限定してsource database、live config、defaultを変更しない。 | +| [soft-start-feedback-reinforcement](https://github.com/Liplus-Project/neuron-graph-rag/wiki/soft-start-feedback-reinforcement) | active | 最初の credited `used` に通常 bounded updateの小さなprovisional fractionを適用し、最初の独立`confirmed`がremainder、後続confirmationがgeometric decayを適用する。v1 snapshot評価の不支持を保持し、baseline-aware successorはfresh initial evidenceからq3 first mutationを導出する。v2 freeze-only PRとsquash後のobserved registrationを分離し、development全gate通過時だけholdoutを一度開く。source database、live config、defaultを変更しない。 | | [github-rag-mcp-replacement-compatibility](https://github.com/Liplus-Project/neuron-graph-rag/wiki/github-rag-mcp-replacement-compatibility) | active | public GitHub repository一つのread-only snapshotをNGR local indexへ接続する。github-rag-mcp `search` の保存済み raw capture と source URL、根拠を比較する。共有 source identity を確認しても最小 doc 検索 path の候補に限り、production github-rag-mcp、MCP authentication / transport、remote deployment、default変更は含まない。 | ## Entry format diff --git a/docs/baseline-aware-soft-start-snapshot-evaluation.md b/docs/baseline-aware-soft-start-snapshot-evaluation.md new file mode 100644 index 0000000..4d45750 --- /dev/null +++ b/docs/baseline-aware-soft-start-snapshot-evaluation.md @@ -0,0 +1,73 @@ +# Baseline-aware soft-start snapshot evaluation + +## 目的とsuccessor境界 + +snapshot comparison v1はdevelopment 7 hard gate中6件を通過したが、`policy-boundaries`を満たさずholdoutを開かなかった。対象edgeがinitial evidence count `1`を持ち、q3/s1がfresh `used_2`でquorum `3`へ到達した一方、v1 gateは空baselineからの`used_3`発火を固定していた。この不支持result、protocol、gate、解釈、private snapshotは凍結したまま保持する。 + +v2はminor tweakやv1再実行ではない。`baseline_aware_soft_start_snapshot_v2` namespace、fresh transaction-consistent private snapshot、新規fixture / manifest / outputを使う。v1 private snapshotとobserved resultを入力にせず、developmentのcredited edge identityもv1 observed developmentから分離する。 + +## Baseline contractとcapacity + +confirmed / corrected relation caseごとに、snapshot上のcredited edgeについて次を結果前に登録する。 + +- weight +- reinforced count +- relation feedback evidence count +- confirmation count + +q3/s1のexpected first mutation eventは次の式だけで導出する。 + +`max(1, relation_feedback_evidence_quorum - registered_initial_evidence_count)` + +event budgetはfresh trace 4回である。derived first mutationはfinal eventより前に発生し、少なくとも一回の後続fresh evidence activationも観測できなければならない。snapshot actual stateと登録baselineの不一致、式との不一致、capacity不足、selected credited path不一致はresult-free preflight failureである。この場合はregistered resultを作らず、failure reportをIssueへ保存して停止する。 + +## 固定armとschedule + +同じfresh snapshotの別cloneへ次の4 armを適用する。 + +1. `control`: audit-only。 +2. `used_q3_s1`: evidence quorum `3`、sibling normalization `1.0`。 +3. `confirmed_r05_s1`: confirmed-only、decay `0.5`、sibling normalization `1.0`。 +4. `soft_start_r025_r05_s1`: provisional ratio `0.25`、decay `0.5`、sibling normalization `1.0`。 + +各caseはbaseline後に、fresh traceで`used_n`、同じtraceに対する`outcome_n`を`n=1..4`の順に実行する。armごとにfresh clone replayを行い、runtime identityを除くsemantic payload一致を要求する。 + +## Result-freeとprivacy + +source databaseはSQLite URI `mode=ro`と`query_only`で開き、backup APIでprivate destinationへ一度だけ複製する。source containerの前後SHA-256一致とsnapshot `integrity_check`を要求し、arm / replayはsnapshot cloneだけを変更する。 + +public artifactはsource locator、capture timestamp、container / snapshot / schema hash、table name、選択row count、public node / edge identifier、query、baseline数値だけを持つ。snapshot本体、node本文、credential、absolute private pathを含めない。writerはexclusive createし、registered resultの上書き、再実行、再集計を拒否する。 + +v1のresult-free commitとobserved commitはmainへのsquash mergeで一つのcommitになり得るため、mainのmanifest初回追加commitだけからpre-observation development output不在を推論しない。これは履歴上証明できなくなった事実だけを弱める境界変更である。v1 verifierは同commitのhistorical protocol blob hash、既存result-free audit literal、同commitに含まれるdevelopment outputを既存exclusive result verifierで検証し、未開封holdoutの不在を引き続き明示検証する。v1 protocol、gate、result、audit、manifestのbytes自体は変更しない。 + +## Hard gate + +1. snapshot / protocol / case / privacy / baseline / receipt / idempotency / semantic replay / exclusive output integrity。 +2. initial evidenceから導出したq3 first mutationと実測の一致、quorum前不変、後続fresh evidence発火。 +3. confirmed-onlyのused時不変、独立confirmedごとの`1.0, 0.5, 0.25, 0.125` decay。 +4. soft-start first used provisional、first confirmed remainder、後続fixed decay。 +5. soft-startのlearning latency先行と最終q3/s1 relation quality non-regression。 +6. corrected cohortのprovisional cost上限、自動negative reinforcement / rollbackなし。 +7. lexical、zero-hop、direct、reverse、unrelated controlのrank / mutation safety。 +8. credited edgeとconfirmation時same-source siblingだけへのmutation、source snapshot不変、fresh-clone replay。 + +このIssueはprotocol、fixture、runner、verifier、tests、docsをregistered output不在のresult-free stateで固定するfreeze-only phaseである。developmentとholdoutは実行せず、squash merge後に別Issueの新規registration commitを使ってdevelopmentを一度だけ実行する。その全8 gate pass時だけholdoutを一度開く。不支持または判定不能を保存してもquery、ratio、case、schedule、metric、gateを変更しない。このphase分離により、freeze commitのoutput不在と後続observed commitをmain history上で別々に検証可能にする。 + +## 解釈境界 + +development不支持または判定不能ではholdoutを開かずlocal cutoverを支持しない。holdoutまで全gate passしても、支持範囲は固定local snapshot上のcandidateに限定する。このIssue内ではsource database、live config、library defaultを変更しない。 + +```bash +python tools/run_baseline_aware_soft_start_snapshot_evaluation.py --probe --snapshot "$SNAPSHOT" +python tools/run_baseline_aware_soft_start_snapshot_evaluation.py --stage development --snapshot "$SNAPSHOT" +python tools/run_baseline_aware_soft_start_snapshot_evaluation.py --verify development +``` + +## 関連 + +- [Requirements](requirements.md) +- [Soft-start snapshot evaluation v1](soft-start-snapshot-evaluation.md) +- [Confirmed-outcome feedback reinforcement](confirmed-outcome-feedback-reinforcement.md) +- [Historical source verification](historical-source-verification.md) +- [Decision Structure](Decision-Structure.md) +- [Issue #106](https://github.com/Liplus-Project/neuron-graph-rag/issues/106) diff --git a/docs/requirements.md b/docs/requirements.md index 016a772..6644e91 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -101,6 +101,7 @@ 82. frozen evaluation の historical source hash は、manifest path の初回追加 commit または manifest が明示する lowercase full 40-hex source / baseline / prior commit の exact blob bytes に対して検証する。後続の committed manifest rewrite、mutable ref / revision expression、未知 commit、非 ancestor commit、manifest bytes差、欠落 path、hash 不一致を fail closed にし、同名 path の current working tree を過去の evidence として扱わない。既存 protocol が明記する raw-first LF / CRLF whole-file alternate だけを維持し、本文差、mixed newline、bare CR、その他の byte 差を拒否する。 83. soft-start feedback reinforcement は明示 opt-in の relation-only candidate とし、最初の新規 `used` で通常 bounded increment の固定 ratio 分だけを credited path へ適用する。最初の独立 `confirmed` は同じ schedule の残量を一回分の通常 increment まで補い、後続 confirmation は固定 decay ratio で加算する。used 時は sibling normalization を行わず、confirmation の actual delta だけを同一 source の uncredited sibling へ配分する。duplicate、lexical、zero-hop、別 source、uncredited edge、negative outcome は変更せず、candidate mechanics の合格だけで default や local serving policy を変更しない。 84. soft-start snapshot evaluation は、local source database を read-only URI と SQLite backup API で一度だけ transaction-consistent snapshot へ複製し、同じ snapshot のfresh clone上で `control`、`used_q3_s1`、`confirmed_r05_s1`、`soft_start_r025_r05_s1` の固定4 armを比較する。query、public node / edge identifier、outcome、3回のfresh trace schedule、checkpoint、metric、hard gate、exclusive output、snapshot / protocol hashを登録result生成前に固定する。source database、live config、snapshot本体、private本文、absolute private pathをpublic artifactへ含めず、developmentを一度だけ実行し、全hard gate通過時だけholdoutを一度開く。不支持または判定不能を保存してもprotocolを調整せず、local cutover、library default、external corpus、production qualityへ自動で一般化しない。 +85. baseline-aware soft-start snapshot evaluation は、v1のprotocol、gate、observed result、private snapshotを変更、再実行、再集計、入力再利用せず、fresh snapshot、新規namespace、新規output、v1 observed developmentと異なるcredited edge identityを使う。各relation caseのinitial weight、reinforced count、evidence count、confirmation countを結果前に登録し、q3/s1のfirst mutationを`max(1, quorum - initial evidence count)`で導出する。導出不能、baseline不一致、event budget内のquorum capacity不足はregistered resultを作らずfailure reportで停止する。v2 protocolはregistered output不在のfreeze-only PRで固定し、そのsquash merge後の別Issueでdevelopmentを一度だけ実行する。全8 hard gate通過時だけholdoutを一度開き、支持結果もlocal cutover候補に限定してsource database、live config、library defaultを変更しない。 75. v3 implementation、prompt、manifest、query override、schema、集約、path audit、hash規則、gate、stop rule、testsをresult-free commitでpushした後、development stage / 4 case packet / 12 responses / resultを各一度だけ生成する。 76. development全12 gate通過時だけholdout stageを一度生成し、異なるfresh 12 judgesで同じgateを評価する。packet、response、resultの上書き、観測後の規則変更、実LLM品質値のCI再生成を拒否する。 @@ -138,3 +139,4 @@ - [Historical source verification](historical-source-verification.md) が frozen manifest path の初回追加 commit、明示 full source commit ID、exact blob、ancestor、path、newline portability、fail-closed 境界を定義する。 - [Confirmed-outcome feedback reinforcement](confirmed-outcome-feedback-reinforcement.md) が confirmed-only と soft-start の明示 policy、永続 schedule、transaction、receipt、default-preserving boundary を定義する。 - [Soft-start snapshot evaluation](soft-start-snapshot-evaluation.md) が transaction-consistent private snapshot、固定4 arm、result-free freeze、privacy、one-time development、conditional holdout、local cutover境界を定義する。 +- [Baseline-aware soft-start snapshot evaluation](baseline-aware-soft-start-snapshot-evaluation.md) がfresh baseline stateからのq3 boundary導出、v1 evidence isolation、capacity preflight、新規one-time result境界を定義する。 diff --git a/src/neuron_graph_rag/baseline_aware_soft_start_snapshot_evaluation.py b/src/neuron_graph_rag/baseline_aware_soft_start_snapshot_evaluation.py new file mode 100644 index 0000000..9bb9b39 --- /dev/null +++ b/src/neuron_graph_rag/baseline_aware_soft_start_snapshot_evaluation.py @@ -0,0 +1,1139 @@ +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import sqlite3 +from collections.abc import Mapping, Sequence +from dataclasses import asdict +from datetime import datetime, timezone +from pathlib import Path +from tempfile import TemporaryDirectory +from typing import Any + +from .config_provenance import effective_config_provenance +from .corpus_integrity import verify_manifest_source_hashes +from .evidence_feedback import EngineConfig, NeuronGraphRAG +from .feedback import FeedbackLedger +from .models import FeedbackReceipt, SourceUseEvent + + +ROOT = Path(__file__).resolve().parents[2] +STEM = "baseline_aware_soft_start_snapshot_v2" +MANIFEST_PATH = ROOT / "tests" / "fixtures" / f"{STEM}.manifest.json" +EDGE_FIELDS = ("source_id", "target_id", "edge_type") +STAGES = ("development", "holdout") +POLICIES = ( + "control", + "used_q3_s1", + "confirmed_r05_s1", + "soft_start_r025_r05_s1", +) +_ABSOLUTE_PRIVATE_PATH = re.compile( + r"(?:[A-Za-z]:[\\/]|/Users/|/home/|\\\\[^\\]+\\[^\\]+)" +) +_CREDENTIAL = re.compile( + r"(?i)(?:gh[pousr]_[A-Za-z0-9_]{20,}|github_pat_[A-Za-z0-9_]{20,}|" + r"(?:token|password|secret|api[_-]?key)\s*[:=]\s*[^\s,}]+)" +) + + +def _encoded(payload: Any) -> bytes: + return (json.dumps(payload, ensure_ascii=False, indent=2) + "\n").encode("utf-8") + + +def _sha256_bytes(raw: bytes) -> str: + return hashlib.sha256(raw).hexdigest() + + +def _sha256(path: Path) -> str: + return _sha256_bytes(path.read_bytes()) + + +def read_json(path: Path) -> Any: + raw = path.read_bytes() + text = raw.decode("utf-8", errors="strict") + payload = json.loads(text) + if text != json.dumps(payload, ensure_ascii=False, indent=2) + "\n": + raise ValueError(f"non-canonical JSON artifact: {path}") + return payload + + +def write_json_exclusive(path: Path, payload: Mapping[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o644) + try: + with os.fdopen(descriptor, "wb") as stream: + stream.write(_encoded(payload)) + stream.flush() + os.fsync(stream.fileno()) + except BaseException: + path.unlink(missing_ok=True) + raise + + +def assert_public_payload(payload: Any) -> None: + """Reject private paths, database text, and common credential shapes.""" + + def walk(value: Any, path: tuple[str, ...]) -> None: + if isinstance(value, Mapping): + for key, item in value.items(): + name = str(key) + if name.lower() in { + "node_text", + "document_text", + "raw_text", + "private_path", + "source_path", + "snapshot_path", + }: + raise ValueError(f"private field is forbidden: {'.'.join((*path, name))}") + walk(item, (*path, name)) + elif isinstance(value, (list, tuple)): + for index, item in enumerate(value): + walk(item, (*path, str(index))) + elif isinstance(value, str): + if value.startswith(("/", "\\")) or _ABSOLUTE_PRIVATE_PATH.search(value): + raise ValueError(f"absolute private path is forbidden: {'.'.join(path)}") + if _CREDENTIAL.search(value): + raise ValueError(f"credential-shaped value is forbidden: {'.'.join(path)}") + + walk(payload, ()) + + +def _schema_identity(connection: sqlite3.Connection) -> dict[str, Any]: + rows = [ + {"name": str(row[0]), "sql": str(row[1])} + for row in connection.execute( + """ + SELECT name, sql FROM sqlite_master + WHERE type = 'table' AND name NOT LIKE 'sqlite_%' + ORDER BY name + """ + ) + ] + return { + "schema_sha256": _sha256_bytes(_encoded(rows)), + "table_names": [row["name"] for row in rows], + } + + +def acquire_transactional_snapshot(source: Path, destination: Path) -> dict[str, Any]: + """Create one exclusive SQLite backup without exposing either local path.""" + source = source.resolve(strict=True) + destination = destination.resolve(strict=False) + if source == destination: + raise ValueError("source and snapshot must differ") + destination.parent.mkdir(parents=True, exist_ok=True) + reservation = os.open(destination, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + os.close(reservation) + source_before = _sha256(source) + captured_at = datetime.now(timezone.utc).replace(microsecond=0).isoformat() + try: + source_connection = sqlite3.connect(source.as_uri() + "?mode=ro", uri=True) + destination_connection = sqlite3.connect(destination) + try: + source_connection.execute("PRAGMA query_only = ON") + source_connection.backup(destination_connection) + destination_connection.commit() + if destination_connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": + raise RuntimeError("snapshot integrity check failed") + schema = _schema_identity(destination_connection) + counts = { + name: int(destination_connection.execute(f"SELECT count(*) FROM {name}").fetchone()[0]) + for name in ("nodes", "edges", "retrievals", "success_feedback", "delayed_outcomes") + } + finally: + destination_connection.close() + source_connection.close() + source_after = _sha256(source) + if source_after != source_before: + raise RuntimeError("source database container changed during snapshot acquisition") + provenance = { + "source_locator": "local_codex_ngr_database", + "source_access": "sqlite-uri-mode-ro-query-only", + "capture_method": "sqlite-backup-api", + "captured_at": captured_at, + "source_container_sha256_before": source_before, + "source_container_sha256_after": source_after, + "snapshot_sha256": _sha256(destination), + "snapshot_size": destination.stat().st_size, + **schema, + "row_counts": counts, + } + assert_public_payload(provenance) + return provenance + except BaseException: + destination.unlink(missing_ok=True) + raise + + +def prove_writer_verifier_round_trip(path: Path) -> None: + payload = { + "probe_id": "temporary-soft-start-snapshot-placeholder", + "identities": ["zulu-placeholder", "alpha-placeholder", "mike-placeholder"], + "semantic_round_trip": True, + } + write_json_exclusive(path, payload) + try: + if read_json(path) != payload: + raise ValueError("placeholder semantics changed") + finally: + path.unlink(missing_ok=True) + + +def _load_protocol() -> tuple[dict[str, Any], dict[str, dict[str, Any]]]: + manifest = read_json(MANIFEST_PATH) + registered = verify_manifest_source_hashes( + ROOT, MANIFEST_PATH, manifest["artifact_sha256"] + ) + artifacts = { + name: json.loads(registered.artifact_bytes[relative].decode("utf-8", errors="strict")) + for name, relative in manifest["protocol_artifacts"].items() + } + assert_public_payload(manifest) + assert_public_payload(artifacts) + return manifest, artifacts + + +def _load_current_protocol() -> tuple[dict[str, Any], dict[str, dict[str, Any]]]: + """Load current bytes for the result-free pre-commit probe only.""" + manifest = read_json(MANIFEST_PATH) + for relative, expected in manifest["artifact_sha256"].items(): + path = ROOT / relative + if _sha256(path) != expected: + raise ValueError(f"current artifact hash mismatch: {relative}") + artifacts = { + name: read_json(ROOT / relative) + for name, relative in manifest["protocol_artifacts"].items() + } + assert_public_payload(manifest) + assert_public_payload(artifacts) + return manifest, artifacts + + +def _config( + arm: Mapping[str, Any], base_engine_config: Mapping[str, Any] | None = None +) -> EngineConfig: + values = dict(base_engine_config or {}) + values.update(arm["engine_config"]) + return EngineConfig(**values) + + +def derive_q3_first_mutation_event(quorum: int, initial_evidence_count: int) -> int: + if quorum < 1 or initial_evidence_count < 0: + raise ValueError("quorum must be positive and initial evidence non-negative") + return max(1, quorum - initial_evidence_count) + + +def _edge_key(value: Mapping[str, Any] | Sequence[str]) -> str: + if isinstance(value, Mapping): + return "|".join(str(value[field]) for field in EDGE_FIELDS) + return "|".join(str(item) for item in value) + + +def _edge_state(engine: NeuronGraphRAG) -> list[dict[str, Any]]: + states: list[dict[str, Any]] = [] + for edge in engine.store.list_edges(): + identity = (edge.source_id, edge.target_id, edge.edge_type) + confirmed = engine.store.connection.execute( + """ + SELECT confirmation_count, base_increment FROM confirmed_edge_state + WHERE source_id = ? AND target_id = ? AND edge_type = ? + """, + identity, + ).fetchone() + soft = engine.store.connection.execute( + """ + SELECT confirmation_count, base_increment, soft_start_ratio + FROM soft_start_edge_state + WHERE source_id = ? AND target_id = ? AND edge_type = ? + """, + identity, + ).fetchone() + states.append( + { + "source_id": edge.source_id, + "target_id": edge.target_id, + "edge_type": edge.edge_type, + "weight": edge.weight, + "reinforced_count": edge.reinforced_count, + "evidence_count": engine.store.feedback_evidence_count(*identity), + "confirmation_count": ( + int(soft["confirmation_count"]) + if soft is not None + else 0 if confirmed is None else int(confirmed["confirmation_count"]) + ), + "base_increment": ( + float(soft["base_increment"]) + if soft is not None + else None if confirmed is None else float(confirmed["base_increment"]) + ), + "soft_start_ratio": None if soft is None else float(soft["soft_start_ratio"]), + } + ) + return sorted(states, key=_edge_key) + + +def _rank(items: Sequence[Mapping[str, Any]], node_id: str, limit: int) -> int: + return next((int(item["rank"]) for item in items if item["node_id"] == node_id), limit + 1) + + +def _metrics(engine: NeuronGraphRAG, case: Mapping[str, Any], now: float) -> dict[str, Any]: + limit = int(case["retrieval_limit"]) + channels = engine.search_channels(str(case["query"]), limit=limit, now=now) + relation = [ + { + "node_id": hit.node.node_id, + "rank": hit.rank, + "score": hit.channel_score, + } + for hit in channels.relation.hits + ] + target_id = str(case["used_node_id"]) + target_rank = _rank(relation, target_id, limit) + target_score = next((float(item["score"]) for item in relation if item["node_id"] == target_id), 0.0) + competitor = max( + (float(item["score"]) for item in relation if item["node_id"] != target_id), + default=0.0, + ) + direct = engine.search(str(case["direct_query"]), limit=limit, now=now + 0.01) + direct_items = [ + {"node_id": hit.node.node_id, "rank": rank} + for rank, hit in enumerate(direct.hits, 1) + ] + reverse = engine.search_channels(str(case["reverse_query"]), limit=limit, now=now + 0.02) + reverse_items = [ + {"node_id": hit.node.node_id, "rank": hit.rank} + for hit in reverse.relation.hits + ] + return { + "target_rank": target_rank, + "target_mrr": 1.0 / target_rank, + "target_hit_at_k": target_rank <= int(case["hit_k"]), + "score_margin": target_score - competitor, + "top_k": [item["node_id"] for item in relation[: int(case["hit_k"])]], + "direct_rank": _rank(direct_items, str(case["direct_node_id"]), limit), + "reverse_rank": _rank(reverse_items, str(case["reverse_node_id"]), limit), + } + + +def _feedback_semantics(feedback: FeedbackReceipt | None) -> dict[str, Any] | None: + if feedback is None: + return None + return { + "channel": feedback.channel, + "reinforced": [asdict(item) for item in feedback.reinforced_edges], + "normalized": [asdict(item) for item in feedback.normalized_sibling_edges], + "evidence": [asdict(item) for item in feedback.evidence], + } + + +def _source_use( + engine: NeuronGraphRAG, + arm_id: str, + trace_id: str, + node_id: str, + key: str, + now: float, +) -> dict[str, Any]: + events = tuple(SourceUseEvent(node_id, stage) for stage in ("selected", "validated", "used")) + if arm_id == "control": + payload = json.dumps( + {"trace_id": trace_id, "events": [asdict(event) for event in events]}, + sort_keys=True, + separators=(",", ":"), + ) + arguments = { + "idempotency_key": key, + "payload_json": payload, + "receipt_id": f"control-{key}", + "trace_id": trace_id, + "created_at": now, + "events": tuple((event.node_id, event.stage) for event in events), + "apply_feedback": None, + "confirmation_candidate": False, + } + receipt = engine.store.record_source_use(**arguments) + replay = engine.store.record_source_use(**arguments) + return { + "stages": [event["stage"] for event in receipt["events"]], + "changed": [bool(event["changed"]) for event in receipt["events"]], + "newly_used_node_ids": receipt["newly_used_node_ids"], + "feedback": None, + "idempotency_replay_equal": replay == receipt, + } + ledger = FeedbackLedger(engine) + receipt = ledger.record_source_use(trace_id, events, idempotency_key=key, now=now) + replay = ledger.record_source_use(trace_id, events, idempotency_key=key, now=now) + return { + "stages": [item.stage for item in receipt.events], + "changed": [item.changed for item in receipt.events], + "newly_used_node_ids": list(receipt.newly_used_node_ids), + "feedback": _feedback_semantics(receipt.feedback), + "idempotency_replay_equal": replay == receipt, + } + + +def _outcome( + engine: NeuronGraphRAG, + arm_id: str, + trace_id: str, + node_id: str, + outcome: str, + key: str, + now: float, +) -> dict[str, Any]: + summary = "fixed registered outcome" + if arm_id == "control": + payload = json.dumps( + { + "trace_id": trace_id, + "node_ids": [node_id], + "outcome": outcome, + "summary": summary, + "external_ref": None, + }, + sort_keys=True, + separators=(",", ":"), + ) + arguments = { + "idempotency_key": key, + "payload_json": payload, + "outcome_id": f"control-{key}", + "trace_id": trace_id, + "node_ids": (node_id,), + "outcome": outcome, + "summary": summary, + "external_ref": None, + "recorded_at": now, + } + receipt = engine.store.record_outcome(**arguments) + replay = engine.store.record_outcome(**arguments) + return { + "outcome": outcome, + "reinforcement_applied": False, + "confirmations": [], + "credited_paths": [], + "normalized": [], + "idempotency_replay_equal": replay == receipt, + } + ledger = FeedbackLedger(engine) + receipt = ledger.record_outcome( + trace_id, + [node_id], + outcome, + summary, + idempotency_key=key, + now=now, + ) + replay = ledger.record_outcome( + trace_id, + [node_id], + outcome, + summary, + idempotency_key=key, + now=now, + ) + return { + "outcome": receipt.outcome, + "reinforcement_applied": receipt.reinforcement_applied, + "confirmations": [asdict(item) for item in receipt.confirmations], + "credited_paths": [asdict(item) for item in receipt.credited_paths], + "normalized": [asdict(item) for item in receipt.normalized_sibling_edges], + "idempotency_replay_equal": replay == receipt, + } + + +def _checkpoint( + engine: NeuronGraphRAG, + case: Mapping[str, Any], + name: str, + now: float, + baseline: Mapping[str, Any] | None, +) -> dict[str, Any]: + metrics = _metrics(engine, case, now) + edges = _edge_state(engine) + if baseline is None: + changed_edges: list[str] = [] + entry: list[str] = [] + exit_items: list[str] = [] + non_target_churn = 0 + else: + previous = {_edge_key(item): item for item in baseline["edges"]} + changed_edges = [ + _edge_key(item) + for item in edges + if any( + item[field] != previous[_edge_key(item)][field] + for field in ("weight", "reinforced_count", "evidence_count", "confirmation_count") + ) + ] + current_top = set(metrics["top_k"]) + baseline_top = set(baseline["metrics"]["top_k"]) + entry = sorted(current_top - baseline_top) + exit_items = sorted(baseline_top - current_top) + excluded = {str(case["used_node_id"]), str(case["source_node_id"])} + non_target_churn = len((current_top ^ baseline_top) - excluded) + return { + "name": name, + "metrics": metrics, + "edges": edges, + "changed_edges": changed_edges, + "top_k_entry": entry, + "top_k_exit": exit_items, + "non_target_churn": non_target_churn, + } + + +def _select_trace(engine: NeuronGraphRAG, case: Mapping[str, Any], now: float) -> str: + channels = engine.search_channels( + str(case["query"]), limit=int(case["retrieval_limit"]), now=now + ) + trace = channels.relation if case["search_surface"] == "relation" else channels.lexical + node_id = str(case["used_node_id"]) + if node_id not in {hit.node.node_id for hit in trace.hits}: + raise RuntimeError("registered used node is absent from its search trace") + if case["search_surface"] == "relation" and case["case_role"] in {"confirmed", "corrected"}: + paths = engine.store.retrieval_paths(trace.trace_id, node_id) + expected = tuple( + str(case["credited_edge"][field]) for field in EDGE_FIELDS + ) + selected = max( + paths, + key=lambda path: (float(path["contribution"]), str(path["seed_id"])), + default=None, + ) + selected_edges = () if selected is None else tuple( + (step["source_id"], step["target_id"], step["edge_type"]) + for step in selected["steps"] + ) + if selected_edges != (expected,): + raise RuntimeError("registered credited edge is not the selected relation path") + return trace.trace_id + + +def _run_case( + engine: NeuronGraphRAG, + arm_id: str, + case: Mapping[str, Any], + case_index: int, +) -> dict[str, Any]: + clock = 100_000.0 + case_index * 10_000.0 + baseline = _checkpoint(engine, case, "baseline", clock, None) + checkpoints = [baseline] + receipts: list[dict[str, Any]] = [] + for iteration in range(1, 5): + trace_id = _select_trace(engine, case, clock + iteration * 100.0) + use = _source_use( + engine, + arm_id, + trace_id, + str(case["used_node_id"]), + f"{arm_id}-{case['case_id']}-use-{iteration}", + clock + iteration * 100.0 + 0.1, + ) + checkpoints.append( + _checkpoint( + engine, + case, + f"used_{iteration}", + clock + iteration * 100.0 + 10.0, + baseline, + ) + ) + outcome = _outcome( + engine, + arm_id, + trace_id, + str(case["used_node_id"]), + str(case["outcome"]), + f"{arm_id}-{case['case_id']}-outcome-{iteration}", + clock + iteration * 100.0 + 20.0, + ) + receipts.append({"iteration": iteration, "source_use": use, "outcome": outcome}) + checkpoints.append( + _checkpoint( + engine, + case, + f"outcome_{iteration}", + clock + iteration * 100.0 + 30.0, + baseline, + ) + ) + return { + "case_id": case["case_id"], + "case_role": case["case_role"], + "checkpoints": checkpoints, + "receipts": receipts, + } + + +def _run_arm( + snapshot: Path, + arm: Mapping[str, Any], + base_engine_config: Mapping[str, Any], + cases: Sequence[Mapping[str, Any]], +) -> dict[str, Any]: + with TemporaryDirectory() as directory: + clone = Path(directory) / "arm.sqlite" + shutil.copyfile(snapshot, clone) + with NeuronGraphRAG(clone, config=_config(arm, base_engine_config)) as engine: + initial = _edge_state(engine) + observed_cases = [ + _run_case(engine, str(arm["arm_id"]), case, index) + for index, case in enumerate(cases) + ] + final = _edge_state(engine) + return { + "arm_id": arm["arm_id"], + "policy": arm["policy"], + "effective_config_provenance": effective_config_provenance( + _config(arm, base_engine_config) + ), + "initial_edges": initial, + "cases": observed_cases, + "final_edges": final, + } + + +def _checkpoint_by_name(case: Mapping[str, Any], name: str) -> Mapping[str, Any]: + return next(item for item in case["checkpoints"] if item["name"] == name) + + +def _case_by_role(arm: Mapping[str, Any], role: str) -> Mapping[str, Any]: + return next(item for item in arm["cases"] if item["case_role"] == role) + + +def _edge_at(checkpoint: Mapping[str, Any], key: str) -> Mapping[str, Any]: + return next(item for item in checkpoint["edges"] if _edge_key(item) == key) + + +def _delta(case: Mapping[str, Any], checkpoint: str, key: str) -> float: + baseline = _edge_at(_checkpoint_by_name(case, "baseline"), key) + current = _edge_at(_checkpoint_by_name(case, checkpoint), key) + return float(current["weight"]) - float(baseline["weight"]) + + +def evaluate_gates( + preflight: Mapping[str, Any], + cases: Sequence[Mapping[str, Any]], + arms: Mapping[str, Mapping[str, Any]], + thresholds: Mapping[str, Any], +) -> dict[str, bool]: + control = arms["control"] + used = arms["used_q3_s1"] + confirmed = arms["confirmed_r05_s1"] + soft = arms["soft_start_r025_r05_s1"] + registered = {str(case["case_id"]): case for case in cases} + all_receipts = [ + receipt + for arm in arms.values() + for case in arm["cases"] + for receipt in case["receipts"] + ] + baseline_matches = True + for arm in arms.values(): + for observed_case in arm["cases"]: + case = registered[str(observed_case["case_id"])] + if case["case_role"] not in {"confirmed", "corrected"}: + continue + edge = _edge_at( + _checkpoint_by_name(observed_case, "baseline"), + _edge_key(case["credited_edge"]), + ) + baseline_matches = baseline_matches and all( + edge[field] == case["registered_initial_state"][field] + for field in ( + "weight", + "reinforced_count", + "evidence_count", + "confirmation_count", + ) + ) + protocol_integrity = bool(preflight["passed"]) and baseline_matches and all( + bool(arm["fresh_clone_replay"]) for arm in arms.values() + ) and all( + receipt["source_use"]["idempotency_replay_equal"] + and receipt["outcome"]["idempotency_replay_equal"] + for receipt in all_receipts + ) + + confirmed_case = _case_by_role(soft, "confirmed") + confirmed_key = _edge_key(registered[str(confirmed_case["case_id"])]["credited_edge"]) + soft_receipts = confirmed_case["receipts"] + provisional = _delta(confirmed_case, "used_1", confirmed_key) + first_total = _delta(confirmed_case, "outcome_1", confirmed_key) + later = [ + receipt["outcome"]["confirmations"][0] + for receipt in soft_receipts[1:] + if receipt["outcome"]["confirmations"] + ] + confirmed_reference = _delta( + _case_by_role(confirmed, "confirmed"), "outcome_1", confirmed_key + ) + tolerance = float(thresholds["numeric_tolerance"]) + soft_schedule = ( + provisional > 0.0 + and first_total <= confirmed_reference + tolerance + and abs(first_total - confirmed_reference) <= tolerance + and [item["multiplier"] for item in later] == [0.5, 0.25, 0.125] + and all( + current["actual_delta"] <= previous["actual_delta"] + for previous, current in zip(later, later[1:]) + ) + ) + + used_case = _case_by_role(used, "confirmed") + confirmed_only_case = _case_by_role(confirmed, "confirmed") + baseline_q3_boundary = True + event_budget = int(thresholds["event_budget"]) + for role in ("confirmed", "corrected"): + q3_case = _case_by_role(used, role) + case = registered[str(q3_case["case_id"])] + edge_key = _edge_key(case["credited_edge"]) + first = int(case["expected_q3_first_mutation_event"]) + deltas = [_delta(q3_case, f"used_{event}", edge_key) for event in range(1, event_budget + 1)] + evidence = [ + int(_edge_at(_checkpoint_by_name(q3_case, f"used_{event}"), edge_key)["evidence_count"]) + for event in range(1, event_budget + 1) + ] + initial_evidence = int(case["registered_initial_state"]["evidence_count"]) + baseline_q3_boundary = baseline_q3_boundary and ( + all(delta == 0.0 for delta in deltas[: first - 1]) + and deltas[first - 1] > 0.0 + and all(current > previous for previous, current in zip(deltas[first - 1 :], deltas[first:])) + and evidence == [initial_evidence + event for event in range(1, event_budget + 1)] + ) + + confirmed_receipts = confirmed_only_case["receipts"] + confirmations = [ + receipt["outcome"]["confirmations"][0] + for receipt in confirmed_receipts + if receipt["outcome"]["confirmations"] + ] + confirmed_used_nonmutating = ( + _delta(confirmed_only_case, "used_1", confirmed_key) == 0.0 + and all( + _delta(confirmed_only_case, f"used_{event}", confirmed_key) + == _delta(confirmed_only_case, f"outcome_{event - 1}", confirmed_key) + for event in range(2, event_budget + 1) + ) + ) + confirmed_boundary = ( + confirmed_used_nonmutating + and [item["multiplier"] for item in confirmations] + == [1.0, 0.5, 0.25, 0.125] + and all(item["actual_delta"] > 0.0 for item in confirmations) + ) + + soft_first = _checkpoint_by_name(confirmed_case, "used_1")["metrics"]["target_mrr"] + used_first = _checkpoint_by_name(used_case, "used_1")["metrics"]["target_mrr"] + confirmed_first = _checkpoint_by_name(confirmed_only_case, "used_1")["metrics"]["target_mrr"] + soft_final = _checkpoint_by_name(confirmed_case, "outcome_4")["metrics"]["target_mrr"] + used_final = _checkpoint_by_name(used_case, "outcome_4")["metrics"]["target_mrr"] + latency_quality = ( + provisional > 0.0 + and _delta(used_case, "used_1", confirmed_key) == 0.0 + and _delta(confirmed_only_case, "used_1", confirmed_key) == 0.0 + and soft_first >= min(used_first, confirmed_first) + and soft_final >= used_final + ) + + corrected = _case_by_role(soft, "corrected") + corrected_config = registered[str(corrected["case_id"])] + corrected_key = _edge_key(corrected_config["credited_edge"]) + corrected_used = _edge_at(_checkpoint_by_name(corrected, "used_1"), corrected_key) + corrected_base = float(corrected_used["base_increment"] or 0.0) + corrected_cost = _delta(corrected, "outcome_4", corrected_key) + negative_cost = ( + corrected_cost > 0.0 + and corrected_cost + <= corrected_base * float(thresholds["negative_provisional_ratio_max"]) + + tolerance + and _delta(corrected, "used_1", corrected_key) == corrected_cost + and not any( + receipt["outcome"]["reinforcement_applied"] for receipt in corrected["receipts"] + ) + ) + + control_roles = {"lexical", "zero_hop"} + control_safety = True + for arm in arms.values(): + for case in arm["cases"]: + baseline = _checkpoint_by_name(case, "baseline") + final = _checkpoint_by_name(case, "outcome_4") + if case["case_role"] in control_roles and final["changed_edges"]: + control_safety = False + if ( + final["metrics"]["direct_rank"] > baseline["metrics"]["direct_rank"] + or final["metrics"]["reverse_rank"] > baseline["metrics"]["reverse_rank"] + ): + control_safety = False + + locality = True + for arm_id, arm in arms.items(): + initial = {_edge_key(item): item for item in arm["initial_edges"]} + final = {_edge_key(item): item for item in arm["final_edges"]} + changed = { + key + for key in initial + if any( + initial[key][field] != final[key][field] + for field in ("weight", "reinforced_count", "evidence_count", "confirmation_count") + ) + } + allowed = set() if arm_id == "control" else { + edge + for case in cases + if case["case_role"] in {"confirmed", "corrected"} + for edge in case["allowed_mutation_edges"] + } + if not changed <= allowed: + locality = False + locality = locality and bool(preflight["snapshot_unchanged"]) + return { + "protocol-integrity": protocol_integrity, + "baseline-aware-q3-boundary": baseline_q3_boundary, + "confirmed-only-boundary": confirmed_boundary, + "soft-start-schedule": soft_schedule, + "learning-latency-and-quality": latency_quality, + "negative-provisional-bound": negative_cost, + "control-rank-and-mutation-safety": control_safety, + "mutation-locality-and-source-isolation": locality, + } + + +def _preflight( + snapshot: Path, + manifest: Mapping[str, Any], + artifacts: Mapping[str, Mapping[str, Any]], + stage: str, +) -> dict[str, Any]: + output = ROOT / str(manifest["outputs"][stage]) + development = ROOT / str(manifest["outputs"]["development"]) + holdout = ROOT / str(manifest["outputs"]["holdout"]) + fixture = artifacts["fixture"] + cases = fixture["stages"][stage] + snapshot_before = _sha256(snapshot) + with sqlite3.connect(snapshot.as_uri() + "?mode=ro", uri=True) as connection: + connection.execute("PRAGMA query_only = ON") + schema = _schema_identity(connection) + row_counts = { + name: int(connection.execute(f"SELECT count(*) FROM {name}").fetchone()[0]) + for name in manifest["snapshot"]["row_counts"] + } + node_ids = { + str(row[0]) for row in connection.execute("SELECT node_id FROM nodes") + } + edge_keys = { + "|".join(str(item) for item in row) + for row in connection.execute( + "SELECT source_id, target_id, edge_type FROM edges" + ) + } + registered_state_matches = True + derivation_matches = True + capacity_available = True + event_budget = int(artifacts["schedule"]["event_budget"]) + quorum = 3 + for case in cases: + if case["case_role"] not in {"confirmed", "corrected"}: + continue + identity = tuple(str(case["credited_edge"][field]) for field in EDGE_FIELDS) + edge = connection.execute( + """ + SELECT weight, reinforced_count FROM edges + WHERE source_id = ? AND target_id = ? AND edge_type = ? + """, + identity, + ).fetchone() + evidence_count = int( + connection.execute( + """ + SELECT count(*) FROM relation_feedback_evidence + WHERE source_id = ? AND target_id = ? AND edge_type = ? + """, + identity, + ).fetchone()[0] + ) + confirmed = connection.execute( + """ + SELECT confirmation_count FROM confirmed_edge_state + WHERE source_id = ? AND target_id = ? AND edge_type = ? + """, + identity, + ).fetchone() + actual = { + "weight": None if edge is None else float(edge[0]), + "reinforced_count": None if edge is None else int(edge[1]), + "evidence_count": evidence_count, + "confirmation_count": 0 if confirmed is None else int(confirmed[0]), + } + expected_event = derive_q3_first_mutation_event(quorum, evidence_count) + registered_state_matches = registered_state_matches and ( + actual == case["registered_initial_state"] + ) + derivation_matches = derivation_matches and ( + expected_event == case["expected_q3_first_mutation_event"] + ) + capacity_available = capacity_available and expected_event < event_budget + registered_nodes = { + str(case[field]) + for case in cases + for field in ("used_node_id", "source_node_id", "direct_node_id", "reverse_node_id") + } + registered_edges = { + _edge_key(case["credited_edge"]) + for case in cases + if case["case_role"] in {"confirmed", "corrected"} + } + trace_eligibility = True + try: + with TemporaryDirectory() as directory: + probe = Path(directory) / "probe.sqlite" + shutil.copyfile(snapshot, probe) + probe_arm = next( + arm + for arm in artifacts["schedule"]["arms"] + if arm["arm_id"] == "soft_start_r025_r05_s1" + ) + with NeuronGraphRAG( + probe, + config=_config( + probe_arm, artifacts["schedule"]["base_engine_config"] + ), + ) as engine: + for index, case in enumerate(cases): + _select_trace(engine, case, 80_000.0 + index) + except Exception: # noqa: BLE001 - the public report records only pass/fail + trace_eligibility = False + checks = { + "snapshot_hash": snapshot_before == manifest["snapshot"]["snapshot_sha256"], + "snapshot_size": snapshot.stat().st_size == manifest["snapshot"]["snapshot_size"], + "snapshot_schema": schema["schema_sha256"] + == manifest["snapshot"]["schema_sha256"] + and schema["table_names"] == manifest["snapshot"]["table_names"], + "snapshot_row_counts": row_counts == manifest["snapshot"]["row_counts"], + "registered_nodes": registered_nodes <= node_ids, + "registered_edges": registered_edges <= edge_keys, + "registered_initial_state": registered_state_matches, + "baseline_derivation": derivation_matches, + "event_budget_capacity": capacity_available, + "v1_development_edge_isolation": stage != "development" + or registered_edges.isdisjoint( + set(manifest["excluded_v1_development_credited_edges"]) + ), + "trace_eligibility": trace_eligibility, + "arm_order": [arm["arm_id"] for arm in artifacts["schedule"]["arms"]] == list(POLICIES), + "case_identity": len({case["case_id"] for case in cases}) == len(cases), + "event_order": artifacts["schedule"]["event_order"] + == [ + "used_1", + "outcome_1", + "used_2", + "outcome_2", + "used_3", + "outcome_3", + "used_4", + "outcome_4", + ], + "registered_output_absent": not output.exists(), + "result_free_stage_order": stage != "development" or not holdout.exists(), + "conditional_holdout": stage == "development" + or ( + development.exists() + and _verified_development_passed(development, manifest, artifacts) + ), + "privacy": True, + "placeholder_round_trip": artifacts["audit"]["placeholder_round_trip_passed"] is True, + } + return { + "checks": checks, + "passed": all(checks.values()), + "snapshot_sha256_before": snapshot_before, + "snapshot_sha256_after": _sha256(snapshot), + "snapshot_unchanged": _sha256(snapshot) == snapshot_before, + } + + +def _gate_ids(gate: Mapping[str, Any]) -> list[str]: + values = [str(item["gate_id"]) for item in gate["gates"]] + if not values or len(values) != len(set(values)): + raise ValueError("gate IDs must be non-empty and unique") + return values + + +def _semantic_hash(payload: Mapping[str, Any]) -> str: + return _sha256_bytes(_encoded(payload)) + + +def _verified_development_passed( + path: Path, + manifest: Mapping[str, Any], + artifacts: Mapping[str, Mapping[str, Any]], +) -> bool: + payload = read_json(path) + verify_observed_payload(payload, manifest, artifacts) + return payload["stage"] == "development" and payload["all_hard_gates_pass"] is True + + +def _interpretation(decision: str) -> dict[str, str]: + summaries = { + "支持": "固定したlocal snapshotとhard gateの範囲でsoft-start cutover候補を支持する。", + "不支持": "固定したlocal snapshotとhard gateの範囲でsoft-startの比較優位は成立しなかった。", + "判定不能": "固定protocolを完走できず、soft-start cutover可否は判定不能である。", + } + return { + "decision": decision, + "summary": summaries[decision], + "scope": "local snapshot限定。external corpus、production quality、library defaultへ一般化せず、source databaseとlive configを変更しない。", + } + + +def verify_observed_payload( + payload: Mapping[str, Any], + manifest: Mapping[str, Any], + artifacts: Mapping[str, Mapping[str, Any]], +) -> None: + assert_public_payload(payload) + if list(payload) != artifacts["result_schema"]["top_level_fields"]: + raise ValueError("observed top-level field order differs from frozen schema") + if payload["protocol_id"] != manifest["protocol_id"]: + raise ValueError("observed protocol identity differs") + if payload["stage"] not in STAGES: + raise ValueError("observed stage differs from registration") + if payload["snapshot"] != manifest["snapshot"]: + raise ValueError("observed snapshot identity differs from registration") + if payload["protocol_hashes"] != manifest["artifact_sha256"]: + raise ValueError("observed protocol hashes differ from registration") + gate_ids = _gate_ids(artifacts["gate"]) + if [item["gate_id"] for item in payload["gates"]] != gate_ids: + raise ValueError("observed gate order differs") + if payload["status"] == "completed": + expected_order = [ + arm["arm_id"] for arm in artifacts["schedule"]["arms"] + ] + if [arm["arm_id"] for arm in payload["arms"]] != expected_order: + raise ValueError("observed arm order differs from registration") + semantic_fields = ( + "arm_id", + "policy", + "effective_config_provenance", + "initial_edges", + "cases", + "final_edges", + ) + for arm in payload["arms"]: + semantic = {field: arm[field] for field in semantic_fields} + if arm["semantic_sha256"] != _semantic_hash(semantic): + raise ValueError("observed arm semantic hash differs") + arms = {arm["arm_id"]: arm for arm in payload["arms"]} + computed = evaluate_gates( + payload["preflight"], + artifacts["fixture"]["stages"][payload["stage"]], + arms, + artifacts["gate"]["thresholds"], + ) + if {item["gate_id"]: item["passed"] for item in payload["gates"]} != computed: + raise ValueError("gate results are not recomputable") + if payload["failure_code"] is not None: + raise ValueError("completed result cannot carry a failure code") + expected = "支持" if all(computed.values()) else "不支持" + elif payload["status"] == "indeterminate": + if payload["arms"] or any(item["passed"] for item in payload["gates"]): + raise ValueError("indeterminate result must not claim observations") + if payload["failure_code"] not in artifacts["result_schema"]["failure_codes"]: + raise ValueError("unknown failure code") + expected = "判定不能" + else: + raise ValueError("unknown result status") + if payload["all_hard_gates_pass"] is not all(item["passed"] for item in payload["gates"]): + raise ValueError("all_hard_gates_pass is not recomputable") + if payload["interpretation_ja"] != _interpretation(expected): + raise ValueError("interpretation differs from frozen decision mapping") + + +def run_registered_stage(stage: str, snapshot: Path) -> Path: + if stage not in STAGES: + raise ValueError("stage must be development or holdout") + manifest, artifacts = _load_protocol() + output = ROOT / str(manifest["outputs"][stage]) + if output.exists(): + raise FileExistsError(f"registered output already exists: {stage}") + preflight = _preflight(snapshot, manifest, artifacts, stage) + if not preflight["passed"]: + raise RuntimeError(f"protocol preflight failed: {preflight['checks']}") + arms: list[dict[str, Any]] = [] + failure_code: str | None = None + try: + cases = artifacts["fixture"]["stages"][stage] + for arm in artifacts["schedule"]["arms"]: + primary = _run_arm( + snapshot, arm, artifacts["schedule"]["base_engine_config"], cases + ) + replay = _run_arm( + snapshot, arm, artifacts["schedule"]["base_engine_config"], cases + ) + primary["semantic_sha256"] = _semantic_hash(primary) + primary["fresh_clone_replay"] = replay == { + key: primary[key] + for key in ("arm_id", "policy", "effective_config_provenance", "initial_edges", "cases", "final_edges") + } + arms.append(primary) + preflight["snapshot_sha256_after"] = _sha256(snapshot) + preflight["snapshot_unchanged"] = ( + preflight["snapshot_sha256_after"] == preflight["snapshot_sha256_before"] + ) + computed = evaluate_gates( + preflight, + cases, + {arm["arm_id"]: arm for arm in arms}, + artifacts["gate"]["thresholds"], + ) + status = "completed" + decision = "支持" if all(computed.values()) else "不支持" + except Exception: # noqa: BLE001 - a one-time protocol failure is evidence + failure_code = "execution-failed" + arms = [] + computed = {gate_id: False for gate_id in _gate_ids(artifacts["gate"])} + status = "indeterminate" + decision = "判定不能" + gate_ids = _gate_ids(artifacts["gate"]) + payload = { + "protocol_id": manifest["protocol_id"], + "stage": stage, + "status": status, + "failure_code": failure_code, + "snapshot": manifest["snapshot"], + "protocol_hashes": manifest["artifact_sha256"], + "preflight": preflight, + "arms": arms, + "gates": [{"gate_id": gate_id, "passed": computed[gate_id]} for gate_id in gate_ids], + "all_hard_gates_pass": all(computed.values()), + "interpretation_ja": _interpretation(decision), + } + verify_observed_payload(payload, manifest, artifacts) + write_json_exclusive(output, payload) + verify_observed_payload(read_json(output), manifest, artifacts) + return output + + +def verify_registered_result(stage: str) -> None: + if stage not in STAGES: + raise ValueError("stage must be development or holdout") + manifest, artifacts = _load_protocol() + verify_observed_payload(read_json(ROOT / str(manifest["outputs"][stage])), manifest, artifacts) + + +def preflight_snapshot(snapshot: Path) -> dict[str, Any]: + manifest, artifacts = _load_current_protocol() + report = { + stage: _preflight(snapshot, manifest, artifacts, stage) + for stage in STAGES + if stage == "development" + } + assert_public_payload(report) + return report diff --git a/tests/fixtures/baseline_aware_soft_start_snapshot_v2.fixture.json b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.fixture.json new file mode 100644 index 0000000..56cb415 --- /dev/null +++ b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.fixture.json @@ -0,0 +1,252 @@ +{ + "protocol_id": "baseline-aware-soft-start-snapshot-v2", + "selection_rule": "Fresh snapshot public identifiers selected without using v1 private snapshot or observed development edge identities; baseline state and q3 first mutation are frozen before registered results.", + "stages": { + "development": [ + { + "case_id": "v2-dev-confirmed-requirements-real-corpus", + "case_role": "confirmed", + "query": "requirements specification feedback interface", + "search_surface": "relation", + "used_node_id": "docs/real-corpus-benchmark.md", + "source_node_id": "docs/requirements.md", + "credited_edge": { + "source_id": "docs/requirements.md", + "target_id": "docs/real-corpus-benchmark.md", + "edge_type": "links_to" + }, + "registered_initial_state": { + "weight": 0.5626442053538984, + "reinforced_count": 1, + "evidence_count": 0, + "confirmation_count": 0 + }, + "expected_q3_first_mutation_event": 3, + "outcome": "confirmed", + "direct_query": "real corpus benchmark", + "direct_node_id": "docs/real-corpus-benchmark.md", + "reverse_query": "requirements specification", + "reverse_node_id": "docs/requirements.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [ + "docs/requirements.md|docs/anchored-bm25-graph-hybrid-experiment.md|links_to", + "docs/requirements.md|docs/anchored-fusion-calibration-experiment.md|links_to", + "docs/requirements.md|docs/blind-llm-channel-selection-experiment.md|links_to", + "docs/requirements.md|docs/d1-corpus-fixture.md|links_to", + "docs/requirements.md|docs/engine-backed-feedback-trajectory-experiment.md|links_to", + "docs/requirements.md|docs/feedback-adaptation-experiment.md|links_to", + "docs/requirements.md|docs/feedback-adaptation-reproduction-experiment.md|links_to", + "docs/requirements.md|docs/independent-retrieval-channels-experiment.md|links_to", + "docs/requirements.md|docs/neural-dynamics-experiment.md|links_to", + "docs/requirements.md|docs/neural-dynamics-local-competition-experiment.md|links_to", + "docs/requirements.md|docs/node-first-blind-selection-experiment.md|links_to", + "docs/requirements.md|docs/optional-mcp-interface.md|links_to", + "docs/requirements.md|docs/real-corpus-benchmark.md|links_to", + "docs/requirements.md|docs/sibling-normalization-controlled-evaluation.md|links_to", + "docs/requirements.md|docs/sibling-relation-feedback-normalization.md|links_to" + ] + }, + { + "case_id": "v2-dev-corrected-readme-github-compatibility", + "case_role": "corrected", + "query": "Neuron Graph RAG prototype", + "search_surface": "relation", + "used_node_id": "docs/github-rag-mcp-replacement-compatibility.md", + "source_node_id": "README.md", + "credited_edge": { + "source_id": "README.md", + "target_id": "docs/github-rag-mcp-replacement-compatibility.md", + "edge_type": "links_to" + }, + "registered_initial_state": { + "weight": 0.5, + "reinforced_count": 0, + "evidence_count": 0, + "confirmation_count": 0 + }, + "expected_q3_first_mutation_event": 3, + "outcome": "corrected", + "direct_query": "GitHub RAG MCP replacement compatibility", + "direct_node_id": "docs/github-rag-mcp-replacement-compatibility.md", + "reverse_query": "Neuron Graph RAG prototype", + "reverse_node_id": "README.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [ + "README.md|docs/anchored-bm25-graph-hybrid-experiment.md|links_to", + "README.md|docs/anchored-fusion-calibration-experiment.md|links_to", + "README.md|docs/blind-llm-channel-selection-experiment.md|links_to", + "README.md|docs/d1-corpus-fixture.md|links_to", + "README.md|docs/feedback-adaptation-experiment.md|links_to", + "README.md|docs/feedback-adaptation-reproduction-experiment.md|links_to", + "README.md|docs/github-rag-mcp-replacement-compatibility.md|links_to", + "README.md|docs/independent-retrieval-channels-experiment.md|links_to", + "README.md|docs/neural-dynamics-experiment.md|links_to", + "README.md|docs/neural-dynamics-local-competition-experiment.md|links_to", + "README.md|docs/node-first-blind-selection-experiment.md|links_to", + "README.md|docs/optional-mcp-interface.md|links_to", + "README.md|docs/real-corpus-benchmark.md|links_to", + "README.md|docs/requirements.md|links_to" + ] + }, + { + "case_id": "v2-dev-lexical-real-corpus-control", + "case_role": "lexical", + "query": "real corpus benchmark", + "search_surface": "lexical", + "used_node_id": "docs/real-corpus-benchmark.md", + "source_node_id": "docs/real-corpus-benchmark.md", + "credited_edge": {}, + "outcome": "confirmed", + "direct_query": "real corpus benchmark", + "direct_node_id": "docs/real-corpus-benchmark.md", + "reverse_query": "real corpus benchmark", + "reverse_node_id": "docs/real-corpus-benchmark.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [] + }, + { + "case_id": "v2-dev-zero-hop-home-control", + "case_role": "zero_hop", + "query": "Neuron Graph RAG home", + "search_surface": "lexical", + "used_node_id": "docs/Home.md", + "source_node_id": "docs/Home.md", + "credited_edge": {}, + "outcome": "confirmed", + "direct_query": "Neuron Graph RAG home", + "direct_node_id": "docs/Home.md", + "reverse_query": "Neuron Graph RAG home", + "reverse_node_id": "docs/Home.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [] + } + ], + "holdout": [ + { + "case_id": "v2-holdout-confirmed-requirements-fusion", + "case_role": "confirmed", + "query": "requirements specification feedback interface", + "search_surface": "relation", + "used_node_id": "docs/anchored-fusion-calibration-experiment.md", + "source_node_id": "docs/requirements.md", + "credited_edge": { + "source_id": "docs/requirements.md", + "target_id": "docs/anchored-fusion-calibration-experiment.md", + "edge_type": "links_to" + }, + "registered_initial_state": { + "weight": 0.553741578432044, + "reinforced_count": 1, + "evidence_count": 0, + "confirmation_count": 0 + }, + "expected_q3_first_mutation_event": 3, + "outcome": "confirmed", + "direct_query": "anchored fusion calibration experiment", + "direct_node_id": "docs/anchored-fusion-calibration-experiment.md", + "reverse_query": "requirements specification", + "reverse_node_id": "docs/requirements.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [ + "docs/requirements.md|docs/anchored-bm25-graph-hybrid-experiment.md|links_to", + "docs/requirements.md|docs/anchored-fusion-calibration-experiment.md|links_to", + "docs/requirements.md|docs/blind-llm-channel-selection-experiment.md|links_to", + "docs/requirements.md|docs/d1-corpus-fixture.md|links_to", + "docs/requirements.md|docs/engine-backed-feedback-trajectory-experiment.md|links_to", + "docs/requirements.md|docs/feedback-adaptation-experiment.md|links_to", + "docs/requirements.md|docs/feedback-adaptation-reproduction-experiment.md|links_to", + "docs/requirements.md|docs/independent-retrieval-channels-experiment.md|links_to", + "docs/requirements.md|docs/neural-dynamics-experiment.md|links_to", + "docs/requirements.md|docs/neural-dynamics-local-competition-experiment.md|links_to", + "docs/requirements.md|docs/node-first-blind-selection-experiment.md|links_to", + "docs/requirements.md|docs/optional-mcp-interface.md|links_to", + "docs/requirements.md|docs/real-corpus-benchmark.md|links_to", + "docs/requirements.md|docs/sibling-normalization-controlled-evaluation.md|links_to", + "docs/requirements.md|docs/sibling-relation-feedback-normalization.md|links_to" + ] + }, + { + "case_id": "v2-holdout-corrected-readme-channels", + "case_role": "corrected", + "query": "Neuron Graph RAG prototype", + "search_surface": "relation", + "used_node_id": "docs/independent-retrieval-channels-experiment.md", + "source_node_id": "README.md", + "credited_edge": { + "source_id": "README.md", + "target_id": "docs/independent-retrieval-channels-experiment.md", + "edge_type": "links_to" + }, + "registered_initial_state": { + "weight": 0.5, + "reinforced_count": 0, + "evidence_count": 0, + "confirmation_count": 0 + }, + "expected_q3_first_mutation_event": 3, + "outcome": "corrected", + "direct_query": "independent retrieval channels experiment", + "direct_node_id": "docs/independent-retrieval-channels-experiment.md", + "reverse_query": "Neuron Graph RAG prototype", + "reverse_node_id": "README.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [ + "README.md|docs/anchored-bm25-graph-hybrid-experiment.md|links_to", + "README.md|docs/anchored-fusion-calibration-experiment.md|links_to", + "README.md|docs/blind-llm-channel-selection-experiment.md|links_to", + "README.md|docs/d1-corpus-fixture.md|links_to", + "README.md|docs/feedback-adaptation-experiment.md|links_to", + "README.md|docs/feedback-adaptation-reproduction-experiment.md|links_to", + "README.md|docs/github-rag-mcp-replacement-compatibility.md|links_to", + "README.md|docs/independent-retrieval-channels-experiment.md|links_to", + "README.md|docs/neural-dynamics-experiment.md|links_to", + "README.md|docs/neural-dynamics-local-competition-experiment.md|links_to", + "README.md|docs/node-first-blind-selection-experiment.md|links_to", + "README.md|docs/optional-mcp-interface.md|links_to", + "README.md|docs/real-corpus-benchmark.md|links_to", + "README.md|docs/requirements.md|links_to" + ] + }, + { + "case_id": "v2-holdout-lexical-github-control", + "case_role": "lexical", + "query": "GitHub RAG MCP replacement compatibility", + "search_surface": "lexical", + "used_node_id": "docs/github-rag-mcp-replacement-compatibility.md", + "source_node_id": "docs/github-rag-mcp-replacement-compatibility.md", + "credited_edge": {}, + "outcome": "confirmed", + "direct_query": "GitHub RAG MCP replacement compatibility", + "direct_node_id": "docs/github-rag-mcp-replacement-compatibility.md", + "reverse_query": "GitHub RAG MCP replacement compatibility", + "reverse_node_id": "docs/github-rag-mcp-replacement-compatibility.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [] + }, + { + "case_id": "v2-holdout-zero-hop-footer-control", + "case_role": "zero_hop", + "query": "Neuron Graph RAG footer", + "search_surface": "lexical", + "used_node_id": "docs/_Footer.md", + "source_node_id": "docs/_Footer.md", + "credited_edge": {}, + "outcome": "confirmed", + "direct_query": "Neuron Graph RAG footer", + "direct_node_id": "docs/_Footer.md", + "reverse_query": "Neuron Graph RAG footer", + "reverse_node_id": "docs/_Footer.md", + "retrieval_limit": 21, + "hit_k": 5, + "allowed_mutation_edges": [] + } + ] + } +} diff --git a/tests/fixtures/baseline_aware_soft_start_snapshot_v2.gate.json b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.gate.json new file mode 100644 index 0000000..3e5ccce --- /dev/null +++ b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.gate.json @@ -0,0 +1,43 @@ +{ + "protocol_id": "baseline-aware-soft-start-snapshot-v2", + "thresholds": { + "negative_provisional_ratio_max": 0.25, + "confirmation_decay_ratio": 0.5, + "numeric_tolerance": 1e-12, + "event_budget": 4 + }, + "gates": [ + { + "gate_id": "protocol-integrity", + "hard": true + }, + { + "gate_id": "baseline-aware-q3-boundary", + "hard": true + }, + { + "gate_id": "confirmed-only-boundary", + "hard": true + }, + { + "gate_id": "soft-start-schedule", + "hard": true + }, + { + "gate_id": "learning-latency-and-quality", + "hard": true + }, + { + "gate_id": "negative-provisional-bound", + "hard": true + }, + { + "gate_id": "control-rank-and-mutation-safety", + "hard": true + }, + { + "gate_id": "mutation-locality-and-source-isolation", + "hard": true + } + ] +} diff --git a/tests/fixtures/baseline_aware_soft_start_snapshot_v2.manifest.json b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.manifest.json new file mode 100644 index 0000000..258433a --- /dev/null +++ b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.manifest.json @@ -0,0 +1,73 @@ +{ + "protocol_id": "baseline-aware-soft-start-snapshot-v2", + "issue": 106, + "source_boundary": "fresh-one-time-transaction-consistent-local-snapshot", + "historical_verification": "manifest-first-addition-commit-exact-blobs", + "predecessor_boundary": "v1 protocol, gate, result, interpretation, and private snapshot are not modified, rerun, recomputed, or used as v2 inputs", + "excluded_v1_development_credited_edges": [ + "docs/requirements.md|docs/optional-mcp-interface.md|links_to", + "docs/sibling-relation-feedback-normalization.md|docs/sibling-normalization-controlled-evaluation.md|links_to" + ], + "snapshot": { + "source_locator": "local_codex_ngr_database", + "source_access": "sqlite-uri-mode-ro-query-only", + "capture_method": "sqlite-backup-api", + "captured_at": "2026-08-21T01:14:55+00:00", + "source_container_sha256_before": "90c664d345aab9a9ccf36943b5b503cf98e9588577db0e7ccd4f07b27cb33fe0", + "source_container_sha256_after": "90c664d345aab9a9ccf36943b5b503cf98e9588577db0e7ccd4f07b27cb33fe0", + "snapshot_sha256": "003f24478f8d18dd34ab6aee5d939202d116f195ac8db33c29e38ab7e0f456d1", + "snapshot_size": 425984, + "schema_sha256": "a4e58aa7675a6d9e3d9dd7fa778d5bbb4d0ec45eecc262e62f87e019a3ccae9f", + "table_names": [ + "activation_state", + "confirmed_edge_state", + "confirmed_relation_feedback", + "confirmed_source_uses", + "delayed_outcome_nodes", + "delayed_outcomes", + "edges", + "feedback_requests", + "nodes", + "relation_feedback_evidence", + "retrieval_channels", + "retrieval_results", + "retrievals", + "source_use_state", + "success_feedback", + "success_nodes" + ], + "row_counts": { + "nodes": 21, + "edges": 30, + "retrievals": 24, + "success_feedback": 10, + "delayed_outcomes": 3 + } + }, + "protocol_artifacts": { + "fixture": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.fixture.json", + "schedule": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.schedule.json", + "gate": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.gate.json", + "result_schema": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-schema.json", + "audit": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-free-audit.json" + }, + "outputs": { + "development": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.development.observed.json", + "holdout": "tests/fixtures/baseline_aware_soft_start_snapshot_v2.holdout.observed.json" + }, + "artifact_sha256": { + "src/neuron_graph_rag/baseline_aware_soft_start_snapshot_evaluation.py": "9532e9e6fe59a48946d529347f38b7c1c6b685a4d3ad212ad5d8232c22bdc012", + "tools/run_baseline_aware_soft_start_snapshot_evaluation.py": "a119742a092aee44f2f7bb135d795f314fc65a623efa60669da20d1fb86cf714", + "tests/test_baseline_aware_soft_start_snapshot_evaluation.py": "ab6647bdf796e43a4a8eb8077b525b7f6e78184739db7067fad75cc93274c0d9", + "tests/test_soft_start_snapshot_evaluation.py": "d576fa0f0e6ceb525532bfba78ee66903651f5730f50fe9a0a09294687fa2dba", + "tests/fixtures/baseline_aware_soft_start_snapshot_v2.fixture.json": "7b55771bb37e68313ba50e7bdf00aa6edab2fabef61a2ade0fb651fdb2e2ae62", + "tests/fixtures/baseline_aware_soft_start_snapshot_v2.schedule.json": "a4f697a2eeb4c2921b7613a24da629f1e39f6ace286b5f1bf2ba1a2b168f18ca", + "tests/fixtures/baseline_aware_soft_start_snapshot_v2.gate.json": "e0ebf1c3a4e0b827377aa7833c2c384fd3a0ba919e61a5f1e1633ca6d46b6998", + "tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-schema.json": "099fcf79eecb207ebbe3750ddac20876b81fb5675f1758e8a560302c49e09604", + "tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-free-audit.json": "c53874177be067783e00350590879f815bc98acb9768c6cd3ebdad9e316827c8", + "docs/baseline-aware-soft-start-snapshot-evaluation.md": "0481abf383c60722a3d4b8416cf166eb4c67fa6f8e425eef63d1e9b2c44317f5", + "docs/requirements.md": "4f4d97f8212a0cf7d35f02b826bbb2a9f08ada5089eced96cfcd168ea4119278", + "docs/Decision-Structure.md": "5889ecbd5e23bd67e3494d30a9ef7b37855eba46f3a807d45c4adfb2bcd7d3df", + "README.md": "cbee26ad9f4e6343786843ac70bdd491fd6c855b3d9dd05e6ca4c73563e92ab2" + } +} diff --git a/tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-free-audit.json b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-free-audit.json new file mode 100644 index 0000000..7cc166c --- /dev/null +++ b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-free-audit.json @@ -0,0 +1,16 @@ +{ + "protocol_id": "baseline-aware-soft-start-snapshot-v2", + "result_free": true, + "registered_outputs_absent": true, + "source_database_write_count": 0, + "fresh_snapshot_acquisition_count": 1, + "v1_private_snapshot_used": false, + "v1_observed_result_used": false, + "v1_development_credited_edge_reused": false, + "snapshot_clone_only": true, + "snapshot_body_private": true, + "private_node_text_registered": false, + "absolute_private_path_registered": false, + "placeholder_round_trip_passed": true, + "prior_frozen_artifacts_modified": false +} diff --git a/tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-schema.json b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-schema.json new file mode 100644 index 0000000..28e78aa --- /dev/null +++ b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.result-schema.json @@ -0,0 +1,21 @@ +{ + "protocol_id": "baseline-aware-soft-start-snapshot-v2", + "top_level_fields": [ + "protocol_id", + "stage", + "status", + "failure_code", + "snapshot", + "protocol_hashes", + "preflight", + "arms", + "gates", + "all_hard_gates_pass", + "interpretation_ja" + ], + "failure_codes": [ + "execution-failed" + ], + "preflight_failure_policy": "no-registered-result-and-failure-report", + "output_policy": "exclusive-create-no-overwrite-no-rerun" +} diff --git a/tests/fixtures/baseline_aware_soft_start_snapshot_v2.schedule.json b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.schedule.json new file mode 100644 index 0000000..00358fb --- /dev/null +++ b/tests/fixtures/baseline_aware_soft_start_snapshot_v2.schedule.json @@ -0,0 +1,86 @@ +{ + "protocol_id": "baseline-aware-soft-start-snapshot-v2", + "base_engine_config": { + "sparse_weight": 0.55, + "dense_weight": 0.45, + "entry_weight": 0.55, + "graph_weight": 0.45, + "seed_count": 3, + "max_hops": 2, + "hop_decay": 0.7, + "activation_half_life_seconds": 3600.0, + "feedback_learning_rate": 0.2, + "sibling_feedback_normalization": 1.0, + "maximum_edge_weight": 2.0, + "maximum_activation": 10.0, + "max_paths_per_node": 4, + "max_propagation_expansions": 10000, + "activation_strategy": "current_positive_additive", + "activation_budget": 1.0, + "inhibition_ratio": 0.0, + "inhibition_top_k": 0, + "query_transmission_floor": 0.4, + "query_transmission_power": 1.0, + "recurrent_steps": 3, + "recurrent_decay": 0.5, + "convergence_tolerance": 1e-09, + "max_active_paths_per_node": 4, + "use_dense_retrieval": true, + "use_graph_propagation": true, + "graph_normalization": "max", + "final_fusion_strategy": "linear", + "rrf_k": 60, + "relation_feedback_evidence_quorum": 1, + "confirmed_outcome_reinforcement": false, + "confirmation_decay_ratio": null, + "soft_start_feedback_reinforcement": false, + "soft_start_feedback_ratio": null + }, + "arms": [ + { + "arm_id": "control", + "policy": "audit-only", + "engine_config": {} + }, + { + "arm_id": "used_q3_s1", + "policy": "baseline-aware-used-evidence-quorum-3-sibling-1.0", + "engine_config": { + "relation_feedback_evidence_quorum": 3 + } + }, + { + "arm_id": "confirmed_r05_s1", + "policy": "confirmed-only-decay-0.5-sibling-1.0", + "engine_config": { + "confirmed_outcome_reinforcement": true, + "confirmation_decay_ratio": 0.5 + } + }, + { + "arm_id": "soft_start_r025_r05_s1", + "policy": "soft-start-0.25-decay-0.5-sibling-1.0", + "engine_config": { + "soft_start_feedback_reinforcement": true, + "soft_start_feedback_ratio": 0.25, + "confirmation_decay_ratio": 0.5 + } + } + ], + "event_order": [ + "used_1", + "outcome_1", + "used_2", + "outcome_2", + "used_3", + "outcome_3", + "used_4", + "outcome_4" + ], + "event_budget": 4, + "q3_first_mutation_derivation": "max(1, relation_feedback_evidence_quorum - registered_initial_evidence_count)", + "capacity_rule": "derived first mutation must occur before the final event so one later fresh evidence activation is observed", + "checkpoint_policy": "baseline and after every registered event", + "fresh_trace_policy": "one new search trace per event iteration", + "fresh_clone_replay": true +} diff --git a/tests/test_baseline_aware_soft_start_snapshot_evaluation.py b/tests/test_baseline_aware_soft_start_snapshot_evaluation.py new file mode 100644 index 0000000..698b314 --- /dev/null +++ b/tests/test_baseline_aware_soft_start_snapshot_evaluation.py @@ -0,0 +1,211 @@ +from __future__ import annotations + +import hashlib +import json +import sqlite3 +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from neuron_graph_rag.baseline_aware_soft_start_snapshot_evaluation import ( + MANIFEST_PATH, + POLICIES, + STEM, + acquire_transactional_snapshot, + assert_public_payload, + derive_q3_first_mutation_event, + prove_writer_verifier_round_trip, + read_json, + run_registered_stage, + verify_registered_result, + write_json_exclusive, +) +from neuron_graph_rag.corpus_integrity import verify_manifest_source_hashes + + +ROOT = Path(__file__).resolve().parents[1] +FIXTURES = ROOT / "tests" / "fixtures" + + +def _fixture(name: str) -> dict[str, object]: + return read_json(FIXTURES / f"{STEM}.{name}.json") + + +class BaselineAwareSoftStartSnapshotFreezeTest(unittest.TestCase): + def test_result_free_artifacts_are_canonical_private_free_and_complete(self) -> None: + names = ( + "fixture", + "schedule", + "gate", + "result-schema", + "result-free-audit", + ) + for name in names: + path = FIXTURES / f"{STEM}.{name}.json" + raw = path.read_bytes() + payload = json.loads(raw.decode("utf-8", errors="strict")) + self.assertNotIn(b"\r", raw) + self.assertEqual( + raw.decode("utf-8"), + json.dumps(payload, ensure_ascii=False, indent=2) + "\n", + ) + self.assertEqual( + payload["protocol_id"], "baseline-aware-soft-start-snapshot-v2" + ) + assert_public_payload(payload) + + schedule = _fixture("schedule") + self.assertEqual( + [arm["arm_id"] for arm in schedule["arms"]], list(POLICIES) + ) + self.assertEqual(schedule["event_budget"], 4) + self.assertEqual( + schedule["event_order"], + [ + "used_1", + "outcome_1", + "used_2", + "outcome_2", + "used_3", + "outcome_3", + "used_4", + "outcome_4", + ], + ) + gate = _fixture("gate") + self.assertEqual(len(gate["gates"]), 8) + self.assertEqual(gate["gates"][1]["gate_id"], "baseline-aware-q3-boundary") + + def test_initial_state_derivation_capacity_and_v1_isolation_are_frozen(self) -> None: + fixture = _fixture("fixture") + manifest = read_json(MANIFEST_PATH) + excluded = set(manifest["excluded_v1_development_credited_edges"]) + development_edges: set[str] = set() + all_case_ids: set[str] = set() + for stage in ("development", "holdout"): + for case in fixture["stages"][stage]: + self.assertNotIn(case["case_id"], all_case_ids) + all_case_ids.add(case["case_id"]) + if case["case_role"] not in {"confirmed", "corrected"}: + continue + initial = case["registered_initial_state"] + expected = derive_q3_first_mutation_event( + 3, initial["evidence_count"] + ) + self.assertEqual(case["expected_q3_first_mutation_event"], expected) + self.assertLess(expected, 4) + key = "|".join( + case["credited_edge"][field] + for field in ("source_id", "target_id", "edge_type") + ) + if stage == "development": + development_edges.add(key) + self.assertTrue(development_edges.isdisjoint(excluded)) + audit = _fixture("result-free-audit") + self.assertFalse(audit["v1_private_snapshot_used"]) + self.assertFalse(audit["v1_observed_result_used"]) + self.assertFalse(audit["v1_development_credited_edge_reused"]) + + def test_manifest_reads_registration_bytes_and_outputs_were_absent(self) -> None: + manifest = read_json(MANIFEST_PATH) + registered = verify_manifest_source_hashes( + ROOT, MANIFEST_PATH, manifest["artifact_sha256"] + ) + self.assertEqual(registered.artifact_sha256, manifest["artifact_sha256"]) + self.assertFalse( + any("soft_start_snapshot_v1" in path for path in manifest["artifact_sha256"]) + ) + for relative in manifest["outputs"].values(): + completed = subprocess.run( + ["git", "cat-file", "-e", f"{registered.source_commit}:{relative}"], + cwd=ROOT, + check=False, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + self.assertNotEqual(completed.returncode, 0) + + def test_preflight_failure_creates_no_registered_result(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + snapshot = root / "snapshot.sqlite" + snapshot.write_bytes(b"not-opened") + output = root / "development.json" + manifest = { + "outputs": { + "development": str(output), + "holdout": str(root / "holdout.json"), + } + } + report = { + "checks": {"registered_initial_state": False}, + "passed": False, + "snapshot_sha256_before": None, + "snapshot_sha256_after": None, + "snapshot_unchanged": False, + } + with patch( + "neuron_graph_rag.baseline_aware_soft_start_snapshot_evaluation._load_protocol", + return_value=(manifest, {}), + ), patch( + "neuron_graph_rag.baseline_aware_soft_start_snapshot_evaluation._preflight", + return_value=report, + ): + with self.assertRaisesRegex(RuntimeError, "protocol preflight failed"): + run_registered_stage("development", snapshot) + self.assertFalse(output.exists()) + + def test_snapshot_acquisition_is_read_only_private_and_exclusive(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = root / "source.sqlite" + snapshot = root / "snapshot.sqlite" + connection = sqlite3.connect(source) + try: + connection.execute("CREATE TABLE nodes(node_id TEXT PRIMARY KEY, text TEXT)") + connection.execute( + "CREATE TABLE edges(source_id TEXT, target_id TEXT, edge_type TEXT)" + ) + connection.execute("CREATE TABLE retrievals(trace_id TEXT)") + connection.execute("CREATE TABLE success_feedback(feedback_id TEXT)") + connection.execute("CREATE TABLE delayed_outcomes(outcome_id TEXT)") + connection.execute("INSERT INTO nodes VALUES ('public-id', 'private body')") + connection.commit() + finally: + connection.close() + before = hashlib.sha256(source.read_bytes()).hexdigest() + provenance = acquire_transactional_snapshot(source, snapshot) + self.assertEqual(before, hashlib.sha256(source.read_bytes()).hexdigest()) + self.assertNotIn("private body", json.dumps(provenance)) + assert_public_payload(provenance) + with self.assertRaises(FileExistsError): + acquire_transactional_snapshot(source, snapshot) + + def test_writer_round_trip_privacy_and_registered_results(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + prove_writer_verifier_round_trip(root / "probe.json") + self.assertFalse((root / "probe.json").exists()) + target = root / "exclusive.json" + write_json_exclusive(target, {"safe": True}) + with self.assertRaises(FileExistsError): + write_json_exclusive(target, {"safe": False}) + for payload in ( + {"node_text": "secret"}, + {"value": "C:\\private\\snapshot.sqlite"}, + {"value": "/private/snapshot.sqlite"}, + {"value": "password=hunter2"}, + ): + with self.assertRaises(ValueError): + assert_public_payload(payload) + if MANIFEST_PATH.exists(): + manifest = read_json(MANIFEST_PATH) + for stage, relative in manifest["outputs"].items(): + if (ROOT / relative).exists(): + verify_registered_result(stage) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_soft_start_snapshot_evaluation.py b/tests/test_soft_start_snapshot_evaluation.py index a0a4956..6838bec 100644 --- a/tests/test_soft_start_snapshot_evaluation.py +++ b/tests/test_soft_start_snapshot_evaluation.py @@ -81,14 +81,14 @@ def test_result_free_artifacts_are_canonical_private_free_and_complete(self) -> self.assertEqual(audit["source_database_write_count"], 0) self.assertTrue(audit["placeholder_round_trip_passed"]) - def test_manifest_reads_registered_commit_bytes_and_registered_outputs_were_absent(self) -> None: + def test_manifest_reads_registered_commit_bytes_and_squashed_outputs_verify(self) -> None: manifest = read_json(MANIFEST_PATH) assert_public_payload(manifest) registered = verify_manifest_source_hashes( ROOT, MANIFEST_PATH, manifest["artifact_sha256"] ) self.assertEqual(registered.artifact_sha256, manifest["artifact_sha256"]) - for relative in manifest["outputs"].values(): + for stage, relative in manifest["outputs"].items(): completed = subprocess.run( [ "git", @@ -101,7 +101,17 @@ def test_manifest_reads_registered_commit_bytes_and_registered_outputs_were_abse stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) - self.assertNotEqual(completed.returncode, 0) + # A squash merge can collapse the result-free and development commits, + # so main history cannot prove that development was absent before it was + # observed. The historical blob and exclusive observed-result verifier + # remain checkable, while unopened holdout absence remains exact. + if stage == "development": + self.assertEqual(completed.returncode, 0) + self.assertTrue((ROOT / relative).exists()) + verify_registered_result(stage) + else: + self.assertNotEqual(completed.returncode, 0) + self.assertFalse((ROOT / relative).exists()) def test_snapshot_acquisition_is_transactional_read_only_and_exclusive(self) -> None: with tempfile.TemporaryDirectory() as directory: diff --git a/tools/run_baseline_aware_soft_start_snapshot_evaluation.py b/tools/run_baseline_aware_soft_start_snapshot_evaluation.py new file mode 100644 index 0000000..2605521 --- /dev/null +++ b/tools/run_baseline_aware_soft_start_snapshot_evaluation.py @@ -0,0 +1,47 @@ +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from tempfile import TemporaryDirectory + +from neuron_graph_rag.baseline_aware_soft_start_snapshot_evaluation import ( + acquire_transactional_snapshot, + preflight_snapshot, + prove_writer_verifier_round_trip, + run_registered_stage, + verify_registered_result, +) + + +def main() -> None: + parser = argparse.ArgumentParser() + actions = parser.add_mutually_exclusive_group(required=True) + actions.add_argument("--acquire", action="store_true") + actions.add_argument("--probe", action="store_true") + actions.add_argument("--stage", choices=("development", "holdout")) + actions.add_argument("--verify", choices=("development", "holdout")) + parser.add_argument("--source", type=Path) + parser.add_argument("--snapshot", type=Path) + args = parser.parse_args() + if args.acquire: + if args.source is None or args.snapshot is None: + parser.error("--acquire requires --source and --snapshot") + print(json.dumps(acquire_transactional_snapshot(args.source, args.snapshot), indent=2)) + elif args.probe: + if args.snapshot is None: + parser.error("--probe requires --snapshot") + with TemporaryDirectory() as directory: + prove_writer_verifier_round_trip(Path(directory) / "placeholder.json") + print(json.dumps(preflight_snapshot(args.snapshot), indent=2)) + elif args.stage: + if args.snapshot is None: + parser.error("--stage requires --snapshot") + print(run_registered_stage(args.stage, args.snapshot)) + else: + verify_registered_result(args.verify) + print(f"{args.verify} verification passed") + + +if __name__ == "__main__": + main()