diff --git a/docs/Decision-Structure.md b/docs/Decision-Structure.md index cd8df6a..7370de5 100644 --- a/docs/Decision-Structure.md +++ b/docs/Decision-Structure.md @@ -16,10 +16,12 @@ | [confirmed-outcome-feedback-reinforcement](https://github.com/Liplus-Project/neuron-graph-rag/wiki/confirmed-outcome-feedback-reinforcement) | superseded | confirmed-only candidate の mechanics と証拠は保持するが、利用直後の小さな適応を残す [soft-start-feedback-reinforcement](https://github.com/Liplus-Project/neuron-graph-rag/wiki/soft-start-feedback-reinforcement) が後続比較の現在候補としてこの判断を supersede する。 | | [outcome-driven-feedback-deactivation](https://github.com/Liplus-Project/neuron-graph-rag/wiki/outcome-driven-feedback-deactivation) | evaluating | 因果帰属できるnegative outcomeだけがsoft-start contributionと同一単位のsibling normalizationを正確に戻し、superseded edgeは削除せずdormant / reactivationで扱う。mechanicsとresult-free freezeを先に固定し、観測と採用判断はsuccessor Issueへ分離する。 | | [frozen-output-round-trip-integrity](https://github.com/Liplus-Project/neuron-graph-rag/wiki/frozen-output-round-trip-integrity) | active | result-free evaluation は canonical gate ID array を唯一の順序正本とし、freeze 前に登録外 placeholder と temporary output で実 writer から実 verifier への非アルファベット順 round-trip を証明する。observed result は exclusive creation し、array の完全性、順序、重複なし、全 gate pass を検証する。 | +| [historical-freeze-verification-boundary](https://github.com/Liplus-Project/neuron-graph-rag/wiki/historical-freeze-verification-boundary) | active | historical source hash は current working tree ではなく、frozen manifest が登録した source commit の exact bytes に対して検証し、repository の正当な後続変更と過去証拠の不変性を時間軸で分離する。 | | [longitudinal-feedback-adaptation](https://github.com/Liplus-Project/neuron-graph-rag/wiki/longitudinal-feedback-adaptation) | active | longitudinal feedback-adaptation は、repository-native controlled corpus v3 の相互に独立した cluster と、その source 文書に明記した 0、1、3、10 credit ceiling を用いる。corpus phase は query、gold、schedule、runner、gate、manifest、result、既定値を定義・変更しない。 | | [single-corpus-real-feedback-validation](https://github.com/Liplus-Project/neuron-graph-rag/wiki/single-corpus-real-feedback-validation) | superseded | [repository-native-controlled-corpus](https://github.com/Liplus-Project/neuron-graph-rag/wiki/repository-native-controlled-corpus) がこの node を supersede する。以後の evaluation は、NGR repository に公開する固定 SHA の controlled corpus を source とし、D1 single-corpus experiment は capacity が増えるまで waiting とする。 | | [repository-native-controlled-corpus](https://github.com/Liplus-Project/neuron-graph-rag/wiki/repository-native-controlled-corpus) | active | repository-native controlled corpus v2 は、固定 SHA の公開 documentation と本文中の明示的な同一 directory 相対 link だけから、node、doc path、source URL、credited edge identity が相互に分離した development / holdout の各 3-edge path を導出する。v1 は provenance として保持する。これは controlled benchmark であり、外部 corpus への一般化、評価 query、gold、result、既定値変更を含まない。 | | [soft-start-feedback-reinforcement](https://github.com/Liplus-Project/neuron-graph-rag/wiki/soft-start-feedback-reinforcement) | active | 最初の credited `used` に通常 bounded updateの小さなprovisional fractionを適用し、最初の独立`confirmed`がremainder、後続confirmationがgeometric decayを適用する。v1 snapshot評価の不支持を保持し、baseline-aware successorはfresh initial evidenceからq3 first mutationを導出する。v2 freeze-only PRとsquash後のobserved registrationを分離し、development全gate通過時だけholdoutを一度開く。source database、live config、defaultを変更しない。 | +| [real-task-shadow-feedback-validation](https://github.com/Liplus-Project/neuron-graph-rag/wiki/real-task-shadow-feedback-validation) | active | confirmed-triggered reinforcement の次の実証は、客観的 outcome を事前登録した Codex 実タスクの non-serving shadow A/B とし、検索品質、誤強化率、Agent end-to-end 効果を default 採用前に測る。 | | [github-rag-mcp-replacement-compatibility](https://github.com/Liplus-Project/neuron-graph-rag/wiki/github-rag-mcp-replacement-compatibility) | active | public GitHub repository一つのread-only snapshotをNGR local indexへ接続する。github-rag-mcp `search` の保存済み raw capture と source URL、根拠を比較する。共有 source identity を確認しても最小 doc 検索 path の候補に限り、production github-rag-mcp、MCP authentication / transport、remote deployment、default変更は含まない。 | | [sqlite-canonical-judgment-graph](https://github.com/Liplus-Project/neuron-graph-rag/wiki/sqlite-canonical-judgment-graph) | active | NGR 自身の判断構造は SQLite の stable identity、revision、lifecycle、typed relation を machine-native 正本とし、raw SQL でなく atomic domain API で変更する。Wiki は移行 fixture の検証後に optional generated view へ下げる。 | diff --git a/docs/decision-wiki-pilot-manifest.json b/docs/decision-wiki-pilot-manifest.json new file mode 100644 index 0000000..8f40964 --- /dev/null +++ b/docs/decision-wiki-pilot-manifest.json @@ -0,0 +1,27 @@ +{ + "export_sha256": "14ac7dc1e22cf696cb865bf4077bd04ab8ddb106b06b48951c3edfbe00fd7c5c", + "format": "ngr-decision-wiki-pilot/v1", + "integrity": { + "dangling_relations": [], + "duplicate_successors": [], + "foreign_key_violations": [], + "sqlite_integrity": "ok", + "supersession_inconsistencies": [] + }, + "judgment_count": 86, + "lifecycle_counts": { + "active": 81, + "archived": 5 + }, + "relation_count": 85, + "repositories": { + "Liplus-Project/liplus-language": { + "commit": "d0c8143d4c126ee7d1e44d914eb87c7b7d771a2e", + "judgment_count": 72 + }, + "Liplus-Project/neuron-graph-rag": { + "commit": "6030b5621130b1b448c350b1a3b14d84fd4331b1", + "judgment_count": 14 + } + } +} diff --git a/docs/decision-wiki-pilot-migration.md b/docs/decision-wiki-pilot-migration.md new file mode 100644 index 0000000..9ca1b92 --- /dev/null +++ b/docs/decision-wiki-pilot-migration.md @@ -0,0 +1,28 @@ +# Decision Wiki pilot migration + +## Boundary + +This pilot imports only entries enumerated by each repository-owned `Decision-Structure.md` index. By default the tool reads the mirror in each Wiki clone; `--liplus-index` and `--ngr-index` select an explicit canonical index when the mirror is behind its repository. Unclassified Wiki pages are not discovered or imported. The Wiki clones, existing search databases, and frozen feedback experiment databases remain read-only. The pilot database is a disposable, dedicated SQLite file and does not switch the source-of-truth boundary. + +Each stable identity is `:`, so equal slugs in different repositories do not collide. Provenance retains the full page body, page SHA-256, repository, Wiki URL, source index state, and exact Wiki commit. The import accepts `supersedes`, `depends on`, `conflicts with`, `refines`, and `informs`; spaces are normalized to underscores in SQLite relation identifiers. + +Index states `active` and `evaluating` import as active. `archived` and `superseded` import as archived. A `supersedes` relation also archives its predecessor and records `superseded_by`. Duplicate identities, missing indexed pages, unknown Decision targets, multiple successors, ambiguous edge declarations, and partial publication fail closed. + +## Reproduction + +Run from the repository root with clean, independently acquired Wiki clones and paths that do not exist yet: + +```powershell +python tools/import_decision_wikis.py ` + --liplus-wiki C:\path\to\liplus-language.wiki ` + --ngr-wiki C:\path\to\neuron-graph-rag.wiki ` + --ngr-index C:\path\to\neuron-graph-rag\docs\Decision-Structure.md ` + --database C:\path\to\pilot\decisions.sqlite ` + --export C:\path\to\pilot\decisions.export.json ` + --backup C:\path\to\pilot\decisions.backup.sqlite ` + --manifest C:\path\to\pilot\decisions.manifest.json +``` + +The command refuses to overwrite any output. It builds all four outputs under temporary sibling names, imports all records in one transaction, checks SQLite and judgment supersession integrity, independently exports the database twice and compares the bytes, verifies the SQLite backup, and records source commits and counts in the manifest. Only after every validation succeeds are the database, export, backup, and manifest published as one bundle. Any build or publication failure removes every temporary and newly published output, leaving no partial set. + +The committed `decision-wiki-pilot-manifest.json` records the completed pilot without committing the SQLite binary, backup, export, or Wiki clones. diff --git a/docs/requirements.md b/docs/requirements.md index b8ab00f..20940af 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -105,6 +105,7 @@ 86. outcome-driven feedback deactivation はsoft-startと同時にだけ有効化できるdefault-off candidateとする。provisional / confirmationごとにcredited加算と同時発生したsame-source sibling normalization減算を一つのsigned mutation journalへ永続化し、因果帰属できる`corrected` / `rolled_back`だけが未反転contributionを基礎weight未満へ下げずexact reversalする。`superseded`はedge、evidence、trace、outcomeを削除せずrelation edgeをdormantにして通常activationから除外し、同じ保存済みcredited pathの後続`confirmed`で再活性化する。duplicate、retry、restart、transaction failure、lexical、zero-hop、別source、uncredited edge、因果帰属不能outcomeは二重減算または局所外mutationを行わない。 87. outcome-driven deactivation evaluation はcontrol / candidate、`corrected` / `rolled_back` / `superseded`、exact credited / sibling inverse、baseline floor、dormancy / reactivation、rank / locality、source isolation、exclusive outputを結果観測前に固定する。protocolはregistered output不在のfreeze-only PRで固定し、そのsquash merge後のsuccessor Issueでdevelopmentを一度だけ実行する。全hard gate通過時だけholdoutを一度開き、観測前後にquery、case、schedule、metric、gate、default、live configを変更しない。 88. NGR 自身の新規 judgment graph は SQLite の stable identity、revision、lifecycle、provenance、typed relation を machine-readable 正本とする。add / update / supersede / archive / restore / hard-delete candidate は raw SQL でなく atomic domain API を通し、stale revision、dangling relation、部分更新、二重 successor を fail closed にする。archive は通常 retrieval から外す論理的忘却、hard delete は履歴参照のない archived candidate だけに許す物理削除として分離する。current graph の deterministic export / atomic import と SQLite backup / integrity-checked restore を維持し、既存 Wiki entry の本番移行は fixture 検証後に分離する。 +89. Li+ / NGR Decision Structure Wiki pilot は各 repository の index が列挙する entry だけを専用の新規 SQLite へ取り込み、repository namespace 付き identity、page 本文、Wiki URL、repository、取得 commit、source state、typed relation を保持する。duplicate identity、unknown relation target、parser ambiguity、partial publication、既存出力の上書きを fail closed にし、SQLite / supersession integrity、deterministic export、backup を検証する。Wiki、既存検索 DB、凍結済み feedback 実験 DB は変更せず、本 pilot だけで正本を切り替えない。 75. v3 implementation、prompt、manifest、query override、schema、集約、path audit、hash規則、gate、stop rule、testsをresult-free commitでpushした後、development stage / 4 case packet / 12 responses / resultを各一度だけ生成する。 76. development全12 gate通過時だけholdout stageを一度生成し、異なるfresh 12 judgesで同じgateを評価する。packet、response、resultの上書き、観測後の規則変更、実LLM品質値のCI再生成を拒否する。 @@ -145,3 +146,4 @@ - [Outcome-driven feedback deactivation](outcome-driven-feedback-deactivation.md) がsigned contribution journal、exact reversal、dormancy / reactivation、default-off境界、result-free freezeを定義する。 - [Baseline-aware soft-start snapshot evaluation](baseline-aware-soft-start-snapshot-evaluation.md) がfresh baseline stateからのq3 boundary導出、v1 evidence isolation、capacity preflight、新規one-time result境界を定義する。 - [Canonical SQLite judgment graph](canonical-sqlite-judgment-graph.md) が judgment source-of-truth、domain write API、logical forgetting、hard-delete boundary、deterministic portability、backup / restore を定義する。 +- [Decision Wiki pilot migration](decision-wiki-pilot-migration.md) が Li+ / NGR Wiki import の対象境界、identity namespace、provenance、fail-closed 条件、再現手順を定義する。 diff --git a/src/neuron_graph_rag/decision_wiki_import.py b/src/neuron_graph_rag/decision_wiki_import.py new file mode 100644 index 0000000..7d58dd4 --- /dev/null +++ b/src/neuron_graph_rag/decision_wiki_import.py @@ -0,0 +1,256 @@ +from __future__ import annotations + +import hashlib +import json +import os +import re +import sqlite3 +import tempfile +from contextlib import closing +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Iterable +from urllib.parse import urlparse + +from .judgments import JudgmentContractError +from .storage import SQLiteStore + + +_SLUG = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$") +_TABLE_LINK = re.compile(r"^\s*\[\s*`?([^]`]+)`?\s*\]\((https://github\.com/([^/]+/[^/]+)/wiki/([^)#]+))\)\s*$") +_HEADING = re.compile(r"^##\s+(.+?)\s*$", re.MULTILINE) +_EDGE_LINE = re.compile( + r"^\s*[-*]\s+(?:\*\*(supersedes|depends on|conflicts with|refines|informs)(?:[^*]*)\*\*|(supersedes|depends on|conflicts with|refines|informs))\s*:?[ \t]*(.*?)\s*$", + re.IGNORECASE, +) +_WIKILINK = re.compile(r"\[\[([a-z0-9]+(?:-[a-z0-9]+)*)\]\]") +_MARKDOWN_LINK = re.compile(r"\[[^]]+\]\(([^)]+)\)") +_RELATION_TYPES = {"supersedes", "depends_on", "conflicts_with", "refines", "informs"} + + +class DecisionWikiImportError(JudgmentContractError): + """A fail-closed Decision Structure migration error.""" + + +@dataclass(frozen=True) +class WikiSource: + repository: str + clone: Path + index: Path + commit: str + + @property + def namespace(self) -> str: + return self.repository.rsplit("/", 1)[-1] + + +def _identity(repository: str, slug: str) -> str: + return f"{repository.rsplit('/', 1)[-1]}:{slug}" + + +def _section(text: str, names: Iterable[str]) -> str | None: + matches = list(_HEADING.finditer(text)) + wanted = {name.casefold() for name in names} + for position, match in enumerate(matches): + if match.group(1).strip().casefold() in wanted: + end = matches[position + 1].start() if position + 1 < len(matches) else len(text) + return text[match.end() : end].strip() + return None + + +def parse_index(source: WikiSource) -> list[dict[str, str]]: + text = source.index.read_text(encoding="utf-8") + rows: list[dict[str, str]] = [] + seen: set[str] = set() + for line in text.splitlines(): + if not line.lstrip().startswith("|"): + continue + cells = [cell.strip() for cell in line.strip().strip("|").split("|")] + if not cells: + continue + link = _TABLE_LINK.fullmatch(cells[0]) + if link is None: + continue + label, url, repository, slug = link.groups() + if repository.casefold() != source.repository.casefold() or label != slug: + raise DecisionWikiImportError(f"ambiguous index identity: {line}") + if _SLUG.fullmatch(slug) is None or slug in seen: + raise DecisionWikiImportError(f"invalid or duplicate index slug: {slug}") + seen.add(slug) + if len(cells) >= 3: + state, resolution = cells[1], cells[2] + elif len(cells) == 2: + state, resolution = "active", cells[1] + else: + raise DecisionWikiImportError(f"index row has no resolution: {slug}") + if not resolution or state not in {"active", "archived", "superseded", "evaluating"}: + raise DecisionWikiImportError(f"unsupported index state or empty resolution: {slug}") + rows.append({"slug": slug, "state": state, "resolution": resolution, "wiki_url": url}) + if not rows: + raise DecisionWikiImportError(f"no Decision Structure entries in {source.index}") + return rows + + +def _target_from_url(url: str, current_repository: str) -> tuple[str, str] | None: + parsed = urlparse(url) + if not parsed.scheme and not parsed.netloc: + slug = parsed.path.strip("/") + return (current_repository, slug) if _SLUG.fullmatch(slug) else None + parts = parsed.path.strip("/").split("/") + if parsed.netloc.casefold() == "github.com" and len(parts) == 4 and parts[2] == "wiki": + return f"{parts[0]}/{parts[1]}", parts[3] + return None + + +def parse_relations(page: str, repository: str) -> list[tuple[str, str, str]]: + edges = _section(page, ("Edges",)) + if edges is None: + return [] + relations: list[tuple[str, str, str]] = [] + for line in edges.splitlines(): + match = _EDGE_LINE.fullmatch(line) + if match is None: + if line.strip() and re.search(r"\b(supersedes|depends on|conflicts with|refines|informs)\b", line, re.I): + raise DecisionWikiImportError(f"ambiguous edge declaration: {line}") + continue + kind = (match.group(1) or match.group(2)).casefold().replace(" ", "_") + if kind not in _RELATION_TYPES: + raise DecisionWikiImportError(f"unsupported relation: {kind}") + body = match.group(3) + if not body or re.match(r"^(none|（.*なし.*）|\(.*none.*\))", body, re.I): + continue + targets = [(repository, slug) for slug in _WIKILINK.findall(body)] + for url in _MARKDOWN_LINK.findall(body): + target = _target_from_url(url, repository) + if target is not None: + targets.append(target) + targets = list(dict.fromkeys(targets)) + if len(targets) > 1: + raise DecisionWikiImportError(f"edge line has multiple targets: {line}") + if targets: + relations.append((kind, targets[0][0], targets[0][1])) + if len(relations) != len(set(relations)): + raise DecisionWikiImportError("duplicate relation") + return sorted(relations) + + +def build_payload(sources: Iterable[WikiSource]) -> tuple[dict[str, Any], dict[str, Any]]: + sources = tuple(sources) + records: list[dict[str, Any]] = [] + indexed: set[str] = set() + source_rows: list[tuple[WikiSource, dict[str, str]]] = [] + for source in sources: + for row in parse_index(source): + identity = _identity(source.repository, row["slug"]) + if identity in indexed: + raise DecisionWikiImportError(f"duplicate identity: {identity}") + indexed.add(identity) + source_rows.append((source, row)) + + incoming_supersession: dict[str, str] = {} + pending: list[tuple[str, str, str]] = [] + for source, row in source_rows: + page_path = source.clone / f'{row["slug"]}.md' + if not page_path.is_file(): + raise DecisionWikiImportError(f"indexed page is missing: {page_path}") + page_bytes = page_path.read_bytes() + try: + page = page_bytes.decode("utf-8") + except UnicodeDecodeError as error: + raise DecisionWikiImportError(f"page is not UTF-8: {page_path}") from error + source_id = _identity(source.repository, row["slug"]) + parsed = parse_relations(page, source.repository) + relations: list[dict[str, str]] = [] + for kind, target_repository, target_slug in parsed: + target_id = _identity(target_repository, target_slug) + if target_id not in indexed: + raise DecisionWikiImportError(f"unknown relation target: {source_id} -> {target_id}") + relations.append({"target_id": target_id, "relation_type": kind}) + pending.append((source_id, target_id, kind)) + if kind == "supersedes": + if target_id in incoming_supersession: + raise DecisionWikiImportError(f"multiple successors for {target_id}") + incoming_supersession[target_id] = source_id + statement = row["resolution"] or _section(page, ("Current resolution", "判断")) + if not statement: + raise DecisionWikiImportError(f"no resolution for {source_id}") + records.append( + { + "judgment_id": source_id, + "revision": 1, + "statement": statement, + "rationale": page, + "provenance": { + "page_body": page, + "page_sha256": hashlib.sha256(page_bytes).hexdigest(), + "repository": source.repository, + "source_state": row["state"], + "wiki_commit": source.commit, + "wiki_url": row["wiki_url"], + }, + "lifecycle": "archived" if row["state"] in {"archived", "superseded"} else "active", + "superseded_by": None, + "relations": relations, + } + ) + by_id = {record["judgment_id"]: record for record in records} + for record in records: + if record["provenance"]["source_state"] == "superseded" and record["judgment_id"] not in incoming_supersession: + raise DecisionWikiImportError(f"superseded entry has no successor: {record['judgment_id']}") + for predecessor, successor in incoming_supersession.items(): + record = by_id[predecessor] + if record["lifecycle"] == "active" and record["provenance"]["source_state"] != "active": + raise DecisionWikiImportError(f"supersession state conflict: {predecessor}") + record["lifecycle"] = "archived" + record["superseded_by"] = successor + records.sort(key=lambda item: item["judgment_id"]) + manifest = { + "format": "ngr-decision-wiki-pilot/v1", + "judgment_count": len(records), + "relation_count": len(pending), + "lifecycle_counts": { + state: sum(record["lifecycle"] == state for record in records) + for state in ("active", "archived") + }, + "repositories": { + source.repository: { + "commit": source.commit, + "judgment_count": sum( + record["provenance"]["repository"] == source.repository for record in records + ), + } + for source in sources + }, + } + return {"format": "ngr-judgment-graph/v1", "judgments": records}, manifest + + +def import_atomically(destination: Path, payload: dict[str, Any]) -> None: + if destination.exists(): + raise FileExistsError(f"refusing to overwrite database: {destination}") + destination.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{destination.name}.", suffix=".tmp", dir=destination.parent + ) + os.close(descriptor) + temporary = Path(temporary_name) + temporary.unlink() + try: + store = SQLiteStore(temporary) + try: + from .judgments import JudgmentGraph + + JudgmentGraph(store).import_graph(payload) + finally: + store.close() + with closing(sqlite3.connect(temporary)) as connection: + if connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": + raise DecisionWikiImportError("SQLite integrity check failed") + os.replace(temporary, destination) + finally: + if temporary.exists(): + temporary.unlink() + + +def deterministic_json(payload: object) -> bytes: + return (json.dumps(payload, ensure_ascii=False, sort_keys=True, indent=2) + "\n").encode("utf-8") diff --git a/tests/test_decision_wiki_import.py b/tests/test_decision_wiki_import.py new file mode 100644 index 0000000..f7c22be --- /dev/null +++ b/tests/test_decision_wiki_import.py @@ -0,0 +1,155 @@ +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +from neuron_graph_rag import NeuronGraphRAG +from neuron_graph_rag.decision_wiki_import import ( + DecisionWikiImportError, + WikiSource, + build_payload, + deterministic_json, + import_atomically, +) +from tools.import_decision_wikis import publish_bundle + + +def _source(root: Path, repository: str, rows: str, pages: dict[str, str]) -> WikiSource: + clone = root / repository.rsplit("/", 1)[-1] + clone.mkdir() + index = clone / "Decision-Structure.md" + index.write_text( + "# Decision Structure\n\n| Node | State | Current resolution |\n" + "| --- | --- | --- |\n" + rows, + encoding="utf-8", + ) + for slug, page in pages.items(): + (clone / f"{slug}.md").write_text(page, encoding="utf-8") + return WikiSource(repository, clone, index, "a" * 40) + + +class DecisionWikiImportTests(unittest.TestCase): + def test_namespace_relations_lifecycle_and_provenance(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + first = _source( + root, "Example/one", + "| [same](https://github.com/Example/one/wiki/same) | archived | Old |\n", + {"same": "## Current resolution\n\nOld\n\n## Edges\n\n- **depends on** none\n"}, + ) + second = _source( + root, "Example/two", + "| [same](https://github.com/Example/two/wiki/same) | active | New |\n" + "| [next](https://github.com/Example/two/wiki/next) | active | Next |\n", + { + "same": "## Current resolution\n\nNew\n\n## Edges\n\n- **informs** [next](next)\n", + "next": "## Current resolution\n\nNext\n\n## Edges\n\n- **supersedes** [same](same)\n", + }, + ) + payload, manifest = build_payload((first, second)) + by_id = {item["judgment_id"]: item for item in payload["judgments"]} + self.assertEqual(set(by_id), {"one:same", "two:same", "two:next"}) + self.assertEqual(by_id["two:same"]["superseded_by"], "two:next") + self.assertEqual(by_id["two:same"]["lifecycle"], "archived") + self.assertEqual(by_id["two:next"]["relations"][0]["relation_type"], "supersedes") + self.assertEqual(by_id["one:same"]["provenance"]["wiki_commit"], "a" * 40) + self.assertEqual(manifest["judgment_count"], 3) + self.assertEqual(manifest["relation_count"], 2) + + def test_unknown_relation_target_fails_closed(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source = _source( + Path(directory), "Example/one", + "| [source](https://github.com/Example/one/wiki/source) | active | Source |\n", + {"source": "## Edges\n\n- **depends on** [missing](missing)\n"}, + ) + with self.assertRaisesRegex(DecisionWikiImportError, "unknown relation target"): + build_payload((source,)) + + def test_duplicate_index_identity_and_ambiguous_edge_fail(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = _source( + root, "Example/one", + "| [same](https://github.com/Example/one/wiki/same) | active | Same |\n" + "| [same](https://github.com/Example/one/wiki/same) | active | Same again |\n", + {"same": "## Edges\n\n- **depends on** none\n"}, + ) + with self.assertRaisesRegex(DecisionWikiImportError, "duplicate"): + build_payload((source,)) + source.index.write_text( + "| [same](https://github.com/Example/one/wiki/same) | active | Same |\n" + "| [other](https://github.com/Example/one/wiki/other) | active | Other |\n", + encoding="utf-8", + ) + (source.clone / "same.md").write_text( + "## Edges\n\n- **depends on** [other](other) and [same](same)\n", encoding="utf-8" + ) + (source.clone / "other.md").write_text("## Edges\n\n- **informs** none\n", encoding="utf-8") + with self.assertRaisesRegex(DecisionWikiImportError, "multiple targets"): + build_payload((source,)) + + def test_atomic_import_refuses_overwrite_and_is_deterministic(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + payload = { + "format": "ngr-judgment-graph/v1", + "judgments": [{ + "judgment_id": "one:a", "revision": 1, "statement": "A", + "rationale": "Reason", "provenance": {"source": "wiki"}, + "lifecycle": "active", "superseded_by": None, "relations": [], + }], + } + destination = root / "pilot.sqlite" + import_atomically(destination, payload) + with NeuronGraphRAG(destination) as engine: + first = deterministic_json(engine.judgments.export()) + second = deterministic_json(engine.judgments.export()) + self.assertEqual(first, second) + before = destination.read_bytes() + with self.assertRaises(FileExistsError): + import_atomically(destination, payload) + self.assertEqual(destination.read_bytes(), before) + broken = root / "broken.sqlite" + invalid = {"format": "ngr-judgment-graph/v1", "judgments": [{"judgment_id": "bad"}]} + with self.assertRaises(Exception): + import_atomically(broken, invalid) + self.assertFalse(broken.exists()) + + def test_bundle_failure_after_export_leaves_no_outputs(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = _source( + root, "Example/one", + "| [source](https://github.com/Example/one/wiki/source) | active | Source |\n", + {"source": "## Current resolution\n\nSource\n\n## Edges\n\n- **informs** none\n"}, + ) + outputs = ( + root / "decisions.sqlite", + root / "decisions.export.json", + root / "decisions.backup.sqlite", + root / "decisions.manifest.json", + ) + + for failure_stage in ("after_export", "published:decisions.sqlite"): + with self.subTest(failure_stage=failure_stage): + def fail(stage: str) -> None: + if stage == failure_stage: + raise RuntimeError("injected bundle failure") + + with self.assertRaisesRegex(RuntimeError, "injected"): + publish_bundle((source,), *outputs, failure_hook=fail) + self.assertEqual([path for path in outputs if path.exists()], []) + self.assertEqual(list(root.glob(".*.tmp")), []) + + publish_bundle((source,), *outputs) + self.assertTrue(all(path.exists() for path in outputs)) + manifest = json.loads(outputs[3].read_text(encoding="utf-8")) + self.assertEqual(manifest["judgment_count"], 1) + self.assertEqual(manifest["integrity"]["sqlite_integrity"], "ok") + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/import_decision_wikis.py b/tools/import_decision_wikis.py new file mode 100644 index 0000000..dc4b9b6 --- /dev/null +++ b/tools/import_decision_wikis.py @@ -0,0 +1,136 @@ +from __future__ import annotations + +import argparse +import hashlib +import os +import sqlite3 +import tempfile +from contextlib import closing +from pathlib import Path +from typing import Callable, Iterable + +from neuron_graph_rag import NeuronGraphRAG +from neuron_graph_rag.decision_wiki_import import ( + WikiSource, + build_payload, + deterministic_json, + import_atomically, +) +try: + from tools.judgment_graph import backup, integrity +except ModuleNotFoundError: # Direct script execution adds tools/ rather than the repository root. + from judgment_graph import backup, integrity + + +def _commit(clone: Path) -> str: + import subprocess + + return subprocess.run( + ["git", "-C", str(clone), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + encoding="utf-8", + ).stdout.strip() + + +def _temporary_sibling(path: Path) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, name = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent) + os.close(descriptor) + temporary = Path(name) + temporary.unlink() + return temporary + + +def _publish_exclusive(temporary: Path, destination: Path) -> None: + os.link(temporary, destination) + temporary.unlink() + + +def publish_bundle( + sources: Iterable[WikiSource], + database: Path, + export_path: Path, + backup_path: Path, + manifest_path: Path, + *, + failure_hook: Callable[[str], None] | None = None, +) -> None: + outputs = (database, export_path, backup_path, manifest_path) + existing = [str(path) for path in outputs if path.exists()] + if existing: + raise FileExistsError("refusing to overwrite existing output: " + ", ".join(existing)) + temporary = {path: _temporary_sibling(path) for path in outputs} + published: list[Path] = [] + try: + payload, manifest = build_payload(sources) + import_atomically(temporary[database], payload) + check = integrity(temporary[database]) + if failure_hook: + failure_hook("after_database") + + with NeuronGraphRAG(temporary[database]) as engine: + first_export = deterministic_json(engine.judgments.export()) + second_export = deterministic_json(engine.judgments.export()) + if first_export != second_export: + raise RuntimeError("deterministic export verification failed") + temporary[export_path].write_bytes(first_export) + if failure_hook: + failure_hook("after_export") + + backup(temporary[database], temporary[backup_path]) + with closing(sqlite3.connect(temporary[backup_path])) as connection: + if connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": + raise RuntimeError("backup integrity check failed") + if failure_hook: + failure_hook("after_backup") + + manifest["integrity"] = check + manifest["export_sha256"] = hashlib.sha256(first_export).hexdigest() + temporary[manifest_path].write_bytes(deterministic_json(manifest)) + if failure_hook: + failure_hook("before_publish") + + for destination in outputs: + _publish_exclusive(temporary[destination], destination) + published.append(destination) + if failure_hook: + failure_hook(f"published:{destination.name}") + except BaseException: + for path in published: + if path.exists(): + path.unlink() + raise + finally: + for path in temporary.values(): + if path.exists(): + path.unlink() + + +def main() -> None: + parser = argparse.ArgumentParser(description="Import indexed Decision Structure Wiki entries") + parser.add_argument("--liplus-wiki", type=Path, required=True) + parser.add_argument("--ngr-wiki", type=Path, required=True) + parser.add_argument("--liplus-index", type=Path) + parser.add_argument("--ngr-index", type=Path) + parser.add_argument("--database", type=Path, required=True) + parser.add_argument("--export", dest="export_path", type=Path, required=True) + parser.add_argument("--backup", dest="backup_path", type=Path, required=True) + parser.add_argument("--manifest", type=Path, required=True) + arguments = parser.parse_args() + sources = ( + WikiSource("Liplus-Project/liplus-language", arguments.liplus_wiki, arguments.liplus_index or arguments.liplus_wiki / "Decision-Structure.md", _commit(arguments.liplus_wiki)), + WikiSource("Liplus-Project/neuron-graph-rag", arguments.ngr_wiki, arguments.ngr_index or arguments.ngr_wiki / "Decision-Structure.md", _commit(arguments.ngr_wiki)), + ) + try: + publish_bundle( + sources, arguments.database, arguments.export_path, + arguments.backup_path, arguments.manifest, + ) + except FileExistsError as error: + parser.error(str(error)) + + +if __name__ == "__main__": + main()