Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f81bdd5
chore: delete the manual rebuild engine and the daemon bulk-rebuild r…
Sinity Sep 5, 2026
01041be
chore: delete the generic repair product; raw convergence keeps one o…
Sinity Sep 5, 2026
bdb4458
chore: retire the storage/index FTS shim and rewrite maintenance docs…
Sinity Sep 5, 2026
93570e8
chore: delete the declared-not-routed candidate-build operation contract
Sinity Sep 5, 2026
5a6eed0
wip: checkpoint of interrupted lane repair-engine-delete
Sinity Sep 5, 2026
7b19c4b
test: retarget surviving proofs at their new owners
Sinity Sep 5, 2026
a1ac09e
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 5, 2026
0bd70ef
chore: close the retired surfaces after the master merge
Sinity Sep 5, 2026
e1f820f
test: drop the proofs the deleted engines owned
Sinity Sep 5, 2026
bde2e4e
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 5, 2026
84adc4e
test: resolve the maintenance Literal and drop the retired failure count
Sinity Sep 5, 2026
253f99c
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 5, 2026
d603062
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 5, 2026
acf07df
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 5, 2026
f09887f
test: point the daemon proofs at converge_materialization
Sinity Sep 5, 2026
b645eaf
test: resolve a real config behind the daemon config seam
Sinity Sep 5, 2026
1a33b6b
docs: name the surviving materialization entry point
Sinity Sep 5, 2026
5ddcbed
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 5, 2026
c576801
Merge remote-tracking branch 'origin/master' into lane/repair-engine-…
Sinity Sep 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion devtools/command_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ def to_dict(self) -> dict[str, object]:
examples=(
"devtools scenario list",
"devtools scenario run archive-smoke --tier 0",
"devtools scenario run rebuild-safety --report-dir .cache/rebuild-safety-report --json",
"devtools scenario run storage-correctness --report-dir .cache/storage-correctness-report --json",
),
),
CommandSpec(
Expand Down
685 changes: 0 additions & 685 deletions devtools/rebuild_safety_scenario.py

This file was deleted.

90 changes: 0 additions & 90 deletions devtools/verification_scenario.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,12 @@
import subprocess
import sys
import time
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol, TextIO

from devtools import repo_root as _get_root
from devtools.cli_boundary import invoke_polylogue_cli
from devtools.rebuild_safety_scenario import (
REBUILD_DIFFERENTIAL_SCENARIO_NAME,
REBUILD_SAFETY_SCENARIO_NAME,
RebuildComparisonResult,
run_rebuild_differential,
run_rebuild_safety,
)
from devtools.storage_correctness_scenario import (
STORAGE_CORRECTNESS_SCENARIO_NAME,
run_storage_correctness,
Expand All @@ -35,7 +27,6 @@
"archive-smoke",
"reader-visual-smoke",
STORAGE_CORRECTNESS_SCENARIO_NAME,
REBUILD_SAFETY_SCENARIO_NAME,
)
Comment on lines 27 to 30

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the scenario inventory test after removing rebuild safety

When tests/unit/devtools/test_verification_scenario.py::test_list_scenarios_reports_live_paths_without_baseline_counts runs, it still evaluates next(...) for an entry named rebuild-safety and asserts that scenario's old payload. Removing it from _SCENARIO_NAMES and list_scenarios() therefore raises StopIteration, so the managed suite fails; remove or replace that expectation with a retained scenario.

Useful? React with 👍 / 👎.

_ARCHIVE_SMOKE_TIER = 0
_READER_VISUAL_SMOKE_PYTEST_ARGS: tuple[str, ...] = ("-m", "pytest", "-q", "tests/visual")
Expand Down Expand Up @@ -128,77 +119,6 @@ def failed_stages(self) -> tuple[str, ...]:
return tuple(name for name, status in self.stage_statuses().items() if status is OutcomeStatus.ERROR)


class RebuildSafetyResult:
"""Direct result wrapper for the derived-tier rebuild verification lane."""

def __init__(self, *, report_dir: Path | None) -> None:
self.report_dir = report_dir
self.safety, self.safety_error = self._run("rebuild-safety", run_rebuild_safety)
self.differential, self.differential_error = self._run("rebuild-differential", run_rebuild_differential)
self._write_report()

@staticmethod
def _run(
name: str, runner: Callable[[], RebuildComparisonResult]
) -> tuple[RebuildComparisonResult | None, str | None]:
try:
return runner(), None
except Exception as exc:
return None, f"{name} failed: {type(exc).__name__}: {exc}"

@staticmethod
def _report(value: RebuildComparisonResult | None, error: str | None) -> str:
if error is not None:
return error
assert value is not None
return value.format_report()

def _write_report(self) -> None:
if self.report_dir is None:
return
self.report_dir.mkdir(parents=True, exist_ok=True)
(self.report_dir / "rebuild-safety.txt").write_text(
f"{self._report(self.safety, self.safety_error)}\n\n"
f"{self._report(self.differential, self.differential_error)}\n",
encoding="utf-8",
)

@property
def scenario_name(self) -> str:
return REBUILD_SAFETY_SCENARIO_NAME

@property
def all_passed(self) -> bool:
return (
self.safety_error is None
and self.differential_error is None
and self.safety is not None
and self.differential is not None
and self.safety.all_passed
and self.differential.all_passed
)

def stage_statuses(self) -> dict[str, OutcomeStatus]:
return {
REBUILD_SAFETY_SCENARIO_NAME: OutcomeStatus.OK
if self.safety_error is None and self.safety is not None and self.safety.all_passed
else OutcomeStatus.ERROR,
REBUILD_DIFFERENTIAL_SCENARIO_NAME: OutcomeStatus.OK
if self.differential_error is None and self.differential is not None and self.differential.all_passed
else OutcomeStatus.ERROR,
}

def failed_stages(self) -> tuple[str, ...]:
return tuple(name for name, status in self.stage_statuses().items() if status is OutcomeStatus.ERROR)

def extra_payload(self) -> dict[str, object]:
payload: dict[str, object] = {
"safety_report": self._report(self.safety, self.safety_error),
"differential_report": self._report(self.differential, self.differential_error),
}
return payload


def get_archive_smoke_checks() -> tuple[ArchiveSmokeCheck, ...]:
"""Return direct CLI checks for the archive-smoke scenario."""
return _ARCHIVE_SMOKE_CHECKS
Expand Down Expand Up @@ -253,11 +173,6 @@ def list_scenarios(*, as_json: bool) -> int:
"command": " ".join((sys.executable, *_READER_VISUAL_SMOKE_PYTEST_ARGS)),
},
storage_correctness_scenario_entry(),
{
"name": REBUILD_SAFETY_SCENARIO_NAME,
"kind": "derived-tier-differential",
"checks": [REBUILD_SAFETY_SCENARIO_NAME, REBUILD_DIFFERENTIAL_SCENARIO_NAME],
},
]
payload = {"scenarios": scenarios}
if as_json:
Expand All @@ -271,9 +186,6 @@ def list_scenarios(*, as_json: bool) -> int:
if name == "storage-correctness":
print(f"{name:<20s} checks: {entry['check_count']}")
continue
if name == REBUILD_SAFETY_SCENARIO_NAME:
print(f"{name:<20s} checks: rebuild safety + differential")
continue
print(f"{name:<20s} tier-0 checks: {entry['tier_0_check_count']}")
return 0

Expand Down Expand Up @@ -479,8 +391,6 @@ def main(argv: list[str] | None = None) -> int:
result: _ScenarioResult
if args.scenario == "storage-correctness":
result = run_storage_correctness(report_dir=args.report_dir)
elif args.scenario == REBUILD_SAFETY_SCENARIO_NAME:
result = RebuildSafetyResult(report_dir=args.report_dir)
else:
result = run_archive_smoke(
tier=args.tier,
Expand Down
15 changes: 1 addition & 14 deletions docs/architecture-hotspots.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,27 +25,14 @@ won't hurt."
| - | --- | --- | --- | --- | --- |
| 1 | Storage read tier | `storage/sqlite/archive_tiers/archive.py` | 11,382 → **11,324** lines (raw-revision/membership governance extracted to `archive_tiers/revision_governance.py`, 2,841 lines, polylogue-1r9c) | `ArchiveStore` — every SELECT-shaped query surface (sessions, messages, blocks, insights reads, search) | `storage/` |
| 2 | API facade | `api/archive.py` | 5,881 lines | `Polylogue.repository`/`.backend` verb surface consumed by CLI/MCP/daemon | `api/` (surface) |
| 3 | Storage repair | `storage/repair.py` | 5,558 lines | `repair_*`/`preview_*`/`run_safe_repairs`/`collect_archive_debt_statuses_sync` — every integrity-repair entrypoint the CLI `check`/`repair` commands and daemon convergence call | `storage/` (maintenance-adjacent, see note below) |
| 3 | Raw convergence | `storage/raw_convergence.py` | 5,558 lines at census time | `converge_raw_materialization` and the raw-authority frontier strategies the daemon's raw drain executes | `storage/` |
| 4 | Daemon HTTP | `daemon/http.py` | 4,609 lines | the daemon's REST/web-shell route table | `daemon/` (surface) |
| 5 | Storage write tier | `storage/sqlite/archive_tiers/write.py` | 4,595 → **4,210** lines (this bead's slice 1 landed) | `write_parsed_session_to_archive` + session/message/block/tag/work-event/phase writers | `storage/` |
| 6 | CLI query dispatch | `cli/archive_query.py`, fn `_execute_archive_query_stdout` | 2,488 lines file / 632-line function (174-805) | the `find`/`read`/`analyze` query-mode stdout path | `cli/` (surface) |
| 7 | Daemon service loop | `daemon/cli.py`, fn `run_daemon_services` | 1,936 lines file / 475-line function (1007-1481) | `polylogued run` — daemon startup and the ~10 concurrent maintenance loops (see `polylogue-9e5.7`'s lock/starvation map) | `daemon/` |
| 8a | ~~MCP read tools~~ **resolved** | ~~`mcp/server_tools.py`, fn `register_read_tools`~~ | 1,286 lines file / 490-line function → **0** (function and its file's other per-tool registrars deleted; `server_tools.py` is now a 19-line passthrough into `mcp/server_cutover.py`, polylogue-t46.8) | was every read-only MCP tool registration; replaced by the six-tool `query`/`read`/`get`/`explain`/`context`/`status` algebra | `mcp/` (surface) |
| 8b | ~~MCP mutation tools~~ **resolved** | ~~`mcp/server_mutation_tools.py`, fn `register_mutation_tools`~~ | 497 lines → **0** (file deleted; logic re-hosted as `_dispatch_write` in `mcp/server_cutover.py`, polylogue-t46.8.3) | was every write MCP tool registration; replaced by the single capability-gated `write(operation=, ...)` transaction | `mcp/` (surface) |

Note on #3: `storage/repair.py`'s placement is itself a case study for the
polylogue-c9y placement doctrine — under the new rule-5 test ("integrity
repair — detecting and fixing rows that violate an invariant the write path
should have prevented") this is squarely `maintenance/` territory by
function, but it lives under `storage/` today because it also owns
low-level SQL the `maintenance/` package doesn't otherwise touch (receipt
files, WAL journal-mode manipulation, quarantine census staging). A future
slice should decide: either `maintenance/` absorbs the orchestration layer
and calls into `storage/` for the SQL primitives (matches the doctrine), or
the doctrine gets a documented exception for repair modules that are
SQL-heavy enough to need `storage/`'s proximity. Not decided in this pass —
flagged as an open question for whichever child bead executes #3's slice.

## Call boundaries and ownership seams

- **#1 (read tier) ← everything reads through it.** `#2` (API facade), the
Expand Down
2 changes: 1 addition & 1 deletion docs/atlas/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The live MCP surface is a twelve-tool operation algebra. Six read tools are alwa
| `emit_decision` | `write` | Append a decision event with evidence references (`polylogue/mcp/declarations/registry.py:181-194`) |
| `judge` | `judge` | Decide assertion candidates (`polylogue/mcp/declarations/registry.py:195-208`) |
| `run` | `write` | Execute saved query or recipe refs (`polylogue/mcp/declarations/registry.py:209-222`) |
| `maintenance` | `maintenance` | Preview, execute, inspect, and rebuild (`polylogue/mcp/declarations/registry.py:223-238`) |
| `maintenance` | `maintenance` | Rebuild derived indexes and inspect or adjudicate operation recovery (`polylogue/mcp/declarations/registry.py:223-238`) |

`write`, `judge`, and `maintenance` are independent booleans, not a role ladder. `run` shares the `write` gate (`polylogue/mcp/declarations/models.py:16-40`; `tests/unit/mcp/test_tool_declarations.py:26-41`).

Expand Down
14 changes: 2 additions & 12 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -571,22 +571,12 @@ Commands:
```text
Usage: polylogue ops doctor [OPTIONS]

Health check with optional maintenance and cleanup previews.
Read-only health check over the archive, runtime, daemon, blobs, and
schemas.

Options:
-f, --format [json] Output format
-v, --verbose Show breakdown by origin
--repair Run safe derived-data maintenance repairs
--cleanup Run destructive archive cleanup for orphaned
or empty persisted data
--target [session_insights|empty_sessions|superseded_raw_snapshots]
Limit maintenance to named targets such as
session_insights, empty_sessions, or
superseded_raw_snapshots
--preview Preview maintenance without executing
(requires --repair or --cleanup)
--vacuum Reclaim unused space after maintenance
(requires --repair or --cleanup)
--deep Run SQLite integrity and expensive orphan
scans (slow on large databases)
--runtime Run environment and runtime verification
Expand Down
15 changes: 8 additions & 7 deletions docs/code-navigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ landmarks in [Internals](internals.md).
The governing rule is simple:

> Put a change in the layer that owns its meaning, then adapt outward. Do not
> start from the CLI, daemon, or a repair command and work inward.
> start from the CLI, daemon, or a maintenance verb and work inward.

## Five-minute mental model

Expand Down Expand Up @@ -39,8 +39,9 @@ CLI / API / MCP / HTTP / rendering surfaces

`polylogued` owns normal writes. `source.db`, `user.db`, and source blob bytes
are durable evidence. `index.db`, `embeddings.db`, insights, FTS, and most
status products are rebuildable. Maintenance code may verify or repair an
invariant, but it is never the normal home for new archive semantics.
status products are rebuildable and converge through the daemon. Maintenance
code verifies invariants or recovers durable evidence; it is never the normal
home for new archive semantics.

## Read the code in this order

Expand All @@ -63,7 +64,7 @@ entire package tree:
declared multi-surface operations.
8. [`polylogue/daemon/convergence.py`](../polylogue/daemon/convergence.py) and
[`convergence_stages.py`](../polylogue/daemon/convergence_stages.py) —
bounded repair of rebuildable products after ingest.
bounded convergence of rebuildable products after ingest.
9. [`polylogue/surfaces/payloads.py`](../polylogue/surfaces/payloads.py) —
provider-neutral response payloads shared by public surfaces.
10. [`docs/plans/layering.yaml`](plans/layering.yaml) — enforced import and
Expand Down Expand Up @@ -144,8 +145,8 @@ enforced boundary authority.

### Verification worlds

- `maintenance/` — fail-closed verification and operator-supervised repair over
typed storage primitives; never the primary write path.
- `maintenance/` — fail-closed verification and guarded recovery over typed
storage primitives; never the primary write path.
- `schemas/` — provider schema observation, inference, validation, and drift.
- `scenarios/` — reusable scenario declarations and executable workload worlds.
- `demo/` — deterministic private-data-free product demonstrations.
Expand All @@ -172,7 +173,7 @@ registry or declaration, not the rendered output.

- **Raw SQL outside `storage/`.** Add or call a storage accessor instead.
- **Normal semantics in `maintenance/`.** Fix the write path; keep maintenance
for diagnosis, one-shot repair, and recovery.
for diagnosis and recovery.
- **Inferring `Provider` from `Origin`.** The mapping is not injective. Preserve
original acquisition evidence at wire boundaries.
- **Surface-specific copies of domain policy.** Put the rule in `archive/`,
Expand Down
8 changes: 3 additions & 5 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ A few keys not shown in the full example above, with their TOML path:
| `ingest_commit_batch_messages` | `sources.ingest_commit_batch_messages` | Messages per commit batch during ingest (default 8000). |
| `ingest_parse_workers` | `POLYLOGUE_INGEST_PARSE_WORKERS` (env only) | Worker count for CPU-bound source parsing. Read from the environment; there is no TOML key. The default adapts to the interpreter — `min(16, cpus-2)` on a free-threaded build (the packaged daemon), `min(8, cpus-1)` under the GIL. Set to `1` to disable pooling. |
| `live_full_ingest_workers` | `sources.live_full_ingest_workers` | Parallel workers for a live full-reingest pass (default 1). |
| `raw_authority_commit_batch_size` | `pipeline.raw_authority.commit_batch_size` | Census-phase commit batch size for raw-materialization repair (polylogue-amg1); unset uses the built-in default, `<=0` disables batching (per-raw commits). |
| `raw_authority_commit_batch_size` | `pipeline.raw_authority.commit_batch_size` | Census-phase commit batch size for raw materialization; unset uses the built-in default, `<=0` disables batching (per-raw commits). |
| `raw_authority_whale_payload_bytes` | `pipeline.raw_authority.whale_payload_bytes` | Escalation-tier payload envelope (bytes) for the daemon whale pass (polylogue-t93b); unset/`<=0` uses the built-in default (8 GiB). Widens the resource-block envelope for one dedicated, stream-safe-gated single-component pass only -- the ordinary fast-path envelope is unaffected. |
| `daemon_parse_stage_workers` | `daemon.raw_materialization.parse_stage_workers` | Worker cap for the daemon-owned pre-parse thread pool (polylogue-m6tp phase (a); always runs -- pre-parses raw-materialization census candidates in a bounded thread pool before the writer hold); unset/`<=0` uses the adaptive `cpu_count - 1` default. |
| `daemon_parse_stage_max_inflight_bytes` | `daemon.raw_materialization.parse_stage_max_inflight_bytes` | Whale-memory budget (bytes) for raw payloads admitted while prefetch parses are in flight; unset/`<=0` uses the adaptive 1/16-physical-RAM default (clamped [64 MiB, 2 GiB]). |
Expand Down Expand Up @@ -400,7 +400,7 @@ Common runtime overrides:
| `POLYLOGUE_CREDENTIAL_PATH` | Drive auth | OAuth client JSON path. |
| `POLYLOGUE_TOKEN_PATH` | Drive auth | OAuth token path. |
| `POLYLOGUE_HOOK_PROVIDER` | `hook_provider` | Force hook-harness detection to `claude-code`/`codex`. |
| `POLYLOGUE_RAW_AUTHORITY_COMMIT_BATCH_SIZE` | `raw_authority_commit_batch_size` | Census-phase commit batch size for raw-materialization repair. |
| `POLYLOGUE_RAW_AUTHORITY_COMMIT_BATCH_SIZE` | `raw_authority_commit_batch_size` | Census-phase commit batch size for raw materialization. |
| `POLYLOGUE_RAW_AUTHORITY_WHALE_PAYLOAD_BYTES` | `raw_authority_whale_payload_bytes` | Escalation-tier payload envelope for the daemon whale pass. |
| `POLYLOGUE_DAEMON_PARSE_STAGE_WORKERS` | `daemon_parse_stage_workers` | Worker cap for the daemon-owned pre-parse thread pool. |
| `POLYLOGUE_DAEMON_PARSE_STAGE_MAX_INFLIGHT_BYTES` | `daemon_parse_stage_max_inflight_bytes` | In-flight raw-payload budget for the prefetch cache. |
Expand Down Expand Up @@ -482,9 +482,7 @@ output.
### Health Checks

- `polylogue ops doctor` validates config, archive root, DB reachability, index status, and Drive credential/token presence.
- `polylogue ops doctor --repair` runs safe derived-data maintenance.
- `polylogue ops doctor --cleanup` runs destructive archive cleanup; preview it first.
- `polylogue ops doctor --repair --vacuum` compacts the database after maintenance.
- `polylogue ops doctor --deep` adds SQLite integrity and exact orphan scans.
- Workstation-specific policy such as cgroup slice placement and hard caps belongs in the host environment, not in the product CLI.

---
Expand Down
Loading