diff --git a/demo/README.md b/demo/README.md index a109cd182b..409236d5ef 100644 --- a/demo/README.md +++ b/demo/README.md @@ -10,21 +10,15 @@ This directory holds self-contained demos that are kept runnable from the repo checkout for reference and experimentation: -- `auto_research/` — the auto-research worker/supervisor showcase (kernels, - worker loop, evidence packets, terminal-result query, demo end-to-end). -- `multi_agent/` — the companion multi-agent launcher showcase (contract, - round ledger, role successor, visible launch policy, wake scheduler). -- `visible_multi_agent_launcher.py` / `visible_multi_agent_tmux.py` — the - tmux-based visible multi-agent launcher used by the demo. +- [`workspace/`](workspace/README.md) — local, repeatable workspace stories + backed by isolated LoopX state. The demos depend on the real LoopX product modules (`loopx.quota`, `loopx.todos`, `loopx.status`, …) and are run by importing the `demo` package from the repo root (examples and smokes already add the repo root to `sys.path`). -These were relocated from the product capability surface -(`loopx/capabilities/auto_research` and `loopx/control_plane/agents/multi_agent`) -to make the demo/non-product boundary explicit. If a piece here matures into a -stable caller contract, it should be promoted back into -`loopx/capabilities//` with a real entrypoint and focused -validation. +The retired tmux multi-agent and auto-research prototypes are available in Git +history. For current collaboration and research examples, see +[`collaboration-delivery`](../examples/collaboration-delivery/README.md) and +[`managed-research-team`](../examples/managed-research-team/README.md). diff --git a/demo/auto_research/README.md b/demo/auto_research/README.md deleted file mode 100644 index 126c8b7650..0000000000 --- a/demo/auto_research/README.md +++ /dev/null @@ -1,493 +0,0 @@ -# Auto-Research Command Path - -This guide is the shortest operator path for running the LoopX auto-research -demo from a clean workspace. It explains what to run, which visible digital -employees appear, what artifacts to inspect, and how to stop or take over. - -Use the deeper showcase and protocol docs only after this path is clear: - -- [Multi-agent product recipe](../../docs/guides/multi-agent-product-recipe.md) -- [Stop, takeover, and state-aware wake walkthrough](../../docs/guides/auto-research-stop-takeover-wake-walkthrough.md) -- [Decentralized auto-research showcase](../../docs/product/use-cases/auto-research/decentralized-auto-research-showcase.md) -- [auto_research_role_state_machine_v0](../../docs/reference/protocols/auto-research-role-state-machine-v0.md) -- [auto_research_role_profile_v0](../../docs/reference/protocols/auto-research-role-profile-v0.md) - -Implementation boundary: auto-research is a thin preset over the generic -multi-agent kernel. `demo/auto_research/preset.py` owns only the -research roles, handoff hints, metric/evidence loop defaults, and seed todo -phrasing. The generic kernel owns the real Codex TUI panes, pane-local A2A -tick, workspace/trust-safe launch, todo/evidence/status protocol, and compact -human status. The developer-facing recipe proof also follows that boundary: -`preset.py` calls the generic `multi_agent.recipe` helper instead of defining -decentralized A2A proof mechanics itself. -Use the [multi-agent product recipe](../../docs/guides/multi-agent-product-recipe.md) when a new -product wants to copy the pattern without copying auto-research code. - -## Promotion Decision - -The validated visible proof is promoted into this existing command path, not -into a second auto-research runner. The public recipe remains: - -- the operator runs one command; -- the user layer supplies only topic, objective, rounds, reasoning effort, and - optional role overrides; -- the auto-research preset supplies research roles, handoff hints, seed todos, - and evidence defaults; -- the generic multi-agent kernel supplies runner, wake, pane-local tick, status, - attach, retry, and stop mechanics. - -A visible proof is ready for this path only when the configured roles open as -real interactive Codex CLI panes, the fixed-prompt wake causes each pane to run -its local A2A tick, role output is summarized through LoopX todo/evidence/status -artifacts, and any live evidence packet is compact and public-safe. The proof -must not depend on raw logs, private artifacts, credentials, local absolute -paths, or a product-specific launcher hidden inside the auto-research preset. - -## User Contract Entrypoint - -The smallest user-facing auto-research contract is one open question: - -```bash -loopx auto-research "" -``` - -This command does not launch panes or claim that research has been completed. -It renders the fixed contract the visible multi-agent run must satisfy: - -- `research_brief`: what has been read, what has not been read, and the claim - boundary; -- `action_plan`: P0/P1/P2 work, capped at five todos; -- `evidence_refs`: code, docs, benchmarks, issues, and pull requests; -- `next_executable_step`: whether the next step can run automatically; -- `gate`: the exact user judgment needed before crossing a boundary. - -This keeps the user layer and the auto-research preset thin. The user supplies -one open question; auto-research supplies a fixed output contract; the generic -kernel owns the runner, real Codex TUI panes, pane-local A2A tick, and -todo/evidence/status protocol. - -When the run must end in a verifiable deliverable rather than only a research -finding, the curator writes `auto_research_delivery_contract_v0`. The envelope -keeps the original wish, assumptions, required artifacts, acceptance criteria, -failure fallbacks, and reentry conditions together with the existing -`research_contract_v0`. Evidence created from that file is bound to its -atomic `wish_id`, `contract_ref`, and `contract_revision` lineage; partial or -mismatched lineage is not eligible for verification. - -After terminal decisions and any required independent reviews exist, build the -read-only delivery receipt: - -```bash -loopx auto-research artifact-receipt \ - --contract delivery-contract.public.json -``` - -The receipt distinguishes `verified`, `partial`, `inconclusive`, -`not_fulfilled`, and `stale`. A single failed attempt is never enough for -`not_fulfilled`; that state requires a terminal retirement decision and any -independent review required by the delivery contract. Non-verified receipts -return the unmet criteria, verified boundary, available fallback artifacts, and -reentry conditions. - -The contract also includes the next one-command launch surface: - -```bash -loopx auto-research start "" --execute -``` - -Without `--execute`, `start` returns the same contract-anchored runner packet as -a dry-run preview. With `--execute`, it creates an isolated research frontier -and starts the visible Codex TUI lanes through the generic multi-agent kernel. -The launcher then broadcasts the fixed pane-local A2A wake prompt so each pane -runs its own `$LOOPX_PANE_A2A_TICK` against LoopX quota/frontier state. Research -evidence must be authored by the visible role after it does real work. This is still -decentralized A2A, not a workflow driver: the broadcaster does not select todos, -run worker turns, or write LoopX state. -The wake summary is state context for the pane to read, not a research result or -skip gate. Later wake rounds still ask each role to check its own LoopX state and -tick when runnable work remains. -The fixed wake is session-scoped: it targets stable pane lane metadata inside -the requested tmux session and must not infer work from mutable Codex pane -titles, stale panes, or other sessions on the host. - -For the first visible demo, prefer a Codex CLI goal/pane-driven surface with a -LoopX state-driven frontier. Do not replace it with a hidden dynamic workflow -driver. The launcher may open panes and broadcast the fixed wake prompt, but -dependency ordering must live in normal LoopX todos, resume gates, quota, and -frontier projection. For example, an evaluator pane should wait on executor -evidence through a resumable todo instead of closing its own review todo when -evidence has not landed yet. - -The user still only supplies one open question; agent ids, pane-local tick -commands, evidence schemas, and runner wiring stay inside the kernel. Pass -`--attach` when you want to skip the default visible role wake and immediately -enter the tmux session. -By default, visible panes open in a stable user-owned workspace under -`~/loopx-auto-research//visible-workspace` so the first screen does not -land in a generated temp directory. Pass `--workspace` to choose a different -scratch directory. - -## Start From A Clean Workspace - -Use a user-owned directory for the visible demo, while keeping LoopX state in -the normal shared control plane. This keeps research scratch files separate -from the LoopX repository but lets every lane read the same registry, quota, -todo, frontier, and rollout-event state. For the smoothest first run, start -from a directory the Codex CLI already trusts, or from a plain non-git demo -directory you own. - -```bash -mkdir -p loopx-auto-research-demo -cd loopx-auto-research-demo -export LOOPX_REGISTRY="$HOME/.codex/loopx/registry.global.json" -export LOOPX_RUNTIME_ROOT="$HOME/.codex/loopx" -``` - -Install or repair the CLI when needed: - -```bash -curl -fsSL https://huangruiteng.github.io/loopx/install.sh | bash -export PATH="$HOME/.local/bin:$PATH" -loopx doctor -``` - -## 0. Prove The Worker-Loop Positive Path - -The fastest honest positive check is the one-question start path. It seeds a -fresh demo-local LoopX goal, lets role-compatible workers read quota/frontier -state, opens visible Codex TUI panes by default, and keeps the first output tied -to the fixed user contract. It is intentionally small and still does not claim -that visible Codex lanes authored the research result unless a compact live -evidence packet is supplied. - -To run the normal human-facing path and open visible panes: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" \ - --execute \ - --replace-existing -``` - -That command is the user-facing UX for auto-research. It creates a fresh -isolated demo goal by default, launches the visible tmux lanes, broadcasts the -fixed decentralized A2A wake prompt, and records compact wake/tick evidence -before user takeover. Each tmux window should open as a real interactive Codex -CLI TUI role, not as a JSON/status stream. Generic launcher internals stay -inside LoopX; the operator does not need to know `demo-e2e`, agent ids, wake -flags, or implementation paths. - -Visible Codex TUI panes default to a stable -`~/loopx-auto-research//visible-workspace`, not to a demo-local git -worktree or generated temp directory. The demo registry, runtime root, queue, -and evidence state stay isolated, but the first screen should not be a -generated worktree trust prompt. If you want an explicit scratch location, pass -`--workspace "$HOME/loopx-auto-research-demo" --create-workspace`; avoid -pointing `--workspace` at the demo-local control-plane directory. - -When this demo is being advanced from a broader productization goal such as -`loopx-meta`, do not change `--goal-id` to that meta goal. Omit `--goal-id` for -an isolated demo goal, or pass a purpose-built research frontier goal when you -want visible lanes to read an existing target. Add `--tracking-goal-id loopx-meta` -only when the caller needs metadata that says which parent goal is tracking the -product work; tracking metadata never drives the visible lane frontier. - -If you want to inspect before opening visible Codex lanes, start with the -read-only dry-run. It tells the operator which command will run the -multi-round positive path: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" -``` - -When the dry-run looks right, run the multi-round positive path: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" \ - --execute -``` - -That command starts visible panes, wakes each pane with the fixed -decentralized A2A prompt, and keeps the current shell available for the compact -JSON result. To attach immediately instead, pass `--attach`; that means -operator takeover first and skips the default visible role wake: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" \ - --execute \ - --attach -``` - -For JSON-only automation, make the headless path explicit: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research start "How should we evaluate autonomous research agents?" \ - --execute \ - --headless -``` - -Visible panes default to the Codex CLI TUI and the product start path stays in -the current shell after recording wake evidence. The pane-local -`$LOOPX_PANE_LOOPX` wrapper is for human-readable LoopX commands inside that TUI, and -`$LOOPX_PANE_LOOPX_JSON` is reserved for redirected machine artifacts. Raw JSON -should not be dumped into the first visible screen; future control surfaces can -render those artifacts separately. - -Expected minimal E2E result: - -- visible launch controls create real Codex TUI panes with role-local - bootstrap prompts; -- each pane can run `$LOOPX_PANE_A2A_TICK` to read its own quota/frontier; -- `pane_local_a2a.worker_turn_configured` is `false` for the default - auto-research preset; -- no dev or holdout metric is claimed unless a lane-authored public-safe - evidence packet is explicitly supplied; -- visible launch controls stay separate from the research result and only prove - that panes can be inspected, stopped, or retried. - -Maintainer acceptance for this path is: - -```bash -python3 examples/auto-research-layered-e2e-acceptance-smoke.py -``` - -That smoke checks the shortest layered contract, the visible Codex TUI runner -contract, todo/frontier routing, and the no-fake-metric boundary without making -the auto-research preset own generic runner machinery. - -Concrete KNN visible command: - -```bash -loopx --format json auto-research start \ - "How can the KNN solver improve exact-neighbor speedup?" \ - --preset knn-demo \ - --language zh \ - --execute -``` - -`--preset knn-demo` is the public signal that LoopX should materialize a small -generated KNN benchmark workspace. The visible panes get `solution.py` as the -only editable file, `task.py`/`eval.py`/`eval.sh` as protected files, and real -commands for `bash eval.sh dev` and `bash eval.sh test`. The open question names -the research topic; it does not smuggle the benchmark baseline into the system -through natural-language inference. - -The generic path without a preset also works: - -```bash -loopx auto-research start \ - "How should we evaluate whether multi-agent auto research creates value?" \ - --execute -``` - -KNN acceptance: - -- the preset materializes the generated benchmark workspace, not an inferred - natural-language baseline; -- panes are named by research role, not backend agent implementation names; -- the first tick is quota/frontier/status, not research completion; -- dev/holdout uplift is absent until visible roles write real public-safe - evidence; -- the public boundary records no raw logs, private artifacts, credentials, or - local absolute paths. - -Remaining boundary: video/showcase evidence should come from visible Codex panes -performing the research work. A headless smoke can validate state plumbing, but -it is not a claim that visible Codex panes authored the research result. - -For maintainer-level rehearsal, `demo-e2e --launch-visible --attach` is still -accepted as a lower-level command. It is not the user-facing product path; the -short `auto-research start "" --execute` command owns the -default visible role wake behavior. - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research demo-e2e \ - --agent-id auto-research-operator \ - --reasoning-effort high \ - --execute \ - --launch-visible \ - --launcher tmux \ - --attach -``` - -If a previous visible rehearsal is still alive, retry with -`--replace-existing` or stop it first: - -```bash -tmux kill-session -t loopx-auto-research -``` - -The one-command path must not record raw logs, private artifacts, credentials, -or local absolute workspace paths. It may write only public-safe LoopX state -that the visible roles authored. - -## 1. Preview The One-Command Start - -The default user path starts from one open question and previews the visible -Codex TUI lane packet without starting processes. - -```bash -loopx --format json auto-research start \ - "How should we evaluate autonomous research agents?" -``` - -When the preview is acceptable, launch the visible lanes: - -```bash -loopx --format json auto-research start \ - "How should we evaluate autonomous research agents?" \ - --execute -``` - -The visible lanes use LoopX state for quota, frontier, todo projection, and -public-safe evidence. The first pane tick is only a guard/frontier read; the -research result must be authored by the visible roles, not by a hidden demo -metric loop. Raw JSON is written only to local artifacts. Lower-level -`demo-e2e` commands remain available for maintainers, but they are not the -user-facing entrypoint. - -## 2. Inspect The Visible Employee Plan - -The supervisor is a host launcher, not a leader agent. Start with the dry-run -packet and inspect it before launching Codex. - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research demo-supervisor \ - --goal-id loopx-auto-research-demo \ - --workspace "$PWD" -``` - -The default visible digital employees are: - -| Pane | Role | What it owns | -| --- | --- | --- | -| `research-curator:research-curator` | Research curator | Keeps the research contract, protected boundary, metric, stop policy, evidence review, and operator gates explicit. | -| `hypothesis-proposer:hypothesis-proposer` | Hypothesis proposer | Turns ideas into todo-backed hypotheses, successor links, and retirement rationale. | -| `research-executor:research-executor` | Research executor | Executes selected hypotheses under an isolated attempt boundary when mutation is required and preserves scored or unscored evidence. | -| `evaluator-promoter:evaluator-promoter` | Evaluator/promoter | Checks holdout/verification evidence, classifies claims, and keeps promotion boundaries explicit. | - -Each Codex TUI role must route through its own quota/frontier/worker-turn path -inside the pane and honor the selected Todo's execution authority. For a `hard_lease` Goal, -an executing summary worker acquires its own exact task lease before writing, -passes that lease into completion, and releases it on failure or unfinished -completion. An existing execution's lease is a conflict, never borrowed. -The supervisor only makes those roles visible and interactive. -The panes share the same LoopX goal surface: registry, runtime root, frontier, -todo projection, and evidence graph. Do not move every pane into an unrelated -empty workspace; isolate only mutating research-executor attempts with a claimed -git worktree or equivalent execution boundary. -Visible Codex TUI panes should default to the caller's workspace or an explicit -user-owned scratch workspace. They should not default into the demo-local -control-plane repository or generated lane worktrees, because that exposes -workspace-trust prompts before the user sees the actual research roles. - -For compatibility or product experiments, `--agent` can still name explicit -lanes, including a separate evaluator-promoter lane. - -## 3. Run The Worker Loop - -The visible panes should do work through the same CLI path a heartbeat worker -uses: each turn re-reads quota, frontier, todo projection, and rollout evidence -before writing anything. -The user-facing terminal should first show the Codex CLI TUI. Role progress, -selected todo, action, and blocker summaries should appear as normal Codex -interaction when the role runs LoopX commands; raw JSON belongs in -`.public.json` artifacts, not on the first visible screen. - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research worker-loop \ - --goal-id loopx-auto-research-demo \ - --agent-id research-curator \ - --agent-id hypothesis-proposer \ - --agent-id research-executor \ - --agent-id evaluator-promoter \ - --max-rounds 4 -``` - -When the dry-run shows the selected lane work is safe, add `--execute` and -`--complete-selected-todo`. This is the smallest real multi-agent loop: it is -state-mediated, not a hidden leader workflow. - -## 3b. Stop, Takeover, And State-Aware Wake - -Operator control stays on this command path: - -- place `workspace/.loopx-auto-research-stop` to exit the next worker-loop round - with `stop_reason = operator_stop_requested` (individual `worker-turn` calls - remain available for takeover); -- pass `--execute --attach` on `auto-research start` for immediate tmux - takeover without a background wake; -- pass `--no-attach --wake-visible-after-launch` when you want a state-aware - wake that skips quiet-completion, empty-frontier, or quota-blocked lanes. - -`--attach` and `--wake-visible-after-launch` cannot be combined. For the full -stop → takeover → resume cycle and the synthetic smokes that pin it, see the -[stop/takeover/wake walkthrough](../../docs/guides/auto-research-stop-takeover-wake-walkthrough.md). - -## 4. Launch A Visible Rehearsal - -Use tmux when available so the user can watch several Codex CLI TUIs in one -place: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research demo-supervisor \ - --goal-id loopx-auto-research-demo \ - --workspace "$PWD" \ - --execute \ - --launcher tmux \ - --attach -``` - -The user can stop a tmux rehearsal with: - -```bash -tmux kill-session -t loopx-auto-research -``` - -Or take over a lane by attaching, interrupting the pane, and continuing from -the visible prompt: - -```bash -tmux attach -t loopx-auto-research -``` - -## 5. Inspect Progress - -Useful read-only checks: - -```bash -loopx --registry "$LOOPX_REGISTRY" --runtime-root "$LOOPX_RUNTIME_ROOT" status -loopx --registry "$LOOPX_REGISTRY" --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research frontier --goal-id loopx-auto-research-demo \ - --agent-id auto-research-operator -``` - -The demo is healthy when the user can identify the active hypothesis, see which -lane owns the next transition, inspect evidence or retry rationale, and stop or -take over before any private material, credential, protected file, or -production action is needed. - -## Boundary - -This command path is for local, visible, user-takeover rehearsals. It is not a -claim that a research result is production-ready, not a public first-screen -approval, and not permission to publish private evidence. Promotion still -requires rollout-backed evidence, held-out checks when relevant, and normal -LoopX gate/writeback rules. diff --git a/demo/auto_research/__init__.py b/demo/auto_research/__init__.py deleted file mode 100644 index 169c9fbf56..0000000000 --- a/demo/auto_research/__init__.py +++ /dev/null @@ -1,16 +0,0 @@ -from __future__ import annotations - -from .core import ( - LIGHTWEIGHT_AUTO_RESEARCH_EVIDENCE_SCHEMA_VERSION, - LIGHTWEIGHT_AUTO_RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION, - lightweight_hypothesis, - run_lightweight_auto_research, -) -__all__ = [ - "LIGHTWEIGHT_AUTO_RESEARCH_EVIDENCE_SCHEMA_VERSION", - "LIGHTWEIGHT_AUTO_RESEARCH_HYPOTHESIS_SCHEMA_VERSION", - "LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION", - "lightweight_hypothesis", - "run_lightweight_auto_research", -] diff --git a/demo/auto_research/artifact_receipt.py b/demo/auto_research/artifact_receipt.py deleted file mode 100644 index bc73daf8e3..0000000000 --- a/demo/auto_research/artifact_receipt.py +++ /dev/null @@ -1,700 +0,0 @@ -from __future__ import annotations - -import json -from collections.abc import Mapping, Sequence -from pathlib import Path -from typing import Any - -from .delivery_contract import ( - AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION, - AutoResearchContractLineage, - auto_research_contract_lineage_matches, - normalize_auto_research_contract_lineage, - normalize_auto_research_delivery_contract, -) -from .research_state import build_research_evidence_graph_from_rollout_events -from .terminal_result_query import build_terminal_result_query -from loopx.agent_registry import registered_agent_ids_from_registry -from loopx.history import load_registry -from loopx.paths import resolve_runtime_root -from loopx.rollout_event_log import load_rollout_events, rollout_event_log_path - - -AUTO_RESEARCH_ARTIFACT_RECEIPT_SCHEMA_VERSION = ( - "auto_research_artifact_receipt_v0" -) -ARTIFACT_RECEIPT_STATUSES = { - "verified", - "partial", - "inconclusive", - "not_fulfilled", - "stale", -} - - -def _details(event: Mapping[str, Any]) -> Mapping[str, Any]: - value = event.get("details") - return value if isinstance(value, Mapping) else {} - - -def _contract_events( - rollout_events: Sequence[Mapping[str, Any]], - *, - wish_id: str, -) -> list[Mapping[str, Any]]: - return [ - event - for event in rollout_events - if str(event.get("classification") or "") - == AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION - and str(_details(event).get("wish_id") or "") == wish_id - ] - - -def _lineage_events( - rollout_events: Sequence[Mapping[str, Any]], - *, - hypothesis_id: str, -) -> list[Mapping[str, Any]]: - return [ - event - for event in rollout_events - if str(_details(event).get("hypothesis_id") or "") == hypothesis_id - and str(event.get("event_kind") or "") - in {"research_hypothesis", "research_evidence"} - ] - - -def _artifact_refs( - events: Sequence[Mapping[str, Any]], - *, - expected_lineage: AutoResearchContractLineage, -) -> list[str]: - refs = { - str(ref) - for event in events - if auto_research_contract_lineage_matches( - _details(event), - expected=expected_lineage, - ) - for ref in event.get("artifact_refs") or [] - if str(ref).strip() - } - return sorted(refs) - - -def _failure_kind(result: Mapping[str, Any]) -> str: - if result.get("decision_state") == "conflict": - return "conflicting_terminal_decisions" - review = result.get("peer_review") - if isinstance(review, Mapping): - review_failure = _review_failure_kind(review) - if review_failure: - return review_failure - decision = result.get("terminal_decision") - if ( - isinstance(decision, Mapping) - and decision.get("outcome") == "retired" - and decision.get("reason") - ): - return str(decision["reason"]) - if result.get("research_status") == "needs_retry": - return "research_retry_pending" - return "terminal_decision_missing" - - -def _review_failure_kind(review: Mapping[str, Any]) -> str | None: - state = str(review.get("state") or "") - verdict = ( - str(review.get("verdict") or "") - if state == "independent_reviewed" - else None - ) - return { - ("conflict", None): "conflicting_peer_reviews", - ("independent_reviewed", "needs_more_evidence"): "needs_more_evidence", - ("independent_reviewed", "reject"): "independent_review_rejected", - ("self_review_only", None): "independent_review_missing", - ("unreviewed", None): "independent_review_missing", - }.get((state, verdict)) - - -def _result_parts( - result: Mapping[str, Any] | None, -) -> tuple[Mapping[str, Any], Mapping[str, Any]]: - if not isinstance(result, Mapping): - return {}, {} - decision = result.get("terminal_decision") - review = result.get("peer_review") - return ( - decision if isinstance(decision, Mapping) else {}, - review if isinstance(review, Mapping) else {}, - ) - - -def _lineage_state( - events: Sequence[Mapping[str, Any]], - *, - expected_lineage: AutoResearchContractLineage, -) -> tuple[list[str], list[Mapping[str, Any]]]: - observed_revisions = sorted( - { - str(_details(event).get("contract_revision") or "") - for event in events - if _details(event).get("contract_revision") - } - ) - current = [ - event - for event in events - if auto_research_contract_lineage_matches( - _details(event), - expected=expected_lineage, - ) - ] - return observed_revisions, current - - -def _result_artifact_refs( - result: Mapping[str, Any] | None, - *, - decision: Mapping[str, Any], - review: Mapping[str, Any], -) -> list[str]: - if not isinstance(result, Mapping) or result.get("decision_state") != "current": - return [] - refs = list(decision.get("evidence_refs") or []) - refs.extend( - str(ref) - for item in review.get("reviews") or [] - if isinstance(item, Mapping) - for ref in item.get("evidence_refs") or [] - ) - return refs - - -def _has_complete_research_lineage( - events: Sequence[Mapping[str, Any]], -) -> bool: - event_kinds = {str(event.get("event_kind") or "") for event in events} - return {"research_hypothesis", "research_evidence"} <= event_kinds - - -def _criterion_artifact_state( - criterion: Mapping[str, Any], - *, - current_lineage: Sequence[Mapping[str, Any]], - expected_lineage: AutoResearchContractLineage, - result: Mapping[str, Any] | None, - decision: Mapping[str, Any], - review: Mapping[str, Any], -) -> tuple[list[str], list[str]]: - if not _has_complete_research_lineage(current_lineage): - return [], sorted(criterion["required_artifact_refs"]) - observed = sorted( - set( - _artifact_refs( - current_lineage, - expected_lineage=expected_lineage, - ) - ) - | set( - _result_artifact_refs( - result, - decision=decision, - review=review, - ) - ) - ) - missing = sorted(set(criterion["required_artifact_refs"]) - set(observed)) - return observed, missing - - -def _criterion_status( - criterion: Mapping[str, Any], - *, - result: Mapping[str, Any] | None, - decision: Mapping[str, Any], - review: Mapping[str, Any], - current_lineage: Sequence[Mapping[str, Any]], - observed_revisions: Sequence[str], - missing_artifacts: Sequence[str], -) -> str: - if observed_revisions and not current_lineage: - return "stale" - if not _has_complete_research_lineage(current_lineage) or result is None: - return "inconclusive" - if result.get("decision_state") == "stale": - return "stale" - if result.get("decision_state") == "conflict": - return "inconclusive" - independent_approved = ( - review.get("state") == "independent_reviewed" - and review.get("verdict") == "approve" - ) - review_satisfied = ( - not criterion["requires_independent_review"] or independent_approved - ) - if decision.get("outcome") == "retired": - return "not_fulfilled" if review_satisfied else "inconclusive" - if decision.get("outcome") != "promoted" or not review_satisfied: - return "inconclusive" - return "partial" if missing_artifacts else "verified" - - -def _criterion_failure_kind( - *, - status: str, - criterion: Mapping[str, Any], - result: Mapping[str, Any] | None, - evidence_node: Mapping[str, Any] | None, - review: Mapping[str, Any], - current_lineage: Sequence[Mapping[str, Any]], -) -> str: - if status == "stale": - return "contract_revision_changed" - if status == "partial": - return "required_artifact_missing" - if ( - status == "inconclusive" - and not _has_complete_research_lineage(current_lineage) - ): - return "contract_evidence_missing" - review_failure = _review_failure_kind(review) - if isinstance(result, Mapping) and _result_has_review_failure( - criterion, - result=result, - review=review, - review_failure=review_failure, - ): - return _failure_kind(result) - if ( - status != "verified" - and isinstance(evidence_node, Mapping) - and evidence_node.get("failure_kind") - ): - return str(evidence_node["failure_kind"]) - if status != "verified" and isinstance(result, Mapping): - return _failure_kind(result) - return "" - - -def _result_has_review_failure( - criterion: Mapping[str, Any], - *, - result: Mapping[str, Any], - review: Mapping[str, Any], - review_failure: str | None, -) -> bool: - return ( - result.get("decision_state") == "conflict" - or review.get("state") == "conflict" - or ( - criterion["requires_independent_review"] - and review_failure == "independent_review_missing" - ) - or review_failure - in {"needs_more_evidence", "independent_review_rejected"} - ) - - -def _criterion_result( - criterion: Mapping[str, Any], - *, - result: Mapping[str, Any] | None, - evidence_node: Mapping[str, Any] | None, - lineage_events: Sequence[Mapping[str, Any]], - expected_lineage: AutoResearchContractLineage, -) -> dict[str, Any]: - observed_revisions, current_lineage = _lineage_state( - lineage_events, - expected_lineage=expected_lineage, - ) - decision, review = _result_parts(result) - observed_artifacts, missing_artifacts = _criterion_artifact_state( - criterion, - current_lineage=current_lineage, - expected_lineage=expected_lineage, - result=result, - decision=decision, - review=review, - ) - - status = _criterion_status( - criterion, - result=result, - decision=decision, - review=review, - current_lineage=current_lineage, - observed_revisions=observed_revisions, - missing_artifacts=missing_artifacts, - ) - failure_kind = _criterion_failure_kind( - status=status, - criterion=criterion, - result=result, - evidence_node=evidence_node, - review=review, - current_lineage=current_lineage, - ) - return { - "criterion_id": criterion["criterion_id"], - "description": criterion["description"], - "hypothesis_id": criterion["hypothesis_id"], - "required": criterion["required"], - "status": status, - "required_artifact_refs": list(criterion["required_artifact_refs"]), - "observed_artifact_refs": observed_artifacts, - "missing_artifact_refs": missing_artifacts, - "decision_state": _decision_state(result), - "terminal_outcome": str(decision.get("outcome") or ""), - "terminal_reason": str(decision.get("reason") or ""), - "review_state": str(review.get("state") or "not_applicable"), - "review_verdict": str(review.get("verdict") or ""), - "failure_kind": failure_kind, - "measurement_scope": _measurement_scope(evidence_node), - "observed_contract_revisions": observed_revisions, - } - - -def _decision_state(result: Mapping[str, Any] | None) -> str: - if not isinstance(result, Mapping): - return "missing" - return str(result.get("decision_state") or "missing") - - -def _measurement_scope(evidence_node: Mapping[str, Any] | None) -> str: - if not isinstance(evidence_node, Mapping): - return "" - return str(evidence_node.get("measurement_scope") or "") - - -def _receipt_status( - criteria: Sequence[Mapping[str, Any]], - *, - missing_required_artifacts: Sequence[str], - contract_state: str, -) -> str: - required = [item for item in criteria if item.get("required") is True] - statuses = {str(item.get("status") or "") for item in required} - if contract_state == "stale" or "stale" in statuses: - return "stale" - if contract_state != "current": - return "inconclusive" - if "not_fulfilled" in statuses: - return "not_fulfilled" - if required and statuses == {"verified"} and not missing_required_artifacts: - return "verified" - if "verified" in statuses or "partial" in statuses: - return "partial" - return "inconclusive" - - -def _failure_feedback( - *, - status: str, - contract_state: str, - criteria: Sequence[Mapping[str, Any]], - missing_required_artifacts: Sequence[str], - fallback_artifact_refs: Sequence[str], - reentry_conditions: Sequence[str], - observed_artifact_refs: Sequence[str], -) -> dict[str, Any] | None: - if status == "verified": - return None - failed = [ - item - for item in criteria - if item.get("required") is True and item.get("status") != "verified" - ] - verified = [ - item - for item in criteria - if item.get("status") == "verified" - ] - fallbacks = sorted(set(fallback_artifact_refs) & set(observed_artifact_refs)) - failure_kinds, derived_reentry = _failure_reentry_details( - failed, - contract_state=contract_state, - missing_required_artifacts=missing_required_artifacts, - ) - summaries = { - "stale": ( - "The delivery contract changed after the recorded research evidence; " - "rerun the affected criteria against the current contract." - ), - "not_fulfilled": ( - "The current evidence supports a terminal conclusion that at least " - "one required criterion cannot be fulfilled under this contract." - ), - "partial": ( - "Some required criteria or artifacts are verified, but the complete " - "delivery contract is not satisfied." - ), - "inconclusive": ( - "The available evidence is insufficient for a verified or terminal " - "not-fulfilled conclusion." - ), - } - return { - "summary": summaries[status], - "failure_kinds": sorted(failure_kinds), - "unmet_criteria": [str(item["criterion_id"]) for item in failed], - "verified_boundary": [str(item["criterion_id"]) for item in verified], - "missing_required_artifact_refs": list(missing_required_artifacts), - "fallback_artifact_refs": fallbacks, - "reentry_conditions": list( - dict.fromkeys([*reentry_conditions, *derived_reentry]) - ), - } - - -def _failure_reentry_details( - failed: Sequence[Mapping[str, Any]], - *, - contract_state: str, - missing_required_artifacts: Sequence[str], -) -> tuple[set[str], list[str]]: - failure_kinds = { - str(item.get("failure_kind") or "") - for item in failed - if item.get("failure_kind") - } - reentry = [ - f"resolve:{item['criterion_id']}:{item['failure_kind']}" - for item in failed - ] - if contract_state == "missing": - failure_kinds.add("contract_record_missing") - reentry.insert(0, "record_current_delivery_contract") - if missing_required_artifacts: - failure_kinds.add("required_artifact_missing") - reentry.extend( - f"provide:{artifact_ref}" - for artifact_ref in missing_required_artifacts - ) - return failure_kinds, reentry - - -def _contract_state( - contract_events: Sequence[Mapping[str, Any]], - *, - expected_lineage: AutoResearchContractLineage, -) -> tuple[str | None, str]: - latest_revision = None - if contract_events: - latest_revision = ( - str(_details(contract_events[-1]).get("contract_revision") or "") - or None - ) - if latest_revision is None: - return None, "missing" - if contract_events and auto_research_contract_lineage_matches( - _details(contract_events[-1]), - expected=expected_lineage, - ): - return latest_revision, "current" - return latest_revision, "stale" - - -def _indexed_records( - records: object, -) -> dict[str, Mapping[str, Any]]: - if not isinstance(records, Sequence): - return {} - return { - str(item.get("hypothesis_id") or ""): item - for item in records - if isinstance(item, Mapping) - } - - -def _acceptance_results( - *, - contract: Mapping[str, Any], - rollout_events: Sequence[Mapping[str, Any]], - results: Mapping[str, Mapping[str, Any]], - evidence_nodes: Mapping[str, Mapping[str, Any]], - expected_lineage: AutoResearchContractLineage, -) -> list[dict[str, Any]]: - return [ - _criterion_result( - criterion, - result=results.get(criterion["hypothesis_id"]), - evidence_node=evidence_nodes.get(criterion["hypothesis_id"]), - lineage_events=_lineage_events( - rollout_events, - hypothesis_id=criterion["hypothesis_id"], - ), - expected_lineage=expected_lineage, - ) - for criterion in contract["acceptance_criteria"] - ] - - -def _artifact_summary( - contract: Mapping[str, Any], - criteria: Sequence[Mapping[str, Any]], -) -> tuple[list[str], list[str]]: - observed = sorted( - { - str(ref) - for criterion in criteria - for ref in criterion["observed_artifact_refs"] - } - ) - missing = sorted( - { - str(artifact["artifact_ref"]) - for artifact in contract["required_artifacts"] - if artifact["required"] - } - - set(observed) - ) - return observed, missing - - -def _verification_summary( - *, - criteria: Sequence[Mapping[str, Any]], - contract: Mapping[str, Any], - evidence_graph_revision: str, -) -> dict[str, Any]: - return { - "required_count": sum(item["required"] for item in criteria), - "verified_count": sum( - item["status"] == "verified" for item in criteria - ), - "not_fulfilled_count": sum( - item["status"] == "not_fulfilled" for item in criteria - ), - "independent_review_required": any( - item["requires_independent_review"] - for item in contract["acceptance_criteria"] - ), - "evidence_graph_revision": evidence_graph_revision, - } - - -def build_auto_research_artifact_receipt( - *, - delivery_contract: Mapping[str, Any], - rollout_events: Sequence[Mapping[str, Any]], - registered_agent_ids: Sequence[str], -) -> dict[str, Any]: - contract = normalize_auto_research_delivery_contract(delivery_contract) - research_contract = contract["research_contract"] - expected_lineage = normalize_auto_research_contract_lineage( - { - "wish_id": contract["wish"]["wish_id"], - "contract_ref": contract["contract_ref"], - "contract_revision": contract["contract_revision"], - }, - field="delivery_contract", - required=True, - ) - contract_events = _contract_events( - rollout_events, - wish_id=contract["wish"]["wish_id"], - ) - latest_revision, contract_state = _contract_state( - contract_events, - expected_lineage=expected_lineage, - ) - evidence_graph = build_research_evidence_graph_from_rollout_events( - goal_id=research_contract["goal_id"], - rollout_events=[dict(event) for event in rollout_events], - ) - query = build_terminal_result_query( - evidence_graph=evidence_graph, - rollout_events=rollout_events, - registered_agent_ids=registered_agent_ids, - include_history=True, - ) - criteria = _acceptance_results( - contract=contract, - rollout_events=rollout_events, - results=_indexed_records(query["results"]), - evidence_nodes=_indexed_records(evidence_graph.get("nodes")), - expected_lineage=expected_lineage, - ) - observed_artifact_refs, missing_required_artifacts = _artifact_summary( - contract, - criteria, - ) - status = _receipt_status( - criteria, - missing_required_artifacts=missing_required_artifacts, - contract_state=contract_state, - ) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_ARTIFACT_RECEIPT_SCHEMA_VERSION, - "goal_id": research_contract["goal_id"], - "wish": contract["wish"], - "contract": { - "contract_ref": contract["contract_ref"], - "contract_revision": contract["contract_revision"], - "latest_recorded_revision": latest_revision, - "state": contract_state, - }, - "status": status, - "artifacts": { - "required": contract["required_artifacts"], - "observed_refs": observed_artifact_refs, - "missing_required_refs": missing_required_artifacts, - }, - "acceptance_results": criteria, - "verification_summary": _verification_summary( - criteria=criteria, - contract=contract, - evidence_graph_revision=query["evidence_graph_revision"], - ), - "failure_feedback": _failure_feedback( - status=status, - contract_state=contract_state, - criteria=criteria, - missing_required_artifacts=missing_required_artifacts, - fallback_artifact_refs=contract["failure_policy"][ - "fallback_artifact_refs" - ], - reentry_conditions=contract["failure_policy"]["reentry_conditions"], - observed_artifact_refs=observed_artifact_refs, - ), - "learning_disposition": ( - "candidate" if status in {"verified", "not_fulfilled"} else "none" - ), - "boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "user_acceptance_inferred": False, - "automatic_skill_promotion": False, - }, - } - - -def load_auto_research_artifact_receipt( - *, - contract_path: str | Path, - registry_path: Path, - runtime_root_arg: str | None, -) -> dict[str, Any]: - raw_contract = json.loads( - Path(contract_path).expanduser().read_text(encoding="utf-8") - ) - if not isinstance(raw_contract, Mapping): - raise ValueError("delivery contract file must contain a JSON object") - contract = normalize_auto_research_delivery_contract(raw_contract) - goal_id = contract["research_contract"]["goal_id"] - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - events = load_rollout_events(rollout_event_log_path(runtime_root, goal_id)) - return build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=events, - registered_agent_ids=registered_agent_ids_from_registry( - registry_path, - goal_id, - ), - ) diff --git a/demo/auto_research/bootstrap_contract.py b/demo/auto_research/bootstrap_contract.py deleted file mode 100644 index 738c9dd040..0000000000 --- a/demo/auto_research/bootstrap_contract.py +++ /dev/null @@ -1,157 +0,0 @@ -from __future__ import annotations - -import shlex - - -AUTO_RESEARCH_LIVE_WORKER_PROOF_SCHEMA_VERSION = "auto_research_visible_worker_proof_v0" - - -def auto_research_contract_command_text(*, cli_bin: str, objective: str) -> str: - return f"{shlex.quote(cli_bin)} auto-research {shlex.quote(str(objective).strip())}" - - -def auto_research_start_command_text( - *, - cli_bin: str, - objective: str, - preset_id: str | None = None, - execute: bool = False, - headless: bool = False, - no_attach: bool = False, - wake_visible_after_launch: bool = False, - output_language: str = "en", -) -> str: - parts = [ - shlex.quote(cli_bin), - "auto-research", - "start", - shlex.quote(str(objective).strip()), - ] - if preset_id: - parts.extend(["--preset", shlex.quote(str(preset_id).strip())]) - if output_language and output_language != "en": - parts.extend(["--language", shlex.quote(output_language)]) - if execute: - parts.append("--execute") - if headless: - parts.append("--headless") - if no_attach: - parts.append("--no-attach") - if wake_visible_after_launch: - parts.append("--wake-visible-after-launch") - return " ".join(parts) - - -def build_auto_research_contract_acceptance( - user_contract: dict[str, object], -) -> dict[str, object]: - command = ( - user_contract.get("command_contract") - if isinstance(user_contract.get("command_contract"), dict) - else {} - ) - required_outputs = ( - command.get("auto_research_required_outputs") - if isinstance(command.get("auto_research_required_outputs"), list) - else [] - ) - output_field_map = { - "research_brief": "research_brief", - "action_plan": "action_plan", - "evidence_refs": "evidence_refs", - "next_executable_step": "next_executable_step", - "gate": "gate", - } - present_outputs = [ - output - for output in required_outputs - if user_contract.get(output_field_map.get(str(output), "")) is not None - ] - action_plan = user_contract.get("action_plan") - gate = user_contract.get("gate") if isinstance(user_contract.get("gate"), dict) else {} - next_step = ( - user_contract.get("next_executable_step") - if isinstance(user_contract.get("next_executable_step"), dict) - else {} - ) - one_click_start = ( - user_contract.get("one_click_start") - if isinstance(user_contract.get("one_click_start"), dict) - else {} - ) - preset_context = ( - user_contract.get("preset_context") - if isinstance(user_contract.get("preset_context"), dict) - else {} - ) - preset_id = str(preset_context.get("preset_id") or "").strip() - expected_start_template = ( - f'loopx auto-research start "" --preset {preset_id} --execute' - if preset_id - else 'loopx auto-research start "" --execute' - ) - checks = { - "one_question_input": user_contract.get("open_question") not in (None, ""), - "required_outputs_present": len(present_outputs) == len(required_outputs), - "action_plan_bounded": isinstance(action_plan, list) and 0 < len(action_plan) <= 5, - "next_step_automatic_by_default": next_step.get("can_run_automatically") is True, - "gate_present": bool(gate.get("user_judgment_needed")), - "canonical_invocation_short": ( - command.get("canonical_invocation") == 'loopx auto-research ""' - ), - "one_click_start_present": ( - one_click_start.get("command_template") == expected_start_template - ), - "one_click_start_uses_generic_kernel": ( - one_click_start.get("uses_generic_kernel") is True - and one_click_start.get("coordination_model") == "decentralized_state_a2a" - ), - } - accepted = all(checks.values()) - return { - "schema_version": "auto_research_user_contract_acceptance_v0", - "accepted": accepted, - "canonical_invocation": command.get("canonical_invocation"), - "required_outputs": required_outputs, - "present_outputs": present_outputs, - "checks": checks, - "missing_outputs": [ - output for output in required_outputs if output not in present_outputs - ], - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - - -def build_auto_research_live_worker_proof(*, launch_visible: bool) -> dict[str, object]: - return { - "schema_version": AUTO_RESEARCH_LIVE_WORKER_PROOF_SCHEMA_VERSION, - "lane_authored_evidence_loaded": False, - "visible_role_participation_required": True, - "visible_role_participation_verified": False, - "visible_role_participation_basis": "not_loaded", - "pane_local_a2a_status_loaded": False, - "pane_local_a2a_status_check_count": 0, - "decentralized_a2a_rounds_verified": False, - "cadence_wake_loaded": False, - "cadence_wake_verified": False, - "visible_lanes_launched": bool(launch_visible), - "visible_lanes_accepted": False, - "evidence_source": "not_loaded", - "reason": ( - "start and demo-e2e share one kernel live-bootstrap contract for visible " - "Codex worker panes; presentation-specific reporting stays outside the " - "auto-research preset." - ), - "next_step": [ - "let each visible pane read its LoopX frontier with a guard tick", - "have each role author public-safe research evidence or successor todos", - "load compact evidence from lane-authored pane artifacts", - ], - "owner_layer": "generic_multi_agent_kernel", - "user_facing_by_default": False, - } diff --git a/demo/auto_research/catalog_entry.py b/demo/auto_research/catalog_entry.py deleted file mode 100644 index 0dde05299b..0000000000 --- a/demo/auto_research/catalog_entry.py +++ /dev/null @@ -1,133 +0,0 @@ -from __future__ import annotations - -from typing import Any - - -AUTO_RESEARCH_CATALOG_ENTRY: dict[str, Any] = { - "id": "auto-research", - "origin": "builtin", - "visibility": "public", - "provider_id": "loopx-core", - "documentation": { - "source_root": "demo/auto_research", - "site_root": "capabilities/auto-research", - "canonical": "README.md", - "aliases": [ - { - "source": "README.md", - "site": "guides/auto-research-command-path.md", - }, - ], - }, - "title": "Decentralized auto-research preset", - "status": "experimental", - "real_world_anchor": ( - "one-question research contracts with role-scoped workers, evidence packets, and a visible frontier" - ), - "user_value": ( - "Launch a thin multi-agent research recipe while reusing LoopX todo, quota, " - "history, evidence, and continuation contracts." - ), - "entry_command": "loopx auto-research ", - "commands": [ - { - "command": "loopx auto-research ", - "purpose": "Render the fixed one-question contract, bounded plan, and exact start command.", - "write_boundary": "stateless contract output only", - }, - { - "command": "loopx auto-research start --execute", - "purpose": "Create an isolated goal and launch visible role-scoped workers through the shared multi-agent runtime.", - "write_boundary": "local goal, workspace, and visible launcher state; no publication, benchmark submission, or protected external action", - }, - { - "command": "loopx auto-research decide --goal-id --hypothesis-id --outcome --reason --agent-id --execute", - "purpose": "Record one terminal research decision after current evidence satisfies the promotion or retirement contract.", - "write_boundary": "one public-safe validation event bound to the current hypothesis evidence revision", - }, - { - "command": "loopx auto-research review --goal-id --hypothesis-id --reviewer-agent-id --verdict --require-independent --execute", - "purpose": "Record a registered-peer review receipt without rewriting the research evidence or terminal decision.", - "write_boundary": "one public-safe validation receipt; independent mode rejects producer and decision-agent self-review", - }, - { - "command": "loopx auto-research results --goal-id [--hypothesis-id ] [--include-history]", - "purpose": "Read exact current and historical terminal outcomes, review state, and Explore readback.", - "write_boundary": "read-only projection over rollout and Explore result events", - }, - { - "command": "loopx auto-research artifact-receipt --contract ", - "purpose": "Fold one versioned research wish contract, current evidence, terminal decisions, and independent reviews into a read-only delivery receipt.", - "write_boundary": "read-only projection; does not infer user acceptance, promote skills, or write project state", - }, - { - "command": "loopx auto-research project-results --goal-id --execute", - "purpose": "Project current terminal outcomes into deterministic existing Explore node and finding events.", - "write_boundary": "goal-scoped Explore result log only; repeated projection is idempotent", - }, - ], - "implemented_protocols": [ - { - "schema_version": "auto_research_user_contract_v0", - "module": "demo.auto_research.user_contract", - "doc": "demo/auto_research/README.md", - }, - { - "schema_version": "decentralized_auto_research_projection_v0", - "module": "demo.auto_research.research_state", - "doc": "docs/reference/protocols/decentralized-auto-research-state-v0.md", - }, - { - "schema_version": "auto_research_terminal_decision_v0", - "module": "demo.auto_research.terminal_result_contract", - "doc": "docs/reference/protocols/decentralized-auto-research-state-v0.md", - }, - { - "schema_version": "auto_research_peer_review_v0", - "module": "demo.auto_research.terminal_result_contract", - "doc": "docs/reference/protocols/auto-research-lane-contract-v1.md", - }, - { - "schema_version": "auto_research_terminal_result_query_v0", - "module": "demo.auto_research.terminal_result_query", - "doc": "docs/reference/protocols/decentralized-auto-research-state-v0.md", - }, - { - "schema_version": "auto_research_terminal_result_projection_v0", - "module": "demo.auto_research.terminal_results", - "doc": "docs/reference/protocols/decentralized-auto-research-state-v0.md", - }, - { - "schema_version": "auto_research_delivery_contract_v0", - "module": "demo.auto_research.delivery_contract", - "doc": "docs/reference/protocols/auto-research-wish-to-artifact-v0.md", - }, - { - "schema_version": "auto_research_artifact_receipt_v0", - "module": "demo.auto_research.artifact_receipt", - "doc": "docs/reference/protocols/auto-research-wish-to-artifact-v0.md", - }, - ], - "smokes": [ - "python3 examples/auto-research-user-contract-entry-smoke.py", - "python3 examples/auto-research-worker-turn-smoke.py", - "python3 examples/auto-research-terminal-results-smoke.py", - "python3 examples/auto-research-terminal-results-cli-smoke.py", - "python3 examples/auto-research-artifact-receipt-smoke.py", - ], - "docs": [ - "demo/auto_research/README.md", - "docs/product/use-cases/auto-research/decentralized-auto-research-showcase.md", - ], - "boundaries": [ - "The preset reuses shared control-plane and multi-agent contracts; it is not a second scheduler or runner.", - "Completion and uplift require role-authored evidence and projected outcomes; the launcher does not manufacture research results.", - "Promotion and retirement candidates are not terminal decisions; explicit decisions bind one hypothesis evidence revision.", - "Same-agent review remains visible but cannot be labeled independent or upgrade a finding to confirmed or refuted.", - "A delivery receipt aggregates existing evidence and never treats one failed attempt as proof that a wish cannot be fulfilled.", - "A not-fulfilled receipt requires a terminal retirement decision plus any independent review required by the delivery contract.", - ], - "next_real_step": ( - "Use the one-question entrypoint for bounded research and preserve every promoted or rejected outcome in canonical evidence." - ), -} diff --git a/demo/auto_research/cli.py b/demo/auto_research/cli.py deleted file mode 100644 index 08968d633a..0000000000 --- a/demo/auto_research/cli.py +++ /dev/null @@ -1,1489 +0,0 @@ -from __future__ import annotations - -import argparse -import json -import re -from collections.abc import Callable -from datetime import datetime, timezone -from pathlib import Path - -from .artifact_receipt import load_auto_research_artifact_receipt -from .human_view import render_auto_research_markdown -from .research_state import ( - build_auto_research_projection, - build_live_auto_research_projection, - load_auto_research_fixture, -) -from .demo_supervisor import build_auto_research_demo_supervisor_plan -from .evidence_packet import load_auto_research_evidence_packet_inputs -from .defaults import ( - AUTO_RESEARCH_DEFAULT_GOAL_ID, - AUTO_RESEARCH_DEFAULT_OBJECTIVE, - AUTO_RESEARCH_KNN_DEMO_PRESET_ID, - AUTO_RESEARCH_SUPPORTED_PRESET_IDS, -) -from .demo_e2e import run_auto_research_demo_e2e -from .knn_demo_workspace import materialize_knn_demo_workspace -from .live_evidence import ( - LIVE_CODEX_E2E_DEFAULT_OUTPUT, - capture_live_codex_e2e_evidence, -) -from .worker_loop import run_auto_research_worker_loop -from .worker_runtime import run_auto_research_worker_turn -from .rollout_append import ( - append_auto_research_rollout_events as _append_auto_research_rollout_events, -) -from .terminal_results import ( - decide_terminal_result, - project_terminal_results, - query_terminal_results, - review_terminal_result, -) -from .user_contract import ( - build_auto_research_user_contract, - infer_auto_research_output_language, - normalize_auto_research_preset_id, -) -from ..multi_agent.visible_launch_policy import ( - make_visible_launcher_callback, - make_visible_wake_callback, - resolve_codex_trust_workspace, - resolve_visible_launch_policy, -) -from loopx.control_plane.runtime.public_safety import public_safe_compact_text -from loopx.history import load_registry -from loopx.paths import resolve_runtime_root -from loopx.quota import build_quota_should_run -from loopx.rollout_event_log import load_rollout_events, rollout_event_log_path -from loopx.status import collect_status -from ..visible_multi_agent_launcher import ( - execute_visible_multi_agent_launcher, -) - - -PrintPayload = Callable[ - [dict[str, object], str, Callable[[dict[str, object]], str]], - None, -] -FormatSelector = Callable[..., str] -AddFormat = Callable[[argparse.ArgumentParser], None] - -AUTO_RESEARCH_DEMO_GOAL_PREFIX = "loopx-auto-research-demo" -AUTO_RESEARCH_START_AGENT_ID = "auto-research-operator" -AUTO_RESEARCH_COMMAND_FAILED_ERROR_CODE = "auto_research_command_failed" -AUTO_RESEARCH_INVALID_INPUT_ERROR_CODE = "auto_research_invalid_input" -AUTO_RESEARCH_IO_FAILED_ERROR_CODE = "auto_research_io_failed" -AUTO_RESEARCH_SUBCOMMANDS = frozenset( - { - "append-evidence", - "artifact-receipt", - "capture-live-evidence", - "contract", - "decide", - "demo-e2e", - "demo-supervisor", - "evidence", - "frontier", - "project-results", - "results", - "review", - "start", - "worker-loop", - "worker-turn", - } -) - - -def _public_auto_research_error(exc: Exception) -> tuple[str, str]: - """Project an internal exception into a stable, public-safe CLI error.""" - - if isinstance(exc, json.JSONDecodeError): - return ( - AUTO_RESEARCH_INVALID_INPUT_ERROR_CODE, - "Auto Research input must be valid JSON.", - ) - if isinstance(exc, OSError): - return ( - AUTO_RESEARCH_IO_FAILED_ERROR_CODE, - "Auto Research could not access a required resource.", - ) - if isinstance(exc, ValueError): - public_message = public_safe_compact_text(str(exc), limit=220) - return ( - AUTO_RESEARCH_INVALID_INPUT_ERROR_CODE, - public_message or "Auto Research input is invalid.", - ) - return ( - AUTO_RESEARCH_COMMAND_FAILED_ERROR_CODE, - "Auto Research command failed unexpectedly.", - ) - - -def rewrite_auto_research_question_argv(argv: list[str]) -> list[str]: - """Map `loopx auto-research ""` to the explicit contract command.""" - - values = list(argv) - try: - command_index = values.index("auto-research") - except ValueError: - return values - - cursor = command_index + 1 - while cursor < len(values): - token = values[cursor] - if token == "--format" and cursor + 1 < len(values): - cursor += 2 - continue - if token.startswith("--format="): - cursor += 1 - continue - break - if cursor >= len(values): - return values - - token = values[cursor] - if token.startswith("-") or token in AUTO_RESEARCH_SUBCOMMANDS: - return values - return values[:cursor] + ["contract"] + values[cursor:] - - -def _demo_goal_suffix(value: object, *, fallback: str = "run") -> str: - text = re.sub(r"[^A-Za-z0-9._-]+", "-", str(value or "").strip()).strip("-._") - return text[:40] or fallback - - -def _default_auto_research_start_workspace(goal_id: str) -> str: - return str( - Path.home() - / "loopx-auto-research" - / _demo_goal_suffix(goal_id, fallback="run") - / "visible-workspace" - ) - - -def _resolve_demo_goal_surface( - *, - goal_id: str | None, - demo_run_id: str | None, - inherit_default_goal: bool, -) -> tuple[str, str]: - if inherit_default_goal and goal_id: - raise ValueError("--inherit-default-goal cannot be combined with --goal-id") - if inherit_default_goal and demo_run_id: - raise ValueError("--inherit-default-goal cannot be combined with --demo-run-id") - if goal_id and demo_run_id: - raise ValueError("--demo-run-id is only used when --goal-id is omitted") - if inherit_default_goal: - return AUTO_RESEARCH_DEFAULT_GOAL_ID, "inherited_default_goal" - if goal_id: - return goal_id, "explicit_goal" - run_id = demo_run_id or datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") - return f"{AUTO_RESEARCH_DEMO_GOAL_PREFIX}-{_demo_goal_suffix(run_id)}", "fresh_demo_goal" - - -def register_auto_research_commands( - subparsers: argparse._SubParsersAction, - add_subcommand_format: AddFormat, -) -> None: - auto_research_parser = subparsers.add_parser( - "auto-research", - help="Project public-safe decentralized auto-research frontiers.", - ) - auto_research_parser.add_argument( - "--format", - dest="auto_research_format", - choices=["markdown", "json"], - help="Output format for the auto-research command group.", - ) - auto_research_sub = auto_research_parser.add_subparsers( - dest="auto_research_command", - required=True, - ) - contract_parser = auto_research_sub.add_parser( - "contract", - help="Render the fixed user-facing auto-research contract for one open question.", - ) - add_subcommand_format(contract_parser) - contract_parser.add_argument("open_question", help="Quoted open research question.") - contract_parser.add_argument( - "--max-todos", - type=int, - default=5, - help="Maximum action-plan todos, capped at 5.", - ) - contract_parser.add_argument( - "--language", - choices=["auto", "zh", "en"], - default="auto", - help="Human-readable output language for the research process.", - ) - contract_parser.add_argument( - "--preset", - choices=AUTO_RESEARCH_SUPPORTED_PRESET_IDS, - help="Explicit domain demo context, e.g. knn-demo materializes a KNN benchmark workspace.", - ) - - start_parser = auto_research_sub.add_parser( - "start", - help="Start visible multi-agent auto-research from one open question.", - ) - add_subcommand_format(start_parser) - start_parser.add_argument("open_question", help="Quoted open research question.") - start_parser.add_argument( - "--language", - choices=["auto", "zh", "en"], - default="auto", - help="Human-readable output language for visible role panes.", - ) - start_parser.add_argument( - "--preset", - choices=AUTO_RESEARCH_SUPPORTED_PRESET_IDS, - help=( - "Explicit domain demo context. Use knn-demo when the baseline/metric " - "come from a generated KNN benchmark workspace rather than the question text." - ), - ) - start_parser.add_argument( - "--execute", - action="store_true", - help=( - "Create an isolated research goal and launch visible Codex TUI lanes. " - "Omit to preview the exact contract, commands, and runner packet." - ), - ) - start_parser.add_argument( - "--headless", - action="store_true", - help="With --execute, run the non-interactive worker-loop proof instead of visible panes.", - ) - start_parser.add_argument( - "--wake-visible-after-launch", - action="store_true", - default=None, - help=( - "After launching visible panes, broadcast the fixed decentralized A2A wake prompt. " - "Omit for the default operator-visible launch where each Codex TUI runs its own first tick." - ), - ) - start_parser.add_argument( - "--no-wake-visible-after-launch", - dest="wake_visible_after_launch", - action="store_false", - help=( - "Keep the post-launch wake disabled while preserving the normal visible launch." - ), - ) - start_parser.add_argument( - "--auto-wake", - action=argparse.BooleanOptionalAction, - default=True, - help=( - "Run a session-scoped background wake loop so successor todos can be picked up " - "without a human sending `loopx multi-agent wake`. Use --no-auto-wake for full " - "manual takeover." - ), - ) - start_parser.add_argument( - "--auto-wake-interval-seconds", - type=float, - default=45.0, - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--no-attach", - action="store_true", - help="With visible --execute, start tmux in the background instead of attaching.", - ) - start_parser.add_argument( - "--attach", - action="store_true", - help=( - "With visible --execute, attach to tmux after launch and skip the default wake so " - "operator takeover happens first." - ), - ) - start_parser.add_argument( - "--demo-run-id", - help="Optional public-safe suffix for the isolated goal id.", - ) - start_parser.add_argument( - "--goal-id", - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--tracking-goal-id", - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--worker-loop-rounds", - type=int, - default=4, - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--configure-visible-worker-turn", - action="store_true", - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--session-name", - default="loopx-auto-research", - help="Public-safe tmux session name for visible lanes.", - ) - start_parser.add_argument( - "--workspace", - help=( - "Directory where visible Codex lanes should start. " - "Omit to use ~/loopx-auto-research//visible-workspace." - ), - ) - start_parser.add_argument( - "--create-workspace", - action="store_true", - help="Create --workspace when it does not already exist.", - ) - start_parser.add_argument( - "--keep-workspace", - action="store_true", - help=argparse.SUPPRESS, - ) - start_parser.add_argument("--cli-bin", default="loopx", help=argparse.SUPPRESS) - start_parser.add_argument("--codex-bin", default="codex", help=argparse.SUPPRESS) - start_parser.add_argument("--tmux-bin", default="tmux", help=argparse.SUPPRESS) - start_parser.add_argument( - "--reasoning-effort", - default="high", - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--launcher", - choices=["auto", "tmux"], - default="auto", - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--replace-existing", - action="store_true", - help=argparse.SUPPRESS, - ) - start_parser.add_argument( - "--codex-trust-workspace", - action=argparse.BooleanOptionalAction, - default=None, - help=argparse.SUPPRESS, - ) - - frontier_parser = auto_research_sub.add_parser( - "frontier", - help="Render a per-agent decentralized research frontier from a public fixture or live LoopX state.", - ) - add_subcommand_format(frontier_parser) - frontier_parser.add_argument( - "--fixture", - help="Path to a decentralized_auto_research_fixture_v0 JSON file.", - ) - frontier_parser.add_argument( - "--goal-id", - help="Goal id for live LoopX quota/status input. Mutually exclusive with --fixture.", - ) - frontier_parser.add_argument( - "--agent-id", - required=True, - help="Agent id whose runnable frontier should be projected.", - ) - - evidence_parser = auto_research_sub.add_parser( - "evidence", - help="Build public-safe research hypothesis/evidence records from metric evaluator outputs.", - ) - add_subcommand_format(evidence_parser) - evidence_parser.add_argument("--contract", required=True, help="Path to research_contract_v0 JSON.") - evidence_parser.add_argument( - "--eval-result", - action="append", - required=True, - help="Path to a public-safe evaluator JSON result. Repeat for dev/holdout or retry evidence.", - ) - evidence_parser.add_argument("--hypothesis-id", required=True) - evidence_parser.add_argument("--todo-id", required=True) - evidence_parser.add_argument("--agent-id", required=True) - evidence_parser.add_argument("--claimed-by", required=True) - evidence_parser.add_argument("--mechanism-family", required=True) - evidence_parser.add_argument("--hypothesis", required=True) - evidence_parser.add_argument("--parent-hypothesis-id") - evidence_parser.add_argument("--grounding-ref", action="append", default=[]) - evidence_parser.add_argument("--novelty-audit-ref") - evidence_parser.add_argument("--branch-ref") - evidence_parser.add_argument("--attempt-start", type=int, default=1) - - append_parser = auto_research_sub.add_parser( - "append-evidence", - help="Append an auto_research_evidence_packet_v0 into the LoopX rollout event log.", - ) - add_subcommand_format(append_parser) - append_parser.add_argument("--packet", required=True, help="Path to auto_research_evidence_packet_v0 JSON.") - append_parser.add_argument( - "--dry-run", - action="store_true", - help="Preview rollout events without appending them.", - ) - append_parser.add_argument( - "--output", - help="Write the append result JSON to this path without recording the path in LoopX state.", - ) - - decide_parser = auto_research_sub.add_parser( - "decide", - help="Record an evidence-bound terminal decision for one Auto Research hypothesis.", - ) - add_subcommand_format(decide_parser) - decide_parser.add_argument("--goal-id", required=True) - decide_parser.add_argument("--hypothesis-id", required=True) - decide_parser.add_argument("--outcome", choices=["promoted", "retired"], required=True) - decide_parser.add_argument( - "--reason", - choices=[ - "contradicted", - "duplicate", - "holdout_validated", - "operator_rejected", - "retry_exhausted", - "superseded", - ], - required=True, - ) - decide_parser.add_argument("--agent-id", required=True) - decide_parser.add_argument("--evidence-ref", action="append", default=[]) - decide_parser.add_argument( - "--execute", - action="store_true", - help="Append the terminal decision. Omit to preview the validated decision.", - ) - - review_parser = auto_research_sub.add_parser( - "review", - help="Record a review receipt bound to the current terminal decision and evidence graph revision.", - ) - add_subcommand_format(review_parser) - review_parser.add_argument("--goal-id", required=True) - review_parser.add_argument("--hypothesis-id", required=True) - review_parser.add_argument("--reviewer-agent-id", required=True) - review_parser.add_argument( - "--verdict", - choices=["approve", "needs_more_evidence", "reject"], - required=True, - ) - review_parser.add_argument("--evidence-ref", action="append", default=[]) - review_parser.add_argument( - "--require-independent", - action="store_true", - help="Reject a reviewer who produced the hypothesis or terminal decision.", - ) - review_parser.add_argument( - "--execute", - action="store_true", - help="Append the review receipt. Omit to preview the validated receipt.", - ) - - results_parser = auto_research_sub.add_parser( - "results", - help="Query current terminal decisions and review state by exact hypothesis id.", - ) - add_subcommand_format(results_parser) - results_parser.add_argument("--goal-id", required=True) - results_parser.add_argument("--hypothesis-id") - results_parser.add_argument("--include-history", action="store_true") - - project_results_parser = auto_research_sub.add_parser( - "project-results", - help="Project terminal Auto Research decisions into the existing Explore result log.", - ) - add_subcommand_format(project_results_parser) - project_results_parser.add_argument("--goal-id", required=True) - project_results_parser.add_argument( - "--execute", - action="store_true", - help="Append the deterministic Explore events. Omit to preview.", - ) - - artifact_receipt_parser = auto_research_sub.add_parser( - "artifact-receipt", - help=( - "Build a read-only Wish-to-Artifact receipt from one delivery " - "contract and the current Auto Research evidence." - ), - ) - add_subcommand_format(artifact_receipt_parser) - artifact_receipt_parser.add_argument( - "--contract", - required=True, - help="Path to auto_research_delivery_contract_v0 JSON.", - ) - - live_evidence_parser = auto_research_sub.add_parser( - "capture-live-evidence", - help="Build compact public-safe live Codex E2E evidence after lane-authored evidence is appended.", - ) - add_subcommand_format(live_evidence_parser) - live_evidence_parser.add_argument( - "--packet", - required=True, - help="Path to the public auto_research_evidence_packet_v0 JSON produced by a visible lane.", - ) - live_evidence_parser.add_argument( - "--append-result", - required=True, - help="Path to the JSON output from a real auto-research append-evidence run.", - ) - live_evidence_parser.add_argument("--agent-id", required=True) - live_evidence_parser.add_argument( - "--lane-count", - type=int, - default=4, - help="Accepted visible lane count to record in the compact live evidence.", - ) - live_evidence_parser.add_argument( - "--visible-lanes-accepted", - action="store_true", - help="Required acknowledgement that the visible lanes were launched and accepted.", - ) - live_evidence_parser.add_argument( - "--output", - default=LIVE_CODEX_E2E_DEFAULT_OUTPUT, - help="Output path for --execute. The path is not recorded in the evidence payload.", - ) - live_evidence_parser.add_argument( - "--execute", - action="store_true", - help="Write the compact evidence JSON to --output. Omit to preview the payload.", - ) - - worker_turn_parser = auto_research_sub.add_parser( - "worker-turn", - help="Run one LoopX-selected auto-research worker turn from quota/frontier.", - ) - add_subcommand_format(worker_turn_parser) - worker_turn_parser.add_argument("--agent-id", required=True) - worker_turn_parser.add_argument( - "--goal-id", - default=AUTO_RESEARCH_DEFAULT_GOAL_ID, - help="Research goal id whose quota/frontier this worker should obey.", - ) - worker_turn_parser.add_argument( - "--objective", - default=AUTO_RESEARCH_DEFAULT_OBJECTIVE, - help="Public-safe objective used by the built-in lightweight metric kernel.", - ) - worker_turn_parser.add_argument( - "--output-dir", - default="auto_research_lightweight_kernel", - help=argparse.SUPPRESS, - ) - worker_turn_parser.add_argument( - "--evidence-dir", - default=".local/auto-research-worker", - help="Local-only evidence output directory inside the worker workspace.", - ) - worker_turn_parser.add_argument( - "--lane-count", - type=int, - default=1, - help="Visible lane count recorded when live evidence is captured.", - ) - worker_turn_parser.add_argument( - "--visible-lanes-accepted", - action="store_true", - help="Acknowledge that the visible lanes were launched and accepted.", - ) - worker_turn_parser.add_argument( - "--live-evidence-output", - default=LIVE_CODEX_E2E_DEFAULT_OUTPUT, - help="Local filename for compact public-safe live evidence.", - ) - worker_turn_parser.add_argument( - "--complete-selected-todo", - action="store_true", - help="After a successful --execute turn, mark the selected LoopX todo done.", - ) - worker_turn_parser.add_argument( - "--execute", - action="store_true", - help="Run the selected worker action. Omit to show the plan from quota/frontier.", - ) - - worker_loop_parser = auto_research_sub.add_parser( - "worker-loop", - help="Run repeated LoopX-selected auto-research worker turns for a visible lane set.", - ) - add_subcommand_format(worker_loop_parser) - worker_loop_parser.add_argument( - "--agent-id", - action="append", - required=True, - help="Agent id to poll in loop order. Repeat for each visible worker lane.", - ) - worker_loop_parser.add_argument( - "--goal-id", - default=AUTO_RESEARCH_DEFAULT_GOAL_ID, - help="Research goal id whose quota/frontier each worker should obey.", - ) - worker_loop_parser.add_argument( - "--objective", - default=AUTO_RESEARCH_DEFAULT_OBJECTIVE, - help="Public-safe objective used by the built-in lightweight metric kernel.", - ) - worker_loop_parser.add_argument( - "--output-dir", - default="auto_research_lightweight_kernel", - help=argparse.SUPPRESS, - ) - worker_loop_parser.add_argument( - "--evidence-dir", - default=".local/auto-research-worker", - help="Local-only evidence output directory inside the worker workspace.", - ) - worker_loop_parser.add_argument( - "--lane-count", - type=int, - help="Visible lane count recorded when live evidence is captured.", - ) - worker_loop_parser.add_argument( - "--visible-lanes-accepted", - action="store_true", - help="Acknowledge that the visible lanes were launched and accepted.", - ) - worker_loop_parser.add_argument( - "--live-evidence-output", - default=LIVE_CODEX_E2E_DEFAULT_OUTPUT, - help="Local filename for compact public-safe live evidence.", - ) - worker_loop_parser.add_argument( - "--complete-selected-todo", - action="store_true", - help="After successful --execute turns, mark selected LoopX todos done.", - ) - worker_loop_parser.add_argument( - "--max-rounds", - type=int, - default=4, - help="Maximum polling rounds across the agent list.", - ) - worker_loop_parser.add_argument( - "--execute", - action="store_true", - help="Run each selected worker action. Omit to preview loop selection.", - ) - - demo_supervisor_parser = auto_research_sub.add_parser( - "demo-supervisor", - help="Plan a dry-run tmux/Codex-CLI supervisor for decentralized auto-research lanes.", - ) - add_subcommand_format(demo_supervisor_parser) - demo_supervisor_parser.add_argument( - "--goal-id", - help=( - "Research goal id whose frontier each lane should inspect. " - "Omit to use a fresh demo-local goal surface." - ), - ) - demo_supervisor_parser.add_argument( - "--demo-run-id", - help="Public-safe suffix for the generated demo goal id when --goal-id is omitted.", - ) - demo_supervisor_parser.add_argument( - "--inherit-default-goal", - action="store_true", - help=f"Opt into the shared internal demo goal {AUTO_RESEARCH_DEFAULT_GOAL_ID}.", - ) - demo_supervisor_parser.add_argument( - "--agent", - action="append", - default=[], - help=( - "Agent/lane pair as agent_id:lane_id. Repeat for each visible lane. " - "Omit to use the default LoopX auto-research demo lane set." - ), - ) - demo_supervisor_parser.add_argument( - "--session-name", - default="loopx-auto-research", - help="Public-safe tmux session name for the generated dry-run script.", - ) - demo_supervisor_parser.add_argument("--cli-bin", default="loopx", help="LoopX CLI executable name.") - demo_supervisor_parser.add_argument("--codex-bin", default="codex", help="Codex CLI executable name.") - demo_supervisor_parser.add_argument("--tmux-bin", default="tmux", help="tmux executable name.") - demo_supervisor_parser.add_argument( - "--reasoning-effort", - default="high", - help="Reasoning effort passed to visible Codex lanes through model_reasoning_effort.", - ) - demo_supervisor_parser.add_argument( - "--language", - choices=["auto", "zh", "en"], - default="auto", - help="Human-readable output language for visible role panes.", - ) - demo_supervisor_parser.add_argument( - "--execute", - action="store_true", - help=( - "Launch visible Codex CLI lanes in tmux. Omit for the default dry-run packet. " - "This only starts local visible terminals; LoopX writeback still happens through normal lane commands." - ), - ) - demo_supervisor_parser.add_argument( - "--launcher", - choices=["auto", "tmux"], - default="auto", - help="Visible process launcher for --execute. auto currently resolves to tmux.", - ) - demo_supervisor_parser.add_argument( - "--attach", - action="store_true", - help="After --execute with tmux, attach to the session. Terminal launcher opens visible windows directly.", - ) - demo_supervisor_parser.add_argument( - "--replace-existing", - action="store_true", - help="With tmux launcher, kill an existing session with the same name before launching.", - ) - demo_supervisor_parser.add_argument( - "--workspace", - help=( - "Directory where visible Codex lanes should start. Defaults to the current directory. " - "For demos, prefer an empty user-owned research workspace that shares LoopX state through " - "--registry/--runtime-root." - ), - ) - demo_supervisor_parser.add_argument( - "--create-workspace", - action="store_true", - help="Create --workspace when it does not already exist.", - ) - demo_supervisor_parser.add_argument( - "--codex-trust-workspace", - action=argparse.BooleanOptionalAction, - default=False, - help=( - "Pass a per-invocation Codex trust config for the selected visible workspace. " - "Default is off for generic supervisor launches." - ), - ) - - demo_e2e_parser = auto_research_sub.add_parser( - "demo-e2e", - help=( - "Run or preview the one-command multi-round lightweight research path and " - "launch visible Codex TUI lanes or a headless worker-loop proof." - ), - ) - add_subcommand_format(demo_e2e_parser) - demo_e2e_parser.add_argument("--agent-id", required=True) - demo_e2e_parser.add_argument( - "--agent", - action="append", - default=[], - help=( - "Optional visible worker lane as agent_id:lane_id:role_id. " - "Omit to use the default four-role worker loop." - ), - ) - demo_e2e_parser.add_argument( - "--goal-id", - help=( - "Research goal id for the demo evidence. " - "Omit to create a fresh isolated demo goal surface." - ), - ) - demo_e2e_parser.add_argument( - "--demo-run-id", - help="Public-safe suffix for the generated demo goal id when --goal-id is omitted.", - ) - demo_e2e_parser.add_argument( - "--inherit-default-goal", - action="store_true", - help=f"Opt into reusing the shared internal demo goal {AUTO_RESEARCH_DEFAULT_GOAL_ID}.", - ) - demo_e2e_parser.add_argument( - "--tracking-goal-id", - help=( - "Optional parent/productization goal id for status writeback context. " - "Visible lanes still inspect --goal-id as the research frontier." - ), - ) - demo_e2e_parser.add_argument( - "--objective", - default=AUTO_RESEARCH_DEFAULT_OBJECTIVE, - help="Compact public-safe research objective for the generated contract.", - ) - demo_e2e_parser.add_argument( - "--language", - choices=["auto", "zh", "en"], - default="auto", - help="Human-readable output language for visible role panes.", - ) - demo_e2e_parser.add_argument( - "--output-dir", - default="auto_research_lightweight_kernel", - help=argparse.SUPPRESS, - ) - demo_e2e_parser.add_argument( - "--execute", - action="store_true", - help=( - "Seed a demo-local LoopX goal queue and, by default, launch visible Codex TUI lanes " - "that work the frontier from zero. Use --headless or --run-worker-loop for the " - "non-interactive worker-loop proof path." - ), - ) - demo_e2e_parser.add_argument( - "--run-worker-loop", - action="store_true", - help=argparse.SUPPRESS, - ) - demo_e2e_parser.add_argument( - "--worker-loop-rounds", - type=int, - default=4, - help="Maximum worker-loop rounds for --execute.", - ) - demo_e2e_parser.add_argument( - "--launch-visible", - action="store_true", - help=( - "With --execute, explicitly launch the visible multi-lane supervisor. " - "This is the default unless --headless is set. " - "Visible panes alone do not make the multi-round kernel a live Codex E2E result." - ), - ) - demo_e2e_parser.add_argument( - "--headless", - action="store_true", - help="With --execute, skip visible tmux/Codex panes and return JSON only.", - ) - demo_e2e_parser.add_argument( - "--live-evidence", - help=( - "Path to compact public-safe live Codex lane evidence. " - "Raw transcripts are not read." - ), - ) - demo_e2e_parser.add_argument( - "--visible-live-evidence-wait-seconds", - type=float, - default=30.0, - help=argparse.SUPPRESS, - ) - demo_e2e_parser.add_argument( - "--configure-visible-worker-turn", - action="store_true", - help=argparse.SUPPRESS, - ) - demo_e2e_parser.add_argument( - "--wake-visible-after-launch", - action="store_true", - help=( - "After starting visible tmux panes, broadcast the fixed pane-local A2A " - "wake prompt. Each pane still runs its own LoopX tick from state." - ), - ) - demo_e2e_parser.add_argument( - "--auto-wake", - action=argparse.BooleanOptionalAction, - default=True, - help=( - "Run a session-scoped background wake loop for visible auto-research lanes. " - "Use --no-auto-wake for full manual takeover." - ), - ) - demo_e2e_parser.add_argument( - "--auto-wake-interval-seconds", - type=float, - default=45.0, - help=argparse.SUPPRESS, - ) - demo_e2e_parser.add_argument( - "--keep-workspace", - action="store_true", - help="Keep the temporary demo workspace after execution. The output payload still redacts its absolute path.", - ) - demo_e2e_parser.add_argument( - "--session-name", - default="loopx-auto-research", - help="Public-safe tmux session name when --launch-visible is set.", - ) - demo_e2e_parser.add_argument("--cli-bin", default="loopx", help="LoopX CLI executable name.") - demo_e2e_parser.add_argument("--codex-bin", default="codex", help="Codex CLI executable name.") - demo_e2e_parser.add_argument("--tmux-bin", default="tmux", help="tmux executable name.") - demo_e2e_parser.add_argument( - "--reasoning-effort", - default="high", - help="Reasoning effort passed to visible Codex lanes through model_reasoning_effort.", - ) - demo_e2e_parser.add_argument( - "--launcher", - choices=["auto", "tmux"], - default="auto", - help="Visible process launcher for --launch-visible. auto currently resolves to tmux.", - ) - demo_e2e_parser.add_argument( - "--attach", - action="store_true", - help="After visible launch with tmux, attach to the session. This is the default for --execute unless --no-attach is set.", - ) - demo_e2e_parser.add_argument( - "--no-attach", - action="store_true", - help="With the default visible --execute launch, start tmux in the background instead of attaching.", - ) - demo_e2e_parser.add_argument( - "--replace-existing", - action="store_true", - help="With tmux launcher, kill an existing session with the same name before launching.", - ) - demo_e2e_parser.add_argument( - "--workspace", - help=( - "Directory where visible Codex lanes should start when --launch-visible is set. " - "Omit to use a demo-owned clean workspace." - ), - ) - demo_e2e_parser.add_argument( - "--create-workspace", - action="store_true", - help="Create --workspace when it does not already exist.", - ) - demo_e2e_parser.add_argument( - "--codex-trust-workspace", - action=argparse.BooleanOptionalAction, - default=None, - help=( - "Pass a per-invocation Codex trust config for the visible workspace. " - "Defaults on only for the demo-owned clean workspace." - ), - ) - - -def _execute_auto_research_demo_supervisor( - payload: dict[str, object], - *, - registry_path: Path, - runtime_root_arg: str | None, - launcher: str, - tmux_bin: str, - cli_bin: str, - codex_bin: str, - attach: bool, - replace_existing: bool, - workspace: str | None, - create_workspace: bool, - codex_trust_workspace: bool, - auto_wake: bool = False, - auto_wake_interval_seconds: float = 45.0, -) -> dict[str, object]: - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - result, chosen, workspace_mode = execute_visible_multi_agent_launcher( - payload=payload, - registry=registry_path, - runtime_root=runtime_root, - requested_launcher=launcher, - tmux_bin=tmux_bin, - cli_bin=cli_bin, - codex_bin=codex_bin, - attach=attach, - replace_existing=replace_existing, - workspace=workspace, - create_workspace=create_workspace, - cwd=Path.cwd(), - codex_trust_workspace=codex_trust_workspace, - auto_wake=auto_wake, - auto_wake_interval_seconds=auto_wake_interval_seconds, - launch_result_schema="auto_research_demo_launch_result_v0", - lane_default="research-lane", - ) - payload["mode"] = "executed_visible_launch" - payload["launch_result"] = result - boundary = payload.get("boundary") - if isinstance(boundary, dict): - boundary.update( - { - "dry_run_plan_only": False, - "starts_tmux": chosen == "tmux", - "opens_terminal": False, - "runs_codex": True, - "writes_loopx_state": False, - "spends_loopx_quota": False, - "external_service_call": False, - "workspace_mode": workspace_mode, - "workspace_write_scope": "selected_visible_workspace_only", - "codex_trust_workspace": codex_trust_workspace, - "codex_trust_scope": ( - "persisted_selected_workspace_and_git_root" - if codex_trust_workspace - else "native_codex_trust_prompt" - ), - "shared_state_route": "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", - "shared_goal_surface": True, - "all_lane_workspace_isolation": False, - "mutation_isolation_policy": ( - "only mutating research-executor attempts require a claimed git worktree " - "or equivalent execution boundary" - ), - } - ) - return payload - - -def handle_auto_research_command( - args: argparse.Namespace, - *, - registry_path: Path, - runtime_root_arg: str | None, - output_format: FormatSelector, - print_payload: PrintPayload, -) -> int: - try: - if args.auto_research_command == "contract": - payload = build_auto_research_user_contract( - args.open_question, - max_todos=args.max_todos, - output_language=args.language, - preset_id=args.preset, - ) - elif args.auto_research_command == "start": - preset_id = normalize_auto_research_preset_id(args.preset) - launch_visible = bool(args.execute and not args.headless) - visible_policy = resolve_visible_launch_policy( - args, - launch_visible=launch_visible, - default_wake_allowed=False, - default_attach_allowed=launch_visible, - attach_wake_conflict_message=( - "--attach cannot be combined with --wake-visible-after-launch; " - "choose operator takeover (--attach) or visible role wake (--no-attach)" - ), - ) - goal_id, goal_surface_mode = _resolve_demo_goal_surface( - goal_id=args.goal_id, - demo_run_id=args.demo_run_id, - inherit_default_goal=False, - ) - output_language = infer_auto_research_output_language( - args.open_question, - output_language=args.language, - ) - - def append_start_evidence(packet_path: str) -> dict[str, object]: - return _append_auto_research_rollout_events( - packet_path=packet_path, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - dry_run=False, - ) - - visible_launcher: Callable[[dict[str, object], Path, str | None, Path], dict[str, object]] | None = None - visible_wake: Callable[[str, list[str]], dict[str, object]] | None = None - codex_trust_visible_workspace = resolve_codex_trust_workspace( - args, - launch_visible=visible_policy.launch_visible, - default=True, - ) - prepared_preset_workspace: dict[str, object] | None = None - - def start_workspace_policy(_default_workspace: Path) -> tuple[str | None, bool, bool]: - nonlocal prepared_preset_workspace - default_start_workspace = args.workspace is None - workspace = ( - _default_auto_research_start_workspace(goal_id) - if default_start_workspace - else args.workspace - ) - create_workspace = True if default_start_workspace else args.create_workspace - if preset_id == AUTO_RESEARCH_KNN_DEMO_PRESET_ID and workspace: - prepared_preset_workspace = materialize_knn_demo_workspace( - workspace, - goal_id=goal_id, - ) - create_workspace = True - return ( - workspace, - create_workspace, - codex_trust_visible_workspace, - ) - - visible_launcher = make_visible_launcher_callback( - launch_visible=visible_policy.launch_visible, - launch_executor=_execute_auto_research_demo_supervisor, - launcher=args.launcher, - tmux_bin=args.tmux_bin, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - attach=visible_policy.attach, - replace_existing=args.replace_existing, - workspace_policy=start_workspace_policy, - auto_wake=bool(visible_policy.launch_visible and args.auto_wake), - auto_wake_interval_seconds=args.auto_wake_interval_seconds, - ) - - if visible_policy.launch_visible: - visible_wake = make_visible_wake_callback(tmux_bin=args.tmux_bin) - - payload = run_auto_research_demo_e2e( - agent_id=AUTO_RESEARCH_START_AGENT_ID, - goal_id=goal_id, - goal_surface_mode=goal_surface_mode, - tracking_goal_id=args.tracking_goal_id, - preset_id=preset_id, - objective=args.open_question, - output_dir="auto_research_lightweight_kernel", - execute=args.execute, - run_worker_loop=bool(args.execute and args.headless), - worker_loop_rounds=args.worker_loop_rounds, - configure_visible_worker_turn=args.configure_visible_worker_turn, - launch_visible=visible_policy.launch_visible, - keep_workspace=args.keep_workspace, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - session_name=args.session_name, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - tmux_bin=args.tmux_bin, - reasoning_effort=args.reasoning_effort, - output_language=output_language, - live_evidence_path=None, - append_evidence=append_start_evidence, - visible_launcher=visible_launcher, - visible_wake=visible_wake, - wake_visible_after_launch=visible_policy.wake_visible_after_launch, - ) - if prepared_preset_workspace: - payload["preset_workspace"] = prepared_preset_workspace - elif args.auto_research_command == "frontier": - if bool(args.fixture) == bool(args.goal_id): - raise ValueError(f"auto-research {args.auto_research_command} requires exactly one of --fixture or --goal-id") - if args.fixture: - fixture = load_auto_research_fixture(args.fixture) - payload = build_auto_research_projection( - fixture, - agent_id=args.agent_id, - ) - else: - status_payload = collect_status( - registry_path=registry_path, - runtime_root_override=runtime_root_arg, - limit=5, - goal_id=args.goal_id, - ) - quota_payload = build_quota_should_run( - status_payload, - goal_id=args.goal_id, - agent_id=args.agent_id, - ) - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - rollout_events = load_rollout_events( - rollout_event_log_path(runtime_root, args.goal_id) - ) - payload = build_live_auto_research_projection( - goal_id=args.goal_id, - agent_id=args.agent_id, - quota_payload=quota_payload, - rollout_events=rollout_events, - ) - elif args.auto_research_command == "evidence": - payload = load_auto_research_evidence_packet_inputs( - contract_path=args.contract, - eval_result_paths=args.eval_result, - hypothesis_id=args.hypothesis_id, - todo_id=args.todo_id, - agent_id=args.agent_id, - claimed_by=args.claimed_by, - mechanism_family=args.mechanism_family, - hypothesis=args.hypothesis, - parent_hypothesis_id=args.parent_hypothesis_id, - grounding_refs=args.grounding_ref, - novelty_audit_ref=args.novelty_audit_ref, - branch_ref=args.branch_ref, - attempt_start=args.attempt_start, - ) - elif args.auto_research_command == "append-evidence": - payload = _append_auto_research_rollout_events( - packet_path=args.packet, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - dry_run=args.dry_run, - ) - if args.output: - output_path = Path(args.output).expanduser() - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text( - json.dumps(payload, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - elif args.auto_research_command == "decide": - payload = decide_terminal_result( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=args.goal_id, - hypothesis_id=args.hypothesis_id, - outcome=args.outcome, - reason=args.reason, - decided_by=args.agent_id, - evidence_refs=args.evidence_ref, - dry_run=not args.execute, - ) - elif args.auto_research_command == "review": - payload = review_terminal_result( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=args.goal_id, - hypothesis_id=args.hypothesis_id, - reviewer_agent_id=args.reviewer_agent_id, - verdict=args.verdict, - evidence_refs=args.evidence_ref, - require_independent=args.require_independent, - dry_run=not args.execute, - ) - elif args.auto_research_command == "results": - payload = query_terminal_results( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=args.goal_id, - hypothesis_id=args.hypothesis_id, - include_history=args.include_history, - ) - elif args.auto_research_command == "project-results": - payload = project_terminal_results( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=args.goal_id, - dry_run=not args.execute, - ) - elif args.auto_research_command == "artifact-receipt": - payload = load_auto_research_artifact_receipt( - contract_path=args.contract, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - ) - elif args.auto_research_command == "capture-live-evidence": - payload = capture_live_codex_e2e_evidence( - packet_path=args.packet, - append_result_path=args.append_result, - agent_id=args.agent_id, - lane_count=args.lane_count, - visible_lanes_accepted=args.visible_lanes_accepted, - ) - if args.execute: - output_path = Path(args.output).expanduser() - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text( - json.dumps(payload, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - elif args.auto_research_command == "worker-turn": - def append_worker_evidence(packet_path: str) -> dict[str, object]: - return _append_auto_research_rollout_events( - packet_path=packet_path, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - dry_run=False, - ) - - payload = run_auto_research_worker_turn( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=args.goal_id, - agent_id=args.agent_id, - objective=args.objective, - workspace=Path.cwd(), - output_dir=args.output_dir, - evidence_dir=args.evidence_dir, - execute=args.execute, - append_evidence=append_worker_evidence if args.execute else None, - lane_count=args.lane_count, - visible_lanes_accepted=args.visible_lanes_accepted, - live_evidence_output=args.live_evidence_output, - complete_selected_todo=args.complete_selected_todo, - ) - elif args.auto_research_command == "worker-loop": - def append_loop_evidence(packet_path: str) -> dict[str, object]: - return _append_auto_research_rollout_events( - packet_path=packet_path, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - dry_run=False, - ) - - payload = run_auto_research_worker_loop( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=args.goal_id, - agent_ids=args.agent_id, - objective=args.objective, - workspace=Path.cwd(), - output_dir=args.output_dir, - evidence_dir=args.evidence_dir, - execute=args.execute, - append_evidence=append_loop_evidence if args.execute else None, - lane_count=args.lane_count, - visible_lanes_accepted=args.visible_lanes_accepted, - live_evidence_output=args.live_evidence_output, - complete_selected_todo=args.complete_selected_todo, - max_rounds=args.max_rounds, - ) - elif args.auto_research_command == "demo-supervisor": - goal_id, goal_surface_mode = _resolve_demo_goal_surface( - goal_id=args.goal_id, - demo_run_id=args.demo_run_id, - inherit_default_goal=args.inherit_default_goal, - ) - payload = build_auto_research_demo_supervisor_plan( - goal_id=goal_id, - agent_specs=args.agent, - session_name=args.session_name, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - tmux_bin=args.tmux_bin, - reasoning_effort=args.reasoning_effort, - output_language=infer_auto_research_output_language( - "", - output_language=args.language, - ), - ) - payload["goal_surface_route"] = { - "schema_version": "auto_research_demo_goal_surface_v0", - "mode": goal_surface_mode, - "goal_id": goal_id, - "fresh_by_default": goal_surface_mode == "fresh_demo_goal", - "reuses_default_internal_goal": goal_id == AUTO_RESEARCH_DEFAULT_GOAL_ID, - "default_internal_goal_id": AUTO_RESEARCH_DEFAULT_GOAL_ID, - } - if args.execute: - payload = _execute_auto_research_demo_supervisor( - payload, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - launcher=args.launcher, - tmux_bin=args.tmux_bin, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - attach=args.attach, - replace_existing=args.replace_existing, - workspace=args.workspace, - create_workspace=args.create_workspace, - codex_trust_workspace=args.codex_trust_workspace, - ) - elif args.auto_research_command == "demo-e2e": - if args.headless and args.launch_visible: - raise ValueError("--headless cannot be combined with --launch-visible") - goal_id, goal_surface_mode = _resolve_demo_goal_surface( - goal_id=args.goal_id, - demo_run_id=args.demo_run_id, - inherit_default_goal=args.inherit_default_goal, - ) - - def append_demo_e2e_evidence(packet_path: str) -> dict[str, object]: - return _append_auto_research_rollout_events( - packet_path=packet_path, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - dry_run=False, - ) - - visible_launcher: Callable[[dict[str, object], Path, str | None, Path], dict[str, object]] | None = None - visible_wake: Callable[[str, list[str]], dict[str, object]] | None = None - auto_visible_launch = bool(args.execute and not args.headless) - launch_visible = bool(args.launch_visible or auto_visible_launch) - visible_policy = resolve_visible_launch_policy( - args, - launch_visible=launch_visible, - default_wake_allowed=auto_visible_launch, - default_attach_allowed=auto_visible_launch, - attach_wake_conflict_message=( - "--wake-visible-after-launch cannot be combined with --attach; " - "wake evidence must be recorded before operator takeover" - ), - ) - def demo_workspace_policy(default_workspace: Path) -> tuple[str | None, bool, bool]: - demo_owned_workspace = args.workspace is None - return ( - str(default_workspace / "visible-user-workspace") - if demo_owned_workspace - else args.workspace, - True if demo_owned_workspace else args.create_workspace, - resolve_codex_trust_workspace( - args, - launch_visible=visible_policy.launch_visible, - default=demo_owned_workspace, - ), - ) - - visible_launcher = make_visible_launcher_callback( - launch_visible=visible_policy.launch_visible, - launch_executor=_execute_auto_research_demo_supervisor, - launcher=args.launcher, - tmux_bin=args.tmux_bin, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - attach=visible_policy.attach, - replace_existing=args.replace_existing, - workspace_policy=demo_workspace_policy, - auto_wake=bool(visible_policy.launch_visible and args.auto_wake), - auto_wake_interval_seconds=args.auto_wake_interval_seconds, - ) - - if visible_policy.launch_visible: - visible_wake = make_visible_wake_callback(tmux_bin=args.tmux_bin) - - run_hidden_worker_loop = bool(args.execute and (args.headless or args.run_worker_loop)) - payload = run_auto_research_demo_e2e( - agent_id=args.agent_id, - goal_id=goal_id, - goal_surface_mode=goal_surface_mode, - agent_specs=args.agent, - tracking_goal_id=args.tracking_goal_id, - objective=args.objective, - output_dir=args.output_dir, - execute=args.execute, - run_worker_loop=run_hidden_worker_loop, - worker_loop_rounds=args.worker_loop_rounds, - configure_visible_worker_turn=args.configure_visible_worker_turn, - launch_visible=visible_policy.launch_visible, - keep_workspace=args.keep_workspace, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - session_name=args.session_name, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - tmux_bin=args.tmux_bin, - reasoning_effort=args.reasoning_effort, - output_language=infer_auto_research_output_language( - args.objective, - output_language=args.language, - ), - live_evidence_path=args.live_evidence, - append_evidence=append_demo_e2e_evidence, - visible_launcher=visible_launcher, - visible_wake=visible_wake, - wake_visible_after_launch=visible_policy.wake_visible_after_launch, - visible_live_evidence_wait_seconds=args.visible_live_evidence_wait_seconds, - ) - else: - raise ValueError( - "auto-research requires the `contract`, `frontier`, `evidence`, " - "`append-evidence`, `decide`, `review`, `results`, " - "`project-results`, `artifact-receipt`, `capture-live-evidence`, " - "`worker-turn`, `worker-loop`, `demo-supervisor`, `demo-e2e`, " - "or `start` subcommand" - ) - except Exception as exc: - error_code, public_error = _public_auto_research_error(exc) - payload = { - "ok": False, - "mode": "auto-research", - "error_code": error_code, - "error": public_error, - } - print_payload(payload, output_format(args, "auto_research_format"), render_auto_research_markdown) - return 0 if payload.get("ok") else 1 diff --git a/demo/auto_research/core.py b/demo/auto_research/core.py deleted file mode 100644 index ebf7e85a77..0000000000 --- a/demo/auto_research/core.py +++ /dev/null @@ -1,11 +0,0 @@ -from __future__ import annotations - -# New lightweight research kernel. Keep this file small: user-facing -# auto-research additions should stay in core/kernel and wrap generic runners. -from .kernel import ( - LIGHTWEIGHT_AUTO_RESEARCH_EVIDENCE_SCHEMA_VERSION, # noqa: F401 - LIGHTWEIGHT_AUTO_RESEARCH_HYPOTHESIS_SCHEMA_VERSION, # noqa: F401 - LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION, # noqa: F401 - lightweight_hypothesis, # noqa: F401 - run_lightweight_auto_research, # noqa: F401 -) diff --git a/demo/auto_research/defaults.py b/demo/auto_research/defaults.py deleted file mode 100644 index b333a04cbc..0000000000 --- a/demo/auto_research/defaults.py +++ /dev/null @@ -1,79 +0,0 @@ -from __future__ import annotations - -from ..multi_agent.contract import build_three_layer_minimality_contract -from .knn_demo_workspace import ( - KNN_DEMO_CONTRACT_FILE, - KNN_DEMO_DEV_EVAL_COMMAND, - KNN_DEMO_EDITABLE_SCOPE, - KNN_DEMO_HOLDOUT_EVAL_COMMAND, - KNN_DEMO_PROTECTED_SCOPE, -) - - -AUTO_RESEARCH_DEFAULT_GOAL_ID = "loopx-auto-research-demo" -AUTO_RESEARCH_DEFAULT_OBJECTIVE = ( - "Improve a bounded research candidate through public-safe multi-agent evidence." -) -AUTO_RESEARCH_PRESET_ID = "auto_research_thin_preset" -AUTO_RESEARCH_KNN_DEMO_PRESET_ID = "knn-demo" -AUTO_RESEARCH_SUPPORTED_PRESET_IDS = (AUTO_RESEARCH_KNN_DEMO_PRESET_ID,) - -AUTO_RESEARCH_KNN_DEMO_CONTEXT = { - "schema_version": "auto_research_preset_context_v0", - "preset_id": AUTO_RESEARCH_KNN_DEMO_PRESET_ID, - "source": "built_in_demo_preset", - "baseline_source": "generated_knn_benchmark_workspace", - "question_text_supplies_baseline": False, - "metric_name": "speedup", - "baseline_metric": 1.0, - "workspace_materializer": "built_in_knn_speedup_workspace_v0", - "benchmark_contract_file": KNN_DEMO_CONTRACT_FILE, - "editable_scope": list(KNN_DEMO_EDITABLE_SCOPE), - "protected_scope": list(KNN_DEMO_PROTECTED_SCOPE), - "dev_eval_command": KNN_DEMO_DEV_EVAL_COMMAND, - "holdout_eval_command": KNN_DEMO_HOLDOUT_EVAL_COMMAND, - "holdout_split": "test", - "claim_boundary": ( - "The KNN baseline is a generated benchmark workspace. Only solution.py is " - "editable; improvement claims require public-safe dev and held-out command output." - ), -} - - -def build_auto_research_layer_contract() -> dict[str, object]: - """Return the reusable layer split for auto-research as a thin preset.""" - - return build_three_layer_minimality_contract( - product_id="auto-research", - preset_id=AUTO_RESEARCH_PRESET_ID, - user_intent_fields=[ - "topic_or_objective", - "rounds", - "role_overrides", - "data_or_eval_entrypoint", - ], - preset_responsibilities=[ - "research_roles", - "handoff_hints", - "metric_contract_hints", - "domain_defaults", - ], - preset_forbidden_mechanics=[ - "multi_agent_runner", - "real_codex_tui_panes", - "workspace_and_trust_safe_launch", - "decentralized_a2a_driver", - "pane_local_a2a_status_check", - "todo_evidence_status_protocol", - "compact_human_status", - "default_loopx_skill_bootstrap", - "fixed_a2a_wake_prompt", - "kernel_default_skill_prompting", - ], - extension_points=[ - "role_overrides", - "metric_adapter", - "evidence_packet_adapter", - "data_or_eval_entrypoint", - ], - ) diff --git a/demo/auto_research/delivery_contract.py b/demo/auto_research/delivery_contract.py deleted file mode 100644 index 4bd8e15246..0000000000 --- a/demo/auto_research/delivery_contract.py +++ /dev/null @@ -1,526 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -from collections.abc import Mapping, Sequence -from typing import Any, NamedTuple - -from .evidence_packet import ( - _compact_public_text, - _compact_public_text_list, - _compact_public_token, - validate_research_contract, -) - - -AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION = ( - "auto_research_delivery_contract_v0" -) -AUTO_RESEARCH_CONTRACT_LINEAGE_SCHEMA_VERSION = ( - "auto_research_contract_lineage_v0" -) - -MAX_ASSUMPTIONS = 8 -MAX_NON_GOALS = 8 -MAX_REQUIRED_ARTIFACTS = 12 -MAX_ACCEPTANCE_CRITERIA = 12 -MAX_REENTRY_CONDITIONS = 8 - -_DELIVERY_CONTRACT_FIELDS = { - "schema_version", - "contract_ref", - "contract_revision", - "wish", - "research_contract", - "required_artifacts", - "acceptance_criteria", - "failure_policy", -} -_WISH_FIELDS = { - "wish_id", - "original_text", - "intent_summary", - "assumptions", - "non_goals", -} -_ARTIFACT_FIELDS = {"artifact_ref", "kind", "description", "required"} -_CRITERION_FIELDS = { - "criterion_id", - "description", - "hypothesis_id", - "required_artifact_refs", - "requires_independent_review", - "required", -} -_FAILURE_POLICY_FIELDS = {"fallback_artifact_refs", "reentry_conditions"} -AUTO_RESEARCH_CONTRACT_LINEAGE_FIELDS = ( - "wish_id", - "contract_ref", - "contract_revision", -) - - -class AutoResearchContractLineage(NamedTuple): - wish_id: str - contract_ref: str - contract_revision: str - - -class AutoResearchContractLineageError(ValueError): - pass - - -def normalize_auto_research_contract_lineage( - value: Mapping[str, Any], - *, - field: str, - required: bool = False, -) -> AutoResearchContractLineage | None: - supplied = { - name - for name in AUTO_RESEARCH_CONTRACT_LINEAGE_FIELDS - if str(value.get(name) or "").strip() - } - if not supplied: - if required: - raise AutoResearchContractLineageError( - f"{field} requires complete contract lineage" - ) - return None - if supplied != set(AUTO_RESEARCH_CONTRACT_LINEAGE_FIELDS): - raise AutoResearchContractLineageError( - f"{field} contract lineage must provide wish_id, contract_ref, " - "and contract_revision together" - ) - try: - return AutoResearchContractLineage( - *( - _compact_public_text( - value.get(name), - field=f"{field}.{name}", - max_len=180, - ) - for name in AUTO_RESEARCH_CONTRACT_LINEAGE_FIELDS - ) - ) - except ValueError as exc: - raise AutoResearchContractLineageError(str(exc)) from exc - - -def auto_research_contract_lineage_matches( - value: Mapping[str, Any], - *, - expected: AutoResearchContractLineage | None, -) -> bool: - try: - return ( - normalize_auto_research_contract_lineage( - value, - field="contract_lineage", - ) - == expected - ) - except ValueError: - return False - - -def auto_research_contract_lineage_dict( - lineage: AutoResearchContractLineage, -) -> dict[str, str]: - return dict(zip(AUTO_RESEARCH_CONTRACT_LINEAGE_FIELDS, lineage)) - - -def _strict_mapping( - value: object, - *, - field: str, - allowed: set[str], -) -> Mapping[str, Any]: - if not isinstance(value, Mapping): - raise ValueError(f"{field} must be an object") - unexpected = sorted(set(value) - allowed) - if unexpected: - raise ValueError( - f"{field} contains unsupported fields: {', '.join(unexpected)}" - ) - return value - - -def _bounded_list( - value: object, - *, - field: str, - maximum: int, -) -> Sequence[Any]: - if not isinstance(value, Sequence) or isinstance(value, (str, bytes)): - raise ValueError(f"{field} must be a list") - if len(value) > maximum: - raise ValueError(f"{field} must contain at most {maximum} items") - return value - - -def _boolean(value: object, *, field: str, default: bool) -> bool: - if value is None: - return default - if not isinstance(value, bool): - raise ValueError(f"{field} must be a boolean") - return value - - -def _canonical_digest(value: Mapping[str, Any]) -> str: - encoded = json.dumps( - value, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - return f"sha256:{hashlib.sha256(encoded).hexdigest()}" - - -def _wish_id(*, goal_id: str, original_text: str) -> str: - encoded = f"{goal_id}\n{original_text}".encode("utf-8") - return f"wish_{hashlib.sha256(encoded).hexdigest()[:16]}" - - -def _normalize_wish( - value: object, - *, - goal_id: str, -) -> dict[str, Any]: - wish = _strict_mapping( - value, - field="delivery_contract.wish", - allowed=_WISH_FIELDS, - ) - original_text = _compact_public_text( - wish.get("original_text"), - field="delivery_contract.wish.original_text", - max_len=500, - ) - derived_wish_id = _wish_id( - goal_id=goal_id, - original_text=original_text, - ) - provided_wish_id = wish.get("wish_id") - if provided_wish_id and provided_wish_id != derived_wish_id: - raise ValueError("delivery_contract.wish.wish_id does not match the wish") - return { - "wish_id": derived_wish_id, - "original_text": original_text, - "intent_summary": _compact_public_text( - wish.get("intent_summary"), - field="delivery_contract.wish.intent_summary", - max_len=300, - ), - "assumptions": _compact_public_text_list( - _bounded_list( - wish.get("assumptions") or [], - field="delivery_contract.wish.assumptions", - maximum=MAX_ASSUMPTIONS, - ), - field="delivery_contract.wish.assumptions", - ), - "non_goals": _compact_public_text_list( - _bounded_list( - wish.get("non_goals") or [], - field="delivery_contract.wish.non_goals", - maximum=MAX_NON_GOALS, - ), - field="delivery_contract.wish.non_goals", - ), - } - - -def _normalize_required_artifacts(value: object) -> list[dict[str, Any]]: - required_artifacts: list[dict[str, Any]] = [] - seen_artifact_refs: set[str] = set() - for index, item in enumerate( - _bounded_list( - value, - field="delivery_contract.required_artifacts", - maximum=MAX_REQUIRED_ARTIFACTS, - ) - ): - artifact = _strict_mapping( - item, - field=f"delivery_contract.required_artifacts[{index}]", - allowed=_ARTIFACT_FIELDS, - ) - artifact_ref = _compact_public_text( - artifact.get("artifact_ref"), - field=f"delivery_contract.required_artifacts[{index}].artifact_ref", - max_len=180, - ) - if artifact_ref in seen_artifact_refs: - raise ValueError( - "delivery_contract.required_artifacts artifact_ref values " - "must be unique" - ) - seen_artifact_refs.add(artifact_ref) - required_artifacts.append( - { - "artifact_ref": artifact_ref, - "kind": _compact_public_token( - artifact.get("kind"), - field=f"delivery_contract.required_artifacts[{index}].kind", - ), - "description": _compact_public_text( - artifact.get("description"), - field=( - "delivery_contract.required_artifacts" - f"[{index}].description" - ), - max_len=300, - ), - "required": _boolean( - artifact.get("required"), - field=( - "delivery_contract.required_artifacts" - f"[{index}].required" - ), - default=True, - ), - } - ) - if not required_artifacts: - raise ValueError("delivery_contract requires at least one artifact") - if not any(item["required"] for item in required_artifacts): - raise ValueError( - "delivery_contract requires at least one required artifact" - ) - return required_artifacts - - -def _normalize_acceptance_criteria( - value: object, - *, - artifact_refs: set[str], -) -> list[dict[str, Any]]: - acceptance_criteria: list[dict[str, Any]] = [] - seen_criterion_ids: set[str] = set() - for index, item in enumerate( - _bounded_list( - value, - field="delivery_contract.acceptance_criteria", - maximum=MAX_ACCEPTANCE_CRITERIA, - ) - ): - criterion = _strict_mapping( - item, - field=f"delivery_contract.acceptance_criteria[{index}]", - allowed=_CRITERION_FIELDS, - ) - criterion_id = _compact_public_token( - criterion.get("criterion_id"), - field=f"delivery_contract.acceptance_criteria[{index}].criterion_id", - ) - if criterion_id in seen_criterion_ids: - raise ValueError( - "delivery_contract.acceptance_criteria criterion_id values " - "must be unique" - ) - seen_criterion_ids.add(criterion_id) - required_refs = _compact_public_text_list( - _bounded_list( - criterion.get("required_artifact_refs") or [], - field=( - "delivery_contract.acceptance_criteria" - f"[{index}].required_artifact_refs" - ), - maximum=MAX_REQUIRED_ARTIFACTS, - ), - field=( - "delivery_contract.acceptance_criteria" - f"[{index}].required_artifact_refs" - ), - ) - unknown_refs = sorted(set(required_refs) - artifact_refs) - if unknown_refs: - raise ValueError( - "acceptance criteria reference undeclared artifacts: " - + ", ".join(unknown_refs) - ) - if len(set(required_refs)) != len(required_refs): - raise ValueError( - "acceptance criteria required_artifact_refs must be unique" - ) - acceptance_criteria.append( - { - "criterion_id": criterion_id, - "description": _compact_public_text( - criterion.get("description"), - field=( - "delivery_contract.acceptance_criteria" - f"[{index}].description" - ), - max_len=300, - ), - "hypothesis_id": _compact_public_token( - criterion.get("hypothesis_id"), - field=( - "delivery_contract.acceptance_criteria" - f"[{index}].hypothesis_id" - ), - ), - "required_artifact_refs": required_refs, - "requires_independent_review": _boolean( - criterion.get("requires_independent_review"), - field=( - "delivery_contract.acceptance_criteria" - f"[{index}].requires_independent_review" - ), - default=True, - ), - "required": _boolean( - criterion.get("required"), - field=( - "delivery_contract.acceptance_criteria" - f"[{index}].required" - ), - default=True, - ), - } - ) - if not acceptance_criteria: - raise ValueError( - "delivery_contract requires at least one acceptance criterion" - ) - if not any(item["required"] for item in acceptance_criteria): - raise ValueError( - "delivery_contract requires at least one required acceptance criterion" - ) - return acceptance_criteria - - -def _normalize_failure_policy( - value: object, - *, - artifact_refs: set[str], -) -> dict[str, list[str]]: - failure_policy = _strict_mapping( - value or {}, - field="delivery_contract.failure_policy", - allowed=_FAILURE_POLICY_FIELDS, - ) - fallback_artifact_refs = _compact_public_text_list( - _bounded_list( - failure_policy.get("fallback_artifact_refs") or [], - field="delivery_contract.failure_policy.fallback_artifact_refs", - maximum=MAX_REQUIRED_ARTIFACTS, - ), - field="delivery_contract.failure_policy.fallback_artifact_refs", - ) - unknown_fallbacks = sorted(set(fallback_artifact_refs) - artifact_refs) - if unknown_fallbacks: - raise ValueError( - "failure policy references undeclared artifacts: " - + ", ".join(unknown_fallbacks) - ) - return { - "fallback_artifact_refs": fallback_artifact_refs, - "reentry_conditions": _compact_public_text_list( - _bounded_list( - failure_policy.get("reentry_conditions") or [], - field="delivery_contract.failure_policy.reentry_conditions", - maximum=MAX_REENTRY_CONDITIONS, - ), - field="delivery_contract.failure_policy.reentry_conditions", - ), - } - - -def normalize_auto_research_delivery_contract( - value: Mapping[str, Any], -) -> dict[str, Any]: - raw = _strict_mapping( - value, - field="delivery_contract", - allowed=_DELIVERY_CONTRACT_FIELDS, - ) - if raw.get("schema_version") != AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION: - raise ValueError( - "delivery_contract.schema_version must be " - f"{AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION}" - ) - research_contract = validate_research_contract( - dict( - _strict_mapping( - raw.get("research_contract"), - field="delivery_contract.research_contract", - allowed={ - "schema_version", - "goal_id", - "research_objective", - "editable_scope", - "protected_scope", - "metric", - "dev_eval", - "holdout_eval", - "promotion_policy", - }, - ) - ) - ) - required_artifacts = _normalize_required_artifacts( - raw.get("required_artifacts") - ) - artifact_refs = { - str(artifact["artifact_ref"]) for artifact in required_artifacts - } - normalized = { - "schema_version": AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION, - "contract_ref": _compact_public_text( - raw.get("contract_ref"), - field="delivery_contract.contract_ref", - max_len=160, - ), - "wish": _normalize_wish( - raw.get("wish"), - goal_id=research_contract["goal_id"], - ), - "research_contract": research_contract, - "required_artifacts": required_artifacts, - "acceptance_criteria": _normalize_acceptance_criteria( - raw.get("acceptance_criteria"), - artifact_refs=artifact_refs, - ), - "failure_policy": _normalize_failure_policy( - raw.get("failure_policy"), - artifact_refs=artifact_refs, - ), - } - contract_revision = _canonical_digest(normalized) - provided_revision = raw.get("contract_revision") - if provided_revision and provided_revision != contract_revision: - raise ValueError( - "delivery_contract.contract_revision does not match normalized content" - ) - normalized["contract_revision"] = contract_revision - return normalized - - -def build_auto_research_contract_lineage( - research_contract: Mapping[str, Any], - *, - delivery_contract: Mapping[str, Any] | None = None, -) -> dict[str, Any]: - if delivery_contract: - normalized_delivery = normalize_auto_research_delivery_contract( - delivery_contract - ) - return { - "schema_version": AUTO_RESEARCH_CONTRACT_LINEAGE_SCHEMA_VERSION, - "contract_revision": normalized_delivery["contract_revision"], - "contract_ref": normalized_delivery["contract_ref"], - "wish_id": normalized_delivery["wish"]["wish_id"], - "delivery_contract_schema_version": ( - AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION - ), - } - return { - "schema_version": AUTO_RESEARCH_CONTRACT_LINEAGE_SCHEMA_VERSION, - "contract_revision": _canonical_digest(dict(research_contract)), - "contract_ref": None, - "wish_id": None, - "delivery_contract_schema_version": None, - } diff --git a/demo/auto_research/demo_e2e.py b/demo/auto_research/demo_e2e.py deleted file mode 100644 index 938e80d0ee..0000000000 --- a/demo/auto_research/demo_e2e.py +++ /dev/null @@ -1,1835 +0,0 @@ -from __future__ import annotations - -import json -import shlex -import tempfile -import time -from collections.abc import Callable, Sequence -from pathlib import Path - -from .bootstrap_contract import ( - auto_research_contract_command_text, - auto_research_start_command_text, - build_auto_research_contract_acceptance, - build_auto_research_live_worker_proof, -) -from .demo_supervisor import build_auto_research_demo_supervisor_plan -from .defaults import AUTO_RESEARCH_DEFAULT_GOAL_ID -from .live_evidence import load_live_codex_e2e_evidence -from .preset import ( - auto_research_seed_action_for_role, - auto_research_seed_title, - default_auto_research_agent_specs, -) -from .rollout_append import append_auto_research_rollout_events -from .user_contract import build_auto_research_user_contract -from .worker_loop import run_auto_research_worker_loop -from .worker_runtime import load_auto_research_worker_frontier -from ..multi_agent.collective_round_ledger import ( - build_multi_agent_collective_round_ledger, -) -from loopx.control_plane.todos.contract import normalize_required_write_scopes - - -AppendEvidence = Callable[[str], dict[str, object]] -VisibleLauncher = Callable[..., dict[str, object]] -VisibleWake = Callable[[str, Sequence[str]], dict[str, object]] - -AUTO_RESEARCH_DEMO_E2E_SCHEMA_VERSION = "auto_research_demo_e2e_result_v0" -AUTO_RESEARCH_SEED_ACTION_CHAIN = ( - "write_research_contract", - "propose_hypothesis", - "run_dev_eval", - "summarize_evidence", -) -AUTO_RESEARCH_SEED_ACTION_ORDER = { - action: index for index, action in enumerate(AUTO_RESEARCH_SEED_ACTION_CHAIN) -} -AUTO_RESEARCH_SEED_PREREQUISITE_ACTION_BY_ACTION = dict( - zip(AUTO_RESEARCH_SEED_ACTION_CHAIN[1:], AUTO_RESEARCH_SEED_ACTION_CHAIN) -) -AUTO_RESEARCH_DEMO_CONTROL_WRITE_SCOPE = ("examples/**", "experiments/**", ".local/**") -AUTO_RESEARCH_DEMO_AVAILABLE_CAPABILITIES = ("benchmark_runner",) - - -def _prepare_visible_demo_workspace_route( - *, - control_project: Path, - lanes: list[dict[str, object]], -) -> dict[str, object]: - """Keep visible Codex TUIs off demo-local git worktrees. - - The demo still shares one LoopX state surface. Mutating attempts can claim - their own execution boundary from inside the role, but the first visible TUI - screen must not start in a freshly-created git worktree because Codex will - stop on a trust prompt before the user sees the auto-research flow. - """ - - (control_project / ".gitignore").write_text(".local/\n", encoding="utf-8") - - peer_count = 0 - for lane in lanes: - agent_id = str(lane.get("agent_id") or "").strip() - if not agent_id: - continue - lane["workspace_role"] = "shared_visible_tui_workspace" - peer_count += 1 - - return { - "schema_version": "auto_research_visible_demo_workspace_route_v0", - "shared_goal_surface": "demo_local_loopx_registry_and_runtime", - "peer_workspace": "visible_codex_tui_workspace", - "default_visible_workspace": "demo_owned_clean_workspace", - "peer_lane_workspace": "visible_codex_tui_workspace", - "peer_lane_worktree_count": 0, - "peer_lane_count": peer_count, - "trust_prompt_avoidance": ( - "demo_owned_clean_workspace_with_persisted_codex_trust_config" - ), - "mutation_isolation_policy": ( - "mutating attempts claim an execution boundary from inside the role; " - "the first visible TUI uses the demo-owned clean workspace unless " - "the operator passes --workspace" - ), - "absolute_paths_recorded": False, - } - - -def _visible_demo_goal_write_scope( - preset_context: dict[str, object] | None, -) -> list[str]: - scopes = list(AUTO_RESEARCH_DEMO_CONTROL_WRITE_SCOPE) - if isinstance(preset_context, dict): - scopes.extend( - normalize_required_write_scopes(preset_context.get("editable_scope")) - ) - return list(dict.fromkeys(scopes)) - - -def _seed_visible_demo_control_plane( - *, - demo_root: Path, - goal_id: str, - objective: str, - supervisor: dict[str, object], - preset_context: dict[str, object] | None = None, -) -> tuple[dict[str, object], Path, str]: - """Create a tiny demo-local LoopX queue for visible workers.""" - - from loopx.bootstrap import bootstrap_project - from loopx.configure_goal import configure_goal - from loopx.todos import add_goal_todo - - control_project = demo_root / "visible-control-plane" - control_registry = demo_root / "visible-control-plane.registry.json" - control_runtime = demo_root / "visible-control-plane.runtime" - bootstrap_project( - project=control_project, - registry_path=control_registry, - runtime_root=control_runtime, - goal_id=goal_id, - objective=objective, - domain="auto-research-demo", - role="agent", - parent_goal_id=None, - state_file=None, - goal_doc=None, - adapter_kind="auto_research_demo_local_queue", - adapter_status="connected", - next_probe=None, - spawn_allowed=True, - max_children=4, - allowed_domains=["auto-research-demo"], - write_scope=_visible_demo_goal_write_scope(preset_context), - force=False, - dry_run=False, - sync_global=False, - ) - lanes = [lane for lane in supervisor.get("lanes") or [] if isinstance(lane, dict)] - agents = sorted( - { - str(lane.get("agent_id") or "").strip() - for lane in lanes - if str(lane.get("agent_id") or "").strip() - } - ) - configure_goal( - registry_path=control_registry, - goal_id=goal_id, - registered_agents=agents, - agent_model="peer_v1", - waiting_on="codex", - orchestration_mode="multi_subagent", - spawn_allowed=True, - execute=True, - ) - registry_payload = json.loads(control_registry.read_text(encoding="utf-8")) - for goal in registry_payload.get("goals", []): - if isinstance(goal, dict) and str(goal.get("id")) == goal_id: - coordination = goal.setdefault("coordination", {}) - if isinstance(coordination, dict): - capabilities = [ - str(value).strip() - for value in coordination.get("available_capabilities", []) - if str(value).strip() - ] - for capability in AUTO_RESEARCH_DEMO_AVAILABLE_CAPABILITIES: - if capability not in capabilities: - capabilities.append(capability) - coordination["available_capabilities"] = capabilities - goal["workspace_guard_policy"] = { - "schema_version": "loopx_workspace_guard_policy_v0", - "peer_independent_worktree_required": False, - "reason": ( - "auto_research_demo_local_queue uses a demo-owned workspace and " - "writes only demo-local LoopX state/evidence; repository edits still " - "require an explicit lane-owned execution boundary" - ), - } - break - control_registry.write_text( - json.dumps(registry_payload, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - - seeded_todos: list[dict[str, object]] = [] - seeded_todo_ids_by_action: dict[str, str] = {} - seed_rows: list[tuple[str, str, str, str]] = [] - for lane in lanes: - agent_id = str(lane.get("agent_id") or "").strip() - role_id = str(lane.get("role_id") or "").strip() - lane_id = str(lane.get("lane_id") or "").strip() - action_kind = auto_research_seed_action_for_role(role_id) - seed_rows.append((action_kind, agent_id, role_id, lane_id)) - seed_rows.sort( - key=lambda row: ( - AUTO_RESEARCH_SEED_ACTION_ORDER.get( - row[0], len(AUTO_RESEARCH_SEED_ACTION_ORDER) - ), - row[1], - ) - ) - for action_kind, agent_id, role_id, lane_id in seed_rows: - title = auto_research_seed_title( - action_kind=action_kind, - role_id=role_id, - lane_id=lane_id, - ) - prerequisite_action = AUTO_RESEARCH_SEED_PREREQUISITE_ACTION_BY_ACTION.get( - action_kind - ) - prerequisite_todo_id = ( - seeded_todo_ids_by_action.get(prerequisite_action) - if prerequisite_action - else None - ) - resume_when = ( - f"todo_done:{prerequisite_todo_id}" - if prerequisite_todo_id - else None - ) - result = add_goal_todo( - registry_path=control_registry, - goal_id=goal_id, - role="agent", - text=f"[P0-auto-research-live] {title}", - task_class="advancement_task", - action_kind=action_kind, - claimed_by=agent_id or None, - resume_when=resume_when, - project=control_project, - ) - todo_id = result.get("todo_id") - if isinstance(todo_id, str) and todo_id: - seeded_todo_ids_by_action.setdefault(action_kind, todo_id) - # Canonical create receipts carry the normalized record in `todo`; - # legacy File creation keeps its metadata on the outer response. - created_todo = result.get("todo") - created_fields = created_todo if isinstance(created_todo, dict) else result - seeded_todos.append( - { - "todo_id": todo_id, - "agent_id": agent_id, - "lane_id": lane_id, - "role_id": role_id, - "action_kind": action_kind, - "resume_when": created_fields.get("resume_when"), - } - ) - - workspace_route = _prepare_visible_demo_workspace_route( - control_project=control_project, - lanes=lanes, - ) - - summary = { - "schema_version": "auto_research_visible_demo_control_plane_v0", - "mode": "demo_local_loopx_queue", - "goal_id": goal_id, - "registry_scope": "demo_local_runtime", - "registered_agent_count": len(agents), - "seeded_todo_count": len(seeded_todos), - "seeded_todos": seeded_todos, - "state_route": "visible_lanes_use_LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", - "workspace_route": workspace_route, - "workspace_guard_policy": { - "peer_independent_worktree_required": False, - "repository_edits_still_require_lane_boundary": True, - }, - "absolute_paths_recorded": False, - "private_artifacts_recorded": False, - } - return summary, control_registry, str(control_runtime) - - -def _command_text( - *, - cli_bin: str, - goal_id: str, - agent_id: str, - execute: bool, - run_worker_loop: bool = False, - launch_visible: bool = False, - headless: bool = False, - attach: bool = False, - no_attach: bool = False, - live_evidence: bool = False, - wake_visible_after_launch: bool = False, - configure_visible_worker_turn: bool = False, - tracking_goal_id: str | None = None, - output_language: str = "en", -) -> str: - parts = [ - shlex.quote(cli_bin), - "--format", - "json", - "auto-research", - "demo-e2e", - "--goal-id", - shlex.quote(goal_id), - "--agent-id", - shlex.quote(agent_id), - ] - if tracking_goal_id: - parts.extend(["--tracking-goal-id", shlex.quote(tracking_goal_id)]) - if output_language and output_language != "en": - parts.extend(["--language", shlex.quote(output_language)]) - if execute: - parts.append("--execute") - if run_worker_loop: - parts.append("--run-worker-loop") - if headless: - parts.append("--headless") - if launch_visible: - parts.append("--launch-visible") - if attach: - parts.append("--attach") - if no_attach: - parts.append("--no-attach") - if live_evidence: - parts.extend(["--live-evidence", ""]) - if wake_visible_after_launch: - parts.append("--wake-visible-after-launch") - if configure_visible_worker_turn: - parts.append("--configure-visible-worker-turn") - return " ".join(parts) - - -def _build_demo_frontier_route_contract( - *, - goal_id: str, - tracking_goal: str, - preset_context: dict[str, object] | None, - goal_surface_mode: str, - reuses_default_internal_goal: bool, -) -> dict[str, object]: - return { - "schema_version": "auto_research_demo_frontier_route_v0", - "goal_surface_mode": goal_surface_mode, - "frontier_goal_id": goal_id, - "tracking_goal_id": tracking_goal or None, - "preset_id": preset_context.get("preset_id") if preset_context else None, - "preset_baseline_source": ( - preset_context.get("baseline_source") if preset_context else None - ), - "tracking_goal_drives_frontier": False, - "visible_lanes_read_goal_id": goal_id, - "fresh_goal_default": goal_surface_mode == "fresh_demo_goal", - "inherits_default_goal": goal_surface_mode == "inherited_default_goal", - "reuses_default_internal_goal": reuses_default_internal_goal, - "default_internal_goal_id": AUTO_RESEARCH_DEFAULT_GOAL_ID, - "dedicated_positive_demo_frontier": not reuses_default_internal_goal, - "reason": ( - "Omitting --goal-id creates an isolated demo goal surface. " - "Use --goal-id to target a specific research frontier, or --inherit-default-goal " - "to intentionally reuse the internal shared demo goal. --tracking-goal-id is metadata " - "for the parent productization goal and must not reroute panes." - ), - } - - -def _build_demo_e2e_commands( - *, - cli_bin: str, - objective: str, - preset_id: str | None, - output_language: str, - goal_id: str, - agent_id: str, - tracking_goal: str, - session_name: str, -) -> dict[str, str]: - tracking_goal_id = tracking_goal or None - return { - "one_question_contract": auto_research_contract_command_text( - cli_bin=cli_bin, - objective=objective, - ), - "one_question_start": auto_research_start_command_text( - cli_bin=cli_bin, - objective=objective, - preset_id=preset_id, - execute=True, - output_language=output_language, - ), - "one_question_start_preview": auto_research_start_command_text( - cli_bin=cli_bin, - objective=objective, - preset_id=preset_id, - output_language=output_language, - ), - "one_question_start_with_visible_wake": auto_research_start_command_text( - cli_bin=cli_bin, - objective=objective, - preset_id=preset_id, - execute=True, - output_language=output_language, - ), - "one_command_worker_loop": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - run_worker_loop=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "headless_worker_loop": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - run_worker_loop=True, - headless=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "start_visible_lanes_without_attach": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - launch_visible=True, - no_attach=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "one_command_visible_wake_demo": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - launch_visible=True, - no_attach=True, - wake_visible_after_launch=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "one_command_visible_worker_turn_validation": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - launch_visible=True, - no_attach=True, - configure_visible_worker_turn=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "one_command_worker_loop_with_visible_lanes": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - run_worker_loop=True, - launch_visible=True, - attach=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "load_live_worker_evidence": _command_text( - cli_bin=cli_bin, - goal_id=goal_id, - agent_id=agent_id, - execute=True, - live_evidence=True, - tracking_goal_id=tracking_goal_id, - output_language=output_language, - ), - "wake_visible_lanes": ( - f"{shlex.quote(cli_bin)} --format json multi-agent wake --session-name " - f"{shlex.quote(session_name)} --execute" - ), - } - - -def _supervisor_summary(supervisor: dict[str, object]) -> dict[str, object]: - product_spec = ( - supervisor.get("product_spec") - if isinstance(supervisor.get("product_spec"), dict) - else {} - ) - runner_contract = ( - supervisor.get("runner_contract") - if isinstance(supervisor.get("runner_contract"), dict) - else {} - ) - pane_local_a2a = ( - runner_contract.get("pane_local_a2a") - if isinstance(runner_contract.get("pane_local_a2a"), dict) - else {} - ) - decentralized_a2a_driver = ( - runner_contract.get("decentralized_a2a_driver") - if isinstance(runner_contract.get("decentralized_a2a_driver"), dict) - else {} - ) - cli_contract = ( - supervisor.get("cli_contract") - if isinstance(supervisor.get("cli_contract"), dict) - else {} - ) - auto_research = ( - supervisor.get("auto_research") - if isinstance(supervisor.get("auto_research"), dict) - else {} - ) - coordination_model = ( - supervisor.get("coordination_model") - if isinstance(supervisor.get("coordination_model"), dict) - else {} - ) - return { - "schema_version": supervisor.get("schema_version"), - "mode": supervisor.get("mode"), - "lane_count": len(supervisor.get("lanes") or []), - "reasoning_contract": supervisor.get("reasoning_contract"), - "uses_generic_runner": bool( - product_spec.get("uses_generic_runner") - or auto_research.get("uses_generic_runner") - ), - "generic_spec_schema": product_spec.get("schema_version"), - "runner_contract_schema": runner_contract.get("schema_version"), - "pane_local_a2a": { - "tick_command": pane_local_a2a.get("tick_command"), - "machine_json_policy": pane_local_a2a.get("machine_json_policy"), - "machine_json_destination": pane_local_a2a.get("machine_json_destination"), - "status_artifact": pane_local_a2a.get("status_artifact"), - "human_default": pane_local_a2a.get("human_default"), - }, - "decentralized_a2a_driver": { - "schema_version": decentralized_a2a_driver.get("schema_version"), - "owner_layer": decentralized_a2a_driver.get("owner_layer"), - "driver_model": decentralized_a2a_driver.get("driver_model"), - "coordination_pattern": decentralized_a2a_driver.get("coordination_pattern"), - "broadcaster_decides_work": ( - decentralized_a2a_driver.get("broadcaster", {}).get("decides_work") - if isinstance(decentralized_a2a_driver.get("broadcaster"), dict) - else None - ), - "pane_decision_owner": ( - decentralized_a2a_driver.get("pane", {}).get("decision_owner") - if isinstance(decentralized_a2a_driver.get("pane"), dict) - else None - ), - "user_and_preset_do_not_own_tick_driver": ( - decentralized_a2a_driver.get("acceptance", {}).get( - "user_and_preset_do_not_own_tick_driver" - ) - if isinstance(decentralized_a2a_driver.get("acceptance"), dict) - else None - ), - }, - "machine_json_policy": cli_contract.get("machine_json_policy"), - "domain_specific_runner_logic": bool(product_spec.get("domain_specific")), - "kernel_boundary": { - "state_bus": auto_research.get("state_bus"), - "presentation_layers_in_kernel": bool( - auto_research.get("presentation_layers_in_kernel") - ), - "coordination_pattern": coordination_model.get("pattern"), - }, - } - - -def _compact_live_worker_evidence(evidence: dict[str, object]) -> dict[str, object]: - return { - "schema_version": "auto_research_live_worker_evidence_summary_v0", - "loaded": True, - "source": evidence.get("source"), - "goal_id": evidence.get("goal_id"), - "agent_id": evidence.get("agent_id"), - "lane_count": evidence.get("lane_count"), - "evidence_event_count": evidence.get("evidence_event_count"), - "result_status": evidence.get("result_status"), - "protected_scope_clean": bool(evidence.get("protected_scope_clean")), - "dev_metric": evidence.get("dev_metric"), - "holdout_metric": evidence.get("holdout_metric"), - "public_boundary": evidence.get("public_boundary"), - } - - -def _compact_visible_pane_a2a_status(artifacts: list[dict[str, object]]) -> dict[str, object]: - lane_outcomes: list[dict[str, object]] = [] - status_check_count = 0 - for raw in artifacts: - checks = raw.get("checks") if isinstance(raw.get("checks"), list) else [] - status_check_count += len(checks) - last = checks[-1] if checks and isinstance(checks[-1], dict) else {} - lane_outcomes.append( - { - "agent_id": raw.get("agent_id"), - "role_id": raw.get("role_id"), - "status": raw.get("status"), - "worker_configured": raw.get("worker_configured") is True, - "worker_label": raw.get("worker_label"), - "selected_todo_id": last.get("selected_todo_id"), - "selected_action": last.get("selected_action"), - "worker_status": last.get("worker_status"), - "counts_as_research_round": False, - } - ) - return { - "schema_version": "auto_research_visible_pane_a2a_status_summary_v0", - "loaded": bool(artifacts), - "source": "visible_pane_a2a_status_artifacts", - "lane_count": len(artifacts), - "status_check_count": status_check_count, - "workflow_driver": False, - "counts_as_collective_research_round": False, - "lane_outcomes": lane_outcomes, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - "local_workspace_path_redacted": True, - }, - } - - -def _build_collective_round_summary( - *, - source: str, - agent_count: int | None, - collective_round_count: int, - dev_metric: float | None, - holdout_metric: float | None, - dev_metric_sequence: list[float] | None = None, - holdout_metric_sequence: list[float] | None = None, - holdout_improvement_count: int | None = None, - evidence_event_count: int | None = None, - expected_lanes: list[dict[str, object]] | None = None, - lane_outcomes: list[dict[str, object]] | None = None, - role_declared_successor_todos: list[dict[str, object]] | None = None, - visible_role_participation_verified: bool = False, - visible_role_participation_basis: str | None = None, -) -> dict[str, object]: - baseline = 1.0 - integrated_evidence: dict[str, object] = { - "dev_metric": dev_metric, - "holdout_metric": holdout_metric, - "evidence_event_count": evidence_event_count, - } - if dev_metric_sequence is not None: - integrated_evidence["dev_metric_sequence"] = list(dev_metric_sequence) - if holdout_metric_sequence is not None: - integrated_evidence["holdout_metric_sequence"] = list(holdout_metric_sequence) - if holdout_improvement_count is not None: - integrated_evidence["holdout_improvement_count"] = holdout_improvement_count - kernel_ledger = build_multi_agent_collective_round_ledger( - source=source, - expected_lanes=expected_lanes, - lane_outcomes=lane_outcomes, - integrated_evidence=integrated_evidence, - role_declared_successor_todos=role_declared_successor_todos, - baseline_metric=baseline, - required_full_participation_round_count=4, - required_holdout_improvement_count=2, - ) - kernel_evidence = ( - kernel_ledger.get("integrated_evidence") - if isinstance(kernel_ledger.get("integrated_evidence"), dict) - else {} - ) - verification = ( - kernel_ledger.get("collective_research_verification") - if isinstance(kernel_ledger.get("collective_research_verification"), dict) - else {} - ) - dev_metric = _numeric_metric(kernel_evidence.get("dev_metric")) - holdout_metric = _numeric_metric(kernel_evidence.get("holdout_metric")) - dev_sequence = _numeric_sequence(kernel_evidence.get("dev_metric_sequence")) - holdout_sequence = _numeric_sequence(kernel_evidence.get("holdout_metric_sequence")) - holdout_improvements = ( - int(kernel_evidence.get("holdout_improvement_count")) - if isinstance(kernel_evidence.get("holdout_improvement_count"), int) - and not isinstance(kernel_evidence.get("holdout_improvement_count"), bool) - else 0 - ) - full_participation_round_count = kernel_ledger.get("full_participation_round_count") - if not isinstance(full_participation_round_count, int) or isinstance( - full_participation_round_count, bool - ): - full_participation_round_count = 0 - full_participation_verified = kernel_ledger.get("full_participation_verified") is True - full_participation_gap = ( - kernel_ledger.get("full_participation_requirement_gap") - if isinstance(kernel_ledger.get("full_participation_requirement_gap"), dict) - else {} - ) - full_participation_count_basis = str( - kernel_ledger.get("full_participation_count_basis") or "" - ) - multi_round_research_verified = ( - verification.get("verified") is True - and verification.get("dev_metric_over_baseline") is True - and holdout_metric is not None - and visible_role_participation_verified - ) - return { - "schema_version": "auto_research_collective_round_summary_v0", - "loaded": collective_round_count > 0 or dev_metric is not None or holdout_metric is not None, - "source": source, - "claim_source": source, - "round_unit": "collective_agent_pass", - "kernel_ledger": kernel_ledger, - "definition": ( - "one collective research round means each configured research lane has " - "one quota/frontier/worker-turn opportunity; pane-local tick loops are " - "reported separately and do not by themselves prove multi-round research" - ), - "visible_role_participation_required": True, - "visible_role_participation_verified": bool(visible_role_participation_verified), - "visible_role_participation_basis": visible_role_participation_basis - or ( - "visible_pane_artifacts_plus_live_evidence" - if visible_role_participation_verified - else "not_visible_pane_verified" - ), - "claim_boundary": ( - "worker-loop summaries are control-plane plumbing evidence only; " - "visible multi-role research claims require visible pane artifacts " - "plus lane-authored public evidence" - ), - "agent_count": agent_count, - "required_collective_round_count": 4, - "required_holdout_improvement_count": 2, - "collective_round_count": collective_round_count, - "full_participation_round_count": full_participation_round_count, - "full_participation_count_basis": full_participation_count_basis, - "full_participation_requirement_gap": full_participation_gap, - "full_participation_verified": full_participation_verified, - "evidence_stage_count": len(dev_sequence) + len(holdout_sequence), - "multi_round_research_verified": multi_round_research_verified, - "improvement_over_rounds": multi_round_research_verified, - "holdout_improvement_verified": holdout_improvements >= 2, - "baseline_metric": baseline, - "dev_metric": dev_metric, - "holdout_metric": holdout_metric, - "dev_metric_sequence": dev_sequence, - "holdout_metric_sequence": holdout_sequence, - "holdout_improvement_count": holdout_improvements, - "holdout_delta_over_dev": ( - holdout_metric - dev_metric - if holdout_metric is not None and dev_metric is not None - else None - ), - "evidence_event_count": evidence_event_count, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - "local_workspace_path_redacted": True, - }, - } - - -def _collective_summary_from_worker_loop( - worker_loop: dict[str, object], - *, - agent_count: int, -) -> dict[str, object]: - turns = worker_loop.get("turns") if isinstance(worker_loop.get("turns"), list) else [] - round_indexes = { - turn.get("round") - for turn in turns - if isinstance(turn, dict) - and turn.get("executed") is True - and isinstance(turn.get("round"), int) - and not isinstance(turn.get("round"), bool) - } - dev_metrics = [ - metric - for turn in turns - if isinstance(turn, dict) - for metric in [_numeric_metric(turn.get("dev_metric"))] - if metric is not None - ] - holdout_metrics = [ - metric - for turn in turns - if isinstance(turn, dict) - for metric in [_numeric_metric(turn.get("holdout_metric"))] - if metric is not None - ] - previous_holdout = 1.0 - holdout_improvement_count = 0 - for metric in holdout_metrics: - if metric > previous_holdout: - holdout_improvement_count += 1 - previous_holdout = metric - successors = [ - successor - for turn in turns - if isinstance(turn, dict) and isinstance(turn.get("successor_todos"), list) - for successor in turn.get("successor_todos", []) - if isinstance(successor, dict) - ] - return _build_collective_round_summary( - source="worker_loop_collective_agent_passes", - agent_count=agent_count, - collective_round_count=len(round_indexes), - dev_metric=dev_metrics[-1] if dev_metrics else None, - holdout_metric=holdout_metrics[-1] if holdout_metrics else None, - dev_metric_sequence=dev_metrics, - holdout_metric_sequence=holdout_metrics, - holdout_improvement_count=holdout_improvement_count, - expected_lanes=[ - {"agent_id": turn.get("agent_id"), "role_id": turn.get("role_id")} - for turn in turns - if isinstance(turn, dict) and turn.get("round") == 1 - ], - lane_outcomes=[turn for turn in turns if isinstance(turn, dict)], - role_declared_successor_todos=successors, - visible_role_participation_verified=False, - visible_role_participation_basis="headless_worker_loop_summary_only", - ) - - -def _collective_summary_from_visible_evidence( - *, - pane_status: dict[str, object], - evidence: dict[str, object], - agent_count: int | None, - expected_lanes: list[dict[str, object]] | None = None, -) -> dict[str, object]: - lane_outcomes = ( - pane_status.get("lane_outcomes") - if isinstance(pane_status.get("lane_outcomes"), list) - else [] - ) - return _build_collective_round_summary( - source="visible_lane_authored_evidence", - agent_count=agent_count, - collective_round_count=0, - dev_metric=_numeric_metric(evidence.get("dev_metric")), - holdout_metric=_numeric_metric(evidence.get("holdout_metric")), - evidence_event_count=( - int(evidence.get("evidence_event_count")) - if isinstance(evidence.get("evidence_event_count"), int) - and not isinstance(evidence.get("evidence_event_count"), bool) - else None - ), - expected_lanes=expected_lanes - or ( - pane_status.get("lanes") if isinstance(pane_status.get("lanes"), list) else None - ), - lane_outcomes=lane_outcomes, - visible_role_participation_verified=True, - visible_role_participation_basis=( - "visible_status_artifacts_plus_lane_authored_evidence" - ), - ) - - -def _discover_visible_pane_a2a_status( - *, - runtime_root_arg: str | None, - session_name: str, - goal_id: str, - wait_seconds: float, -) -> dict[str, object] | None: - if not runtime_root_arg: - return None - runtime_root = Path(runtime_root_arg) - artifact_root = runtime_root / "visible-launcher-artifacts" / session_name - - deadline = time.monotonic() + max(0.0, wait_seconds) - while True: - artifacts = [] - if artifact_root.is_dir(): - for candidate in sorted(artifact_root.glob("*/pane-a2a-status.public.json")): - try: - raw = json.loads(candidate.read_text(encoding="utf-8")) - except Exception: - continue - if not isinstance(raw, dict): - continue - if raw.get("schema_version") != "pane_local_a2a_status_check_v0": - continue - if raw.get("source") != "pane_local_a2a_status_check": - continue - if raw.get("goal_id") != goal_id: - continue - boundary = raw.get("public_boundary") if isinstance(raw.get("public_boundary"), dict) else {} - if any( - boundary.get(key) is not False - for key in ( - "raw_logs_recorded", - "private_artifacts_recorded", - "absolute_paths_recorded", - "credentials_recorded", - ) - ): - continue - artifacts.append(raw) - if artifacts: - return _compact_visible_pane_a2a_status(artifacts) - if time.monotonic() >= deadline: - return None - time.sleep(0.5) - - -def _discover_visible_live_evidence( - *, - runtime_root_arg: str | None, - session_name: str, - goal_id: str, - wait_seconds: float, -) -> dict[str, object] | None: - if not runtime_root_arg: - return None - runtime_root = Path(runtime_root_arg) - artifact_root = runtime_root / "visible-launcher-artifacts" / session_name - - deadline = time.monotonic() + max(0.0, wait_seconds) - while True: - candidates = [] - if artifact_root.is_dir(): - for artifact_name in ( - "live-codex-e2e-evidence.public.json", - "live_codex_e2e_evidence.public.json", - ): - candidates.extend(artifact_root.glob(f"*/{artifact_name}")) - candidates = sorted(set(candidates)) - loaded: list[dict[str, object]] = [] - for candidate in candidates: - try: - raw = json.loads(candidate.read_text(encoding="utf-8")) - except Exception: - continue - if not isinstance(raw, dict) or raw.get("goal_id") != goal_id: - continue - lane_agent_id = str(raw.get("agent_id") or "").strip() - if not lane_agent_id: - continue - try: - loaded.append( - load_live_codex_e2e_evidence( - evidence_path=str(candidate), - goal_id=goal_id, - agent_id=lane_agent_id, - ) - ) - except ValueError: - continue - for evidence in loaded: - if evidence.get("holdout_metric") is not None: - merged = dict(evidence) - if merged.get("dev_metric") is None: - for other in loaded: - if other.get("dev_metric") is not None: - merged["dev_metric"] = other.get("dev_metric") - break - return merged - if loaded and time.monotonic() >= deadline: - return loaded[0] - if time.monotonic() >= deadline: - return None - time.sleep(0.5) - - -def _load_live_worker_evidence_into_payload( - *, - payload: dict[str, object], - evidence: dict[str, object], - evidence_source: str, -) -> None: - payload["live_worker_evidence"] = _compact_live_worker_evidence(evidence) - visible_proof = payload["visible_worker_proof"] - if isinstance(visible_proof, dict): - visible_proof["lane_authored_evidence_loaded"] = True - visible_proof["visible_lanes_launched"] = True - visible_proof["visible_lanes_accepted"] = True - visible_proof["evidence_source"] = evidence_source - - -def _load_visible_pane_a2a_status_into_payload( - *, - payload: dict[str, object], - status: dict[str, object], -) -> None: - payload["visible_pane_a2a_status"] = status - visible_proof = payload["visible_worker_proof"] - if isinstance(visible_proof, dict): - visible_proof["pane_local_a2a_status_loaded"] = bool(status.get("loaded")) - visible_proof["pane_local_a2a_status_check_count"] = status.get( - "status_check_count" - ) - visible_proof["decentralized_a2a_rounds_verified"] = False - - -def _load_collective_research_rounds_into_payload( - *, - payload: dict[str, object], - rounds: dict[str, object], -) -> None: - payload["collective_research_rounds"] = rounds - visible_proof = payload["visible_worker_proof"] - if isinstance(visible_proof, dict): - visible_proof["collective_research_rounds_loaded"] = bool(rounds.get("loaded")) - visible_proof["collective_research_rounds_source"] = rounds.get("source") - visible_proof["visible_role_participation_required"] = ( - rounds.get("visible_role_participation_required") is not False - ) - visible_proof["visible_role_participation_verified"] = ( - rounds.get("visible_role_participation_verified") is True - ) - visible_proof["visible_role_participation_basis"] = rounds.get( - "visible_role_participation_basis" - ) - visible_proof["collective_research_round_count"] = rounds.get( - "collective_round_count" - ) - visible_proof["holdout_improvement_count"] = rounds.get( - "holdout_improvement_count" - ) - visible_proof["decentralized_a2a_rounds_verified"] = bool( - rounds.get("multi_round_research_verified") - and rounds.get("visible_role_participation_verified") is True - ) - - -def _load_visible_wake_into_payload( - *, - payload: dict[str, object], - wake: dict[str, object], -) -> None: - driver = ( - wake.get("driver_contract") - if isinstance(wake.get("driver_contract"), dict) - else {} - ) - prompt_submit_checks = wake.get("prompt_submit_checks") or [] - pane_input_ready_checks = wake.get("pane_input_ready_checks") or [] - not_ready_reasons = sorted( - { - str(check.get("not_ready_reason")) - for check in pane_input_ready_checks - if isinstance(check, dict) and check.get("not_ready_reason") - } - ) - prompt_delivered = bool(prompt_submit_checks) - payload["visible_wake"] = { - "schema_version": wake.get("schema_version"), - "mode": wake.get("mode"), - "session_name": wake.get("session_name"), - "target_lanes": wake.get("target_lanes"), - "prompt_hash": wake.get("prompt_hash"), - "coordination_model": wake.get("coordination_model"), - "wakeup_model": wake.get("wakeup_model"), - "workflow_driver": bool(wake.get("workflow_driver")), - "broadcaster_reads_frontier": bool(wake.get("broadcaster_reads_frontier")), - "broadcaster_reads_todo_readiness": bool( - wake.get("broadcaster_reads_todo_readiness") - ), - "broadcaster_selects_todo": bool(wake.get("broadcaster_selects_todo")), - "pane_decision_owner": wake.get("pane_decision_owner"), - "pane_input_ready_verified": wake.get("pane_input_ready_verified") is True, - "pane_input_ready_checks": pane_input_ready_checks, - "pane_input_ready_timeout_seconds": wake.get("pane_input_ready_timeout_seconds"), - "ready_lanes": wake.get("ready_lanes") or [], - "not_ready_lanes": wake.get("not_ready_lanes") or [], - "not_ready_reasons": not_ready_reasons, - "prompt_submit_checks": prompt_submit_checks, - "prompt_delivered": prompt_delivered, - "prompt_delivery": wake.get("prompt_delivery"), - "auto_wake_backoff_recommended": wake.get("auto_wake_backoff_recommended") is True, - "driver_contract_schema": driver.get("schema_version"), - "driver_owner_layer": driver.get("owner_layer"), - "boundary": wake.get("boundary"), - } - visible_proof = payload["visible_worker_proof"] - if isinstance(visible_proof, dict): - visible_proof["cadence_wake_loaded"] = True - prompt_delivery = wake.get("prompt_delivery") - cadence_wake_pending_reason = None - if not prompt_delivered: - if prompt_delivery == "skipped_no_input_ready_panes": - cadence_wake_pending_reason = "pane_not_ready" - elif prompt_delivery == "skipped_terminal_pane_backoff": - cadence_wake_pending_reason = "terminal_backoff" - visible_proof["cadence_wake_verified"] = ( - wake.get("mode") == "execute" - and wake.get("coordination_model") == "decentralized_state_a2a" - and wake.get("wakeup_model") == "fixed_prompt_broadcast" - and wake.get("workflow_driver") is False - and wake.get("broadcaster_reads_frontier") is False - and wake.get("broadcaster_selects_todo") is False - and prompt_delivery - in { - "tmux_paste_buffer_after_codex_tui_first_turn_ready", - "tmux_paste_buffer_after_ready_subset", - } - and prompt_delivered - ) - visible_proof["cadence_wake_prompt_delivered"] = prompt_delivered - visible_proof["cadence_wake_pending_reason"] = cadence_wake_pending_reason - visible_proof["cadence_wake_not_ready_reasons"] = not_ready_reasons - - -def _numeric_metric(value: object) -> float | None: - if isinstance(value, bool): - return None - if isinstance(value, (int, float)): - return float(value) - return None - - -def _numeric_sequence(value: object) -> list[float]: - if not isinstance(value, list): - return [] - metrics: list[float] = [] - for item in value: - metric = _numeric_metric(item) - if metric is not None: - metrics.append(metric) - return metrics - - -def _metric_improvement_count(metrics: list[float], *, baseline: float) -> int: - previous = baseline - count = 0 - for metric in metrics: - if metric > previous: - count += 1 - previous = metric - return count - - -def _build_visible_readiness(payload: dict[str, object]) -> dict[str, object]: - proof = ( - payload.get("visible_worker_proof") - if isinstance(payload.get("visible_worker_proof"), dict) - else {} - ) - pane_status = ( - payload.get("visible_pane_a2a_status") - if isinstance(payload.get("visible_pane_a2a_status"), dict) - else {} - ) - collective_rounds = ( - payload.get("collective_research_rounds") - if isinstance(payload.get("collective_research_rounds"), dict) - else {} - ) - evidence = ( - payload.get("live_worker_evidence") - if isinstance(payload.get("live_worker_evidence"), dict) - else {} - ) - wake = payload.get("visible_wake") if isinstance(payload.get("visible_wake"), dict) else {} - supervisor = payload.get("supervisor") if isinstance(payload.get("supervisor"), dict) else {} - contract_acceptance = ( - payload.get("contract_acceptance") - if isinstance(payload.get("contract_acceptance"), dict) - else {} - ) - driver = ( - supervisor.get("decentralized_a2a_driver") - if isinstance(supervisor.get("decentralized_a2a_driver"), dict) - else {} - ) - baseline = 1.0 - dev_sequence = _numeric_sequence(evidence.get("dev_metric_sequence")) or _numeric_sequence( - collective_rounds.get("dev_metric_sequence") - ) - holdout_sequence = _numeric_sequence(evidence.get("holdout_metric_sequence")) or _numeric_sequence( - collective_rounds.get("holdout_metric_sequence") - ) - dev_metric = dev_sequence[-1] if dev_sequence else _numeric_metric(evidence.get("dev_metric")) - holdout_metric = ( - holdout_sequence[-1] if holdout_sequence else _numeric_metric(evidence.get("holdout_metric")) - ) - if dev_metric is None: - dev_metric = _numeric_metric(collective_rounds.get("dev_metric")) - if holdout_metric is None: - holdout_metric = _numeric_metric(collective_rounds.get("holdout_metric")) - holdout_improvement_count = ( - collective_rounds.get("holdout_improvement_count") - if isinstance(collective_rounds.get("holdout_improvement_count"), int) - and not isinstance(collective_rounds.get("holdout_improvement_count"), bool) - else _metric_improvement_count(holdout_sequence, baseline=baseline) - ) - best_metric = holdout_metric if holdout_metric is not None else dev_metric - best_source = ( - "final_holdout" - if holdout_metric is not None - else "final_dev" - if dev_metric is not None - else None - ) - protected_scope_clean = evidence.get("protected_scope_clean") is True - checks = { - "user_contract_accepted": contract_acceptance.get("accepted") is True, - "visible_lanes_accepted": proof.get("visible_lanes_accepted") is True, - "cadence_wake_verified": proof.get("cadence_wake_verified") is True, - "pane_local_status_loaded": pane_status.get("loaded") is True, - "collective_research_multi_round_verified": ( - collective_rounds.get("multi_round_research_verified") is True - ), - "visible_role_participation_verified": ( - proof.get("visible_role_participation_verified") is True - ), - "lane_authored_evidence_loaded": proof.get("lane_authored_evidence_loaded") is True, - "protected_scope_clean": protected_scope_clean, - "positive_metric_over_baseline": ( - best_metric is not None and best_metric > baseline - ), - "workflow_driver_false": ( - pane_status.get("workflow_driver") is False - and wake.get("workflow_driver") is False - ), - "kernel_driver_contract_loaded": ( - driver.get("owner_layer") == "generic_multi_agent_kernel" - and driver.get("user_and_preset_do_not_own_tick_driver") is True - ), - } - ready = all(checks.values()) - return { - "schema_version": "auto_research_visible_readiness_v0", - "ready": ready, - "readiness_level": "ready" if ready else "collecting_evidence", - "one_command": payload.get("commands", {}).get("one_question_start_with_visible_wake") - or payload.get("commands", {}).get("one_command_visible_wake_demo") - if isinstance(payload.get("commands"), dict) - else None, - "coordination_pattern": "decentralized_state_a2a", - "wake_model": wake.get("wakeup_model"), - "workflow_model": driver.get("driver_model") - or "todo_readiness_edge_plus_fixed_retry", - "driver_owner_layer": driver.get("owner_layer"), - "auto_research_preset_role": "thin_domain_defaults_only", - "user_contract_invocation": contract_acceptance.get("canonical_invocation"), - "leader_agent_required": False, - "manual_artifact_inspection_required": not ready, - "checks": checks, - "missing_requirements": [key for key, passed in checks.items() if not passed], - "contract_acceptance": { - "accepted": contract_acceptance.get("accepted") is True, - "required_outputs": contract_acceptance.get("required_outputs"), - "present_outputs": contract_acceptance.get("present_outputs"), - "one_click_start": contract_acceptance.get("checks", {}).get("one_click_start_present") - if isinstance(contract_acceptance.get("checks"), dict) - else False, - }, - "pane_status": { - "scope": "pane_local_state_check", - "status_check_count": pane_status.get("status_check_count"), - "lane_count": pane_status.get("lane_count"), - "counts_as_collective_research_round": False, - }, - "collective_research_rounds": { - "scope": collective_rounds.get("round_unit") or "collective_agent_pass", - "count": collective_rounds.get("collective_round_count"), - "multi_round_verified": collective_rounds.get( - "multi_round_research_verified" - ) - is True, - "visible_role_participation_verified": collective_rounds.get( - "visible_role_participation_verified" - ) - is True, - "claim_source": collective_rounds.get("claim_source") - or collective_rounds.get("source"), - "full_participation_round_count": collective_rounds.get( - "full_participation_round_count" - ), - "full_participation_count_basis": collective_rounds.get( - "full_participation_count_basis" - ), - "full_participation_requirement_gap": collective_rounds.get( - "full_participation_requirement_gap" - ), - "stages": collective_rounds.get("stages") or [], - "holdout_improvement_count": holdout_improvement_count, - }, - "improvement_summary": { - "baseline_metric": baseline, - "dev_metric_sequence": dev_sequence or ([] if dev_metric is None else [dev_metric]), - "holdout_metric_sequence": holdout_sequence - or ([] if holdout_metric is None else [holdout_metric]), - "holdout_improvement_count": holdout_improvement_count, - "final_dev_metric": dev_metric, - "final_holdout_metric": holdout_metric, - "best_metric": best_metric, - "best_metric_source": best_source, - "improved_over_baseline": best_metric is not None and best_metric > baseline, - "holdout_delta_over_dev": ( - holdout_metric - dev_metric - if holdout_metric is not None and dev_metric is not None - else None - ), - }, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - - -def _load_visible_readiness_into_payload(payload: dict[str, object]) -> None: - payload["visible_readiness"] = _build_visible_readiness(payload) - - -def run_auto_research_demo_e2e( - *, - agent_id: str, - goal_id: str, - tracking_goal_id: str | None, - objective: str, - preset_id: str | None = None, - output_dir: str, - execute: bool, - launch_visible: bool, - keep_workspace: bool, - registry_path: Path, - runtime_root_arg: str | None, - session_name: str, - cli_bin: str, - codex_bin: str, - tmux_bin: str, - reasoning_effort: str, - output_language: str, - live_evidence_path: str | None, - append_evidence: AppendEvidence, - visible_launcher: VisibleLauncher | None = None, - visible_wake: VisibleWake | None = None, - wake_visible_after_launch: bool = False, - goal_surface_mode: str = "explicit_goal", - agent_specs: Sequence[str] | None = None, - run_worker_loop: bool = False, - worker_loop_rounds: int = 4, - configure_visible_worker_turn: bool = False, - visible_live_evidence_wait_seconds: float = 30.0, -) -> dict[str, object]: - if launch_visible and not execute: - raise ValueError("--launch-visible requires --execute") - if run_worker_loop and not execute: - raise ValueError("--run-worker-loop requires --execute") - if worker_loop_rounds < 1: - raise ValueError("--worker-loop-rounds must be >= 1") - if launch_visible and visible_launcher is None: - raise ValueError("--launch-visible requires a visible launcher callback") - if wake_visible_after_launch and not launch_visible: - raise ValueError("--wake-visible-after-launch requires --launch-visible") - if wake_visible_after_launch and visible_wake is None: - raise ValueError("--wake-visible-after-launch requires a visible wake callback") - if live_evidence_path and not execute: - raise ValueError("--live-evidence requires --execute") - - tracking_goal = tracking_goal_id.strip() if isinstance(tracking_goal_id, str) else "" - if tracking_goal == goal_id: - tracking_goal = "" - reuses_default_internal_goal = goal_id == AUTO_RESEARCH_DEFAULT_GOAL_ID - effective_agent_specs = list(agent_specs or []) - if (run_worker_loop or launch_visible) and not effective_agent_specs: - effective_agent_specs = default_auto_research_agent_specs() - user_contract = build_auto_research_user_contract( - objective, - output_language=output_language, - preset_id=preset_id, - ) - preset_context = ( - user_contract.get("preset_context") - if isinstance(user_contract.get("preset_context"), dict) - else None - ) - contract_acceptance = build_auto_research_contract_acceptance(user_contract) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=goal_id, - open_question=objective, - preset_context=preset_context, - agent_specs=effective_agent_specs, - session_name=session_name, - cli_bin=cli_bin, - codex_bin=codex_bin, - tmux_bin=tmux_bin, - reasoning_effort=reasoning_effort, - output_language=output_language, - configure_visible_worker_turn=configure_visible_worker_turn, - ) - execution_kind = ( - "loopx_worker_loop" - if execute and run_worker_loop - else "visible_worker_launch" - if execute and launch_visible - else "worker_loop_preview" - ) - result_source = ( - "loopx_worker_loop_public_evidence" - if execute and run_worker_loop - else "visible_worker_launcher" - if execute and launch_visible - else "dry_run_preview" - ) - payload: dict[str, object] = { - "ok": True, - "schema_version": AUTO_RESEARCH_DEMO_E2E_SCHEMA_VERSION, - "mode": "execute" if execute else "dry_run", - "execution_kind": execution_kind, - "result_source": result_source, - "goal_id": goal_id, - "tracking_goal_id": tracking_goal or None, - "route_contract": _build_demo_frontier_route_contract( - goal_id=goal_id, - tracking_goal=tracking_goal, - preset_context=preset_context, - goal_surface_mode=goal_surface_mode, - reuses_default_internal_goal=reuses_default_internal_goal, - ), - "agent_id": agent_id, - "reasoning_effort": reasoning_effort, - "output_language": output_language, - "user_contract": user_contract, - "preset_context": preset_context, - "contract_acceptance": contract_acceptance, - "commands": _build_demo_e2e_commands( - cli_bin=cli_bin, - objective=objective, - preset_id=preset_id, - output_language=output_language, - goal_id=goal_id, - agent_id=agent_id, - tracking_goal=tracking_goal, - session_name=session_name, - ), - "supervisor": _supervisor_summary(supervisor), - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - "local_workspace_path_redacted": True, - "writes_loopx_state": bool(execute), - "launches_visible_lanes": bool(launch_visible), - "visible_codex_sessions_recorded": False, - }, - "visible_worker_proof": build_auto_research_live_worker_proof( - launch_visible=launch_visible, - ), - "visible_worker_turn_validation": { - "schema_version": "auto_research_visible_worker_turn_validation_v0", - "configured": bool(configure_visible_worker_turn), - "counts_as_research_evidence_without_role_output": False, - "manual_research_required_is_blocker_not_progress": True, - }, - } - if not execute: - payload["worker_loop_preview"] = { - "executed": False, - "result_source": "dry_run_preview", - "expected_steps": ( - "seed a demo-local LoopX queue, let role-compatible workers claim " - "frontier todos, write public-safe evidence, append rollout events, " - "and read compact evidence from state" - ), - "coordination_pattern": "decentralized_state_a2a", - } - return payload - - tmp_obj: tempfile.TemporaryDirectory[str] | None = None - if keep_workspace or launch_visible: - demo_root = Path(tempfile.mkdtemp(prefix="loopx-auto-research-demo-e2e.")) - else: - tmp_obj = tempfile.TemporaryDirectory(prefix="loopx-auto-research-demo-e2e.") - demo_root = Path(tmp_obj.name) - try: - visible_control: dict[str, object] | None = None - visible_registry_path: Path | None = None - visible_runtime_root_arg: str | None = None - - def ensure_visible_control_plane() -> tuple[dict[str, object], Path, str | None]: - nonlocal visible_control, visible_registry_path, visible_runtime_root_arg - if visible_control is None or visible_registry_path is None: - ( - visible_control, - visible_registry_path, - visible_runtime_root_arg, - ) = _seed_visible_demo_control_plane( - demo_root=demo_root, - goal_id=goal_id, - objective=objective, - supervisor=supervisor, - preset_context=preset_context, - ) - payload["visible_control_plane"] = visible_control - return visible_control, visible_registry_path, visible_runtime_root_arg - - visible_control, visible_registry_path, visible_runtime_root_arg = ensure_visible_control_plane() - if run_worker_loop: - worker_agent_ids = [ - str(lane.get("agent_id") or "").strip() - for lane in supervisor.get("lanes") or [] - if isinstance(lane, dict) and str(lane.get("agent_id") or "").strip() - ] - - def append_worker_evidence(packet_path: str) -> dict[str, object]: - return append_auto_research_rollout_events( - packet_path=packet_path, - registry_path=visible_registry_path, - runtime_root_arg=visible_runtime_root_arg, - dry_run=False, - ) - - worker_loop = run_auto_research_worker_loop( - registry_path=visible_registry_path, - runtime_root_arg=visible_runtime_root_arg, - goal_id=goal_id, - agent_ids=worker_agent_ids, - objective=objective, - workspace=demo_root / "visible-control-plane", - output_dir=output_dir, - execute=True, - append_evidence=append_worker_evidence, - lane_count=len(worker_agent_ids), - visible_lanes_accepted=True, - complete_selected_todo=True, - max_rounds=worker_loop_rounds, - ) - payload["worker_loop"] = worker_loop - collective_rounds = _collective_summary_from_worker_loop( - worker_loop, - agent_count=len(worker_agent_ids), - ) - _load_collective_research_rounds_into_payload( - payload=payload, - rounds=collective_rounds, - ) - turns = worker_loop.get("turns") if isinstance(worker_loop.get("turns"), list) else [] - dev_metrics = [ - metric - for turn in turns - if isinstance(turn, dict) - for metric in [_numeric_metric(turn.get("dev_metric"))] - if metric is not None - ] - holdout_metrics = [ - metric - for turn in turns - if isinstance(turn, dict) - for metric in [_numeric_metric(turn.get("holdout_metric"))] - if metric is not None - ] - dev_metric = dev_metrics[-1] if dev_metrics else None - holdout_metric = holdout_metrics[-1] if holdout_metrics else None - positive_result = holdout_metric is not None or dev_metric is not None - payload["tonight_experience"] = { - "schema_version": "auto_research_tonight_experience_v0", - "ready": positive_result, - "one_command": payload["commands"]["one_command_worker_loop"], - "goal_id": goal_id, - "goal_surface_mode": goal_surface_mode, - "coordination_pattern": "decentralized_state_a2a", - "workflow_model": "state_projected_frontier_not_dynamic_workflow", - "driver_role": "polling_driver_only", - "leader_agent_required": False, - "worker_loop_round_count": worker_loop.get("round_count"), - "collective_research_round_count": collective_rounds.get( - "collective_round_count" - ), - "collective_multi_round_verified": collective_rounds.get( - "multi_round_research_verified" - ), - "executed_turn_count": worker_loop.get("executed_turn_count"), - "completed_turn_count": worker_loop.get("completed_turn_count"), - "selected_actions": worker_loop.get("selected_actions"), - "dev_metric_sequence": dev_metrics, - "holdout_metric_sequence": holdout_metrics, - "holdout_improvement_count": collective_rounds.get( - "holdout_improvement_count" - ), - "dev_metric": dev_metric, - "holdout_metric": holdout_metric, - "positive_result": positive_result, - "positive_result_basis": ( - "public_safe_dev_and_holdout_evidence" - if holdout_metric is not None - else "public_safe_dev_evidence" - if dev_metric is not None - else "requires_visible_lane_authored_evidence" - ), - "state_surfaces": [ - "demo-local LoopX registry", - "quota/frontier selection", - "todo completion", - "rollout event log", - ], - "worker_contract": ( - "Each role reads the shared LoopX state surface, accepts only its projected " - "frontier item, writes public-safe evidence, and completes the selected todo." - ), - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "visible_codex_lane_authored": False, - }, - } - if launch_visible and visible_launcher is not None: - ( - visible_control, - visible_registry_path, - visible_runtime_root_arg, - ) = ensure_visible_control_plane() - visible_payload = visible_launcher( - dict(supervisor), - visible_registry_path, - visible_runtime_root_arg, - demo_root, - ) - launch_result = ( - visible_payload.get("launch_result") - if isinstance(visible_payload.get("launch_result"), dict) - else {} - ) - visible_acceptance = ( - launch_result.get("visible_acceptance") - if isinstance(launch_result.get("visible_acceptance"), dict) - else {} - ) - payload["visible_launch"] = { - "mode": visible_payload.get("mode"), - "launch_result": launch_result, - "boundary": visible_payload.get("boundary"), - } - visible_proof = payload["visible_worker_proof"] - if isinstance(visible_proof, dict): - visible_proof["visible_lanes_launched"] = True - visible_proof["visible_lanes_accepted"] = bool(visible_acceptance.get("accepted")) - visible_proof["evidence_source"] = "visible_launcher" - if wake_visible_after_launch and visible_wake is not None: - session = str(launch_result.get("session_name") or session_name) - workspace = demo_root / "visible-control-plane" - - # Build lane_id → agent_id mapping from the supervisor lanes. - lane_agent_lookup: dict[str, str] = {} - for lane in supervisor.get("lanes") or []: - if not isinstance(lane, dict): - continue - lid = str(lane.get("lane_id") or "").strip() - aid = str(lane.get("agent_id") or "").strip() - if lid and aid: - lane_agent_lookup[lid] = aid - - # Filter: only wake lanes that have a selected runnable todo and - # are not in quiet-completion. - ready_lanes: list[str] = [] - skipped_lanes: list[dict[str, object]] = [] - for raw_lane in launch_result.get("started_lanes") or []: - lane_id = str(raw_lane).strip() if raw_lane else "" - if not lane_id: - continue - agent_id = lane_agent_lookup.get(lane_id) - if not agent_id: - skipped_lanes.append({ - "lane_id": lane_id, - "reason": "no_agent_mapping", - }) - continue - try: - frontier = load_auto_research_worker_frontier( - registry_path=visible_registry_path, - runtime_root_arg=visible_runtime_root_arg, - goal_id=goal_id, - agent_id=agent_id, - workspace=workspace, - ) - except Exception: - skipped_lanes.append({ - "lane_id": lane_id, - "agent_id": agent_id, - "reason": "frontier_load_failed", - "error_code": "FRONTIER_LOAD_FAILED", - }) - continue - frontier_data = ( - frontier.get("frontier") - if isinstance(frontier.get("frontier"), dict) - else {} - ) - selected = frontier_data.get("selected") - completion = ( - frontier_data.get("completion") - if isinstance(frontier_data.get("completion"), dict) - else None - ) - quota = ( - frontier.get("quota") - if isinstance(frontier.get("quota"), dict) - else {} - ) - if completion and completion.get("quiet_completion_allowed") is True: - skipped_lanes.append({ - "lane_id": lane_id, - "agent_id": agent_id, - "reason": "quiet_completion_allowed", - }) - continue - if not selected: - skipped_lanes.append({ - "lane_id": lane_id, - "agent_id": agent_id, - "reason": "no_selected_todo", - "runnable_count": frontier_data.get("runnable_count"), - "blocked_count": frontier_data.get("blocked_count"), - }) - continue - if quota.get("should_run") is False: - skipped_lanes.append({ - "lane_id": lane_id, - "agent_id": agent_id, - "reason": "quota_should_run_false", - "quota_state": quota.get("state"), - }) - continue - ready_lanes.append(lane_id) - - if not ready_lanes: - wake_payload: dict[str, object] = { - "ok": True, - "schema_version": "multi_agent_pane_a2a_wakeup_v0", - "mode": "no_op_all_filtered", - "session_name": session, - "target_lanes": [], - "prompt": "", - "prompt_hash": "", - "coordination_model": "decentralized_state_a2a", - "wakeup_model": "state_aware_filter_no_ready_lanes", - "workflow_driver": False, - "broadcaster_reads_frontier": False, - "broadcaster_reads_todo_readiness": False, - "broadcaster_selects_todo": False, - "prompt_delivery": "skipped_no_ready_lanes", - "prompt_delivered": False, - "auto_wake_backoff_recommended": False, - } - else: - wake_payload = visible_wake(session, ready_lanes) - if skipped_lanes: - wake_payload = dict(wake_payload) - wake_payload["state_aware_filter"] = { - "schema_version": "auto_research_state_aware_wake_filter_v0", - "total_started_lanes": len( - [ - s for s in (launch_result.get("started_lanes") or []) - if str(s).strip() - ] - ), - "ready_lane_count": len(ready_lanes), - "skipped_lane_count": len(skipped_lanes), - "skipped_lanes": skipped_lanes, - } - _load_visible_wake_into_payload( - payload=payload, - wake=wake_payload, - ) - pane_status = _discover_visible_pane_a2a_status( - runtime_root_arg=visible_runtime_root_arg, - session_name=str(launch_result.get("session_name") or session_name), - goal_id=goal_id, - wait_seconds=visible_live_evidence_wait_seconds, - ) - if pane_status is not None: - _load_visible_pane_a2a_status_into_payload( - payload=payload, - status=pane_status, - ) - collective_rounds = None - live_evidence = _discover_visible_live_evidence( - runtime_root_arg=visible_runtime_root_arg, - session_name=str(launch_result.get("session_name") or session_name), - goal_id=goal_id, - wait_seconds=visible_live_evidence_wait_seconds, - ) - if live_evidence is not None: - _load_live_worker_evidence_into_payload( - payload=payload, - evidence=live_evidence, - evidence_source="visible_launcher_artifact", - ) - if collective_rounds is None and pane_status is not None: - collective_rounds = _collective_summary_from_visible_evidence( - pane_status=pane_status, - evidence=payload["live_worker_evidence"], - agent_count=( - int(pane_status.get("lane_count")) - if isinstance(pane_status.get("lane_count"), int) - and not isinstance(pane_status.get("lane_count"), bool) - else None - ), - expected_lanes=[ - { - "agent_id": lane.get("agent_id"), - "lane_id": lane.get("lane_id"), - "role_id": lane.get("role_id"), - } - for lane in supervisor.get("lanes") or [] - if isinstance(lane, dict) - ], - ) - _load_collective_research_rounds_into_payload( - payload=payload, - rounds=collective_rounds, - ) - _load_visible_readiness_into_payload(payload) - payload["workspace_retained"] = keep_workspace or launch_visible - if live_evidence_path: - live_evidence = load_live_codex_e2e_evidence( - evidence_path=live_evidence_path, - goal_id=goal_id, - agent_id=agent_id, - ) - _load_live_worker_evidence_into_payload( - payload=payload, - evidence=live_evidence, - evidence_source="live_worker_evidence", - ) - return payload - finally: - if tmp_obj is not None: - tmp_obj.cleanup() diff --git a/demo/auto_research/demo_supervisor.py b/demo/auto_research/demo_supervisor.py deleted file mode 100644 index 2b49b2e21e..0000000000 --- a/demo/auto_research/demo_supervisor.py +++ /dev/null @@ -1,186 +0,0 @@ -"""Thin auto-research visible-lane plan built on the generic TUI runner.""" - -from __future__ import annotations - -from collections.abc import Iterable -from typing import Any - -from .defaults import ( - AUTO_RESEARCH_DEFAULT_GOAL_ID, - build_auto_research_layer_contract, -) -from .preset import ( - AUTO_RESEARCH_PRESET_SCHEMA_VERSION, - build_auto_research_preset_role, - build_auto_research_preset_summary, - auto_research_lane_specs, -) -from ..visible_multi_agent_launcher import build_visible_multi_agent_payload_from_spec - - -AUTO_RESEARCH_DEMO_SUPERVISOR_SCHEMA_VERSION = "auto_research_demo_supervisor_plan_v0" - - -def build_visible_worker_turn_command( - *, - goal_id: str = AUTO_RESEARCH_DEFAULT_GOAL_ID, - agent_id: str, - lane_count: int = 1, -) -> str: - """Return a pane-local worker-turn hook that still obeys visible evidence gates. - - The hook is exported through ``LOOPX_PANE_WORKER_TURN`` and executed by the - pane shell, so it must be a plain argv-style command without shell - metacharacters. Goal, agent, and lane values are baked in as literal safe - tokens; the open question lives in the role profile/preset context rather - than in the hook. - """ - - clean_goal = str(goal_id or AUTO_RESEARCH_DEFAULT_GOAL_ID).strip() - clean_agent = str(agent_id or "").strip() - if not clean_agent: - raise ValueError("build_visible_worker_turn_command requires agent_id") - count = max(1, int(lane_count or 1)) - return " ".join( - [ - "loopx", - "--format", - "json", - "auto-research", - "worker-turn", - "--goal-id", - clean_goal, - "--agent-id", - clean_agent, - "--lane-count", - str(count), - "--visible-lanes-accepted", - "--complete-selected-todo", - "--execute", - ] - ) - - -def build_auto_research_demo_supervisor_plan( - *, - goal_id: str = AUTO_RESEARCH_DEFAULT_GOAL_ID, - open_question: object | None = None, - preset_context: dict[str, object] | None = None, - agent_specs: Iterable[str] | None = None, - session_name: str = "loopx-auto-research", - cli_bin: str = "loopx", - codex_bin: str = "codex", - tmux_bin: str = "tmux", - reasoning_effort: str = "high", - output_language: str = "en", - configure_visible_worker_turn: bool = False, -) -> dict[str, Any]: - """Build the auto-research visible TUI plan from a small role spec. - - This is intentionally only a launcher/spec packet. Research truth flows - through each pane's LoopX quota/frontier/todo/evidence commands. The packet - keeps presentation-specific reporting out of the auto-research kernel. - """ - - goal = str(goal_id).strip() or AUTO_RESEARCH_DEFAULT_GOAL_ID - lanes = auto_research_lane_specs(agent_specs) - worker_turn_configured = bool(configure_visible_worker_turn) - roles = [ - build_auto_research_preset_role( - lane=lane, - goal_id=goal, - reasoning_effort=reasoning_effort, - output_language=output_language, - open_question=open_question, - preset_context=preset_context, - ) - for lane in lanes - ] - if worker_turn_configured: - for role in roles: - role["worker_turn_command"] = build_visible_worker_turn_command( - goal_id=goal, - agent_id=str(role.get("agent_id") or "").strip(), - lane_count=len(lanes), - ) - role_profile = role.get("role_profile") - if isinstance(role_profile, dict): - role_profile["visible_worker_turn_configured"] = True - - payload = build_visible_multi_agent_payload_from_spec( - { - "goal_id": goal, - "session_name": session_name, - "reasoning_effort": reasoning_effort, - "roles": roles, - }, - tmux_bin=tmux_bin, - cli_bin=cli_bin, - codex_bin=codex_bin, - ) - payload.pop("acceptance", None) - runner_contract = ( - payload.get("runner_contract") - if isinstance(payload.get("runner_contract"), dict) - else {} - ) - driver_contract = ( - runner_contract.get("decentralized_a2a_driver") - if isinstance(runner_contract.get("decentralized_a2a_driver"), dict) - else {} - ) - payload.update( - { - "schema_version": AUTO_RESEARCH_DEMO_SUPERVISOR_SCHEMA_VERSION, - "mode": "dry_run", - "layer_minimality_contract": build_auto_research_layer_contract(), - "preset": build_auto_research_preset_summary( - role_count=len(roles), - output_language=output_language, - role_specs=[ - f"{lane['agent_id']}:{lane['lane_id']}:{lane['role_id']}" - for lane in lanes - ], - ), - "auto_research": { - "schema_version": AUTO_RESEARCH_PRESET_SCHEMA_VERSION, - "uses_generic_runner": True, - "surface_count": len(roles), - "state_bus": "loopx_registry_runtime_todo_quota_frontier", - "kernel_driver": "decentralized_a2a_driver", - "kernel_driver_schema": driver_contract.get("schema_version"), - "delegated_kernel_mechanics": [ - "fixed_prompt_wakeup", - "pane_local_a2a_status_check", - "todo_evidence_status_protocol", - ], - "worker_turn_owner": "generic_multi_agent_kernel", - "visible_worker_turn_configured": worker_turn_configured, - "output_language": output_language, - "presentation_layers_in_kernel": False, - }, - "coordination_model": { - "leader_agent_required": False, - "source_of_truth": [ - "quota_should_run", - "agent_scoped_frontier", - "todo_claims", - "rollout_event_log", - ], - "pattern": "decentralized_state_a2a", - }, - "one_click_demo": { - "schema_version": "auto_research_one_click_demo_v1", - "mode": "visible_codex_tui_lanes", - "default_safe": True, - "expected_visible_result": [ - "one tmux window with tiled role panes", - "each pane starts a real interactive Codex CLI TUI", - "each role reads its projected frontier and authors a visible research artifact", - "$LOOPX_PANE_A2A_TICK is a guard/status check, not a fake research writer", - "successor todos flow through LoopX state only after role-authored evidence exists", - ], - }, - } - ) - return payload diff --git a/demo/auto_research/evidence_packet.py b/demo/auto_research/evidence_packet.py deleted file mode 100644 index 4b563ca832..0000000000 --- a/demo/auto_research/evidence_packet.py +++ /dev/null @@ -1,1018 +0,0 @@ -"""Public-safe auto-research evidence packets and rollout events.""" - -from __future__ import annotations - -import json -import hashlib -import math -import re -from pathlib import Path -from typing import Any, Iterable - -from loopx.rollout_event_log import build_rollout_event - - -RESEARCH_CONTRACT_SCHEMA_VERSION = "research_contract_v0" -AUTO_RESEARCH_BENCHMARK_CONTRACT_SCHEMA_VERSION = "auto_research_benchmark_contract_v0" -RESEARCH_HYPOTHESIS_SCHEMA_VERSION = "research_hypothesis_v0" -RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION = "research_evidence_event_v0" -AUTO_RESEARCH_EVIDENCE_PACKET_SCHEMA_VERSION = "auto_research_evidence_packet_v0" -AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION = "auto_research_rollout_append_v0" - -HYPOTHESIS_STATUSES = { - "proposed", - "active", - "running", - "needs_retry", - "supported", - "contradicted", - "promoted", - "retired", -} -EVIDENCE_STATUSES = { - "scored", - "failed_to_run", - "guardrail_failed", - "inconclusive", -} -METRIC_DIRECTIONS = {"maximize", "minimize"} -NEGATIVE_PRIMARY_METRIC_STATUSES = {"failed", "regressed"} -RETRY_PRIMARY_METRIC_STATUSES = {"inconclusive"} -RESEARCH_FAILURE_KINDS = { - "data_or_measurement_gap", - "guardrail_or_protected_boundary", - "mechanism_contradicted", - "overfit_or_nonreplication", - "retry_exhausted", - "unrecoverable_data_gap", -} - -_TOKEN_RE = re.compile(r"^[A-Za-z][A-Za-z0-9_.:-]{0,95}$") -_SHA256_RE = re.compile(r"^[A-Fa-f0-9]{64}$") -_ABSOLUTE_PATH_RE = re.compile( - r"(^|[\s:=])(?:" - + "/" + "Users/" - + "|/" + "private/" - + "|/" + "tmp/" - + "|~[/\\s]|[A-Za-z]:\\\\)" -) -_URL_OR_REMOTE_PATH_RE = re.compile(r"(?i)\b(?:file|s3|gs|tos|hdfs)://") -_PRIVATE_MARKER_TERMS = [ - "author" + "ization:", - r"bearer\s+[A-Za-z0-9._-]+", - r"api[_-]?" + "key", - "pass" + "word", - "sec" + "ret", - r"begin (?:rsa |open)?private " + "key", - "lark" + "office", - r"fei" + r"shu\.cn", - "byte" + "dance", -] -_PRIVATE_MARKER_RE = re.compile(r"(?i)(" + "|".join(_PRIVATE_MARKER_TERMS) + ")") - - -def _compact_public_text(value: Any, *, field: str, max_len: int = 240) -> str: - text = " ".join(str(value or "").strip().split()) - if not text: - raise ValueError(f"{field} must be non-empty") - if len(text) > max_len: - raise ValueError(f"{field} is too long for a compact public-safe field") - if ".." in text.replace("...", ""): - raise ValueError(f"{field} must not contain parent-directory markers") - if _ABSOLUTE_PATH_RE.search(text) or text.startswith(("/", "~")): - raise ValueError(f"{field} must use a public alias, not a local/private path") - if _URL_OR_REMOTE_PATH_RE.search(text): - raise ValueError(f"{field} must use a public alias, not a raw remote path") - if _PRIVATE_MARKER_RE.search(text): - raise ValueError(f"{field} contains a private or credential-like marker") - return text - - -def _compact_public_token(value: Any, *, field: str) -> str: - token = _compact_public_text(value, field=field, max_len=96) - if not _TOKEN_RE.match(token): - raise ValueError( - f"{field} must be a compact public token using letters, digits, dot, colon, dash, or underscore" - ) - return token - - -def _compact_public_text_list(values: Iterable[Any] | None, *, field: str) -> list[str]: - return [_compact_public_text(value, field=f"{field}[]") for value in values or []] - - -def _compact_optional_failure_kind(value: Any, *, field: str) -> str | None: - if value is None or str(value).strip() == "": - return None - kind = _compact_public_token(value, field=field) - if kind not in RESEARCH_FAILURE_KINDS: - choices = ", ".join(sorted(RESEARCH_FAILURE_KINDS)) - raise ValueError(f"{field} must be one of {choices}") - return kind - - -def _finite_float(value: float | int | str | None, *, field: str) -> float | None: - if value is None: - return None - number = float(value) - if not math.isfinite(number): - raise ValueError(f"{field} must be finite") - return number - - -def _json_obj(value: Any, *, field: str) -> dict[str, Any]: - if not isinstance(value, dict): - raise ValueError(f"{field} must be a JSON object") - return value - - -def _json_list(value: Any, *, field: str) -> list[Any]: - if not isinstance(value, list): - raise ValueError(f"{field} must be a JSON list") - return value - - -def _metric_improved( - *, - value: float | None, - baseline: float | None, - direction: str, -) -> bool: - if value is None or baseline is None: - return False - return value > baseline if direction == "maximize" else value < baseline - - -def _metric_rank_key(value: float | None, *, direction: str) -> float: - if value is None: - return float("-inf") - return value if direction == "maximize" else -value - - -def validate_research_contract(contract: dict[str, Any]) -> dict[str, Any]: - schema = _compact_public_token(contract.get("schema_version"), field="research_contract.schema_version") - if schema != RESEARCH_CONTRACT_SCHEMA_VERSION: - raise ValueError(f"research_contract.schema_version must be {RESEARCH_CONTRACT_SCHEMA_VERSION}") - metric = _json_obj(contract.get("metric"), field="research_contract.metric") - direction = _compact_public_token(metric.get("direction"), field="metric.direction") - if direction not in METRIC_DIRECTIONS: - raise ValueError("metric.direction must be maximize or minimize") - return { - "schema_version": schema, - "goal_id": _compact_public_token(contract.get("goal_id"), field="goal_id"), - "research_objective": _compact_public_text(contract.get("research_objective"), field="research_objective"), - "editable_scope": _compact_public_text_list(contract.get("editable_scope"), field="editable_scope"), - "protected_scope": _compact_public_text_list(contract.get("protected_scope"), field="protected_scope"), - "metric": { - "name": _compact_public_token(metric.get("name"), field="metric.name"), - "direction": direction, - "baseline": _finite_float(metric.get("baseline"), field="metric.baseline"), - }, - "dev_eval": _compact_public_text(contract.get("dev_eval"), field="dev_eval"), - "holdout_eval": _compact_public_text(contract.get("holdout_eval"), field="holdout_eval"), - "promotion_policy": _compact_public_token(contract.get("promotion_policy"), field="promotion_policy"), - } - - -def _command_artifact_ref(command: Any, *, field: str) -> str: - command_text = _compact_public_text(command, field=field, max_len=120) - compact = re.sub(r"[^A-Za-z0-9_.:-]+", "-", command_text).strip("-") - return _compact_public_text(f"command:{compact}", field=f"{field}.artifact_ref", max_len=140) - - -def _hash_file(path: Path) -> str | None: - if not path.is_file(): - return None - return hashlib.sha256(path.read_bytes()).hexdigest() - - -def _protected_scope_clean_from_hashes( - *, - contract_root: Path | None, - protected_scope_sha256: dict[str, Any], -) -> bool | None: - if not contract_root or not protected_scope_sha256: - return None - for relative_path, expected in protected_scope_sha256.items(): - rel = _compact_public_text(relative_path, field="protected_scope_sha256.path", max_len=120) - if rel.startswith(("/", "~")) or ".." in rel: - raise ValueError("protected_scope_sha256 paths must be relative public aliases") - expected_hash = _compact_public_text(expected, field="protected_scope_sha256.hash", max_len=64) - if not _SHA256_RE.match(expected_hash): - raise ValueError("protected_scope_sha256.hash must be a sha256 hex digest") - if _hash_file(contract_root / rel) != expected_hash: - return False - return True - - -def _normalize_research_contract_for_packet( - contract: dict[str, Any], - *, - contract_root: Path | None = None, -) -> tuple[dict[str, Any], dict[str, Any] | None]: - schema = _compact_public_token(contract.get("schema_version"), field="research_contract.schema_version") - if schema == RESEARCH_CONTRACT_SCHEMA_VERSION: - return contract, None - if schema != AUTO_RESEARCH_BENCHMARK_CONTRACT_SCHEMA_VERSION: - raise ValueError(f"research_contract.schema_version must be {RESEARCH_CONTRACT_SCHEMA_VERSION}") - - metric = _json_obj(contract.get("metric"), field="benchmark_contract.metric") - normalized = { - "schema_version": RESEARCH_CONTRACT_SCHEMA_VERSION, - "goal_id": contract.get("goal_id") or contract.get("preset_id") or "auto_research_benchmark", - "research_objective": ( - contract.get("research_objective") - or contract.get("objective") - or contract.get("claim_boundary") - or "Improve the benchmark metric inside the declared scope." - ), - "editable_scope": contract.get("editable_scope") or [], - "protected_scope": contract.get("protected_scope") or [], - "metric": { - "name": metric.get("name"), - "direction": metric.get("direction"), - "baseline": metric.get("baseline"), - }, - "dev_eval": contract.get("dev_eval") or contract.get("dev_eval_command"), - "holdout_eval": contract.get("holdout_eval") or contract.get("holdout_eval_command"), - "promotion_policy": contract.get("promotion_policy") or "heldout_improvement_required", - } - benchmark_context = { - "schema_version": schema, - "preset_id": contract.get("preset_id"), - "contract_root": contract_root, - "protected_scope_clean": _protected_scope_clean_from_hashes( - contract_root=contract_root, - protected_scope_sha256=_json_obj( - contract.get("protected_scope_sha256") or {}, - field="benchmark_contract.protected_scope_sha256", - ), - ), - } - return validate_research_contract(normalized), benchmark_context - - -def _primary_metric_status( - *, - ok: bool, - value: float | None, - baseline: float | None, - direction: str, -) -> str: - if not ok: - return "failed" - if value is None or baseline is None: - return "inconclusive" - if _metric_improved(value=value, baseline=baseline, direction=direction): - return "improved" - if value == baseline: - return "flat" - return "regressed" - - -def _normalize_benchmark_eval_result( - result: dict[str, Any], - *, - contract: dict[str, Any], - benchmark_context: dict[str, Any] | None, -) -> dict[str, Any]: - if "metric" in result: - return result - if "score" not in result and "metric_name" not in result: - return result - - raw_split = _compact_public_token(result.get("split") or "dev", field="eval_result.split") - split = "holdout" if raw_split == "test" else raw_split - metric = contract["metric"] - direction = _compact_public_token( - result.get("direction") or metric["direction"], - field="eval_result.direction", - ) - if direction not in METRIC_DIRECTIONS: - raise ValueError("eval_result.direction must be maximize or minimize") - value = _finite_float(result.get("score"), field="eval_result.score") - baseline = _finite_float( - result.get("baseline_score", metric["baseline"]), - field="eval_result.baseline_score", - ) - ok = bool(result.get("ok")) - command = contract["holdout_eval"] if split == "holdout" else contract["dev_eval"] - protected_scope_clean = result.get("protected_scope_clean") - if protected_scope_clean is None and benchmark_context: - protected_scope_clean = benchmark_context.get("protected_scope_clean") - if protected_scope_clean is None: - protected_scope_clean = False - return { - "schema_version": "auto_research_benchmark_eval_result_projection_v0", - "split": split, - "metric": { - "name": result.get("metric_name") or metric["name"], - "direction": direction, - "value": value, - "baseline": baseline, - }, - "eval_status": "scored" if ok else "guardrail_failed", - "primary_metric_status": _primary_metric_status( - ok=ok, - value=value, - baseline=baseline, - direction=direction, - ), - "failure_kind": result.get("failure_kind"), - "measurement_scope": result.get("measurement_scope"), - "remediation_attempt": result.get("remediation_attempt"), - "artifact_refs": result.get("artifact_refs") or [ - f"public_eval:{split}:{result.get('metric_name') or metric['name']}", - _command_artifact_ref(command, field=f"eval_result.{split}.command"), - ], - "protected_scope_clean": bool(protected_scope_clean), - "no_upload": result.get("no_upload", True), - } - - -def validate_research_hypothesis(item: dict[str, Any]) -> dict[str, Any]: - schema = _compact_public_token(item.get("schema_version"), field="hypothesis.schema_version") - if schema != RESEARCH_HYPOTHESIS_SCHEMA_VERSION: - raise ValueError(f"hypothesis.schema_version must be {RESEARCH_HYPOTHESIS_SCHEMA_VERSION}") - status = _compact_public_token(item.get("status"), field="hypothesis.status") - if status not in HYPOTHESIS_STATUSES: - raise ValueError(f"hypothesis.status must be one of {', '.join(sorted(HYPOTHESIS_STATUSES))}") - normalized = { - "schema_version": schema, - "hypothesis_id": _compact_public_token(item.get("hypothesis_id"), field="hypothesis_id"), - "parent_hypothesis_id": ( - _compact_public_token(item.get("parent_hypothesis_id"), field="parent_hypothesis_id") - if item.get("parent_hypothesis_id") - else None - ), - "todo_id": _compact_public_token(item.get("todo_id"), field="todo_id"), - "claimed_by": _compact_public_token(item.get("claimed_by"), field="claimed_by"), - "mechanism_family": _compact_public_text(item.get("mechanism_family"), field="mechanism_family"), - "hypothesis": _compact_public_text(item.get("hypothesis"), field="hypothesis"), - "status": status, - "grounding_refs": _compact_public_text_list(item.get("grounding_refs"), field="grounding_refs"), - "novelty_audit_ref": ( - _compact_public_text(item.get("novelty_audit_ref"), field="novelty_audit_ref") - if item.get("novelty_audit_ref") - else None - ), - "blocked_by": _compact_public_text_list(item.get("blocked_by"), field="blocked_by"), - } - normalized.update( - _normalized_record_contract_lineage( - item, - field="hypothesis", - ) - ) - return normalized - - -def _normalized_record_contract_lineage( - item: dict[str, Any], - *, - field: str, -) -> dict[str, str]: - from .delivery_contract import ( - auto_research_contract_lineage_dict, - normalize_auto_research_contract_lineage, - ) - - lineage = normalize_auto_research_contract_lineage(item, field=field) - return auto_research_contract_lineage_dict(lineage) if lineage else {} - - -def validate_research_evidence_event(item: dict[str, Any]) -> dict[str, Any]: - schema = _compact_public_token(item.get("schema_version"), field="evidence.schema_version") - if schema != RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION: - raise ValueError(f"evidence.schema_version must be {RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION}") - eval_status = _compact_public_token(item.get("eval_status"), field="eval_status") - if eval_status not in EVIDENCE_STATUSES: - raise ValueError(f"eval_status must be one of {', '.join(sorted(EVIDENCE_STATUSES))}") - metric = _json_obj(item.get("metric"), field="evidence.metric") - direction = _compact_public_token(metric.get("direction"), field="evidence.metric.direction") - if direction not in METRIC_DIRECTIONS: - raise ValueError("evidence.metric.direction must be maximize or minimize") - failure_kind = _compact_optional_failure_kind( - item.get("failure_kind"), - field="failure_kind", - ) - measurement_scope = ( - _compact_public_text(item.get("measurement_scope"), field="measurement_scope") - if item.get("measurement_scope") - else None - ) - remediation_attempt = bool(item.get("remediation_attempt")) - if failure_kind == "data_or_measurement_gap" and not measurement_scope: - raise ValueError("data or measurement gaps require measurement_scope") - if remediation_attempt and failure_kind != "data_or_measurement_gap": - raise ValueError( - "remediation attempts require data_or_measurement_gap failure_kind" - ) - normalized = { - "schema_version": schema, - "hypothesis_id": _compact_public_token(item.get("hypothesis_id"), field="hypothesis_id"), - "todo_id": _compact_public_token(item.get("todo_id"), field="todo_id"), - "agent_id": _compact_public_token(item.get("agent_id"), field="agent_id"), - "attempt": int(item.get("attempt") or 1), - "split": _compact_public_token(item.get("split"), field="split"), - "metric": { - "name": _compact_public_token(metric.get("name"), field="evidence.metric.name"), - "value": _finite_float(metric.get("value"), field="evidence.metric.value"), - "direction": direction, - }, - "baseline_metric": _finite_float(item.get("baseline_metric"), field="baseline_metric"), - "eval_status": eval_status, - "primary_metric_status": _compact_public_token( - item.get("primary_metric_status"), - field="primary_metric_status", - ), - "failure_kind": failure_kind, - "measurement_scope": measurement_scope, - "remediation_attempt": remediation_attempt, - "artifact_refs": _compact_public_text_list(item.get("artifact_refs"), field="artifact_refs"), - "protected_scope_clean": bool(item.get("protected_scope_clean")), - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - } - normalized.update( - _normalized_record_contract_lineage( - item, - field="evidence", - ) - ) - return normalized - - -def _load_json_object(path: str | Path, *, field: str) -> dict[str, Any]: - return _json_obj(json.loads(Path(path).expanduser().read_text(encoding="utf-8")), field=field) - - -def _eval_result_to_evidence_event( - result: dict[str, Any], - *, - contract: dict[str, Any], - hypothesis_id: str, - todo_id: str, - agent_id: str, - attempt: int, - branch_ref: str | None = None, - benchmark_context: dict[str, Any] | None = None, -) -> dict[str, Any]: - result = _json_obj(result, field="eval_result") - result = _normalize_benchmark_eval_result( - result, - contract=contract, - benchmark_context=benchmark_context, - ) - if result.get("no_upload") is False: - raise ValueError("eval_result.no_upload must not be false for public auto-research evidence") - metric = _json_obj(result.get("metric"), field="eval_result.metric") - artifact_refs = _compact_public_text_list(result.get("artifact_refs"), field="eval_result.artifact_refs") - if branch_ref: - artifact_refs.append(f"branch:{_compact_public_text(branch_ref, field='branch_ref', max_len=160)}") - event = { - "schema_version": RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION, - "hypothesis_id": hypothesis_id, - "todo_id": todo_id, - "agent_id": agent_id, - "attempt": attempt, - "split": result.get("split"), - "metric": { - "name": metric.get("name"), - "value": metric.get("value"), - "direction": metric.get("direction"), - }, - "baseline_metric": metric.get("baseline", result.get("baseline_metric")), - "eval_status": result.get("eval_status"), - "primary_metric_status": result.get("primary_metric_status"), - "failure_kind": result.get("failure_kind"), - "measurement_scope": result.get("measurement_scope"), - "remediation_attempt": result.get("remediation_attempt"), - "artifact_refs": artifact_refs, - "protected_scope_clean": bool(result.get("protected_scope_clean")), - } - return validate_research_evidence_event(event) - - -def _derive_hypothesis_status(events: list[dict[str, Any]]) -> str: - if any( - not event["protected_scope_clean"] - or event["eval_status"] == "guardrail_failed" - or event["primary_metric_status"] in NEGATIVE_PRIMARY_METRIC_STATUSES - for event in events - ): - return "contradicted" - if any( - event["eval_status"] in {"failed_to_run", "inconclusive"} - or event["primary_metric_status"] in RETRY_PRIMARY_METRIC_STATUSES - for event in events - ): - return "needs_retry" - if any( - _metric_improved( - value=event["metric"]["value"], - baseline=event["baseline_metric"], - direction=event["metric"]["direction"], - ) - for event in events - if event["eval_status"] == "scored" - ): - return "supported" - return "active" - - -def _is_negative_evidence_event(event: dict[str, Any]) -> bool: - return ( - not event["protected_scope_clean"] - or event["eval_status"] == "guardrail_failed" - or event["primary_metric_status"] in NEGATIVE_PRIMARY_METRIC_STATUSES - ) - - -def _is_retry_evidence_event(event: dict[str, Any]) -> bool: - return ( - event["eval_status"] in {"failed_to_run", "inconclusive"} - or event["primary_metric_status"] in RETRY_PRIMARY_METRIC_STATUSES - ) - - -def _normalize_packet_contract( - contract: dict[str, Any], - *, - contract_root: str | Path | None, -) -> tuple[dict[str, Any], dict[str, Any] | None, dict[str, Any], dict[str, Any] | None]: - from .delivery_contract import ( - AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION, - build_auto_research_contract_lineage, - normalize_auto_research_delivery_contract, - ) - - delivery_contract: dict[str, Any] | None = None - if ( - contract.get("schema_version") - == AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION - ): - delivery_contract = normalize_auto_research_delivery_contract(contract) - contract = delivery_contract["research_contract"] - normalized_contract, benchmark_context = _normalize_research_contract_for_packet( - contract, - contract_root=Path(contract_root).expanduser() if contract_root else None, - ) - return ( - normalized_contract, - delivery_contract, - build_auto_research_contract_lineage( - normalized_contract, - delivery_contract=delivery_contract, - ), - benchmark_context, - ) - - -def _apply_contract_lineage( - *, - hypothesis: dict[str, Any], - evidence_events: list[dict[str, Any]], - lineage: dict[str, Any], -) -> None: - if not lineage.get("wish_id"): - return - values = _normalized_record_contract_lineage( - lineage, - field="contract_lineage", - ) - hypothesis.update(values) - for event in evidence_events: - event.update(values) - - -def build_auto_research_evidence_packet( - *, - contract: dict[str, Any], - eval_results: list[dict[str, Any]], - hypothesis_id: str, - todo_id: str, - agent_id: str, - claimed_by: str, - mechanism_family: str, - hypothesis: str, - parent_hypothesis_id: str | None = None, - grounding_refs: list[str] | None = None, - novelty_audit_ref: str | None = None, - branch_ref: str | None = None, - attempt_start: int = 1, - contract_root: str | Path | None = None, -) -> dict[str, Any]: - """Build public-safe research hypothesis/evidence records from eval outputs.""" - - contract, delivery_contract, contract_lineage, benchmark_context = ( - _normalize_packet_contract( - contract, - contract_root=contract_root, - ) - ) - if not eval_results: - raise ValueError("at least one eval result is required") - hypothesis_token = _compact_public_token(hypothesis_id, field="hypothesis_id") - todo_token = _compact_public_token(todo_id, field="todo_id") - agent_token = _compact_public_token(agent_id, field="agent_id") - events = [ - _eval_result_to_evidence_event( - result, - contract=contract, - hypothesis_id=hypothesis_token, - todo_id=todo_token, - agent_id=agent_token, - attempt=attempt_start + index, - branch_ref=branch_ref, - benchmark_context=benchmark_context, - ) - for index, result in enumerate(eval_results) - ] - status = _derive_hypothesis_status(events) - hypothesis_node = validate_research_hypothesis( - { - "schema_version": RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - "hypothesis_id": hypothesis_token, - "parent_hypothesis_id": parent_hypothesis_id, - "todo_id": todo_token, - "claimed_by": claimed_by, - "mechanism_family": mechanism_family, - "hypothesis": hypothesis, - "status": status, - "grounding_refs": grounding_refs or [], - "novelty_audit_ref": novelty_audit_ref, - "blocked_by": ( - ["evidence_or_boundary_guardrail_failed"] - if status == "contradicted" - else [] - ), - } - ) - _apply_contract_lineage( - hypothesis=hypothesis_node, - evidence_events=events, - lineage=contract_lineage, - ) - negative_count = len([event for event in events if _is_negative_evidence_event(event)]) - packet = { - "ok": True, - "schema_version": AUTO_RESEARCH_EVIDENCE_PACKET_SCHEMA_VERSION, - "research_contract": contract, - "hypothesis": hypothesis_node, - "evidence_events": events, - "summary": { - "goal_id": contract["goal_id"], - "hypothesis_id": hypothesis_node["hypothesis_id"], - "todo_id": hypothesis_node["todo_id"], - "status": hypothesis_node["status"], - "evidence_event_count": len(events), - "splits": sorted({event["split"] for event in events}), - "negative_evidence_count": negative_count, - "needs_retry_count": len([event for event in events if _is_retry_evidence_event(event)]), - "protected_scope_clean": all(event["protected_scope_clean"] for event in events), - }, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "source": "public_eval_result_projection", - }, - } - if delivery_contract: - packet["delivery_contract"] = delivery_contract - packet["contract_lineage"] = contract_lineage - return packet - - -def load_auto_research_evidence_packet_inputs( - *, - contract_path: str | Path, - eval_result_paths: list[str | Path], - **kwargs: Any, -) -> dict[str, Any]: - resolved_contract_path = Path(contract_path).expanduser() - return build_auto_research_evidence_packet( - contract=_load_json_object(resolved_contract_path, field="research_contract_file"), - eval_results=[ - _load_json_object(path, field="eval_result_file") - for path in eval_result_paths - ], - contract_root=resolved_contract_path.parent, - **kwargs, - ) - - -def load_auto_research_evidence_packet(path: str | Path) -> dict[str, Any]: - return validate_auto_research_evidence_packet( - _load_json_object(path, field="auto_research_evidence_packet_file") - ) - - -def _delivery_contract_for_packet( - payload: dict[str, Any], - *, - research_contract: dict[str, Any], -) -> tuple[dict[str, Any] | None, dict[str, Any]]: - from .delivery_contract import ( - build_auto_research_contract_lineage, - normalize_auto_research_delivery_contract, - ) - - delivery_contract = ( - normalize_auto_research_delivery_contract( - _json_obj( - payload.get("delivery_contract"), - field="delivery_contract", - ) - ) - if payload.get("delivery_contract") is not None - else None - ) - expected_lineage = build_auto_research_contract_lineage( - research_contract, - delivery_contract=delivery_contract, - ) - provided_lineage = payload.get("contract_lineage") - if provided_lineage is not None and provided_lineage != expected_lineage: - raise ValueError( - "contract_lineage must match the normalized delivery contract" - ) - return delivery_contract, expected_lineage - - -def _validate_packet_lineage( - *, - delivery_contract: dict[str, Any] | None, - expected_lineage: dict[str, Any], - hypothesis: dict[str, Any], - evidence_events: list[dict[str, Any]], -) -> None: - if not delivery_contract: - return - from .delivery_contract import ( - auto_research_contract_lineage_matches, - normalize_auto_research_contract_lineage, - ) - - expected = normalize_auto_research_contract_lineage( - expected_lineage, - field="contract_lineage", - required=True, - ) - if not auto_research_contract_lineage_matches( - hypothesis, - expected=expected, - ): - raise ValueError( - "delivery hypothesis lineage must match the normalized contract" - ) - for event in evidence_events: - if not auto_research_contract_lineage_matches( - event, - expected=expected, - ): - raise ValueError( - "delivery evidence lineage must match the normalized contract" - ) - - -def _validate_packet_event_links( - *, - hypothesis: dict[str, Any], - evidence_events: list[dict[str, Any]], -) -> None: - if not evidence_events: - raise ValueError( - "auto_research_evidence_packet requires at least one evidence event" - ) - for event in evidence_events: - if event["hypothesis_id"] != hypothesis["hypothesis_id"]: - raise ValueError( - "evidence event hypothesis_id must match packet hypothesis" - ) - if event["todo_id"] != hypothesis["todo_id"]: - raise ValueError( - "evidence event todo_id must match packet hypothesis" - ) - - -def _validate_packet_public_boundary(payload: dict[str, Any]) -> None: - public_boundary = _json_obj( - payload.get("public_boundary"), - field="public_boundary", - ) - if public_boundary.get("raw_logs_recorded") or public_boundary.get( - "private_artifacts_recorded" - ): - raise ValueError( - "auto_research_evidence_packet must not record raw logs or " - "private artifacts" - ) - - -def validate_auto_research_evidence_packet(payload: dict[str, Any]) -> dict[str, Any]: - payload = _json_obj(payload, field="auto_research_evidence_packet") - schema = _compact_public_token(payload.get("schema_version"), field="schema_version") - if schema != AUTO_RESEARCH_EVIDENCE_PACKET_SCHEMA_VERSION: - raise ValueError(f"schema_version must be {AUTO_RESEARCH_EVIDENCE_PACKET_SCHEMA_VERSION}") - if payload.get("ok") is False: - raise ValueError("auto_research_evidence_packet.ok must not be false") - contract = validate_research_contract(_json_obj(payload.get("research_contract"), field="research_contract")) - delivery_contract, expected_lineage = _delivery_contract_for_packet( - payload, - research_contract=contract, - ) - hypothesis = validate_research_hypothesis(_json_obj(payload.get("hypothesis"), field="hypothesis")) - evidence_events = [ - validate_research_evidence_event(_json_obj(item, field="evidence_events[]")) - for item in _json_list(payload.get("evidence_events"), field="evidence_events") - ] - _validate_packet_event_links( - hypothesis=hypothesis, - evidence_events=evidence_events, - ) - _validate_packet_lineage( - delivery_contract=delivery_contract, - expected_lineage=expected_lineage, - hypothesis=hypothesis, - evidence_events=evidence_events, - ) - _validate_packet_public_boundary(payload) - normalized = { - "ok": True, - "schema_version": schema, - "research_contract": contract, - "hypothesis": hypothesis, - "evidence_events": evidence_events, - "summary": { - "goal_id": contract["goal_id"], - "hypothesis_id": hypothesis["hypothesis_id"], - "todo_id": hypothesis["todo_id"], - "status": hypothesis["status"], - "evidence_event_count": len(evidence_events), - "splits": sorted({event["split"] for event in evidence_events}), - "negative_evidence_count": len( - [event for event in evidence_events if _is_negative_evidence_event(event)] - ), - "needs_retry_count": len([event for event in evidence_events if _is_retry_evidence_event(event)]), - "protected_scope_clean": all(event["protected_scope_clean"] for event in evidence_events), - }, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "source": "public_eval_result_projection", - }, - } - if delivery_contract: - normalized["delivery_contract"] = delivery_contract - normalized["contract_lineage"] = expected_lineage - return normalized - - -def build_auto_research_rollout_events( - packet: dict[str, Any], - *, - recorded_at: str | None = None, -) -> list[dict[str, Any]]: - packet = validate_auto_research_evidence_packet(packet) - contract = packet["research_contract"] - hypothesis = packet["hypothesis"] - summary = packet["summary"] - lineage = packet.get("contract_lineage") or {} - goal_id = contract["goal_id"] - hypothesis_id = hypothesis["hypothesis_id"] - claimed_by = hypothesis["claimed_by"] - source_refs = [ - {"kind": "grounding", "id": ref} - for ref in hypothesis.get("grounding_refs") or [] - ] - if hypothesis.get("novelty_audit_ref"): - source_refs.append({"kind": "novelty_audit", "id": hypothesis["novelty_audit_ref"]}) - hypothesis_details = { - "hypothesis_id": hypothesis_id, - "parent_hypothesis_id": hypothesis.get("parent_hypothesis_id") or "", - "mechanism_family": hypothesis["mechanism_family"], - "evidence_event_count": summary["evidence_event_count"], - "negative_evidence_count": summary["negative_evidence_count"], - "needs_retry_count": summary["needs_retry_count"], - "protected_scope_clean": summary["protected_scope_clean"], - } - if lineage.get("wish_id"): - hypothesis_details.update( - { - "wish_id": lineage["wish_id"], - "contract_ref": lineage["contract_ref"], - "contract_revision": lineage["contract_revision"], - } - ) - events = [] - if lineage.get("wish_id"): - events.append( - build_rollout_event( - goal_id=goal_id, - event_kind="validation", - status="recorded", - classification=packet["delivery_contract"]["schema_version"], - labels=[ - "auto_research", - "delivery_contract", - "wish_to_artifact", - ], - summary=( - "Auto Research delivery contract recorded for " - f"{lineage['wish_id']} at {lineage['contract_revision']}." - ), - artifact_refs=[lineage["contract_ref"]], - details={ - "wish_id": lineage["wish_id"], - "contract_ref": lineage["contract_ref"], - "contract_revision": lineage["contract_revision"], - "required_artifact_count": len( - packet["delivery_contract"]["required_artifacts"] - ), - "acceptance_criterion_count": len( - packet["delivery_contract"]["acceptance_criteria"] - ), - }, - recorded_at=recorded_at, - ) - ) - events.append( - build_rollout_event( - goal_id=goal_id, - event_kind="research_hypothesis", - agent_id=claimed_by, - todo_id=hypothesis["todo_id"], - lane_id=f"agent:{claimed_by}", - agent_role="auto_research_lane", - status=hypothesis["status"], - classification=RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - labels=[ - "auto_research", - "research_hypothesis", - hypothesis["status"], - hypothesis["mechanism_family"], - ], - summary=( - f"auto-research hypothesis {hypothesis_id} status={hypothesis['status']}: " - f"{hypothesis['hypothesis']}" - ), - source_refs=source_refs, - details=hypothesis_details, - recorded_at=recorded_at, - ) - ) - for evidence in packet["evidence_events"]: - metric = evidence["metric"] - evidence_details = { - "hypothesis_id": evidence["hypothesis_id"], - "attempt": evidence["attempt"], - "split": evidence["split"], - "metric_name": metric["name"], - "metric_value": metric["value"], - "metric_direction": metric["direction"], - "baseline_metric": evidence["baseline_metric"], - "primary_metric_status": evidence["primary_metric_status"], - "failure_kind": evidence["failure_kind"] or "", - "measurement_scope": evidence["measurement_scope"] or "", - "remediation_attempt": evidence["remediation_attempt"], - "eval_status": evidence["eval_status"], - "protected_scope_clean": evidence["protected_scope_clean"], - } - if lineage.get("wish_id"): - evidence_details.update( - { - "wish_id": lineage["wish_id"], - "contract_ref": lineage["contract_ref"], - "contract_revision": lineage["contract_revision"], - } - ) - events.append( - build_rollout_event( - goal_id=goal_id, - event_kind="research_evidence", - agent_id=evidence["agent_id"], - todo_id=evidence["todo_id"], - run_id=f"{evidence['hypothesis_id']}:{evidence['attempt']}:{evidence['split']}", - lane_id=f"agent:{evidence['agent_id']}", - agent_role="auto_research_lane", - status=evidence["eval_status"], - classification=RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION, - labels=[ - "auto_research", - "research_evidence", - evidence["split"], - evidence["eval_status"], - evidence["primary_metric_status"], - ], - summary=( - f"auto-research evidence {evidence['hypothesis_id']} " - f"split={evidence['split']} status={evidence['primary_metric_status']} " - f"value={metric['value']}" - ), - artifact_refs=evidence["artifact_refs"], - details=evidence_details, - recorded_at=recorded_at, - ) - ) - return events diff --git a/demo/auto_research/human_view.py b/demo/auto_research/human_view.py deleted file mode 100644 index f7f31b1b4a..0000000000 --- a/demo/auto_research/human_view.py +++ /dev/null @@ -1,831 +0,0 @@ -"""Human-readable auto-research renderers. - -The auto-research kernel and state readers emit machine payloads. This module -keeps the optional Markdown view separate so production logic does not depend -on legacy presentation helpers. -""" - -from __future__ import annotations - -from .artifact_receipt import AUTO_RESEARCH_ARTIFACT_RECEIPT_SCHEMA_VERSION -from .evidence_packet import AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION -from .preset import build_auto_research_minimal_a2a_recipe -from .user_contract import AUTO_RESEARCH_USER_CONTRACT_SCHEMA_VERSION - - -AUTO_RESEARCH_DEMO_E2E_SCHEMA_VERSION = "auto_research_demo_e2e_result_v0" - - -def _render_failure(payload: dict[str, object]) -> str: - lines = ["# LoopX Auto Research", "", "- ok: `False`"] - if payload.get("error_code"): - lines.append(f"- error_code: `{payload.get('error_code')}`") - if payload.get("error"): - lines.append(f"- error: `{payload.get('error')}`") - return "\n".join(lines) + "\n" - - -def _join_or_none(values: object) -> str: - if isinstance(values, list) and values: - return ", ".join(str(item) for item in values) - return "none" - - -def _dict_value(payload: dict[str, object], key: str) -> dict[str, object]: - value = payload.get(key) - return value if isinstance(value, dict) else {} - - -def _string_value(payload: dict[str, object], key: str) -> str | None: - value = payload.get(key) - if isinstance(value, str) and value.strip(): - return value - return None - - -def _metric_sequence(value: object) -> str: - if isinstance(value, list) and value: - return " -> ".join(str(item) for item in value) - return "none" - - -_ROLE_DESCRIPTIONS = { - "research_curator": "research-curator / 研究策展：界定问题、指标和证据边界", - "hypothesis_proposer": "hypothesis-proposer / 假设生成：提出下一轮可验证假设", - "research_executor": "research-executor / 研究执行：运行 dev/holdout 证据", - "evaluator_promoter": "evaluator-promoter / 评估推进：判断提升并触发后续 todo", -} - -_AGENT_ROLE_IDS = { - "research-curator": "research_curator", - "hypothesis-proposer": "hypothesis_proposer", - "research-executor": "research_executor", - "evaluator-promoter": "evaluator_promoter", -} - - -def _role_description(agent_id: object, role_id: object) -> str: - role = str(role_id or "").strip() - agent = str(agent_id or "").strip() - if not role: - role = _AGENT_ROLE_IDS.get(agent, "") - label = _ROLE_DESCRIPTIONS.get(role) - if label: - return label - return agent or role or "unknown-role" - - -def _role_description_from_spec_line(raw: object) -> str: - parts = [part.strip() for part in str(raw).split(":")] - agent_id = parts[0] if parts else "" - role_id = parts[2] if len(parts) >= 3 else "" - return _role_description(agent_id, role_id) - - -def _render_research_roles( - *, - minimal_recipe: dict[str, object], - collective_rounds: dict[str, object], -) -> list[str]: - kernel = _dict_value(collective_rounds, "kernel_ledger") - lanes = kernel.get("expected_lanes") if isinstance(kernel.get("expected_lanes"), list) else [] - role_lines: list[str] = [] - for lane in lanes: - if isinstance(lane, dict): - role_lines.append( - f"- {_role_description(lane.get('agent_id'), lane.get('role_id'))}" - ) - if not role_lines: - for raw in minimal_recipe.get("preset_recipe_lines") or []: - role_lines.append(f"- {_role_description_from_spec_line(raw)}") - if not role_lines: - return [] - return ["", "## Research Roles / 研究角色", "", *role_lines] - - -def _render_collective_round_summary( - *, - worker_loop: dict[str, object], - collective_rounds: dict[str, object], -) -> list[str]: - if not collective_rounds: - return [] - kernel = _dict_value(collective_rounds, "kernel_ledger") - outcomes = kernel.get("lane_outcomes") if isinstance(kernel.get("lane_outcomes"), list) else [] - by_round: dict[int, list[dict[str, object]]] = {} - for outcome in outcomes: - if not isinstance(outcome, dict): - continue - try: - round_index = int(outcome.get("round") or 0) - except (TypeError, ValueError): - continue - if round_index > 0: - by_round.setdefault(round_index, []).append(outcome) - - completed_turns = worker_loop.get("completed_turn_count") - expected_turns = kernel.get("lane_outcome_count") - participation_gap = ( - collective_rounds.get("full_participation_requirement_gap") - if isinstance(collective_rounds.get("full_participation_requirement_gap"), dict) - else {} - ) - lines = [ - "", - "## Collective Rounds / 集体研究轮次", - "", - ( - f"- verified: `{collective_rounds.get('multi_round_research_verified')}`; " - f"rounds: `{collective_rounds.get('collective_round_count')}`; " - f"full_participation_rounds: `{collective_rounds.get('full_participation_round_count')}`; " - f"basis: `{collective_rounds.get('full_participation_count_basis')}`; " - f"completed_turns: `{completed_turns}/{expected_turns}`" - ), - ( - f"- claim_source: `{collective_rounds.get('claim_source') or collective_rounds.get('source')}`; " - f"visible_role_participation_verified: " - f"`{collective_rounds.get('visible_role_participation_verified')}`" - ), - f"- claim_boundary: {collective_rounds.get('claim_boundary')}", - ( - f"- role_cycle_gap: `{participation_gap.get('shortfall_by_agent') or {}}` " - f"(required `{participation_gap.get('required_count')}`)" - ), - ( - f"- dev_metric_sequence: `{_metric_sequence(collective_rounds.get('dev_metric_sequence'))}`; " - f"holdout_metric_sequence: `{_metric_sequence(collective_rounds.get('holdout_metric_sequence'))}`" - ), - ( - f"- holdout_improvement_count: `{collective_rounds.get('holdout_improvement_count')}` " - f"(required `{collective_rounds.get('required_holdout_improvement_count')}`)" - ), - ( - "- round_semantics: one round is one quota/frontier opportunity for each " - "research role; lanes without a selected todo are shown as no-op instead of hidden." - ), - "", - ] - for round_index in sorted(by_round): - executed: list[str] = [] - noops: list[str] = [] - for outcome in by_round[round_index]: - agent_id = str(outcome.get("agent_id") or "").strip() - action = str(outcome.get("selected_action") or "").strip() - if outcome.get("executed"): - metric = "" - if outcome.get("dev_metric") is not None: - metric = f" dev={outcome.get('dev_metric')}" - if outcome.get("holdout_metric") is not None: - metric = f" holdout={outcome.get('holdout_metric')}" - executed.append(f"{agent_id}:{action}{metric}") - else: - noops.append(agent_id) - executed_text = "; ".join(executed) if executed else "none" - noops_text = "; ".join(noops) if noops else "none" - lines.append(f"- Round {round_index}: executed `{executed_text}`; no_op `{noops_text}`") - return lines - - -def _render_operator_commands(payload: dict[str, object]) -> list[str]: - commands = _dict_value(payload, "commands") - user_contract = _dict_value(payload, "user_contract") or payload - one_click_start = _dict_value(user_contract, "one_click_start") - visible_launch = _dict_value(payload, "visible_launch") - launch_result = _dict_value(visible_launch, "launch_result") - default_start = ( - _string_value(one_click_start, "command") - or _string_value(commands, "one_question_start_with_visible_wake") - or _string_value(commands, "one_question_start") - ) - takeover_start = _string_value(one_click_start, "operator_takeover_command") - attach_semantics = _string_value(one_click_start, "attach_semantics") - attach_command = _string_value(launch_result, "attach_command") - stop_command = _string_value(launch_result, "stop_command") - lines = ["", "## Operator Commands", ""] - if default_start: - lines.append(f"- visible role start: `{default_start}`") - if takeover_start: - lines.append(f"- immediate takeover: `{takeover_start}`") - if attach_semantics: - lines.append(f"- attach_semantics: {attach_semantics}") - if attach_command: - lines.append(f"- tmux attach: `{attach_command}`") - if stop_command: - lines.append(f"- tmux stop: `{stop_command}`") - if len(lines) == 3: - return [] - return lines - - -def _render_user_contract(payload: dict[str, object]) -> str: - brief = _dict_value(payload, "research_brief") - plan = payload.get("action_plan") if isinstance(payload.get("action_plan"), list) else [] - evidence = _dict_value(payload, "evidence_refs") - one_click_start = _dict_value(payload, "one_click_start") - next_step = _dict_value(payload, "next_executable_step") - gate = _dict_value(payload, "gate") - preset_context = _dict_value(payload, "preset_context") - gates = gate.get("user_judgment_needed") if isinstance(gate.get("user_judgment_needed"), list) else [] - lines = [ - "# LoopX Auto Research", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- question: {payload.get('open_question')}", - "", - "## Research Brief", - "", - f"- read: `{_join_or_none(brief.get('read'))}`", - f"- not_read: `{_join_or_none(brief.get('not_read'))}`", - f"- claim_boundary: {brief.get('claim_boundary')}", - "", - ] - if preset_context: - lines.extend( - [ - "## Preset Context", - "", - f"- preset_id: `{preset_context.get('preset_id')}`", - f"- baseline_source: `{preset_context.get('baseline_source')}`", - f"- question_text_supplies_baseline: `{preset_context.get('question_text_supplies_baseline')}`", - f"- metric_name: `{preset_context.get('metric_name')}`", - f"- baseline_metric: `{preset_context.get('baseline_metric')}`", - f"- benchmark_contract_file: `{preset_context.get('benchmark_contract_file')}`", - f"- editable_scope: `{_join_or_none(preset_context.get('editable_scope'))}`", - f"- protected_scope: `{_join_or_none(preset_context.get('protected_scope'))}`", - f"- dev_eval_command: `{preset_context.get('dev_eval_command')}`", - f"- holdout_eval_command: `{preset_context.get('holdout_eval_command')}`", - f"- claim_boundary: {preset_context.get('claim_boundary')}", - "", - ] - ) - lines.extend(["## Action Plan", ""]) - if not plan: - lines.append("- none") - for item in plan: - if not isinstance(item, dict): - continue - lines.append( - f"- {item.get('priority')}: {item.get('todo')} " - f"(`{item.get('owner_layer')}`)" - ) - lines.extend( - [ - "", - "## Evidence Refs", - "", - f"- code: `{_join_or_none(evidence.get('code'))}`", - f"- docs: `{_join_or_none(evidence.get('docs'))}`", - f"- benchmarks: `{_join_or_none(evidence.get('benchmarks'))}`", - f"- issues: `{_join_or_none(evidence.get('issues'))}`", - f"- pull_requests: `{_join_or_none(evidence.get('pull_requests'))}`", - "", - "## One-Click Start", - "", - f"- command: `{one_click_start.get('command')}`", - f"- operator_takeover: `{one_click_start.get('operator_takeover_command')}`", - f"- preview: `{one_click_start.get('preview_command')}`", - f"- starts: `{one_click_start.get('starts')}`", - f"- coordination_model: `{one_click_start.get('coordination_model')}`", - "", - ] - ) - lines.extend(_render_operator_commands(payload)) - lines.extend( - [ - "", - "## Next Executable Step", - "", - f"- can_run_automatically: `{next_step.get('can_run_automatically')}`", - f"- step: {next_step.get('step')}", - "", - "## Gate", - "", - ] - ) - if not gates: - lines.append("- none") - for item in gates: - lines.append(f"- {item}") - lines.append(f"- default_without_user_gate: {gate.get('default_without_user_gate')}") - return "\n".join(lines) + "\n" - - -def render_auto_research_projection_markdown(payload: dict[str, object]) -> str: - if not payload.get("ok"): - return _render_failure(payload) - if "frontier" not in payload or "evidence_graph" not in payload: - return _render_generic(payload) - frontier = _dict_value(payload, "frontier") - graph = _dict_value(payload, "evidence_graph") - selected = _dict_value(frontier, "selected") - lines = [ - "# LoopX Auto Research Frontier", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- agent_id: `{frontier.get('agent_id')}`", - f"- selected: `{selected.get('hypothesis_id') if isinstance(selected, dict) else 'none'}`", - f"- hypotheses: `{graph.get('hypothesis_count')}`", - f"- evidence events: `{graph.get('evidence_event_count')}`", - f"- best dev metric: `{graph.get('best_dev_metric')}`", - f"- best holdout metric: `{graph.get('best_holdout_metric')}`", - f"- promotion candidates: `{len(frontier.get('promotion_candidates') or [])}`", - f"- retirement candidates: `{len(frontier.get('retirement_candidates') or [])}`", - ] - return "\n".join(lines) + "\n" - - -def _render_worker_turn(payload: dict[str, object]) -> str: - frontier_packet = _dict_value(payload, "frontier") - quota = _dict_value(frontier_packet, "quota") - frontier = _dict_value(frontier_packet, "frontier") - selected = _dict_value(frontier, "selected") - completion = _dict_value(payload, "completion") - append = _dict_value(payload, "append") - live_evidence = _dict_value(payload, "live_evidence") - artifact = _dict_value(payload, "artifact") - artifacts = _dict_value(payload, "artifacts") - lines = [ - "# LoopX Auto Research Worker Turn", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- mode: `{payload.get('mode')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- agent_id: `{payload.get('agent_id')}`", - f"- quota_should_run: `{quota.get('should_run')}`", - f"- quota_state: `{quota.get('state')}`", - f"- user_action_required: `{quota.get('user_action_required')}`", - f"- selected_todo: `{payload.get('selected_todo_id') or selected.get('todo_id')}`", - f"- selected_action: `{payload.get('selected_action') or selected.get('allowed_action')}`", - f"- selected_title: {selected.get('title')}", - f"- blocker: `{payload.get('blocker')}`", - f"- blocker_detail: {payload.get('blocker_detail')}", - f"- executed: `{payload.get('executed')}`", - f"- would_execute: `{payload.get('would_execute')}`", - f"- completion_status: `{completion.get('status')}`", - f"- completion_executed: `{completion.get('executed')}`", - f"- artifact: `{artifact.get('filename') or artifacts.get('evidence_packet')}`", - f"- artifact_status: `{payload.get('artifact_status') or payload.get('packet_status')}`", - f"- dev_metric: `{payload.get('dev_metric')}`", - f"- holdout_metric: `{payload.get('holdout_metric')}`", - f"- appended_count: `{append.get('appended_count')}`", - f"- live_evidence_written: `{live_evidence.get('written')}`", - "- public_boundary: raw_logs=`False`, private_artifacts=`False`, paths=`local-only`", - ] - return "\n".join(lines) + "\n" - - -def _render_worker_loop(payload: dict[str, object]) -> str: - turns = payload.get("turns") if isinstance(payload.get("turns"), list) else [] - lines = [ - "# LoopX Auto Research Worker Loop", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- mode: `{payload.get('mode')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- round_count: `{payload.get('round_count')}`", - f"- max_rounds: `{payload.get('max_rounds')}`", - f"- stop_reason: `{payload.get('stop_reason')}`", - f"- turn_count: `{payload.get('turn_count')}`", - f"- executed_turn_count: `{payload.get('executed_turn_count')}`", - f"- completed_turn_count: `{payload.get('completed_turn_count')}`", - f"- selected_actions: `{', '.join(str(action) for action in payload.get('selected_actions') or [])}`", - "", - "## Turns", - "", - ] - if not turns: - lines.append("- none") - for turn in turns: - if not isinstance(turn, dict): - continue - lines.append( - f"- round `{turn.get('round')}` agent `{turn.get('agent_id')}` " - f"role `{turn.get('role_id')}` " - f"mode `{turn.get('mode')}` action `{turn.get('selected_action')}` " - f"executed `{turn.get('executed')}` completion `{turn.get('completion_status')}` " - f"dev `{turn.get('dev_metric')}` holdout `{turn.get('holdout_metric')}`" - ) - return "\n".join(lines) + "\n" - - -def _render_demo_e2e(payload: dict[str, object]) -> str: - worker_loop = _dict_value(payload, "worker_loop") - user_contract = _dict_value(payload, "user_contract") - language_payload = _dict_value(user_contract, "output_language") - tonight = _dict_value(payload, "tonight_experience") - supervisor = _dict_value(payload, "supervisor") - preset = _dict_value(supervisor, "preset") - minimal_recipe = _dict_value(preset, "minimal_a2a_recipe") - if not minimal_recipe: - minimal_recipe = build_auto_research_minimal_a2a_recipe( - open_question=user_contract.get("open_question"), - output_language=str(language_payload.get("resolved") or payload.get("output_language") or "en"), - ) - route = _dict_value(payload, "route_contract") - preset_context = _dict_value(payload, "preset_context") - live = _dict_value(payload, "visible_worker_proof") - pane_status = _dict_value(payload, "visible_pane_a2a_status") - collective_rounds = _dict_value(payload, "collective_research_rounds") - readiness = _dict_value(payload, "visible_readiness") - contract_acceptance = _dict_value(payload, "contract_acceptance") - commands = _dict_value(payload, "commands") - improvement = _dict_value(readiness, "improvement_summary") - participation_gap = _dict_value(collective_rounds, "full_participation_requirement_gap") - role_cycle_shortfall = participation_gap.get("shortfall_by_agent") or {} - lines = [ - "# LoopX Auto Research Minimal E2E Demo", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- mode: `{payload.get('mode')}`", - f"- execution_kind: `{payload.get('execution_kind')}`", - f"- result_source: `{payload.get('result_source')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- tracking_goal_id: `{payload.get('tracking_goal_id')}`", - f"- frontier_goal_id: `{route.get('frontier_goal_id')}`", - f"- preset_id: `{preset_context.get('preset_id')}`", - f"- preset_baseline_source: `{preset_context.get('baseline_source')}`", - f"- baseline_metric: `{preset_context.get('baseline_metric')}`", - f"- benchmark_contract_file: `{preset_context.get('benchmark_contract_file')}`", - f"- editable_scope: `{_join_or_none(preset_context.get('editable_scope'))}`", - f"- protected_scope: `{_join_or_none(preset_context.get('protected_scope'))}`", - f"- dev_eval_command: `{preset_context.get('dev_eval_command')}`", - f"- holdout_eval_command: `{preset_context.get('holdout_eval_command')}`", - f"- agent_id: `{payload.get('agent_id')}`", - f"- user_contract_accepted: `{contract_acceptance.get('accepted')}`", - f"- one_question_contract: `{commands.get('one_question_contract')}`", - f"- one_question_start: `{commands.get('one_question_start')}`", - f"- minimal_a2a_user_plus_preset_lines: `{minimal_recipe.get('user_plus_preset_line_count')}`", - f"- minimal_a2a_shared_kernel_counted: `{minimal_recipe.get('shared_kernel_counted_as_recipe_lines')}`", - f"- reasoning_effort: `{payload.get('reasoning_effort')}`", - f"- worker_loop_executed_turns: `{worker_loop.get('executed_turn_count')}`", - f"- worker_loop_completed_turns: `{worker_loop.get('completed_turn_count')}`", - f"- worker_loop_selected_actions: `{worker_loop.get('selected_actions')}`", - f"- worker_loop_stop_reason: `{worker_loop.get('stop_reason')}`", - f"- tonight_ready: `{tonight.get('ready')}`", - f"- tonight_coordination_pattern: `{tonight.get('coordination_pattern')}`", - f"- tonight_dev_metric: `{tonight.get('dev_metric')}`", - f"- tonight_holdout_metric: `{tonight.get('holdout_metric')}`", - f"- tonight_positive_result: `{tonight.get('positive_result')}`", - f"- visible_lanes_launched: `{live.get('visible_lanes_launched')}`", - f"- visible_lanes_accepted: `{live.get('visible_lanes_accepted')}`", - f"- visible_role_participation_verified: `{live.get('visible_role_participation_verified')}`", - f"- visible_role_participation_basis: `{live.get('visible_role_participation_basis')}`", - f"- visible_pane_status_checks: `{pane_status.get('status_check_count')}`", - f"- pane_status_counts_as_research_rounds: `{pane_status.get('counts_as_collective_research_round')}`", - f"- collective_research_claim_source: `{collective_rounds.get('claim_source') or collective_rounds.get('source')}`", - f"- collective_research_rounds: `{collective_rounds.get('collective_round_count')}`", - f"- collective_research_rounds_verified: `{collective_rounds.get('multi_round_research_verified')}`", - f"- collective_research_role_cycle_gap: `{role_cycle_shortfall}`", - f"- holdout_metric_sequence: `{collective_rounds.get('holdout_metric_sequence')}`", - f"- holdout_improvement_count: `{collective_rounds.get('holdout_improvement_count')}`", - f"- collective_a2a_rounds_verified: `{live.get('decentralized_a2a_rounds_verified')}`", - f"- visible_readiness_ready: `{readiness.get('ready')}`", - f"- visible_readiness_level: `{readiness.get('readiness_level')}`", - f"- visible_best_metric: `{improvement.get('best_metric')}`", - f"- visible_holdout_delta_over_dev: `{improvement.get('holdout_delta_over_dev')}`", - f"- supervisor_lanes: `{supervisor.get('lane_count')}`", - ] - if preset_context: - lines.extend( - [ - "", - "## Preset Context", - "", - f"- preset_id: `{preset_context.get('preset_id')}`", - f"- baseline_source: `{preset_context.get('baseline_source')}`", - f"- question_text_supplies_baseline: `{preset_context.get('question_text_supplies_baseline')}`", - f"- metric_name: `{preset_context.get('metric_name')}`", - f"- baseline_metric: `{preset_context.get('baseline_metric')}`", - f"- protected_scope: `{_join_or_none(preset_context.get('protected_scope'))}`", - f"- claim_boundary: {preset_context.get('claim_boundary')}", - ] - ) - lines.extend( - _render_research_roles( - minimal_recipe=minimal_recipe, - collective_rounds=collective_rounds, - ) - ) - lines.extend( - _render_collective_round_summary( - worker_loop=worker_loop, - collective_rounds=collective_rounds, - ) - ) - if minimal_recipe: - lines.extend( - [ - "", - "## Minimal A2A Recipe", - "", - f"- user_plus_preset_lines: `{minimal_recipe.get('user_plus_preset_line_count')}`", - f"- user_lines: `{minimal_recipe.get('user_line_count')}`", - f"- preset_role_spec_lines: `{minimal_recipe.get('preset_role_spec_line_count')}`", - f"- shared_kernel_counted: `{minimal_recipe.get('shared_kernel_counted_as_recipe_lines')}`", - f"- coordination_model: `{minimal_recipe.get('coordination_model')}`", - ] - ) - for line in minimal_recipe.get("preset_recipe_lines") or []: - lines.append(f"- preset_role: {_role_description_from_spec_line(line)}") - lines.extend(_render_operator_commands(payload)) - return "\n".join(lines) + "\n" - - -def _render_rollout_append(payload: dict[str, object]) -> str: - lines = [ - "# LoopX Auto Research Rollout Append", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- dry_run: `{payload.get('dry_run')}`", - f"- events: `{payload.get('event_count')}`", - f"- appended: `{payload.get('appended_count')}`", - f"- would_append: `{payload.get('would_append_count')}`", - f"- skipped_existing: `{payload.get('skipped_existing_count')}`", - ] - return "\n".join(lines) + "\n" - - -def _render_live_evidence(payload: dict[str, object]) -> str: - visible = payload.get("visible_lanes") if isinstance(payload.get("visible_lanes"), dict) else {} - evidence = payload.get("lane_evidence") if isinstance(payload.get("lane_evidence"), dict) else {} - lines = [ - "# LoopX Auto Research Live Evidence", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- agent_id: `{payload.get('agent_id')}`", - f"- source: `{payload.get('source')}`", - f"- visible_lanes_accepted: `{visible.get('accepted')}`", - f"- lane_count: `{visible.get('lane_count')}`", - f"- evidence_events: `{evidence.get('evidence_event_count')}`", - f"- result_status: `{evidence.get('result_status')}`", - f"- protected_scope_clean: `{evidence.get('protected_scope_clean')}`", - ] - return "\n".join(lines) + "\n" - - -def _render_supervisor(payload: dict[str, object]) -> str: - lanes = payload.get("lanes") if isinstance(payload.get("lanes"), list) else [] - preset = _dict_value(payload, "preset") - minimal_recipe = _dict_value(preset, "minimal_a2a_recipe") - route = _dict_value(payload, "goal_surface_route") - boundary = _dict_value(payload, "boundary") - coordination = _dict_value(payload, "coordination_model") - runner = _dict_value(payload, "runner_contract") - tmux_lifecycle = _dict_value(runner, "tmux_lifecycle") - one_click = _dict_value(payload, "one_click_demo") - cli_contract = _dict_value(payload, "cli_contract") - lines = [ - "# LoopX Auto Research Demo Supervisor", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- mode: `{payload.get('mode')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- session_name: `{payload.get('session_name')}`", - f"- lane_count: `{len(lanes)}`", - f"- frontier_goal_id: `{route.get('frontier_goal_id')}`", - f"- leader_agent_required: `{coordination.get('leader_agent_required')}`", - f"- coordination_pattern: `{coordination.get('pattern')}`", - f"- starts_tmux: `{boundary.get('starts_tmux')}`", - f"- runs_codex: `{boundary.get('runs_codex')}`", - f"- attach: `{tmux_lifecycle.get('attach_command')}`", - "- start_script: `machine_json_only`", - "", - "## Role Profiles", - "", - ] - if not lanes: - lines.append("- none") - for lane in lanes: - if not isinstance(lane, dict): - continue - profile = _dict_value(lane, "role_profile") - lines.append( - f"- `{lane.get('lane_id')}` agent `{lane.get('agent_id')}` role `{lane.get('role_id')}`" - ) - lines.append( - f" - required_worker_playbook: `{profile.get('required_skill')}`" - ) - lines.append(f" - skill_distribution: `{profile.get('skill_distribution')}`") - lines.append(f" - worker_skill_source: `{profile.get('worker_skill_source')}`") - lines.extend(["", "## Lane Timeline", ""]) - if not lanes: - lines.append("- none") - for lane in lanes: - if not isinstance(lane, dict): - continue - timeline = lane.get("lane_timeline") if isinstance(lane.get("lane_timeline"), list) else [] - lines.append(f"- `{lane.get('lane_id')}`: `{' -> '.join(str(item) for item in timeline)}`") - lines.extend( - [ - "", - "## Minimal A2A Recipe", - "", - "- canonical_recipe_ref: `preset.minimal_a2a_recipe`", - f"- user_plus_preset_lines: `{minimal_recipe.get('user_plus_preset_line_count')}`", - f"- user_lines: `{minimal_recipe.get('user_line_count')}`", - f"- preset_role_spec_lines: `{minimal_recipe.get('preset_role_spec_line_count')}`", - f"- shared_kernel_counted: `{minimal_recipe.get('shared_kernel_counted_as_recipe_lines')}`", - f"- coordination_model: `{minimal_recipe.get('coordination_model')}`", - "", - "", - "## One-Click Dry Run", - "", - f"- mode: `{one_click.get('mode')}`", - "- command: `loopx auto-research start \"\" --execute`", - "- operator_takeover: `loopx auto-research start \"\" --execute --attach`", - f"- machine_json_policy: `{cli_contract.get('machine_json_policy')}`", - f"- tmux attach: `{tmux_lifecycle.get('attach_command')}`", - f"- tmux stop: `{tmux_lifecycle.get('stop_command')}`", - ] - ) - return "\n".join(lines) + "\n" - - -def _render_evidence_packet(payload: dict[str, object]) -> str: - hypothesis = _dict_value(payload, "hypothesis") - summary = _dict_value(payload, "evidence_summary") - lines = [ - "# LoopX Auto Research Evidence Packet", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- hypothesis: `{hypothesis.get('hypothesis_id')}`", - f"- todo: `{hypothesis.get('todo_id')}`", - f"- status: `{hypothesis.get('status')}`", - f"- evidence events: `{summary.get('evidence_event_count')}`", - f"- splits: `{', '.join(summary.get('splits', []))}`", - f"- negative evidence: `{summary.get('negative_evidence_count')}`", - f"- protected scope clean: `{summary.get('protected_scope_clean')}`", - ] - return "\n".join(lines) + "\n" - - -def _render_terminal_result(payload: dict[str, object]) -> str: - decision = _dict_value(payload, "decision") - review = _dict_value(payload, "review") - lines = [ - "# LoopX Auto Research Terminal Result", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- dry_run: `{payload.get('dry_run')}`", - f"- appended: `{payload.get('appended')}`", - f"- reused: `{payload.get('reused')}`", - ] - if decision: - lines.extend( - [ - f"- hypothesis_id: `{decision.get('hypothesis_id')}`", - f"- outcome: `{decision.get('outcome')}`", - f"- reason: `{decision.get('reason')}`", - f"- evidence_graph_revision: `{decision.get('evidence_graph_revision')}`", - ] - ) - if review: - lines.extend( - [ - f"- hypothesis_id: `{review.get('hypothesis_id')}`", - f"- verdict: `{review.get('verdict')}`", - f"- independent: `{review.get('independent')}`", - f"- evidence_graph_revision: `{review.get('evidence_graph_revision')}`", - ] - ) - return "\n".join(lines) + "\n" - - -def _render_terminal_result_query(payload: dict[str, object]) -> str: - lines = [ - "# LoopX Auto Research Results", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- evidence_graph_revision: `{payload.get('evidence_graph_revision')}`", - f"- result_count: `{payload.get('result_count')}`", - "", - "## Results", - "", - ] - results = payload.get("results") if isinstance(payload.get("results"), list) else [] - if not results: - lines.append("- none") - for result in results: - if not isinstance(result, dict): - continue - decision = _dict_value(result, "terminal_decision") - review = _dict_value(result, "peer_review") - lines.extend( - [ - f"- `{result.get('hypothesis_id')}`: {result.get('hypothesis')}", - f" - research_status: `{result.get('research_status')}`", - f" - decision_state: `{result.get('decision_state')}`", - f" - terminal_outcome: `{decision.get('outcome')}`", - f" - terminal_reason: `{decision.get('reason')}`", - f" - finding_status: `{result.get('finding_status')}`", - f" - review_state: `{review.get('state')}`", - f" - review_verdict: `{review.get('verdict')}`", - f" - independent_review: `{review.get('independent')}`", - ] - ) - return "\n".join(lines) + "\n" - - -def _render_terminal_result_projection(payload: dict[str, object]) -> str: - projection = _dict_value(payload, "explore_projection") - counts = _dict_value(projection, "counts") - return "\n".join( - [ - "# LoopX Auto Research Result Projection", - "", - f"- schema: `{payload.get('schema_version')}`", - f"- goal_id: `{payload.get('goal_id')}`", - f"- dry_run: `{payload.get('dry_run')}`", - f"- result_count: `{payload.get('result_count')}`", - f"- event_count: `{payload.get('event_count')}`", - f"- appended_count: `{payload.get('appended_count')}`", - f"- reused_count: `{payload.get('reused_count')}`", - f"- node_count: `{counts.get('node_count')}`", - f"- finding_count: `{counts.get('finding_count')}`", - ] - ) + "\n" - - -def _render_artifact_receipt(payload: dict[str, object]) -> str: - wish = _dict_value(payload, "wish") - contract = _dict_value(payload, "contract") - artifacts = _dict_value(payload, "artifacts") - feedback = _dict_value(payload, "failure_feedback") - lines = [ - "# LoopX Auto Research Artifact Receipt", - "", - f"- goal_id: `{payload.get('goal_id')}`", - f"- wish_id: `{wish.get('wish_id')}`", - f"- status: `{payload.get('status')}`", - f"- contract_state: `{contract.get('state')}`", - f"- contract_revision: `{contract.get('contract_revision')}`", - f"- observed_artifacts: `{len(artifacts.get('observed_refs') or [])}`", - f"- missing_required_artifacts: `{len(artifacts.get('missing_required_refs') or [])}`", - ] - if feedback: - lines.extend( - [ - "", - "## Feedback", - "", - str(feedback.get("summary") or ""), - "", - f"- failure_kinds: `{_join_or_none(feedback.get('failure_kinds'))}`", - f"- unmet_criteria: `{_join_or_none(feedback.get('unmet_criteria'))}`", - f"- verified_boundary: `{_join_or_none(feedback.get('verified_boundary'))}`", - f"- fallback_artifacts: `{_join_or_none(feedback.get('fallback_artifact_refs'))}`", - f"- reentry_conditions: `{_join_or_none(feedback.get('reentry_conditions'))}`", - ] - ) - return "\n".join(lines) + "\n" - - -def _render_generic(payload: dict[str, object]) -> str: - lines = [ - "# LoopX Auto Research", - "", - f"- ok: `{payload.get('ok')}`", - f"- schema: `{payload.get('schema_version')}`", - f"- mode: `{payload.get('mode')}`", - f"- goal_id: `{payload.get('goal_id')}`", - ] - return "\n".join(lines) + "\n" - - -def render_auto_research_markdown(payload: dict[str, object]) -> str: - if not payload.get("ok"): - return _render_failure(payload) - schema = payload.get("schema_version") - if schema == "auto_research_worker_turn_v0": - return _render_worker_turn(payload) - if schema == AUTO_RESEARCH_USER_CONTRACT_SCHEMA_VERSION: - return _render_user_contract(payload) - if schema == "auto_research_worker_loop_v0": - return _render_worker_loop(payload) - if schema == AUTO_RESEARCH_DEMO_E2E_SCHEMA_VERSION: - return _render_demo_e2e(payload) - if schema == "auto_research_demo_supervisor_plan_v0": - return _render_supervisor(payload) - if schema == AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION: - return _render_rollout_append(payload) - if schema == "auto_research_live_codex_lane_e2e_evidence_v0": - return _render_live_evidence(payload) - if schema == "auto_research_evidence_packet_v0": - return _render_evidence_packet(payload) - if schema in { - "auto_research_terminal_decision_v0", - "auto_research_peer_review_v0", - }: - return _render_terminal_result(payload) - if schema == "auto_research_terminal_result_query_v0": - return _render_terminal_result_query(payload) - if schema == "auto_research_terminal_result_projection_v0": - return _render_terminal_result_projection(payload) - if schema == AUTO_RESEARCH_ARTIFACT_RECEIPT_SCHEMA_VERSION: - return _render_artifact_receipt(payload) - return render_auto_research_projection_markdown(payload) diff --git a/demo/auto_research/kernel.py b/demo/auto_research/kernel.py deleted file mode 100644 index 161e5d1c21..0000000000 --- a/demo/auto_research/kernel.py +++ /dev/null @@ -1,191 +0,0 @@ -from __future__ import annotations - -from collections.abc import Callable, Iterable, Mapping -from typing import Any - - -LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION = "auto_research_lightweight_result_v0" -LIGHTWEIGHT_AUTO_RESEARCH_HYPOTHESIS_SCHEMA_VERSION = "auto_research_lightweight_hypothesis_v0" -LIGHTWEIGHT_AUTO_RESEARCH_EVIDENCE_SCHEMA_VERSION = "auto_research_lightweight_evidence_v0" - -MetricEvaluator = Callable[[dict[str, Any], str], Mapping[str, Any]] - - -def _token(value: object, *, default: str = "unknown") -> str: - text = str(value or "").strip() - return text or default - - -def _number(value: object, *, default: float | None = None) -> float | None: - if value is None: - return default - try: - number = float(value) - except (TypeError, ValueError) as exc: - raise ValueError(f"metric value is not numeric: {value!r}") from exc - if number != number or number in {float("inf"), float("-inf")}: - raise ValueError(f"metric value is not finite: {value!r}") - return number - - -def _improved(value: float | None, baseline: float, *, direction: str) -> bool: - if value is None: - return False - if direction == "minimize": - return value < baseline - if direction != "maximize": - raise ValueError("direction must be maximize or minimize") - return value > baseline - - -def lightweight_hypothesis( - *, - hypothesis_id: str, - todo_id: str, - claimed_by: str, - text: str, - candidate_key: str, -) -> dict[str, Any]: - return { - "schema_version": LIGHTWEIGHT_AUTO_RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - "hypothesis_id": _token(hypothesis_id, default="hypothesis"), - "todo_id": _token(todo_id, default="todo"), - "claimed_by": _token(claimed_by, default="agent"), - "hypothesis": _token(text, default="untitled hypothesis"), - "candidate_key": _token(candidate_key, default="candidate"), - } - - -def _evidence_event( - *, - hypothesis: dict[str, Any], - split: str, - result: Mapping[str, Any], - baseline: float, - direction: str, -) -> dict[str, Any]: - metric = _number(result.get("metric")) - exact = bool(result.get("exact", True)) - clean = bool(result.get("protected_scope_clean", True)) - status = "improved" if exact and clean and _improved(metric, baseline, direction=direction) else "not_improved" - return { - "schema_version": LIGHTWEIGHT_AUTO_RESEARCH_EVIDENCE_SCHEMA_VERSION, - "hypothesis_id": hypothesis["hypothesis_id"], - "todo_id": hypothesis["todo_id"], - "claimed_by": hypothesis["claimed_by"], - "candidate_key": hypothesis["candidate_key"], - "split": _token(split, default="dev"), - "metric": metric, - "baseline": baseline, - "direction": direction, - "exact": exact, - "protected_scope_clean": clean, - "status": status, - "result_source": _token(result.get("result_source"), default="metric_evaluator"), - "strategy": _token(result.get("strategy"), default=hypothesis["candidate_key"]), - "artifact_refs": [ - _token(item, default="artifact") - for item in result.get("artifact_refs", []) - if str(item).strip() - ][:8], - } - - -def _rank_key(event: dict[str, Any]) -> float: - value = event["metric"] - return float("-inf") if value is None else float(value) - - -def _best_positive( - evidence: list[dict[str, Any]], - *, - direction: str, -) -> dict[str, Any] | None: - positives = [event for event in evidence if event["status"] == "improved"] - if not positives: - return None - if direction == "minimize": - return min(positives, key=_rank_key) - return max(positives, key=_rank_key) - - -def run_lightweight_auto_research( - *, - goal_id: str, - hypotheses: Iterable[dict[str, Any]], - evaluate: MetricEvaluator, - baseline: float = 1.0, - direction: str = "maximize", - dev_split: str = "dev", - holdout_split: str = "holdout", - max_dev_rounds: int | None = None, -) -> dict[str, Any]: - """Run a small auto-research loop: dev attempts, best candidate, holdout. - - This kernel deliberately avoids board, frontier, launcher, and artifact - projection concerns. Those layers can wrap the result without growing the - research decision core. - """ - - base = _number(baseline) - if base is None: - raise ValueError("baseline is required") - if direction not in {"maximize", "minimize"}: - raise ValueError("direction must be maximize or minimize") - items = list(hypotheses) - if not items: - raise ValueError("at least one hypothesis is required") - limit = len(items) if max_dev_rounds is None else max(0, min(max_dev_rounds, len(items))) - - evidence: list[dict[str, Any]] = [] - by_id = {item["hypothesis_id"]: item for item in items} - for hypothesis in items[:limit]: - event = _evidence_event( - hypothesis=hypothesis, - split=dev_split, - result=evaluate(hypothesis, dev_split), - baseline=base, - direction=direction, - ) - evidence.append(event) - - best_dev = _best_positive(evidence, direction=direction) - selected = by_id.get(best_dev["hypothesis_id"]) if best_dev else None - holdout_event = None - if selected is not None: - holdout_event = _evidence_event( - hypothesis=selected, - split=holdout_split, - result=evaluate(selected, holdout_split), - baseline=base, - direction=direction, - ) - evidence.append(holdout_event) - - holdout_positive = bool(holdout_event and holdout_event["status"] == "improved") - dev_positive = best_dev is not None - decision = ( - "validated_positive" - if holdout_positive - else "dev_supported" - if dev_positive - else "continue_research" - ) - return { - "ok": True, - "schema_version": LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION, - "goal_id": _token(goal_id, default="auto-research"), - "candidate_count": len(items), - "dev_round_count": limit, - "evidence_event_count": len(evidence), - "selected_hypothesis_id": selected["hypothesis_id"] if selected else None, - "decision": decision, - "dev_metric": best_dev["metric"] if best_dev else None, - "holdout_metric": holdout_event["metric"] if holdout_event else None, - "evidence": evidence, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "raw_source_bodies_recorded": False, - }, - } diff --git a/demo/auto_research/knn_demo_workspace.py b/demo/auto_research/knn_demo_workspace.py deleted file mode 100644 index 578f347774..0000000000 --- a/demo/auto_research/knn_demo_workspace.py +++ /dev/null @@ -1,322 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -import stat -from pathlib import Path - - -KNN_DEMO_WORKSPACE_SCHEMA_VERSION = "loopx_knn_demo_workspace_v0" - -KNN_DEMO_EDITABLE_SCOPE = ["solution.py"] -KNN_DEMO_PROTECTED_SCOPE = ["task.py", "eval.py", "eval.sh"] -KNN_DEMO_CONTRACT_FILE = "research_contract.public.json" -KNN_DEMO_DEV_EVAL_COMMAND = "bash eval.sh dev" -KNN_DEMO_HOLDOUT_EVAL_COMMAND = "bash eval.sh test" - - -README_MD = """# LoopX KNN Auto-Research Demo - -Goal: improve exact k-nearest-neighbor search speed while preserving correctness. - -Editable scope: -- `solution.py` - -Protected scope: -- `task.py` -- `eval.py` -- `eval.sh` - -Commands: -- `bash eval.sh dev` -- `bash eval.sh test` - -Evidence rule: -- Record the command, split, score, mechanism, and changed file before claiming an improvement. -- Do not edit protected files to improve the score. - -Recording-friendly artifact checklist: -- Curator: metric, editable/protected boundary, and promotion rule. -- Hypothesis proposer: two exact-KNN speedup hypotheses with mechanism and risk. -- Executor: baseline dev score, changed mechanism, post-change dev score, and held-out score when dev improves. -- Evaluator: split-aware verdict and the next role todo or retry reason. -""" - - -AGENTS_MD = """# KNN Demo Agent Rules - -Only edit `solution.py`. - -Do not edit `task.py`, `eval.py`, or `eval.sh`. - -Run `bash eval.sh dev` for development evidence and `bash eval.sh test` for held-out evidence. - -For visible auto-research, do not stop at a status/tick summary. Leave one compact -public-safe artifact, todo update, or evidence packet that another role can use. -""" - - -TASK_PY = '''from __future__ import annotations - -from random import Random - - -K = 7 -DIM = 12 - - -def _sizes(split: str) -> tuple[int, int, int]: - if split == "test": - return 4, 1800, 100 - return 4, 1400, 80 - - -def _seed(split: str) -> int: - return 9001 if split == "test" else 1009 - - -def generate_instances(split: str): - split = "test" if str(split).strip() == "test" else "dev" - count, database_size, query_size = _sizes(split) - rng = Random(_seed(split)) - for _ in range(count): - database = [ - [rng.uniform(-1.0, 1.0) for _ in range(DIM)] - for _ in range(database_size) - ] - queries = [ - [rng.uniform(-1.0, 1.0) for _ in range(DIM)] - for _ in range(query_size) - ] - yield {"database": database, "queries": queries, "k": K} - - -def squared_distance(left, right) -> float: - return sum((a - b) * (a - b) for a, b in zip(left, right)) - - -def reference_solve(problem): - database = problem["database"] - k = int(problem["k"]) - output = [] - for query in problem["queries"]: - ranked = sorted( - (squared_distance(row, query), index) - for index, row in enumerate(database) - ) - output.append([index for _distance, index in ranked[:k]]) - return output - - -def is_solution(problem, candidate) -> bool: - return candidate == reference_solve(problem) -''' - - -SOLUTION_PY = '''from __future__ import annotations - - -def _squared_distance(left, right) -> float: - return sum((a - b) * (a - b) for a, b in zip(left, right)) - - -def solve(problem): - """Baseline exact KNN: full-sort every query.""" - - database = problem["database"] - k = int(problem["k"]) - output = [] - for query in problem["queries"]: - ranked = sorted( - (_squared_distance(row, query), index) - for index, row in enumerate(database) - ) - output.append([index for _distance, index in ranked[:k]]) - return output -''' - - -EVAL_PY = '''from __future__ import annotations - -import importlib -import json -import statistics -import sys -import time - -from task import generate_instances, is_solution, reference_solve - - -def _median_runtime(fn, problem, *, trials: int) -> tuple[float, object]: - timings = [] - result = None - for _ in range(trials): - start = time.perf_counter() - result = fn(problem) - timings.append(time.perf_counter() - start) - return statistics.median(timings), result - - -def evaluate(split: str) -> dict[str, object]: - split = "test" if str(split).strip() == "test" else "dev" - solution = importlib.import_module("solution") - trials = 2 if split == "test" else 3 - speedups = [] - for problem in generate_instances(split): - reference_runtime, _reference = _median_runtime( - reference_solve, - problem, - trials=trials, - ) - candidate_runtime, candidate = _median_runtime( - solution.solve, - problem, - trials=trials, - ) - if not is_solution(problem, candidate): - return { - "ok": False, - "split": split, - "score": 0.0, - "metric_name": "speedup", - "error": "candidate output does not match exact reference", - } - speedups.append(reference_runtime / max(candidate_runtime, 1e-12)) - return { - "ok": True, - "split": split, - "score": round(float(statistics.median(speedups)), 6), - "metric_name": "speedup", - "direction": "maximize", - "baseline_score": 1.0, - "editable_scope": ["solution.py"], - "protected_scope": ["task.py", "eval.py", "eval.sh"], - } - - -def main(argv: list[str]) -> int: - payload = evaluate(argv[1] if len(argv) > 1 else "dev") - print(f"score: {payload['score']:.6f}") - print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) - return 0 if payload.get("ok") else 1 - - -if __name__ == "__main__": - raise SystemExit(main(sys.argv)) -''' - - -EVAL_SH = """#!/usr/bin/env bash -set -euo pipefail -python3 eval.py "${1:-dev}" -""" - - -GITIGNORE = """__pycache__/ -*.pyc -.loopx/ -""" - - -def _sha256_text(content: str) -> str: - return hashlib.sha256(content.encode("utf-8")).hexdigest() - - -def _contract_payload(*, goal_id: str | None = None) -> dict[str, object]: - payload: dict[str, object] = { - "schema_version": "auto_research_benchmark_contract_v0", - "preset_id": "knn-demo", - "objective": "Improve exact KNN speedup while preserving correctness.", - "metric": { - "name": "speedup", - "direction": "maximize", - "baseline": 1.0, - }, - "editable_scope": list(KNN_DEMO_EDITABLE_SCOPE), - "protected_scope": list(KNN_DEMO_PROTECTED_SCOPE), - "protected_scope_sha256": { - "task.py": _sha256_text(TASK_PY), - "eval.py": _sha256_text(EVAL_PY), - "eval.sh": _sha256_text(EVAL_SH), - }, - "dev_eval_command": KNN_DEMO_DEV_EVAL_COMMAND, - "holdout_eval_command": KNN_DEMO_HOLDOUT_EVAL_COMMAND, - "claim_boundary": ( - "Claims require command output from the generated benchmark workspace; " - "do not edit protected files." - ), - } - if goal_id: - payload["goal_id"] = goal_id - return payload - - -def _write_if_missing(path: Path, content: str, *, executable: bool = False) -> bool: - if path.exists(): - if executable: - path.chmod(path.stat().st_mode | stat.S_IXUSR) - return False - path.write_text(content, encoding="utf-8", newline="\n") - if executable: - path.chmod(path.stat().st_mode | stat.S_IXUSR) - return True - - -def materialize_knn_demo_workspace(workspace: str | Path, *, goal_id: str | None = None) -> dict[str, object]: - """Create the real KNN demo benchmark workspace if it is missing.""" - - workspace_path = Path(workspace).expanduser() - created = not workspace_path.exists() - workspace_path.mkdir(parents=True, exist_ok=True) - - files = { - "README.md": README_MD, - "AGENTS.md": AGENTS_MD, - "task.py": TASK_PY, - "solution.py": SOLUTION_PY, - "eval.py": EVAL_PY, - "eval.sh": EVAL_SH, - ".gitignore": GITIGNORE, - } - written: list[str] = [] - kept: list[str] = [] - for relative_path, content in files.items(): - path = workspace_path / relative_path - did_write = _write_if_missing( - path, - content, - executable=relative_path == "eval.sh", - ) - (written if did_write else kept).append(relative_path) - - contract_path = workspace_path / KNN_DEMO_CONTRACT_FILE - contract_path.write_text( - json.dumps(_contract_payload(goal_id=goal_id), ensure_ascii=False, indent=2, sort_keys=True) - + "\n", - encoding="utf-8", - ) - marker_path = workspace_path / ".loopx-knn-demo-workspace.public.json" - marker = { - "schema_version": KNN_DEMO_WORKSPACE_SCHEMA_VERSION, - "preset_id": "knn-demo", - "goal_id": goal_id or "", - "workspace_created": created, - "contract_file": KNN_DEMO_CONTRACT_FILE, - "editable_scope": list(KNN_DEMO_EDITABLE_SCOPE), - "protected_scope": list(KNN_DEMO_PROTECTED_SCOPE), - "dev_eval_command": KNN_DEMO_DEV_EVAL_COMMAND, - "holdout_eval_command": KNN_DEMO_HOLDOUT_EVAL_COMMAND, - "files_written": written, - "files_kept": kept, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - marker_path.write_text( - json.dumps(marker, ensure_ascii=False, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - return marker diff --git a/demo/auto_research/live_evidence.py b/demo/auto_research/live_evidence.py deleted file mode 100644 index f3e9eb1538..0000000000 --- a/demo/auto_research/live_evidence.py +++ /dev/null @@ -1,248 +0,0 @@ -from __future__ import annotations - -import json -from pathlib import Path - -from .evidence_packet import ( - AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION, - load_auto_research_evidence_packet, - validate_auto_research_evidence_packet, -) - - -LIVE_CODEX_E2E_EVIDENCE_SCHEMA_VERSION = "auto_research_live_codex_lane_e2e_evidence_v0" -LIVE_CODEX_E2E_DEFAULT_OUTPUT = "live-codex-e2e-evidence.public.json" - - -def _require_dict(payload: dict[str, object], key: str) -> dict[str, object]: - value = payload.get(key) - if not isinstance(value, dict): - raise ValueError(f"live evidence field `{key}` must be an object") - return value - - -def _require_bool(value: object, *, field: str, expected: bool = True) -> None: - if value is not expected: - raise ValueError(f"live evidence field `{field}` must be {expected}") - - -def _require_positive_int(value: object, *, field: str) -> int: - if not isinstance(value, int) or value <= 0: - raise ValueError(f"live evidence field `{field}` must be a positive integer") - return value - - -def _assert_live_evidence_public_safe(payload: dict[str, object]) -> None: - text = json.dumps(payload, sort_keys=True) - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - if leaked: - raise ValueError("live evidence must be compact and public-safe; forbidden material detected") - - -def _load_json_object(path: str | Path, *, field: str) -> dict[str, object]: - try: - payload = json.loads(Path(path).expanduser().read_text(encoding="utf-8")) - except Exception as exc: - raise ValueError(f"{field} must be readable JSON") from exc - if not isinstance(payload, dict): - raise ValueError(f"{field} root must be an object") - return payload - - -def _metric_by_split(packet: dict[str, object], split: str) -> object: - for event in packet.get("evidence_events") or []: - if not isinstance(event, dict) or event.get("split") != split: - continue - metric = event.get("metric") - if isinstance(metric, dict): - return metric.get("value") - return None - - -def _evidence_event_agent_ids(packet: dict[str, object]) -> list[str]: - agent_ids = [] - for event in packet.get("evidence_events") or []: - if not isinstance(event, dict): - continue - agent_id = str(event.get("agent_id") or "").strip() - if agent_id: - agent_ids.append(agent_id) - return sorted(set(agent_ids)) - - -def build_live_codex_e2e_evidence_from_packet( - *, - packet: dict[str, object], - append_result: dict[str, object], - agent_id: str, - lane_count: int, - visible_lanes_accepted: bool, -) -> dict[str, object]: - packet = validate_auto_research_evidence_packet(packet) - if append_result.get("schema_version") != AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION: - raise ValueError(f"append result schema_version must be {AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION}") - if append_result.get("dry_run") is not False: - raise ValueError("append result must come from a real append-evidence run") - summary = packet["summary"] - goal_id = summary["goal_id"] - if append_result.get("goal_id") != goal_id: - raise ValueError("append result goal_id must match the evidence packet") - event_agent_ids = _evidence_event_agent_ids(packet) - if event_agent_ids != [str(agent_id)]: - raise ValueError("agent_id must match every packet evidence event agent_id") - if not isinstance(lane_count, int) or lane_count <= 0: - raise ValueError("lane_count must be a positive integer") - if not visible_lanes_accepted: - raise ValueError("live evidence capture requires accepted visible lanes") - if append_result.get("appended_count", 0) <= 0: - raise ValueError("append result must include at least one fresh appended event") - counts = append_result.get("counts_by_kind") - if not isinstance(counts, dict) or int(counts.get("research_evidence") or 0) <= 0: - raise ValueError("append result must include at least one research_evidence event") - if summary.get("status") != "supported": - raise ValueError("live evidence can only be captured from a supported packet") - if summary.get("protected_scope_clean") is not True: - raise ValueError("live evidence requires protected_scope_clean=true") - if int(summary.get("negative_evidence_count") or 0) != 0: - raise ValueError("live evidence requires zero negative evidence") - if int(summary.get("needs_retry_count") or 0) != 0: - raise ValueError("live evidence requires zero retry-needed evidence") - payload = { - "ok": True, - "schema_version": LIVE_CODEX_E2E_EVIDENCE_SCHEMA_VERSION, - "source": "live_codex_lane_output", - "goal_id": goal_id, - "agent_id": agent_id, - "visible_lanes": { - "launched": True, - "accepted": True, - "lane_count": lane_count, - }, - "lane_evidence": { - "lane_authored": True, - "evidence_source": "live_codex_lane_output", - "append_status": "appended_to_loopx_state", - "evidence_event_count": int(summary["evidence_event_count"]), - "result_status": summary["status"], - "protected_scope_clean": True, - "dev_metric": _metric_by_split(packet, "dev"), - "holdout_metric": _metric_by_split(packet, "holdout"), - }, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - "local_workspace_path_redacted": True, - }, - } - _assert_live_evidence_public_safe(payload) - return payload - - -def capture_live_codex_e2e_evidence( - *, - packet_path: str, - append_result_path: str, - agent_id: str, - lane_count: int, - visible_lanes_accepted: bool, -) -> dict[str, object]: - packet = load_auto_research_evidence_packet(packet_path) - append_result = _load_json_object(append_result_path, field="append_result_file") - return build_live_codex_e2e_evidence_from_packet( - packet=packet, - append_result=append_result, - agent_id=agent_id, - lane_count=lane_count, - visible_lanes_accepted=visible_lanes_accepted, - ) - - -def load_live_codex_e2e_evidence( - *, - evidence_path: str, - goal_id: str, - agent_id: str, -) -> dict[str, object]: - try: - raw = Path(evidence_path).read_text(encoding="utf-8") - payload = json.loads(raw) - except Exception as exc: - raise ValueError("live evidence must be readable JSON") from exc - if not isinstance(payload, dict): - raise ValueError("live evidence root must be an object") - if payload.get("schema_version") != LIVE_CODEX_E2E_EVIDENCE_SCHEMA_VERSION: - raise ValueError( - "live evidence schema_version must be " - f"{LIVE_CODEX_E2E_EVIDENCE_SCHEMA_VERSION}" - ) - if payload.get("source") != "live_codex_lane_output": - raise ValueError("live evidence source must be live_codex_lane_output") - if payload.get("goal_id") != goal_id: - raise ValueError("live evidence goal_id does not match the demo goal") - if payload.get("agent_id") != agent_id: - raise ValueError("live evidence agent_id does not match the demo agent") - - visible = _require_dict(payload, "visible_lanes") - lane_evidence = _require_dict(payload, "lane_evidence") - boundary = _require_dict(payload, "public_boundary") - _require_bool(visible.get("launched"), field="visible_lanes.launched") - _require_bool(visible.get("accepted"), field="visible_lanes.accepted") - lane_count = _require_positive_int(visible.get("lane_count"), field="visible_lanes.lane_count") - _require_bool(lane_evidence.get("lane_authored"), field="lane_evidence.lane_authored") - if lane_evidence.get("evidence_source") != "live_codex_lane_output": - raise ValueError("lane_evidence.evidence_source must be live_codex_lane_output") - if lane_evidence.get("append_status") != "appended_to_loopx_state": - raise ValueError("lane_evidence.append_status must be appended_to_loopx_state") - evidence_event_count = _require_positive_int( - lane_evidence.get("evidence_event_count"), - field="lane_evidence.evidence_event_count", - ) - if lane_evidence.get("result_status") != "supported": - raise ValueError("lane_evidence.result_status must be supported") - _require_bool( - lane_evidence.get("protected_scope_clean"), - field="lane_evidence.protected_scope_clean", - ) - for key in ( - "raw_logs_recorded", - "private_artifacts_recorded", - "absolute_paths_recorded", - "credentials_recorded", - ): - _require_bool(boundary.get(key), field=f"public_boundary.{key}", expected=False) - _require_bool( - boundary.get("local_workspace_path_redacted"), - field="public_boundary.local_workspace_path_redacted", - ) - _assert_live_evidence_public_safe(payload) - return { - "schema_version": payload["schema_version"], - "source": payload["source"], - "goal_id": goal_id, - "agent_id": agent_id, - "lane_count": lane_count, - "evidence_event_count": evidence_event_count, - "result_status": lane_evidence.get("result_status"), - "protected_scope_clean": True, - "dev_metric": lane_evidence.get("dev_metric"), - "holdout_metric": lane_evidence.get("holdout_metric"), - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - "local_workspace_path_redacted": True, - }, - } diff --git a/demo/auto_research/preset.py b/demo/auto_research/preset.py deleted file mode 100644 index 5c0a98f0b9..0000000000 --- a/demo/auto_research/preset.py +++ /dev/null @@ -1,176 +0,0 @@ -from __future__ import annotations - -import shlex -from collections.abc import Iterable - -from .defaults import AUTO_RESEARCH_DEFAULT_GOAL_ID -from .role_profiles import ( - AUTO_RESEARCH_DEFAULT_LANES, - AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS, # noqa: F401 - AUTO_RESEARCH_REQUIRED_SKILL, - AUTO_RESEARCH_ROLE_PROFILE_SCHEMA_VERSION, - AUTO_RESEARCH_WORKER_SKILL_SOURCE, - auto_research_role_id, - auto_research_role_id_for_action, # noqa: F401 - auto_research_role_profile, - auto_research_seed_action_for_role, # noqa: F401 - auto_research_seed_title, # noqa: F401 - auto_research_successor_specs_for_action, # noqa: F401 - knn_demo_visible_first_steps, -) -from ..multi_agent.recipe import ( - build_minimal_decentralized_a2a_recipe, - parse_multi_agent_role_spec_lines, -) - - -AUTO_RESEARCH_PRESET_SCHEMA_VERSION = "auto_research_thin_preset_v0" -AUTO_RESEARCH_MINIMAL_A2A_RECIPE_SCHEMA_VERSION = "auto_research_minimal_a2a_recipe_v0" - - -def default_auto_research_agent_specs() -> list[str]: - return [ - f"{agent}:{lane}:{role}" - for agent, lane, role, _scope in AUTO_RESEARCH_DEFAULT_LANES - ] - - -def _quoted_open_question(open_question: object | None) -> str: - question = str(open_question or "").strip() - if not question: - return '""' - return shlex.quote(question) - - -def build_auto_research_minimal_a2a_recipe( - *, - open_question: object | None = None, - output_language: str = "en", - role_specs: Iterable[object] | None = None, -) -> dict[str, object]: - """Return the public line-count claim for the thin auto-research preset. - - The count is deliberately about declarative recipe lines, not the shared - LoopX kernel implementation. That keeps the public claim reusable and - honest: other products can replace the four role specs without owning the - runner, fixed wake prompt, pane-local tick, or state protocol. - """ - - language = str(output_language or "en").strip() - language_flag = f" --language {shlex.quote(language)}" if language != "en" else "" - user_line = ( - "loopx auto-research start " - f"{_quoted_open_question(open_question)}{language_flag} --execute" - ) - raw_role_specs = role_specs or default_auto_research_agent_specs() - return build_minimal_decentralized_a2a_recipe( - schema_version=AUTO_RESEARCH_MINIMAL_A2A_RECIPE_SCHEMA_VERSION, - product_id="auto-research", - claim=( - "one user command plus the default four-line auto-research role spec " - "starts decentralized A2A on the shared LoopX multi-agent kernel" - ), - claim_boundary=( - "line count covers user intent and auto-research preset defaults only; " - "the reusable kernel owns visible process launch, fixed wake prompt, " - "pane-local quota/frontier tick, todo/evidence/status protocol, " - "and public artifact routing" - ), - user_recipe_lines=[user_line], - preset_recipe_lines=raw_role_specs, - ) - - -def auto_research_lane_specs(agent_specs: Iterable[str] | None) -> list[dict[str, str]]: - default_scope = { - lane: scope for _agent, lane, _role, scope in AUTO_RESEARCH_DEFAULT_LANES - } - return parse_multi_agent_role_spec_lines( - agent_specs=agent_specs, - default_agent_specs=default_auto_research_agent_specs(), - resolve_role_id=lambda raw_role, index: auto_research_role_id( - raw_role, - index=index, - ), - default_scope_by_lane=default_scope, - ) - - -def build_auto_research_preset_role( - *, - lane: dict[str, str], - goal_id: str = AUTO_RESEARCH_DEFAULT_GOAL_ID, - reasoning_effort: str = "high", - output_language: str = "en", - open_question: object | None = None, - preset_context: dict[str, object] | None = None, -) -> dict[str, object]: - role_id = lane["role_id"] - agent_id = lane["agent_id"] - role_profile = auto_research_role_profile( - role_id=role_id, - goal_id=goal_id, - agent_id=agent_id, - ) - role_profile["output_language"] = output_language - if str(open_question or "").strip(): - role_profile["open_question"] = str(open_question).strip() - if preset_context: - role_profile["preset_context"] = dict(preset_context) - if str(preset_context.get("preset_id") or "") == "knn-demo": - role_profile["visible_first_steps"] = knn_demo_visible_first_steps(role_id) - return { - "agent_id": agent_id, - "lane_id": lane["lane_id"], - "role_id": role_id, - "scope": lane["scope"], - "responsibility": lane["scope"], - "output_language": output_language, - "role_profile_ref": f"{AUTO_RESEARCH_ROLE_PROFILE_SCHEMA_VERSION}:{role_id}", - "role_profile": role_profile, - "skill": { - "name": AUTO_RESEARCH_REQUIRED_SKILL, - "source": AUTO_RESEARCH_WORKER_SKILL_SOURCE, - }, - "handoff_hints": role_profile.get("handoff") or [], - "reasoning_effort": reasoning_effort, - } - - -def build_auto_research_preset_summary( - *, - role_count: int, - open_question: object | None = None, - output_language: str = "en", - role_specs: Iterable[object] | None = None, -) -> dict[str, object]: - return { - "schema_version": AUTO_RESEARCH_PRESET_SCHEMA_VERSION, - "minimal_a2a_recipe": build_auto_research_minimal_a2a_recipe( - open_question=open_question, - output_language=output_language, - role_specs=role_specs, - ), - "owns": [ - "research_roles", - "handoff_hints", - "metric_contract_hints", - "domain_defaults", - ], - "forbidden": [ - "multi_agent_runner", - "real_codex_tui_panes", - "workspace_and_trust_safe_launch", - "decentralized_a2a_driver", - "pane_local_a2a_status_check", - "todo_evidence_status_protocol", - "compact_human_status", - "default_loopx_skill_bootstrap", - "fixed_a2a_wake_prompt", - "kernel_default_skill_prompting", - ], - "worker_skill_scope": "role_specific_semantics_and_successor_todos_only", - "successor_routing": "role_profile_successor_todos_with_target_agent", - "role_count": role_count, - "default_agent_specs": default_auto_research_agent_specs(), - } diff --git a/demo/auto_research/research_state.py b/demo/auto_research/research_state.py deleted file mode 100644 index 8a9bd948ff..0000000000 --- a/demo/auto_research/research_state.py +++ /dev/null @@ -1,1181 +0,0 @@ -"""Small read-model helpers for auto-research worker state. - -This module turns LoopX quota/todo state and rollout evidence events into the -minimal frontier, evidence graph, and decision-candidate shapes that worker -turns need. It intentionally avoids demo boards, starter packs, and other -legacy presentation code. -""" - -from __future__ import annotations - -import json -import re -from pathlib import Path -from typing import Any - -from .delivery_contract import ( - AutoResearchContractLineageError, - auto_research_contract_lineage_dict, - auto_research_contract_lineage_matches, - normalize_auto_research_contract_lineage, -) -from .evidence_packet import ( - METRIC_DIRECTIONS, - RESEARCH_FAILURE_KINDS, - RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION, - RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - _compact_public_text, - _compact_public_token, - _derive_hypothesis_status, - _finite_float, - _is_negative_evidence_event, - _is_retry_evidence_event, - _json_list, - _json_obj, - _metric_improved, - _metric_rank_key, - validate_research_contract, - validate_research_evidence_event, - validate_research_hypothesis, -) -from .preset import AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS - - -AUTO_RESEARCH_FIXTURE_SCHEMA_VERSION = "decentralized_auto_research_fixture_v0" -AUTO_RESEARCH_PROJECTION_SCHEMA_VERSION = "decentralized_auto_research_projection_v0" -RESEARCH_EVIDENCE_GRAPH_SCHEMA_VERSION = "research_evidence_graph_v0" -RESEARCH_FRONTIER_SCHEMA_VERSION = "decentralized_research_frontier_v0" -AUTO_RESEARCH_COMPLETION_STATUS_SCHEMA_VERSION = "auto_research_completion_status_v0" -ROLLOUT_EVIDENCE_GRAPH_SOURCE_KIND = "loopx_rollout_event_log" -AUTO_RESEARCH_ACTION_ALIASES = { - "run_read_only_adapter_tick": "run_dev_eval", -} - - -def load_auto_research_fixture(path: str | Path) -> dict[str, Any]: - payload = json.loads(Path(path).expanduser().read_text(encoding="utf-8")) - return validate_auto_research_fixture(payload) - - -def validate_auto_research_fixture(payload: dict[str, Any]) -> dict[str, Any]: - payload = _json_obj(payload, field="fixture") - schema = _compact_public_token(payload.get("schema_version"), field="schema_version") - if schema != AUTO_RESEARCH_FIXTURE_SCHEMA_VERSION: - raise ValueError(f"schema_version must be {AUTO_RESEARCH_FIXTURE_SCHEMA_VERSION}") - - contract = validate_research_contract(_json_obj(payload.get("research_contract"), field="research_contract")) - hypotheses = [ - validate_research_hypothesis(_json_obj(item, field="hypotheses[]")) - for item in _json_list(payload.get("hypotheses"), field="hypotheses") - ] - evidence_events = [ - validate_research_evidence_event(_json_obj(item, field="evidence_events[]")) - for item in _json_list(payload.get("evidence_events"), field="evidence_events") - ] - - hypothesis_ids = {item["hypothesis_id"] for item in hypotheses} - todo_ids = {item["todo_id"] for item in hypotheses if item.get("todo_id")} - for item in evidence_events: - if item["hypothesis_id"] not in hypothesis_ids: - raise ValueError(f"evidence references unknown hypothesis_id {item['hypothesis_id']}") - if item.get("todo_id") and item["todo_id"] not in todo_ids: - raise ValueError(f"evidence references unknown todo_id {item['todo_id']}") - - agents = [ - _compact_public_token(value, field="agents[]") - for value in _json_list(payload.get("agents"), field="agents") - ] - - return { - "schema_version": schema, - "generated_at": _compact_public_text(payload.get("generated_at"), field="generated_at"), - "research_contract": contract, - "agents": agents, - "hypotheses": hypotheses, - "evidence_events": evidence_events, - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - } - - -def _compact_optional_token(value: Any, *, field: str, default: str) -> str: - if value is None or str(value).strip() == "": - return default - return _compact_public_token(value, field=field) - - -def _compact_optional_text(value: Any, *, field: str, default: str, max_len: int = 240) -> str: - if value is None or str(value).strip() == "": - return default - text = " ".join(str(value).strip().split()) - if len(text) > max_len: - prefix = text[: max_len - 1].rstrip() - if prefix.endswith(".") and not prefix.endswith(".."): - text = prefix - else: - text = prefix.rstrip(".") + "." - # Strip trailing dot-runs so public text validation does not flag them - # as parent-directory markers (text may arrive with "..." from upstream). - while text.endswith(".."): - text = text[:-1] - return _compact_public_text(text, field=field, max_len=max_len) - - -def _live_hypothesis_id(todo_id: str) -> str: - suffix = re.sub(r"[^A-Za-z0-9_:-]+", "_", todo_id.replace("todo_", "", 1)) - return _compact_public_token(f"hyp_{suffix}", field="live.hypothesis_id") - - -def _claimed_by_current_or_unclaimed(item: dict[str, Any], *, agent_id: str) -> bool: - claimed_by = str(item.get("claimed_by") or "").strip() - return not claimed_by or claimed_by == agent_id - - -def _public_todo_items(value: Any) -> list[dict[str, Any]]: - return [item for item in value or [] if isinstance(item, dict) and item.get("todo_id")] - - -def _is_runnable_advancement_todo(item: dict[str, Any]) -> bool: - task_class = str(item.get("task_class") or "advancement_task").strip() - if task_class and task_class != "advancement_task": - return False - status = str(item.get("status") or "open").strip() - if status and status not in {"active", "open", "needs_retry"}: - return False - resume_when = str(item.get("resume_when") or "").strip() - if resume_when and item.get("resume_ready") is False: - return False - return True - - -def _unique_todo_candidates(items: list[dict[str, Any]]) -> list[dict[str, Any]]: - selected: list[dict[str, Any]] = [] - seen: set[str] = set() - for item in items: - todo_id = str(item.get("todo_id") or "").strip() - if not todo_id or todo_id in seen: - continue - seen.add(todo_id) - selected.append(item) - return selected - - -def _quota_payload_todo_candidates( - quota_payload: dict[str, Any], -) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: - gate = quota_payload.get("capability_gate") - if not isinstance(gate, dict): - gate = {} - runnable = _public_todo_items(gate.get("runnable_candidates")) - blocked = _public_todo_items(gate.get("blocked_candidates")) - - agent_summary = quota_payload.get("agent_todo_summary") - if not isinstance(agent_summary, dict): - return _unique_todo_candidates(runnable), _unique_todo_candidates(blocked) - - for key in ( - "active_next_action_executable_items", - "first_executable_items", - "claimed_advancement_open_items", - "unclaimed_advancement_open_items", - ): - runnable.extend( - item - for item in _public_todo_items(agent_summary.get(key)) - if _is_runnable_advancement_todo(item) - ) - - claim_scope = agent_summary.get("claim_scope") - if not isinstance(claim_scope, dict): - claim_scope = {} - for key in ("claimed_by_others_items", "blocked_claimed_items", "blocked_items"): - blocked.extend(_public_todo_items(agent_summary.get(key))) - blocked.extend(_public_todo_items(claim_scope.get(key))) - - return _unique_todo_candidates(runnable), _unique_todo_candidates(blocked) - - -def normalize_auto_research_action(action: object) -> str: - raw = str(action or "").strip() - return AUTO_RESEARCH_ACTION_ALIASES.get(raw, raw) - - -def _todo_frontier_item( - item: dict[str, Any], - *, - default_agent_id: str, - blocked_by: str | None = None, -) -> dict[str, Any]: - todo_id = _compact_public_token(item.get("todo_id"), field="live.todo_id") - claimed_by = _compact_optional_token( - item.get("claimed_by"), - field="live.claimed_by", - default=default_agent_id, - ) - status = _compact_optional_token(item.get("status"), field="live.status", default="open") - summary = { - "hypothesis_id": _live_hypothesis_id(todo_id), - "todo_id": todo_id, - "claimed_by": claimed_by, - "status": "active" if status == "open" else status, - "mechanism_family": _compact_optional_text( - item.get("action_kind") or item.get("task_class") or "advancement_task", - field="live.mechanism_family", - default="advancement_task", - max_len=96, - ), - "source_kind": "todo_item_v0", - "title": _compact_optional_text( - item.get("title") or item.get("text"), - field="live.title", - default=todo_id, - max_len=220, - ), - } - unblocks_todo_id = _compact_optional_token( - item.get("unblocks_todo_id"), - field="live.unblocks_todo_id", - default="", - ) - if unblocks_todo_id: - summary["unblocks_todo_id"] = unblocks_todo_id - resume_when = _compact_optional_text( - item.get("resume_when"), - field="live.resume_when", - default="", - max_len=160, - ) - if resume_when: - summary["resume_when"] = resume_when - if blocked_by: - summary["blocked_by"] = _compact_public_text(blocked_by, field="live.blocked_by", max_len=160) - else: - summary["allowed_action"] = _compact_optional_text( - normalize_auto_research_action(item.get("action_kind") or "advance_todo"), - field="live.allowed_action", - default="advance_todo", - max_len=96, - ) - return summary - - -def _rollout_source_refs(event: dict[str, Any]) -> tuple[list[str], str | None]: - grounding_refs: list[str] = [] - novelty_audit_ref: str | None = None - for index, ref in enumerate(event.get("source_refs") or []): - if not isinstance(ref, dict): - continue - kind = str(ref.get("kind") or "").strip() - ref_id = ref.get("id") - if not ref_id: - continue - if kind == "grounding": - grounding_refs.append( - _compact_public_text(ref_id, field=f"rollout.source_refs[{index}].id") - ) - elif kind == "novelty_audit" and novelty_audit_ref is None: - novelty_audit_ref = _compact_public_text( - ref_id, - field=f"rollout.source_refs[{index}].id", - ) - return grounding_refs, novelty_audit_ref - - -def _rollout_hypothesis_text(event: dict[str, Any], details: dict[str, Any]) -> str: - if details.get("hypothesis"): - return _compact_public_text(details["hypothesis"], field="rollout.details.hypothesis") - summary = str(event.get("summary") or "") - prefix = "auto-research hypothesis " - if summary.startswith(prefix) and ": " in summary: - return _compact_public_text( - summary.split(": ", 1)[1], - field="rollout.summary.hypothesis", - ) - fallback = f"Evidence-backed hypothesis {details.get('hypothesis_id') or event.get('todo_id')}" - return _compact_public_text(fallback, field="rollout.summary.hypothesis") - - -def _rollout_contract_lineage(details: dict[str, Any]) -> dict[str, Any]: - lineage = normalize_auto_research_contract_lineage( - details, - field="rollout.details", - ) - return auto_research_contract_lineage_dict(lineage) if lineage else {} - - -def _hypothesis_blocked_by( - *, - status: object, - negative_count: int, - retry_count: int, -) -> list[str]: - if str(status) == "contradicted" or negative_count: - return ["evidence_or_boundary_guardrail_failed"] - if str(status) == "needs_retry" or retry_count: - return ["needs_retry_evidence"] - return [] - - -def _research_hypothesis_from_rollout_event(event: dict[str, Any]) -> dict[str, Any] | None: - if str(event.get("event_kind") or "") != "research_hypothesis": - return None - if str(event.get("classification") or "") != RESEARCH_HYPOTHESIS_SCHEMA_VERSION: - return None - details = _json_obj(event.get("details") or {}, field="rollout.hypothesis.details") - grounding_refs, novelty_audit_ref = _rollout_source_refs(event) - negative_count = int(details.get("negative_evidence_count") or 0) - retry_count = int(details.get("needs_retry_count") or 0) - status = details.get("status") or event.get("status") or "active" - return validate_research_hypothesis( - { - "schema_version": RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - "hypothesis_id": details.get("hypothesis_id"), - "parent_hypothesis_id": details.get("parent_hypothesis_id") or None, - "todo_id": event.get("todo_id"), - "claimed_by": event.get("agent_id") or "unknown_agent", - "mechanism_family": details.get("mechanism_family") or "rollout_imported", - "hypothesis": _rollout_hypothesis_text(event, details), - "status": status, - "grounding_refs": grounding_refs, - "novelty_audit_ref": novelty_audit_ref, - "blocked_by": _hypothesis_blocked_by( - status=status, - negative_count=negative_count, - retry_count=retry_count, - ), - **_rollout_contract_lineage(details), - } - ) - - -def _research_evidence_from_rollout_event(event: dict[str, Any]) -> dict[str, Any] | None: - if str(event.get("event_kind") or "") != "research_evidence": - return None - if str(event.get("classification") or "") != RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION: - return None - details = _json_obj(event.get("details") or {}, field="rollout.evidence.details") - return validate_research_evidence_event( - { - "schema_version": RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION, - "hypothesis_id": details.get("hypothesis_id"), - "todo_id": event.get("todo_id"), - "agent_id": event.get("agent_id") or "unknown_agent", - "attempt": details.get("attempt") or 1, - "split": details.get("split"), - "metric": { - "name": details.get("metric_name"), - "value": details.get("metric_value"), - "direction": details.get("metric_direction"), - }, - "baseline_metric": details.get("baseline_metric"), - "eval_status": details.get("eval_status") or event.get("status"), - "primary_metric_status": details.get("primary_metric_status") or "inconclusive", - "failure_kind": details.get("failure_kind") or None, - "measurement_scope": details.get("measurement_scope") or None, - "remediation_attempt": bool(details.get("remediation_attempt")), - "artifact_refs": event.get("artifact_refs") or [], - "protected_scope_clean": bool(details.get("protected_scope_clean")), - **_rollout_contract_lineage(details), - } - ) - - -def _synthetic_hypothesis_from_evidence(events: list[dict[str, Any]]) -> dict[str, Any]: - first = events[0] - status = _derive_hypothesis_status(events) - blocked_by = [] - if status == "contradicted": - blocked_by.append("evidence_or_boundary_guardrail_failed") - elif status == "needs_retry": - blocked_by.append("needs_retry_evidence") - return validate_research_hypothesis( - { - "schema_version": RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - "hypothesis_id": first["hypothesis_id"], - "parent_hypothesis_id": None, - "todo_id": first["todo_id"], - "claimed_by": first["agent_id"], - "mechanism_family": "rollout_evidence_only", - "hypothesis": f"Evidence-backed hypothesis {first['hypothesis_id']}", - "status": status, - "grounding_refs": [], - "novelty_audit_ref": None, - "blocked_by": blocked_by, - "wish_id": first.get("wish_id"), - "contract_ref": first.get("contract_ref"), - "contract_revision": first.get("contract_revision"), - } - ) - - -def _best_metric(events: list[dict[str, Any]], *, split: str, direction: str) -> float | None: - values = [ - event["metric"]["value"] - for event in events - if event["split"] == split and event["eval_status"] == "scored" - ] - if not values: - return None - return max(values, key=lambda value: _metric_rank_key(value, direction=direction)) - - -def _default_failure_kind( - *, - status: str, - events: list[dict[str, Any]], -) -> str | None: - explicit = [event for event in events if event.get("failure_kind")] - if explicit: - return str(explicit[-1]["failure_kind"]) - if any(not event["protected_scope_clean"] for event in events): - return "guardrail_or_protected_boundary" - if status == "needs_retry": - return None - return "mechanism_contradicted" - - -def _failure_measurement_scope(events: list[dict[str, Any]]) -> str | None: - values = [ - str(event.get("measurement_scope") or "").strip() - for event in events - if event.get("measurement_scope") - ] - return values[-1] if values else None - - -def _ancestor_hypothesis_ids( - hypothesis_id: str, - *, - hypotheses: dict[str, dict[str, Any]], -) -> set[str]: - ancestor_ids: set[str] = set() - current_id = hypothesis_id - while current_id and current_id not in ancestor_ids: - ancestor_ids.add(current_id) - current = hypotheses.get(current_id) or {} - current_id = str(current.get("parent_hypothesis_id") or "").strip() - return ancestor_ids - - -def build_research_evidence_graph_from_records( - *, - goal_id: str, - hypotheses: list[dict[str, Any]], - evidence_events: list[dict[str, Any]], - metric_name: str, - metric_direction: str, - baseline_metric: float | None, - source_kind: str = "public_records", -) -> dict[str, Any]: - goal = _compact_public_token(goal_id, field="goal_id") - direction = _compact_public_token(metric_direction, field="metric.direction") - if direction not in METRIC_DIRECTIONS: - raise ValueError("metric.direction must be maximize or minimize") - name = _compact_public_token(metric_name, field="metric.name") - source = _compact_public_token(source_kind, field="source_kind") - hypotheses = [validate_research_hypothesis(dict(item)) for item in hypotheses] - events = [validate_research_evidence_event(dict(event)) for event in evidence_events] - baseline = _finite_float(baseline_metric, field="baseline_metric") - scored_events = [event for event in events if event["eval_status"] == "scored"] - best_dev = _best_metric(scored_events, split="dev", direction=direction) - best_holdout = _best_metric(scored_events, split="holdout", direction=direction) - hypotheses_by_id = {item["hypothesis_id"]: item for item in hypotheses} - events_by_hypothesis: dict[str, list[dict[str, Any]]] = {} - for event in events: - events_by_hypothesis.setdefault(event["hypothesis_id"], []).append(event) - nodes = [] - for item in hypotheses: - item_events = events_by_hypothesis.get(item["hypothesis_id"], []) - item_scored_events = [event for event in item_events if event["eval_status"] == "scored"] - item_best_dev = _best_metric(item_scored_events, split="dev", direction=direction) - item_best_holdout = _best_metric(item_scored_events, split="holdout", direction=direction) - item_artifact_refs = sorted( - { - ref - for event in item_events - for ref in event.get("artifact_refs", []) - if ref - } - ) - item_splits = sorted({event["split"] for event in item_events if event.get("split")}) - item_negative_count = len([event for event in item_events if _is_negative_evidence_event(event)]) - item_retry_count = len([event for event in item_events if _is_retry_evidence_event(event)]) - item_failure_kind = ( - _default_failure_kind(status=item["status"], events=item_events) - if item["status"] in {"contradicted", "retired", "needs_retry"} - or item_negative_count - else None - ) - lineage_ids = _ancestor_hypothesis_ids( - item["hypothesis_id"], - hypotheses=hypotheses_by_id, - ) - remediation_attempt_count = len( - [ - event - for event in events - if event["hypothesis_id"] in lineage_ids - and event.get("remediation_attempt") - ] - ) - nodes.append( - { - "hypothesis_id": item["hypothesis_id"], - "parent_hypothesis_id": item["parent_hypothesis_id"], - "todo_id": item["todo_id"], - "claimed_by": item["claimed_by"], - "mechanism_family": item["mechanism_family"], - "hypothesis": item["hypothesis"], - "status": item["status"], - "grounding_refs": item["grounding_refs"], - "novelty_audit_ref": item["novelty_audit_ref"], - "artifact_refs": item_artifact_refs, - "splits": item_splits, - "evidence_event_count": len(item_events), - "best_dev_metric": item_best_dev, - "best_holdout_metric": item_best_holdout, - "dev_improved": _metric_improved( - value=item_best_dev, - baseline=baseline, - direction=direction, - ), - "holdout_improved": _metric_improved( - value=item_best_holdout, - baseline=baseline, - direction=direction, - ), - "negative_evidence_count": item_negative_count, - "needs_retry_count": item_retry_count, - "failure_kind": item_failure_kind, - "measurement_scope": _failure_measurement_scope(item_events), - "remediation_attempt_count": remediation_attempt_count, - "source_kind": source, - } - ) - return { - "schema_version": RESEARCH_EVIDENCE_GRAPH_SCHEMA_VERSION, - "goal_id": goal, - "hypothesis_count": len(hypotheses), - "evidence_event_count": len(events), - "todo_ids": sorted({item["todo_id"] for item in hypotheses}), - "agent_ids": sorted({item["claimed_by"] for item in hypotheses}), - "metric": { - "name": name, - "direction": direction, - "baseline": baseline, - }, - "baseline_metric": baseline, - "best_dev_metric": best_dev, - "best_holdout_metric": best_holdout, - "holdout_improved": _metric_improved(value=best_holdout, baseline=baseline, direction=direction), - "negative_evidence_count": len([event for event in events if _is_negative_evidence_event(event)]), - "needs_retry_count": len( - [event for event in events if _is_retry_evidence_event(event)] - ) + len([item for item in hypotheses if item["status"] == "needs_retry"]), - "remediation_attempt_count": len( - [event for event in events if event.get("remediation_attempt")] - ), - "nodes": nodes, - "source_kind": source, - } - - -def build_research_evidence_graph_from_rollout_events( - *, - goal_id: str, - rollout_events: list[dict[str, Any]], -) -> dict[str, Any]: - goal = _compact_public_token(goal_id, field="goal_id") - hypotheses_by_id, evidence_events = _research_records_from_rollout_events( - rollout_events - ) - - events_by_hypothesis: dict[str, list[dict[str, Any]]] = {} - for evidence in evidence_events: - events_by_hypothesis.setdefault(evidence["hypothesis_id"], []).append(evidence) - for hypothesis_id, events in events_by_hypothesis.items(): - if hypothesis_id not in hypotheses_by_id: - hypotheses_by_id[hypothesis_id] = _synthetic_hypothesis_from_evidence(events) - evidence_events = [ - evidence - for evidence in evidence_events - if auto_research_contract_lineage_matches( - evidence, - expected=normalize_auto_research_contract_lineage( - hypotheses_by_id.get(evidence["hypothesis_id"], {}), - field="hypothesis", - ), - ) - ] - - first_metric_event = evidence_events[0] if evidence_events else None - metric = first_metric_event["metric"] if first_metric_event else {} - graph = build_research_evidence_graph_from_records( - goal_id=goal, - hypotheses=list(hypotheses_by_id.values()), - evidence_events=evidence_events, - metric_name=metric.get("name") or "research_metric", - metric_direction=metric.get("direction") or "maximize", - baseline_metric=first_metric_event.get("baseline_metric") if first_metric_event else None, - source_kind=ROLLOUT_EVIDENCE_GRAPH_SOURCE_KIND, - ) - _apply_contract_lineage_to_graph( - graph, - hypotheses_by_id=hypotheses_by_id, - ) - return graph - - -def _research_records_from_rollout_events( - rollout_events: list[dict[str, Any]], -) -> tuple[dict[str, dict[str, Any]], list[dict[str, Any]]]: - hypotheses_by_id: dict[str, dict[str, Any]] = {} - evidence_events: list[dict[str, Any]] = [] - invalid_lineage_hypothesis_ids: set[str] = set() - for event in rollout_events: - try: - hypothesis = _research_hypothesis_from_rollout_event(event) - except AutoResearchContractLineageError: - invalid_lineage_hypothesis_ids.add( - str((event.get("details") or {}).get("hypothesis_id") or "") - ) - continue - if hypothesis: - hypotheses_by_id[hypothesis["hypothesis_id"]] = hypothesis - continue - try: - evidence = _research_evidence_from_rollout_event(event) - except AutoResearchContractLineageError: - invalid_lineage_hypothesis_ids.add( - str((event.get("details") or {}).get("hypothesis_id") or "") - ) - continue - if evidence: - evidence_events.append(evidence) - invalid_lineage_hypothesis_ids.discard("") - for hypothesis_id in invalid_lineage_hypothesis_ids: - hypotheses_by_id.pop(hypothesis_id, None) - evidence_events = [ - event - for event in evidence_events - if event["hypothesis_id"] not in invalid_lineage_hypothesis_ids - ] - return hypotheses_by_id, evidence_events - - -def _apply_contract_lineage_to_graph( - graph: dict[str, Any], - *, - hypotheses_by_id: dict[str, dict[str, Any]], -) -> None: - for node in graph.get("nodes") or []: - if not isinstance(node, dict): - continue - hypothesis = hypotheses_by_id.get(str(node.get("hypothesis_id") or "")) - if not hypothesis: - continue - lineage = normalize_auto_research_contract_lineage( - hypothesis, - field="hypothesis", - ) - if not lineage: - continue - node.update(auto_research_contract_lineage_dict(lineage)) - - -def build_research_evidence_graph(fixture: dict[str, Any]) -> dict[str, Any]: - fixture = validate_auto_research_fixture(fixture) - contract = fixture["research_contract"] - return build_research_evidence_graph_from_records( - goal_id=contract["goal_id"], - hypotheses=fixture["hypotheses"], - evidence_events=fixture["evidence_events"], - metric_name=contract["metric"]["name"], - metric_direction=contract["metric"]["direction"], - baseline_metric=contract["metric"]["baseline"], - source_kind="public_fixture", - ) - - -def build_research_decision_candidates(evidence_graph: dict[str, Any]) -> dict[str, list[dict[str, Any]]]: - graph = _json_obj(evidence_graph, field="evidence_graph") - metric = graph.get("metric") if isinstance(graph.get("metric"), dict) else {} - direction = str(metric.get("direction") or "maximize") - if direction not in METRIC_DIRECTIONS: - direction = "maximize" - baseline = _finite_float(metric.get("baseline"), field="evidence_graph.metric.baseline") - source_kind = _compact_optional_token( - graph.get("source_kind"), - field="evidence_graph.source_kind", - default="unknown_source", - ) - promotion_candidates: list[dict[str, Any]] = [] - dev_promotion_candidates: list[dict[str, Any]] = [] - validated_promotion_candidates: list[dict[str, Any]] = [] - retirement_candidates: list[dict[str, Any]] = [] - for raw_node in graph.get("nodes") or []: - if not isinstance(raw_node, dict): - continue - hypothesis_id = _compact_public_token(raw_node.get("hypothesis_id"), field="node.hypothesis_id") - todo_id = _compact_public_token(raw_node.get("todo_id"), field="node.todo_id") - status = _compact_optional_token(raw_node.get("status"), field="node.status", default="active") - dev_metric = _finite_float(raw_node.get("best_dev_metric"), field="node.best_dev_metric") - holdout_metric = _finite_float(raw_node.get("best_holdout_metric"), field="node.best_holdout_metric") - negative_count = int(raw_node.get("negative_evidence_count") or 0) - evidence_count = int(raw_node.get("evidence_event_count") or 0) - dev_improved = bool(raw_node.get("dev_improved")) or _metric_improved( - value=dev_metric, - baseline=baseline, - direction=direction, - ) - holdout_improved = bool(raw_node.get("holdout_improved")) or _metric_improved( - value=holdout_metric, - baseline=baseline, - direction=direction, - ) - failure_kind = str(raw_node.get("failure_kind") or "") - retry_exhausted = status == "needs_retry" and failure_kind == "retry_exhausted" - if status in {"contradicted", "retired"} or negative_count > 0 or retry_exhausted: - if failure_kind not in RESEARCH_FAILURE_KINDS: - failure_kind = ( - "guardrail_or_protected_boundary" - if status == "contradicted" and negative_count - else "mechanism_contradicted" - ) - remediation_attempt_count = int( - raw_node.get("remediation_attempt_count") or 0 - ) - remediation_allowed = ( - failure_kind == "data_or_measurement_gap" - and remediation_attempt_count < 1 - ) - retirement_candidates.append( - { - "hypothesis_id": hypothesis_id, - "todo_id": todo_id, - "status": status, - "negative_evidence_count": negative_count, - "evidence_event_count": evidence_count, - "reason": ( - "retry_exhausted" - if retry_exhausted - else "negative_or_guardrail_evidence" - if negative_count - else f"status:{status}" - ), - "failure_kind": failure_kind, - "measurement_scope": raw_node.get("measurement_scope"), - "remediation_attempt_count": remediation_attempt_count, - "remediation_attempt_limit": 1, - "remediation_allowed": remediation_allowed, - "next_outcome": ( - "remediate_data_measurement" - if remediation_allowed - else "propose_failure_successor" - ), - "source_kind": source_kind, - } - ) - continue - if status in {"supported", "promoted"} or dev_improved: - requires = ["boundary_scan"] - requires.append("promotion_decision" if holdout_improved else "holdout_eval") - candidate = { - "hypothesis_id": hypothesis_id, - "todo_id": todo_id, - "status": status, - "dev_metric": dev_metric, - "holdout_metric": holdout_metric, - "evidence_event_count": evidence_count, - "requires": requires, - "source_kind": source_kind, - } - promotion_candidates.append(candidate) - if holdout_improved: - validated_promotion_candidates.append(candidate) - else: - dev_promotion_candidates.append(candidate) - return { - "dev_promotion_candidates": dev_promotion_candidates, - "validated_promotion_candidates": validated_promotion_candidates, - "promotion_candidates": promotion_candidates, - "retirement_candidates": retirement_candidates, - } - - -def build_research_failure_continuation_resolution( - decision_candidates: dict[str, list[dict[str, Any]]], - *, - selected_todo_id: str | None = None, - selected_lineage_todo_id: str | None = None, - allow_unbound_singleton: bool = False, - require_selected_failure_match: bool = False, -) -> dict[str, Any]: - candidates = [ - dict(candidate) - for candidate in decision_candidates.get("retirement_candidates") or [] - if isinstance(candidate, dict) - ] - lineage_todo_id = str(selected_lineage_todo_id or "").strip() - selected_id = str(selected_todo_id or "").strip() - selection_bound = bool(lineage_todo_id) if require_selected_failure_match else bool( - lineage_todo_id or selected_id - ) - selected_parent_todo_id = ( - lineage_todo_id - if require_selected_failure_match - else lineage_todo_id or selected_id - ) - matched = ( - [ - candidate - for candidate in candidates - if str(candidate.get("todo_id") or "").strip() == selected_parent_todo_id - ] - if selection_bound - else [] - ) - candidate = ( - matched[0] - if selection_bound and len(matched) == 1 - else candidates[0] - if allow_unbound_singleton and not selection_bound and len(candidates) == 1 - else None - ) - continuation = ( - { - "hypothesis_id": candidate["hypothesis_id"], - "source_todo_id": candidate["todo_id"], - "failure_kind": candidate["failure_kind"], - "measurement_scope": candidate.get("measurement_scope"), - "remediation_attempt_count": candidate["remediation_attempt_count"], - "remediation_attempt_limit": candidate["remediation_attempt_limit"], - "next_outcome": candidate["next_outcome"], - "monitor_allowed": False, - } - if candidate - else None - ) - return { - "failure_continuation": continuation, - "retirement_candidate_count": len(candidates), - "matched_candidate_count": len(matched), - "lineage_todo_id": lineage_todo_id or None, - "selection_bound": selection_bound, - "ambiguous": not selection_bound and len(candidates) > 1, - "unresolved": require_selected_failure_match - and bool(candidates) - and (not lineage_todo_id or len(matched) != 1), - } - - -def build_research_failure_continuation( - decision_candidates: dict[str, list[dict[str, Any]]], - *, - selected_todo_id: str | None = None, - selected_lineage_todo_id: str | None = None, - allow_unbound_singleton: bool = True, - require_selected_failure_match: bool = False, -) -> dict[str, Any] | None: - resolution = build_research_failure_continuation_resolution( - decision_candidates, - selected_todo_id=selected_todo_id, - selected_lineage_todo_id=selected_lineage_todo_id, - allow_unbound_singleton=allow_unbound_singleton, - require_selected_failure_match=require_selected_failure_match, - ) - return resolution["failure_continuation"] - - -def _holdout_metric_sequence_from_graph(evidence_graph: dict[str, Any]) -> list[float]: - sequence: list[float] = [] - for raw_node in evidence_graph.get("nodes") or []: - if not isinstance(raw_node, dict): - continue - value = raw_node.get("best_holdout_metric") - if value is None or isinstance(value, bool): - continue - number = _finite_float(value, field="node.best_holdout_metric") - if number is not None: - sequence.append(number) - return sequence - - -def _metric_sequence_improvement_count( - sequence: list[float], - *, - baseline: float | None, - direction: str, -) -> int: - if baseline is None: - return 0 - count = 0 - previous = baseline - minimize = direction == "minimize" - for metric in sequence: - improved = metric < previous if minimize else metric > previous - if improved: - count += 1 - previous = metric - return count - - -def build_auto_research_completion_status( - evidence_graph: dict[str, Any], - decision_candidates: dict[str, list[dict[str, Any]]] | None = None, -) -> dict[str, Any]: - graph = _json_obj(evidence_graph, field="evidence_graph") - decisions = decision_candidates or build_research_decision_candidates(graph) - metric = graph.get("metric") if isinstance(graph.get("metric"), dict) else {} - direction = str(metric.get("direction") or "maximize") - if direction not in METRIC_DIRECTIONS: - direction = "maximize" - baseline = _finite_float(metric.get("baseline"), field="evidence_graph.metric.baseline") - holdout_sequence = _holdout_metric_sequence_from_graph(graph) - holdout_improvement_count = _metric_sequence_improvement_count( - holdout_sequence, - baseline=baseline, - direction=direction, - ) - required_holdout_improvement_count = AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS - dev_pending_count = len(decisions.get("dev_promotion_candidates") or []) - validated_count = len(decisions.get("validated_promotion_candidates") or []) - promotion_count = len(decisions.get("promotion_candidates") or []) - retirement_count = len(decisions.get("retirement_candidates") or []) - failure_resolution = build_research_failure_continuation_resolution( - decisions, - allow_unbound_singleton=True, - ) - failure_continuation = failure_resolution["failure_continuation"] - - status = "active" - next_action = "continue_frontier" - required_actions: list[str] = [] - quiet_completion_allowed = False - reason = "frontier_still_active" - if validated_count and holdout_improvement_count >= required_holdout_improvement_count: - status = "target_reached" - next_action = "quiet_completion" - quiet_completion_allowed = True - reason = "required_holdout_improvements_reached" - elif dev_pending_count: - status = "holdout_eval_required" - next_action = "run_holdout_eval" - required_actions = ["holdout_eval", "boundary_scan"] - reason = "dev_promotion_candidate_pending_holdout" - elif validated_count or promotion_count: - status = "promotion_review_required" - next_action = "review_promotion_readiness" - required_actions = ["boundary_scan", "promotion_decision"] - reason = "validated_promotion_candidate_pending_decision" - elif retirement_count: - status = "failure_successor_required" - next_action = str( - (failure_continuation or {}).get("next_outcome") - or "propose_failure_successor" - ) - required_actions = [next_action] - reason = "retirement_candidate_requires_successor_or_exhaustion" - - return { - "schema_version": AUTO_RESEARCH_COMPLETION_STATUS_SCHEMA_VERSION, - "status": status, - "next_action": next_action, - "reason": reason, - "quiet_completion_allowed": quiet_completion_allowed, - "required_actions": required_actions, - "required_holdout_improvement_count": required_holdout_improvement_count, - "holdout_improvement_count": holdout_improvement_count, - "holdout_metric_sequence": holdout_sequence, - "validated_promotion_candidate_count": validated_count, - "dev_candidate_pending_holdout_count": dev_pending_count, - "promotion_candidate_count": promotion_count, - "retirement_candidate_count": retirement_count, - "failure_continuation": failure_continuation, - "failure_continuation_ambiguous": failure_resolution["ambiguous"], - } - - -def build_auto_research_projection( - fixture: dict[str, Any], - *, - agent_id: str, -) -> dict[str, Any]: - fixture = validate_auto_research_fixture(fixture) - agent = _compact_public_token(agent_id, field="agent_id") - contract = fixture["research_contract"] - hypotheses = fixture["hypotheses"] - evidence_graph = build_research_evidence_graph(fixture) - decision_candidates = build_research_decision_candidates(evidence_graph) - completion = build_auto_research_completion_status(evidence_graph, decision_candidates) - - runnable_statuses = {"active", "needs_retry"} - selected = None - blocked: list[dict[str, Any]] = [] - runnable: list[dict[str, Any]] = [] - for item in hypotheses: - item_summary = { - "hypothesis_id": item["hypothesis_id"], - "todo_id": item["todo_id"], - "claimed_by": item["claimed_by"], - "status": item["status"], - "mechanism_family": item["mechanism_family"], - } - if item["claimed_by"] == agent and item["status"] in runnable_statuses and not item["blocked_by"]: - runnable.append(item_summary | {"allowed_action": "run_dev_attempt"}) - selected = selected or runnable[-1] - elif item["claimed_by"] != agent and item["status"] in runnable_statuses: - blocked.append(item_summary | {"blocked_by": f"claimed_by:{item['claimed_by']}"}) - elif item["blocked_by"]: - blocked.append(item_summary | {"blocked_by": ",".join(item["blocked_by"])}) - - frontier = { - "schema_version": RESEARCH_FRONTIER_SCHEMA_VERSION, - "goal_id": contract["goal_id"], - "agent_id": agent, - "selected": selected, - "runnable": runnable, - "blocked": blocked, - "promotion_candidates": decision_candidates["promotion_candidates"], - "retirement_candidates": decision_candidates["retirement_candidates"], - "completion": completion, - } - return { - "ok": True, - "schema_version": AUTO_RESEARCH_PROJECTION_SCHEMA_VERSION, - "source_schema_version": fixture["schema_version"], - "frontier": frontier, - "evidence_graph": evidence_graph, - "completion": completion, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "source": "public_fixture", - }, - } - - -def build_live_auto_research_projection( - *, - goal_id: str, - agent_id: str, - quota_payload: dict[str, Any], - rollout_events: list[dict[str, Any]] | None = None, -) -> dict[str, Any]: - goal = _compact_public_token(goal_id, field="goal_id") - agent = _compact_public_token(agent_id, field="agent_id") - if not quota_payload.get("ok"): - raise ValueError("quota payload must be ok for live auto-research projection") - - runnable_candidates, blocked_candidates = _quota_payload_todo_candidates(quota_payload) - selected_raw = quota_payload.get("agent_lane_next_action") - selected = None - if ( - isinstance(selected_raw, dict) - and selected_raw.get("todo_id") - and _claimed_by_current_or_unclaimed(selected_raw, agent_id=agent) - ): - selected = _todo_frontier_item(selected_raw, default_agent_id=agent) - else: - for item in runnable_candidates: - if _claimed_by_current_or_unclaimed(item, agent_id=agent): - selected = _todo_frontier_item(item, default_agent_id=agent) - break - - runnable: list[dict[str, Any]] = [] - seen_todos: set[str] = set() - for item in runnable_candidates: - if not item.get("todo_id") or not _claimed_by_current_or_unclaimed(item, agent_id=agent): - continue - summary = _todo_frontier_item(item, default_agent_id=agent) - if summary["todo_id"] in seen_todos: - continue - seen_todos.add(summary["todo_id"]) - runnable.append(summary) - - blocked: list[dict[str, Any]] = [] - other_claimed_context = [ - item - for item in runnable_candidates - if item.get("todo_id") and not _claimed_by_current_or_unclaimed(item, agent_id=agent) - ] - for item in [*other_claimed_context, *blocked_candidates][:12]: - if not item.get("todo_id"): - continue - claimed_by = item.get("claimed_by") - status = item.get("status") or "blocked" - reason = f"claimed_by:{claimed_by}" if claimed_by and claimed_by != agent else f"status:{status}" - blocked.append(_todo_frontier_item(item, default_agent_id=agent, blocked_by=reason)) - - todo_graph = { - "schema_version": RESEARCH_EVIDENCE_GRAPH_SCHEMA_VERSION, - "goal_id": goal, - "hypothesis_count": len(runnable) + len(blocked), - "evidence_event_count": 0, - "todo_ids": sorted({item["todo_id"] for item in [*runnable, *blocked]}), - "agent_ids": sorted({item["claimed_by"] for item in [*runnable, *blocked]}), - "metric": { - "name": "runnable_hypotheses", - "direction": "maximize", - "baseline": 0.0, - }, - "baseline_metric": None, - "best_dev_metric": None, - "best_holdout_metric": None, - "holdout_improved": False, - "negative_evidence_count": 0, - "needs_retry_count": 0, - "nodes": [ - { - "hypothesis_id": item["hypothesis_id"], - "parent_hypothesis_id": None, - "todo_id": item["todo_id"], - "claimed_by": item["claimed_by"], - "status": item["status"], - "source_kind": item["source_kind"], - } - for item in [*runnable, *blocked] - ], - "source_kind": "loopx_live_quota_status", - } - rollout_graph = build_research_evidence_graph_from_rollout_events( - goal_id=goal, - rollout_events=rollout_events or [], - ) - evidence_graph = ( - rollout_graph - if rollout_graph["evidence_event_count"] or rollout_graph["hypothesis_count"] - else todo_graph - ) - decisions = build_research_decision_candidates(evidence_graph) - completion = build_auto_research_completion_status(evidence_graph, decisions) - frontier = { - "schema_version": RESEARCH_FRONTIER_SCHEMA_VERSION, - "goal_id": goal, - "agent_id": agent, - "selected": selected, - "runnable": runnable, - "blocked": blocked, - "promotion_candidates": decisions["promotion_candidates"], - "retirement_candidates": decisions["retirement_candidates"], - "completion": completion, - "source_kind": "loopx_live_quota_status", - } - return { - "ok": True, - "schema_version": AUTO_RESEARCH_PROJECTION_SCHEMA_VERSION, - "source_schema_version": "loopx_live_quota_status_v0", - "frontier": frontier, - "evidence_graph": evidence_graph, - "decision_candidates": decisions, - "completion": completion, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "source": ( - "live_quota_status_and_rollout_event_log" - if evidence_graph.get("source_kind") == ROLLOUT_EVIDENCE_GRAPH_SOURCE_KIND - else "live_quota_status_projection" - ), - }, - } diff --git a/demo/auto_research/role_profiles.py b/demo/auto_research/role_profiles.py deleted file mode 100644 index 442ee9ef1f..0000000000 --- a/demo/auto_research/role_profiles.py +++ /dev/null @@ -1,395 +0,0 @@ -from __future__ import annotations - -AUTO_RESEARCH_ROLE_PROFILE_SCHEMA_VERSION = "auto_research_role_profile_v0" -AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS = 2 -AUTO_RESEARCH_REQUIRED_SKILL = "loopx-auto-research" -AUTO_RESEARCH_WORKER_SKILL_SOURCE = ( - "demo/auto_research/worker_skill/SKILL.md" -) -AUTO_RESEARCH_HOLDOUT_SUCCESSOR_TEXT = "[P0-auto-research-live] Run held-out validation for the dev-supported hypothesis from {source_todo_id}, append public-safe evidence, and summarize promotion readiness." -AUTO_RESEARCH_VALIDATED_SUMMARY_SUCCESSOR_TEXT = "[P0-auto-research-live] Summarize held-out validation from {source_todo_id}, promotion readiness, and the public claim boundary for the supported hypothesis." -AUTO_RESEARCH_REFINED_HYPOTHESIS_SUCCESSOR_TEXT = "[P0-auto-research-live] Grow the next evidence-backed hypothesis from the validated branch {source_todo_id} and route a second dev attempt." -AUTO_RESEARCH_REFINED_DEV_SUCCESSOR_TEXT = "[P0-auto-research-live] Run the refined hypothesis from {source_todo_id} on the dev split, append public-safe evidence, and hand off the second holdout check." -AUTO_RESEARCH_CURATOR_REVIEW_SUCCESSOR_TEXT = "[P0-auto-research-live] Re-check the research contract and protected scope after {source_todo_id}, then keep the next collective round honest." -AUTO_RESEARCH_HYPOTHESIS_FRONTIER_REVIEW_SUCCESSOR_TEXT = "[P0-auto-research-live] Review the current hypothesis frontier after {source_todo_id}, record whether another candidate is needed, and keep the round participation visible." -AUTO_RESEARCH_PROMOTION_READINESS_REVIEW_SUCCESSOR_TEXT = "[P0-auto-research-live] Review promotion readiness after {source_todo_id}, record the current evidence gap, and wait for the holdout handoff." -AUTO_RESEARCH_DATA_MEASUREMENT_REMEDIATION_SUCCESSOR_TEXT = "[P0-auto-research-live] Repair the one declared data or measurement gap from {source_todo_id} without changing thresholds, horizons, assets, or acceptance gates; return to evidence evaluation." -AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_TEXT = "[P0-auto-research-live] From the retired evidence linked to {source_todo_id}, propose one changed-mechanism hypothesis or explicitly exhaust the frontier; do not relax parameters or create a future-data monitor." -AUTO_RESEARCH_SUCCESSOR_TODO_COMMAND_TEMPLATE = ( - "loopx todo add --goal-id {goal_id_shell} --role agent " - "--text {text_shell} --task-class {task_class_shell} " - "--action-kind {action_kind_shell} --claimed-by {target_agent_id_shell} " - "--unblocks-todo-id {source_todo_id_shell}" -) - - -def _condition(path: str, op: str, value: object, fail_reason: str) -> dict[str, object]: - return {"path": path, "op": op, "value": value, "fail_reason": fail_reason} - - -def _all(*conditions: dict[str, object]) -> dict[str, object]: - return {"all": list(conditions)} - - -AUTO_RESEARCH_HOLDOUT_SUCCESSOR_CONDITION = _all( - _condition("decision_summary.dev_candidate_pending_holdout_count", "gt", 0, "no_dev_promotion_candidate") -) -AUTO_RESEARCH_VALIDATED_SUMMARY_SUCCESSOR_CONDITION = _all( - _condition("decision_summary.validated_promotion_candidate_count", "gt", 0, "no_validated_promotion_candidate") -) -AUTO_RESEARCH_NEXT_HYPOTHESIS_SUCCESSOR_CONDITION = _all( - _condition("decision_summary.validated_promotion_candidate_count", "gt", 0, "no_validated_promotion_candidate"), - _condition( - "decision_summary.holdout_improvement_count", - "lt", - AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS, - "target_holdout_improvements_reached", - ), -) -AUTO_RESEARCH_REFINED_DEV_SUCCESSOR_CONDITION = _all( - _condition("decision_summary.holdout_improvement_count", "gt", 0, "initial_seed_dev_already_exists"), - _condition( - "decision_summary.holdout_improvement_count", - "lt", - AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS, - "target_holdout_improvements_reached", - ), - _condition("decision_summary.dev_candidate_pending_holdout_count", "eq", 0, "dev_candidate_already_pending_holdout"), -) -AUTO_RESEARCH_FAILURE_REMEDIATION_SUCCESSOR_CONDITION = _all( - _condition( - "decision_summary.failure_continuation.next_outcome", - "eq", - "remediate_data_measurement", - "failure_remediation_not_required", - ), -) -AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_CONDITION = _all( - _condition( - "decision_summary.failure_continuation.next_outcome", - "eq", - "propose_failure_successor", - "failure_proposer_successor_not_required", - ), -) - -AUTO_RESEARCH_CONTINUATION_POLICY = { - "schema_version": "auto_research_continuation_policy_v0", - "successor_source": "role_profile.successor_todos", - "required_holdout_improvement_count": AUTO_RESEARCH_REQUIRED_HOLDOUT_IMPROVEMENTS, - "unmet_target_rule": ( - "if validated_promotion_candidate_count>0 and holdout_improvement_count is below " - "required_holdout_improvement_count, create or link the role-declared successor" - ), - "no_followup_rule": ( - "no-follow-up is valid only after the target is reached, a blocker/user gate is " - "projected, or a proposer explicitly exhausts an evidence-backed retired frontier" - ), -} - - -def _successor( - after_action: str, - condition: dict[str, object], - target_agent_id: str, - target_role_id: str, - action_kind: str, - text: str, -) -> dict[str, object]: - return { - "after_action": after_action, - "condition": condition, - "target_agent_id": target_agent_id, - "target_role_id": target_role_id, - "task_class": "advancement_task", - "action_kind": action_kind, - "text": text, - "todo_command_template": AUTO_RESEARCH_SUCCESSOR_TODO_COMMAND_TEMPLATE, - } - - -AUTO_RESEARCH_DEFAULT_LANES = ( - ("research-curator", "research-curator", "research_curator", "Frame the question, metric, protected boundary, and first research todo."), - ("hypothesis-proposer", "hypothesis-proposer", "hypothesis_proposer", "Grow the hypothesis frontier and route the next bounded research attempt."), - ("research-executor", "research-executor", "research_executor", "Run dev/holdout evidence for selected hypotheses and append public-safe evidence."), - ("evaluator-promoter", "evaluator-promoter", "evaluator_promoter", "Prune or promote claims, summarize validation, and trigger the next research round."), -) - -AUTO_RESEARCH_ROLE_PROFILE_ORDER = ("research_curator", "hypothesis_proposer", "research_executor", "evaluator_promoter") - -AUTO_RESEARCH_ROLE_PROFILE_ALIASES = { - "research-curator": "research_curator", - "curator": "research_curator", - "hypothesis-proposer": "hypothesis_proposer", - "hypothesis_mapper": "hypothesis_proposer", - "hypothesis-mapper": "hypothesis_proposer", - "mapper": "hypothesis_proposer", - "hypothesis-runner": "hypothesis_proposer", - "research-executor": "research_executor", - "research_executor": "research_executor", - "evidence_runner": "research_executor", - "evidence-runner": "research_executor", - "runner": "research_executor", - "evaluator-promoter": "evaluator_promoter", - "evaluator_promoter": "evaluator_promoter", - "evidence_verifier": "evaluator_promoter", - "evidence-verifier": "evaluator_promoter", - "verifier": "evaluator_promoter", - "evidence-promoter": "evaluator_promoter", -} - -AUTO_RESEARCH_ROLE_PROFILES: dict[str, dict[str, object]] = { - "research_curator": { - "phase": "contract", - "allowed_actions": [ - "write_research_contract", - "review_research_contract", - "review_terminal_decision", - ], - "write_scope": [ - "auto_research_delivery_contract_v0", - "research_contract_v0", - "auto_research_peer_review_v0", - "todo_item_v0", - ], - "handoff": ["Create the first hypothesis-proposer todo."], - "successor_todos": [ - _successor("review_research_contract", AUTO_RESEARCH_NEXT_HYPOTHESIS_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "propose_hypothesis", AUTO_RESEARCH_REFINED_HYPOTHESIS_SUCCESSOR_TEXT), - ], - }, - "hypothesis_proposer": { - "phase": "hypothesis", - "allowed_actions": [ - "propose_hypothesis", - "propose_failure_successor", - "review_hypothesis_frontier", - ], - "write_scope": ["research_hypothesis_v0", "todo_item_v0"], - "handoff": ["Create or unblock a research-executor todo."], - "successor_todos": [ - _successor("propose_hypothesis", AUTO_RESEARCH_REFINED_DEV_SUCCESSOR_CONDITION, "research-executor", "research_executor", "run_dev_eval", AUTO_RESEARCH_REFINED_DEV_SUCCESSOR_TEXT), - _successor("propose_hypothesis", AUTO_RESEARCH_REFINED_DEV_SUCCESSOR_CONDITION, "evaluator-promoter", "evaluator_promoter", "review_promotion_readiness", AUTO_RESEARCH_PROMOTION_READINESS_REVIEW_SUCCESSOR_TEXT), - ], - }, - "research_executor": { - "phase": "evidence", - "allowed_actions": [ - "run_dev_eval", - "run_holdout_eval", - "remediate_data_measurement", - ], - "write_scope": ["auto_research_evidence_packet_v0", "rollout_event_log"], - "handoff": [ - "Append scored evidence, complete the selected todo, and create the declared successor todo when the role profile says another split is due." - ], - "successor_todos": [ - _successor("run_dev_eval", AUTO_RESEARCH_HOLDOUT_SUCCESSOR_CONDITION, "research-executor", "research_executor", "run_holdout_eval", AUTO_RESEARCH_HOLDOUT_SUCCESSOR_TEXT), - _successor("run_holdout_eval", AUTO_RESEARCH_VALIDATED_SUMMARY_SUCCESSOR_CONDITION, "evaluator-promoter", "evaluator_promoter", "write_evaluation_summary", AUTO_RESEARCH_VALIDATED_SUMMARY_SUCCESSOR_TEXT), - ], - }, - "evaluator_promoter": { - "phase": "verify", - "allowed_actions": [ - "summarize_evidence", - "write_evaluation_summary", - "classify_evidence", - "review_promotion_readiness", - "record_terminal_decision", - ], - "write_scope": [ - "research_evidence_graph_v0", - "auto_research_terminal_decision_v0", - "todo_item_v0", - ], - "handoff": ["Add a role-declared successor todo when evidence needs another bounded split."], - "successor_todos": [ - _successor("summarize_evidence", AUTO_RESEARCH_HOLDOUT_SUCCESSOR_CONDITION, "research-curator", "research_curator", "review_research_contract", AUTO_RESEARCH_CURATOR_REVIEW_SUCCESSOR_TEXT), - _successor("summarize_evidence", AUTO_RESEARCH_HOLDOUT_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "review_hypothesis_frontier", AUTO_RESEARCH_HYPOTHESIS_FRONTIER_REVIEW_SUCCESSOR_TEXT), - _successor("summarize_evidence", AUTO_RESEARCH_NEXT_HYPOTHESIS_SUCCESSOR_CONDITION, "research-curator", "research_curator", "review_research_contract", AUTO_RESEARCH_CURATOR_REVIEW_SUCCESSOR_TEXT), - _successor("write_evaluation_summary", AUTO_RESEARCH_NEXT_HYPOTHESIS_SUCCESSOR_CONDITION, "research-curator", "research_curator", "review_research_contract", AUTO_RESEARCH_CURATOR_REVIEW_SUCCESSOR_TEXT), - _successor("review_promotion_readiness", AUTO_RESEARCH_NEXT_HYPOTHESIS_SUCCESSOR_CONDITION, "research-curator", "research_curator", "review_research_contract", AUTO_RESEARCH_CURATOR_REVIEW_SUCCESSOR_TEXT), - _successor("review_promotion_readiness", AUTO_RESEARCH_HOLDOUT_SUCCESSOR_CONDITION, "research-curator", "research_curator", "review_research_contract", AUTO_RESEARCH_CURATOR_REVIEW_SUCCESSOR_TEXT), - _successor("review_promotion_readiness", AUTO_RESEARCH_HOLDOUT_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "review_hypothesis_frontier", AUTO_RESEARCH_HYPOTHESIS_FRONTIER_REVIEW_SUCCESSOR_TEXT), - _successor("classify_evidence", AUTO_RESEARCH_FAILURE_REMEDIATION_SUCCESSOR_CONDITION, "research-executor", "research_executor", "remediate_data_measurement", AUTO_RESEARCH_DATA_MEASUREMENT_REMEDIATION_SUCCESSOR_TEXT), - _successor("summarize_evidence", AUTO_RESEARCH_FAILURE_REMEDIATION_SUCCESSOR_CONDITION, "research-executor", "research_executor", "remediate_data_measurement", AUTO_RESEARCH_DATA_MEASUREMENT_REMEDIATION_SUCCESSOR_TEXT), - _successor("write_evaluation_summary", AUTO_RESEARCH_FAILURE_REMEDIATION_SUCCESSOR_CONDITION, "research-executor", "research_executor", "remediate_data_measurement", AUTO_RESEARCH_DATA_MEASUREMENT_REMEDIATION_SUCCESSOR_TEXT), - _successor("review_promotion_readiness", AUTO_RESEARCH_FAILURE_REMEDIATION_SUCCESSOR_CONDITION, "research-executor", "research_executor", "remediate_data_measurement", AUTO_RESEARCH_DATA_MEASUREMENT_REMEDIATION_SUCCESSOR_TEXT), - _successor("classify_evidence", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "propose_failure_successor", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_TEXT), - _successor("summarize_evidence", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "propose_failure_successor", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_TEXT), - _successor("write_evaluation_summary", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "propose_failure_successor", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_TEXT), - _successor("review_promotion_readiness", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_CONDITION, "hypothesis-proposer", "hypothesis_proposer", "propose_failure_successor", AUTO_RESEARCH_FAILURE_PROPOSER_SUCCESSOR_TEXT), - ], - }, -} - -AUTO_RESEARCH_ACTION_ROLE_IDS = { - "write_research_contract": "research_curator", - "review_research_contract": "research_curator", - "propose_hypothesis": "hypothesis_proposer", - "propose_failure_successor": "hypothesis_proposer", - "review_hypothesis_frontier": "hypothesis_proposer", - "run_dev_eval": "research_executor", - "run_holdout_eval": "research_executor", - "remediate_data_measurement": "research_executor", - "write_evidence": "research_executor", - "classify_evidence": "evaluator_promoter", - "summarize_evidence": "evaluator_promoter", - "write_evaluation_summary": "evaluator_promoter", - "review_promotion_readiness": "evaluator_promoter", - "record_terminal_decision": "evaluator_promoter", - "review_terminal_decision": "research_curator", -} - -AUTO_RESEARCH_SEED_TITLES = { - "write_research_contract": "Write the public-safe research contract for the shared demo hypothesis.", - "propose_hypothesis": "Map the first shared idea into a todo-backed research hypothesis.", - "propose_failure_successor": "Propose one changed-mechanism successor from retired evidence or explicitly exhaust the research frontier.", - "claim_attempt": "Claim one visible attempt boundary for the selected hypothesis.", - "run_dev_eval": "Run the selected hypothesis on the dev split, write public-safe evidence, append it, and capture live evidence.", - "run_holdout_eval": "Run held-out validation for the dev-supported hypothesis, append public-safe evidence, and summarize promotion readiness.", - "remediate_data_measurement": "Repair one declared data or measurement gap without changing the research mechanism or acceptance gates.", - "write_evaluation_summary": "Verify the evidence packet and open the next validation or promotion gate.", - "summarize_evidence": "Summarize dev evidence and hand off holdout validation when the hypothesis is supported.", - "review_research_contract": "Re-check the research contract and protected scope for the next collective round.", - "review_hypothesis_frontier": "Review the hypothesis frontier and record whether another bounded candidate is needed.", - "review_promotion_readiness": "Review promotion readiness and record the current evidence gap.", - "record_terminal_decision": "Record one evidence-revision-bound promoted or retired result after the required gate.", - "review_terminal_decision": "Independently review one terminal decision without rewriting research evidence.", -} - -KNN_DEMO_VISIBLE_FIRST_STEP_COMMON = ( - "Read research_contract.public.json before claiming scope or metric facts.", - "Do not treat $LOOPX_PANE_A2A_TICK output as research evidence.", - ( - "Before claiming progress, leave one role-specific public-safe artifact, " - "todo update, or evidence packet that another pane can use." - ), -) - -KNN_DEMO_VISIBLE_FIRST_STEPS_BY_ROLE = { - "research_curator": ( - ( - "Write a compact contract note: metric direction, editable file, " - "protected files, dev/test gates, and promotion rule." - ), - ( - "If the first hypothesis todo is missing, add a hypothesis-proposer " - "todo for two exact-KNN speedup hypotheses and cite the contract note." - ), - ), - "hypothesis_proposer": ( - ( - "Inspect solution.py, task.py, and eval.py; produce a two-row " - "hypothesis table with mechanism, expected speed source, correctness " - "risk, and eval command." - ), - ( - "Route exactly one dev-attempt todo to research-executor and keep " - "the rejected/backup hypothesis visible as a retry candidate." - ), - ), - "research_executor": ( - ( - "Run `bash eval.sh dev` before editing and save the last JSON line " - "as baseline dev evidence." - ), - ( - "Edit only solution.py for one exact-KNN optimization, rerun " - "`bash eval.sh dev`, and record the command, score, and diff summary." - ), - ( - "After a dev improvement, run `bash eval.sh test`, save both JSON " - "outputs, and feed the real evaluator JSON to `loopx auto-research evidence`." - ), - ), - "evaluator_promoter": ( - ( - "Only when your own evaluator todo is runnable, read executor evidence " - "and classify it; if no todo is selected, wait for research-executor " - "evidence instead of closing the evaluator todo." - ), - ( - "Write a verdict with split label, metric direction, protected-scope " - "cleanliness, and next successor/retry todo; do not promote without " - "`bash eval.sh test` output." - ), - ), -} - - -def auto_research_role_id(raw_role: str, *, index: int) -> str: - raw = raw_role.strip() - role_id = AUTO_RESEARCH_ROLE_PROFILE_ALIASES.get(raw, raw) - if role_id in AUTO_RESEARCH_ROLE_PROFILES: - return role_id - if raw: - allowed = ", ".join(AUTO_RESEARCH_ROLE_PROFILE_ORDER) - raise ValueError(f"unknown auto-research role_id {raw!r}; expected one of {allowed}") - return AUTO_RESEARCH_ROLE_PROFILE_ORDER[(index - 1) % len(AUTO_RESEARCH_ROLE_PROFILE_ORDER)] - - -def auto_research_role_profile(*, role_id: str, goal_id: str, agent_id: str) -> dict[str, object]: - base = dict(AUTO_RESEARCH_ROLE_PROFILES[role_id]) - return { - "schema_version": AUTO_RESEARCH_ROLE_PROFILE_SCHEMA_VERSION, - "goal_id": goal_id, - "agent_id": agent_id, - "role_id": role_id, - "required_skill": AUTO_RESEARCH_REQUIRED_SKILL, - "worker_skill_source": AUTO_RESEARCH_WORKER_SKILL_SOURCE, - "skill_distribution": "worker_local", - "worker_skill_scope": "role_specific_semantics_and_successor_todos_only", - "default_kernel_skills_owner": "generic_multi_agent_kernel", - "fixed_a2a_wake_prompt_owner": "generic_multi_agent_kernel", - "continuation_policy": dict(AUTO_RESEARCH_CONTINUATION_POLICY), - "stop_conditions": [ - "quota says should_run=false or user_action_required=true", - "frontier has no selected todo for this agent", - "selected action is outside this role profile", - "work would require raw logs, credentials, protected scope, or private material", - ], - **base, - } - - -def auto_research_role_id_for_action(action: str) -> str: - return AUTO_RESEARCH_ACTION_ROLE_IDS.get(action, "") - - -def auto_research_successor_specs_for_action(*, role_id: str, action: str) -> list[dict[str, object]]: - profile = AUTO_RESEARCH_ROLE_PROFILES.get(role_id) or {} - specs = profile.get("successor_todos") if isinstance(profile, dict) else [] - if not isinstance(specs, list): - return [] - return [ - dict(spec) - for spec in specs - if isinstance(spec, dict) and str(spec.get("after_action") or "") == action - ] - - -def auto_research_seed_action_for_role(role_id: str) -> str: - profile = AUTO_RESEARCH_ROLE_PROFILES.get(role_id) or {} - actions = profile.get("allowed_actions") if isinstance(profile, dict) else [] - action = str((actions or ["advance_todo"])[0]) - if role_id == "research_executor": - return "run_dev_eval" - if role_id == "evaluator_promoter": - return "summarize_evidence" - return action - - -def auto_research_seed_title(*, action_kind: str, role_id: str, lane_id: str) -> str: - return AUTO_RESEARCH_SEED_TITLES.get( - action_kind, - f"Run one role-compatible auto-research action for {role_id or lane_id}.", - ) - - -def knn_demo_visible_first_steps(role_id: str) -> list[str]: - """Return compact KNN-demo role hints; execution stays in visible panes.""" - - return [ - *KNN_DEMO_VISIBLE_FIRST_STEP_COMMON, - *KNN_DEMO_VISIBLE_FIRST_STEPS_BY_ROLE.get(role_id, ()), - ] diff --git a/demo/auto_research/rollout_append.py b/demo/auto_research/rollout_append.py deleted file mode 100644 index 73fae7c96a..0000000000 --- a/demo/auto_research/rollout_append.py +++ /dev/null @@ -1,97 +0,0 @@ -from __future__ import annotations - -from collections import Counter -import json -from pathlib import Path -from typing import Mapping - -from .evidence_packet import ( - AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION, - build_auto_research_rollout_events, - load_auto_research_evidence_packet, -) -from loopx.history import load_registry -from loopx.paths import resolve_runtime_root -from loopx.rollout_event_log import ( - append_rollout_event, - load_rollout_events, - rollout_event_log_path, -) - - -def _event_content_key(event: Mapping[str, object]) -> str: - """Identify the durable research content, not this append observation.""" - - content = { - key: value - for key, value in event.items() - if key not in {"event_id", "recorded_at"} - } - return json.dumps(content, sort_keys=True, ensure_ascii=True) - - -def append_auto_research_rollout_events( - *, - packet_path: str, - registry_path: Path, - runtime_root_arg: str | None, - dry_run: bool, -) -> dict[str, object]: - packet = load_auto_research_evidence_packet(packet_path) - goal_id = packet["research_contract"]["goal_id"] - events = build_auto_research_rollout_events(packet) - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - log_path = rollout_event_log_path(runtime_root, goal_id) - existing_events = load_rollout_events(log_path) - existing_ids = { - str(event.get("event_id")) for event in existing_events if event.get("event_id") - } - existing_content_ids = { - _event_content_key(event): str(event["event_id"]) - for event in existing_events - if event.get("event_id") - } - appended_ids: list[str] = [] - skipped_ids: list[str] = [] - effective_ids: list[str] = [] - for event in events: - event_id = str(event["event_id"]) - content_key = _event_content_key(event) - existing_content_id = existing_content_ids.get(content_key) - if existing_content_id: - skipped_ids.append(existing_content_id) - effective_ids.append(existing_content_id) - continue - if event_id in existing_ids: - skipped_ids.append(event_id) - effective_ids.append(event_id) - continue - if not dry_run: - append_rollout_event(log_path, event) - existing_ids.add(event_id) - existing_content_ids[content_key] = event_id - appended_ids.append(event_id) - effective_ids.append(event_id) - counts_by_kind = Counter(str(event.get("event_kind") or "") for event in events) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_ROLLOUT_APPEND_SCHEMA_VERSION, - "goal_id": goal_id, - "dry_run": dry_run, - "event_count": len(events), - "appended_count": 0 if dry_run else len(appended_ids), - "would_append_count": len(appended_ids), - "skipped_existing_count": len(skipped_ids), - "event_ids": effective_ids, - "appended_event_ids": [] if dry_run else appended_ids, - "skipped_existing_event_ids": skipped_ids, - "counts_by_kind": dict(sorted(counts_by_kind.items())), - "packet_summary": packet["summary"], - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "source": "loopx_rollout_event_log", - }, - } diff --git a/demo/auto_research/terminal_result_contract.py b/demo/auto_research/terminal_result_contract.py deleted file mode 100644 index f7fe717b9b..0000000000 --- a/demo/auto_research/terminal_result_contract.py +++ /dev/null @@ -1,422 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -from typing import Any, Mapping, Sequence - -from .evidence_packet import ( - _compact_public_text_list, - _compact_public_token, -) -from .research_state import build_research_decision_candidates -from loopx.rollout_event_log import build_rollout_event - - -AUTO_RESEARCH_TERMINAL_DECISION_SCHEMA_VERSION = ( - "auto_research_terminal_decision_v0" -) -AUTO_RESEARCH_PEER_REVIEW_SCHEMA_VERSION = "auto_research_peer_review_v0" -AUTO_RESEARCH_TERMINAL_RESULT_QUERY_SCHEMA_VERSION = ( - "auto_research_terminal_result_query_v0" -) - -TERMINAL_OUTCOMES = {"promoted", "retired"} -RETIREMENT_REASONS = { - "contradicted", - "duplicate", - "operator_rejected", - "retry_exhausted", - "superseded", -} -PEER_REVIEW_VERDICTS = {"approve", "needs_more_evidence", "reject"} - - -def _canonical_digest(value: Mapping[str, Any]) -> str: - encoded = json.dumps( - value, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - return hashlib.sha256(encoded).hexdigest() - - -def research_evidence_graph_revision(evidence_graph: Mapping[str, Any]) -> str: - return f"sha256:{_canonical_digest(dict(evidence_graph))}" - - -def hypothesis_evidence_revision( - evidence_graph: Mapping[str, Any], - hypothesis_id: str, -) -> str: - node = _hypothesis_node(evidence_graph, hypothesis_id) - scoped_evidence = { - "goal_id": str(evidence_graph.get("goal_id") or ""), - "metric": dict(evidence_graph.get("metric") or {}), - "node": node, - } - return f"sha256:{_canonical_digest(scoped_evidence)}" - - -def _stable_ref(prefix: str, values: Sequence[object]) -> str: - encoded = json.dumps( - [str(value or "") for value in values], - ensure_ascii=True, - separators=(",", ":"), - ).encode("utf-8") - return f"{prefix}_{hashlib.sha256(encoded).hexdigest()[:16]}" - - -def _event_classification(event: Mapping[str, Any]) -> str: - return str(event.get("classification") or "") - - -def _event_details(event: Mapping[str, Any]) -> dict[str, Any]: - details = event.get("details") - return dict(details) if isinstance(details, Mapping) else {} - - -def _event_decision_ref(event: Mapping[str, Any]) -> str: - causality = event.get("causality") - if not isinstance(causality, Mapping): - return "" - return str(causality.get("decision_id") or "") - - -def _hypothesis_node( - evidence_graph: Mapping[str, Any], - hypothesis_id: str, -) -> dict[str, Any]: - matches = [ - dict(node) - for node in evidence_graph.get("nodes") or [] - if isinstance(node, Mapping) - and str(node.get("hypothesis_id") or "") == hypothesis_id - ] - if len(matches) != 1: - raise ValueError( - f"hypothesis_id must resolve to exactly one evidence graph node: {hypothesis_id}" - ) - return matches[0] - - -def _terminal_decision_events( - rollout_events: Sequence[Mapping[str, Any]], -) -> list[dict[str, Any]]: - return [ - dict(event) - for event in rollout_events - if _event_classification(event) - == AUTO_RESEARCH_TERMINAL_DECISION_SCHEMA_VERSION - ] - - -def _peer_review_events( - rollout_events: Sequence[Mapping[str, Any]], -) -> list[dict[str, Any]]: - return [ - dict(event) - for event in rollout_events - if _event_classification(event) == AUTO_RESEARCH_PEER_REVIEW_SCHEMA_VERSION - ] - - -def _decision_view(event: Mapping[str, Any]) -> dict[str, Any]: - details = _event_details(event) - return { - "decision_ref": _event_decision_ref(event), - "event_id": str(event.get("event_id") or ""), - "recorded_at": str(event.get("recorded_at") or ""), - "goal_id": str(event.get("goal_id") or ""), - "hypothesis_id": str(details.get("hypothesis_id") or ""), - "outcome": str(details.get("outcome") or ""), - "reason": str(details.get("reason") or ""), - "decided_by": str(event.get("agent_id") or ""), - "evidence_graph_revision": str( - details.get("evidence_graph_revision") or "" - ), - "hypothesis_evidence_revision": str( - details.get("hypothesis_evidence_revision") or "" - ), - "evidence_refs": list(event.get("artifact_refs") or []), - } - - -def _review_view( - event: Mapping[str, Any], - *, - registered_agent_ids: Sequence[str], -) -> dict[str, Any]: - details = _event_details(event) - reviewer_agent_id = str(event.get("agent_id") or "") - decision_agent_id = str(details.get("decision_agent_id") or "") - producer_agent_id = str(details.get("producer_agent_id") or "") - registered = set(registered_agent_ids) - independently_separated = reviewer_agent_id not in { - decision_agent_id, - producer_agent_id, - } - return { - "review_ref": _event_decision_ref(event), - "event_id": str(event.get("event_id") or ""), - "recorded_at": str(event.get("recorded_at") or ""), - "goal_id": str(event.get("goal_id") or ""), - "hypothesis_id": str(details.get("hypothesis_id") or ""), - "decision_ref": str(details.get("decision_ref") or ""), - "verdict": str(details.get("verdict") or ""), - "reviewer_agent_id": reviewer_agent_id, - "decision_agent_id": decision_agent_id, - "producer_agent_id": producer_agent_id, - "independent": bool(details.get("independent")) - and reviewer_agent_id in registered - and independently_separated, - "evidence_graph_revision": str( - details.get("evidence_graph_revision") or "" - ), - "hypothesis_evidence_revision": str( - details.get("hypothesis_evidence_revision") or "" - ), - "evidence_refs": list(event.get("artifact_refs") or []), - } - - -def _validate_terminal_decision( - *, - evidence_graph: Mapping[str, Any], - hypothesis_id: str, - outcome: str, - reason: str, -) -> dict[str, Any]: - node = _hypothesis_node(evidence_graph, hypothesis_id) - normalized_outcome = _compact_public_token(outcome, field="outcome") - if normalized_outcome not in TERMINAL_OUTCOMES: - raise ValueError( - "outcome must be one of: " + ", ".join(sorted(TERMINAL_OUTCOMES)) - ) - normalized_reason = _compact_public_token(reason, field="reason") - decisions = build_research_decision_candidates(dict(evidence_graph)) - validated_ids = { - str(item.get("hypothesis_id") or "") - for item in decisions.get("validated_promotion_candidates") or [] - } - retirement_ids = { - str(item.get("hypothesis_id") or "") - for item in decisions.get("retirement_candidates") or [] - } - if normalized_outcome == "promoted": - if normalized_reason != "holdout_validated": - raise ValueError("promoted decisions require reason=holdout_validated") - if hypothesis_id not in validated_ids: - raise ValueError( - "promoted decisions require a validated holdout promotion candidate" - ) - else: - if normalized_reason not in RETIREMENT_REASONS: - raise ValueError( - "retired decision reason must be one of: " - + ", ".join(sorted(RETIREMENT_REASONS)) - ) - if normalized_reason == "contradicted" and hypothesis_id not in retirement_ids: - raise ValueError( - "retired reason=contradicted requires negative or guardrail evidence" - ) - return node - - -def build_terminal_decision_event( - *, - evidence_graph: Mapping[str, Any], - hypothesis_id: str, - outcome: str, - reason: str, - decided_by: str, - evidence_refs: Sequence[str] | None = None, - recorded_at: str | None = None, -) -> dict[str, Any]: - hypothesis = _compact_public_token(hypothesis_id, field="hypothesis_id") - decider = _compact_public_token(decided_by, field="decided_by") - node = _validate_terminal_decision( - evidence_graph=evidence_graph, - hypothesis_id=hypothesis, - outcome=outcome, - reason=reason, - ) - normalized_outcome = _compact_public_token(outcome, field="outcome") - normalized_reason = _compact_public_token(reason, field="reason") - graph_revision = research_evidence_graph_revision(evidence_graph) - hypothesis_revision = hypothesis_evidence_revision( - evidence_graph, - hypothesis, - ) - refs = _compact_public_text_list(evidence_refs, field="evidence_refs") - decision_ref = _stable_ref( - "research_decision", - [ - evidence_graph.get("goal_id"), - hypothesis, - hypothesis_revision, - ], - ) - return build_rollout_event( - goal_id=str(evidence_graph.get("goal_id") or ""), - event_kind="validation", - agent_id=decider, - todo_id=str(node.get("todo_id") or ""), - agent_role="evaluator_promoter", - decision_id=decision_ref, - from_state=str(node.get("status") or ""), - to_state=normalized_outcome, - status=normalized_outcome, - classification=AUTO_RESEARCH_TERMINAL_DECISION_SCHEMA_VERSION, - labels=[ - "auto_research", - "terminal_decision", - normalized_outcome, - normalized_reason, - ], - summary=( - f"Auto Research terminal decision {hypothesis}: " - f"{normalized_outcome} ({normalized_reason})." - ), - artifact_refs=refs, - details={ - "hypothesis_id": hypothesis, - "outcome": normalized_outcome, - "reason": normalized_reason, - "evidence_graph_revision": graph_revision, - "hypothesis_evidence_revision": hypothesis_revision, - }, - recorded_at=recorded_at, - ) - - -def _current_terminal_decision( - *, - rollout_events: Sequence[Mapping[str, Any]], - hypothesis_id: str, - hypothesis_evidence_revision: str, -) -> dict[str, Any]: - current = [ - _decision_view(event) - for event in _terminal_decision_events(rollout_events) - if str(_event_details(event).get("hypothesis_id") or "") == hypothesis_id - and str( - _event_details(event).get("hypothesis_evidence_revision") or "" - ) - == hypothesis_evidence_revision - ] - signatures = { - ( - item["decision_ref"], - item["outcome"], - item["reason"], - item["decided_by"], - ) - for item in current - } - if len(signatures) > 1: - raise ValueError( - f"conflicting terminal decisions for current evidence revision: {hypothesis_id}" - ) - if not current: - raise ValueError( - f"no terminal decision exists for current evidence revision: {hypothesis_id}" - ) - return current[-1] - - -def build_peer_review_event( - *, - evidence_graph: Mapping[str, Any], - rollout_events: Sequence[Mapping[str, Any]], - hypothesis_id: str, - reviewer_agent_id: str, - verdict: str, - evidence_refs: Sequence[str] | None = None, - require_independent: bool = False, - registered_agent_ids: Sequence[str] | None = None, - recorded_at: str | None = None, -) -> dict[str, Any]: - hypothesis = _compact_public_token(hypothesis_id, field="hypothesis_id") - reviewer = _compact_public_token( - reviewer_agent_id, - field="reviewer_agent_id", - ) - normalized_verdict = _compact_public_token(verdict, field="verdict") - if normalized_verdict not in PEER_REVIEW_VERDICTS: - raise ValueError( - "verdict must be one of: " - + ", ".join(sorted(PEER_REVIEW_VERDICTS)) - ) - registered = { - _compact_public_token(agent_id, field="registered_agent_ids[]") - for agent_id in registered_agent_ids or [] - } - if reviewer not in registered: - raise ValueError( - f"reviewer_agent_id is not registered for this research goal: {reviewer}" - ) - node = _hypothesis_node(evidence_graph, hypothesis) - graph_revision = research_evidence_graph_revision(evidence_graph) - hypothesis_revision = hypothesis_evidence_revision( - evidence_graph, - hypothesis, - ) - decision = _current_terminal_decision( - rollout_events=rollout_events, - hypothesis_id=hypothesis, - hypothesis_evidence_revision=hypothesis_revision, - ) - producer = str(node.get("claimed_by") or "") - decision_agent = str(decision.get("decided_by") or "") - independent = reviewer not in {producer, decision_agent} - if require_independent and not independent: - raise ValueError( - "independent peer review requires a reviewer distinct from both " - "the hypothesis producer and terminal decision agent" - ) - refs = _compact_public_text_list(evidence_refs, field="evidence_refs") - review_ref = _stable_ref( - "research_review", - [ - evidence_graph.get("goal_id"), - hypothesis, - decision["decision_ref"], - hypothesis_revision, - reviewer, - normalized_verdict, - ], - ) - return build_rollout_event( - goal_id=str(evidence_graph.get("goal_id") or ""), - event_kind="validation", - agent_id=reviewer, - todo_id=str(node.get("todo_id") or ""), - agent_role="evaluator_promoter", - decision_id=review_ref, - source_event_id=str(decision.get("event_id") or ""), - status=normalized_verdict, - classification=AUTO_RESEARCH_PEER_REVIEW_SCHEMA_VERSION, - labels=[ - "auto_research", - "peer_review", - normalized_verdict, - "independent" if independent else "self_review", - ], - summary=( - f"Auto Research review {hypothesis}: {normalized_verdict}; " - f"independent={str(independent).lower()}." - ), - artifact_refs=refs, - details={ - "hypothesis_id": hypothesis, - "decision_ref": decision["decision_ref"], - "verdict": normalized_verdict, - "decision_agent_id": decision_agent, - "producer_agent_id": producer, - "independent": independent, - "evidence_graph_revision": graph_revision, - "hypothesis_evidence_revision": hypothesis_revision, - }, - recorded_at=recorded_at, - ) diff --git a/demo/auto_research/terminal_result_projection.py b/demo/auto_research/terminal_result_projection.py deleted file mode 100644 index a239402abe..0000000000 --- a/demo/auto_research/terminal_result_projection.py +++ /dev/null @@ -1,171 +0,0 @@ -from __future__ import annotations - -import hashlib -from datetime import datetime, timezone -from typing import Any, Mapping - -from loopx.capabilities.explore.result_log import ( - build_explore_edge_event, - build_explore_finding_event, - build_explore_node_event, -) - - -def _now_iso() -> str: - return ( - datetime.now(timezone.utc) - .replace(microsecond=0) - .isoformat() - .replace("+00:00", "Z") - ) - - -def explore_result_ids(hypothesis_id: str) -> tuple[str, str]: - suffix = hashlib.sha256(hypothesis_id.encode("utf-8")).hexdigest()[:16] - return f"research_{suffix}", f"finding_{suffix}" - - -def _projection_timestamp(record: Mapping[str, Any]) -> str: - decision = record.get("terminal_decision") - review = record.get("peer_review") - review_rows = review.get("reviews") if isinstance(review, Mapping) else [] - if isinstance(review_rows, list) and review_rows: - return str(review_rows[-1].get("recorded_at") or _now_iso()) - if isinstance(decision, Mapping): - return str(decision.get("recorded_at") or _now_iso()) - history = record.get("decision_history") - if isinstance(history, list) and history: - return str(history[-1].get("recorded_at") or _now_iso()) - return "1970-01-01T00:00:00Z" - - -def build_terminal_result_explore_events( - *, - query: Mapping[str, Any], -) -> list[dict[str, Any]]: - goal_id = str(query.get("goal_id") or "") - node_ids = { - str(record.get("hypothesis_id") or ""): explore_result_ids( - str(record.get("hypothesis_id") or "") - )[0] - for record in query.get("results") or [] - if isinstance(record, Mapping) - and ( - record.get("finding_status") - or record.get("decision_state") != "pending" - ) - } - events: list[dict[str, Any]] = [] - for raw_record in query.get("results") or []: - if not isinstance(raw_record, Mapping): - continue - record = dict(raw_record) - if ( - record["decision_state"] == "pending" - and not record.get("finding_status") - ): - continue - hypothesis_id = str(record["hypothesis_id"]) - node_id, finding_id = explore_result_ids(hypothesis_id) - decision = record.get("terminal_decision") - review = record.get("peer_review") - timestamp = _projection_timestamp(record) - if record["decision_state"] == "conflict": - node_status = "blocked" - blocked_reason = "Conflicting terminal decisions exist for the current evidence revision." - elif record["decision_state"] == "stale": - node_status = "exploring" - blocked_reason = None - elif isinstance(decision, Mapping) and decision.get("outcome") == "promoted": - node_status = "resolved" - blocked_reason = None - elif record["decision_state"] == "pending": - node_status = "exploring" - blocked_reason = None - else: - node_status = "dead_end" - blocked_reason = None - evidence_refs = [] - if isinstance(decision, Mapping): - evidence_refs.extend(decision.get("evidence_refs") or []) - if isinstance(review, Mapping): - for row in review.get("reviews") or []: - if isinstance(row, Mapping): - evidence_refs.extend(row.get("evidence_refs") or []) - events.append( - build_explore_node_event( - goal_id=goal_id, - node_id=node_id, - node_kind="hypothesis", - title=str(record.get("hypothesis") or hypothesis_id), - status=node_status, - summary=( - f"Auto Research hypothesis {hypothesis_id}; " - f"decision_state={record['decision_state']}; " - f"review_state={review.get('state') if isinstance(review, Mapping) else 'not_applicable'}." - ), - blocked_reason=blocked_reason, - parent_id=node_ids.get(str(record.get("parent_hypothesis_id") or "")), - agent_id=str(record.get("producer_agent_id") or ""), - evidence_refs=sorted(set(evidence_refs)), - tags=[ - "auto_research", - str(record.get("mechanism_family") or "research"), - record["decision_state"], - ], - recorded_at=timestamp, - ) - ) - parent = str(record.get("parent_hypothesis_id") or "") - if parent and parent in node_ids: - events.append( - build_explore_edge_event( - goal_id=goal_id, - from_node=node_ids[parent], - to_node=node_id, - edge_type="leads_to", - summary="A later Auto Research hypothesis extends the earlier research path.", - recorded_at=timestamp, - ) - ) - finding_status = record.get("finding_status") - if not finding_status: - continue - outcome = ( - str(decision.get("outcome") or "") - if isinstance(decision, Mapping) - else "stale" - ) - reason = ( - str(decision.get("reason") or "") - if isinstance(decision, Mapping) - else "evidence_revision_changed" - ) - events.append( - build_explore_finding_event( - goal_id=goal_id, - finding_id=finding_id, - node_id=node_id, - title=str(record.get("hypothesis") or hypothesis_id), - status=str(finding_status), - summary=( - f"Terminal outcome={outcome}; reason={reason}; " - f"review_state={review.get('state') if isinstance(review, Mapping) else 'not_applicable'}; " - f"review_verdict={review.get('verdict') if isinstance(review, Mapping) else None}." - ), - agent_id=str( - decision.get("decided_by") - if isinstance(decision, Mapping) - else record.get("producer_agent_id") - ), - evidence_refs=sorted(set(evidence_refs)), - tags=[ - "auto_research", - hypothesis_id, - outcome, - str(finding_status), - ], - recorded_at=timestamp, - ) - ) - return events diff --git a/demo/auto_research/terminal_result_query.py b/demo/auto_research/terminal_result_query.py deleted file mode 100644 index ce0b9f99ad..0000000000 --- a/demo/auto_research/terminal_result_query.py +++ /dev/null @@ -1,233 +0,0 @@ -from __future__ import annotations - -from typing import Any, Mapping, Sequence - -from .evidence_packet import _compact_public_token -from .terminal_result_contract import ( - AUTO_RESEARCH_TERMINAL_RESULT_QUERY_SCHEMA_VERSION, - _decision_view, - _event_details, - _peer_review_events, - _review_view, - _terminal_decision_events, - hypothesis_evidence_revision, - research_evidence_graph_revision, -) - - -def _review_resolution( - *, - reviews: Sequence[Mapping[str, Any]], - decision_ref: str, - hypothesis_revision: str, - registered_agent_ids: Sequence[str], -) -> dict[str, Any]: - matching = [ - _review_view( - review, - registered_agent_ids=registered_agent_ids, - ) - for review in reviews - if str(_event_details(review).get("decision_ref") or "") == decision_ref - and str( - _event_details(review).get("hypothesis_evidence_revision") or "" - ) - == hypothesis_revision - ] - independent = [item for item in matching if item["independent"]] - verdicts = {item["verdict"] for item in independent} - if len(verdicts) > 1: - return { - "state": "conflict", - "independent": True, - "verdict": None, - "reviews": matching, - } - if independent: - return { - "state": "independent_reviewed", - "independent": True, - "verdict": independent[-1]["verdict"], - "reviews": matching, - } - if matching: - return { - "state": "self_review_only", - "independent": False, - "verdict": matching[-1]["verdict"], - "reviews": matching, - } - return { - "state": "unreviewed", - "independent": False, - "verdict": None, - "reviews": [], - } - - -def _result_record( - *, - evidence_graph: Mapping[str, Any], - node: Mapping[str, Any], - graph_revision: str, - decisions: Sequence[Mapping[str, Any]], - reviews: Sequence[Mapping[str, Any]], - registered_agent_ids: Sequence[str], -) -> dict[str, Any]: - hypothesis_id = str(node.get("hypothesis_id") or "") - hypothesis_revision = hypothesis_evidence_revision( - evidence_graph, - hypothesis_id, - ) - history = [ - _decision_view(event) - for event in decisions - if str(_event_details(event).get("hypothesis_id") or "") == hypothesis_id - ] - current = [ - item - for item in history - if item["hypothesis_evidence_revision"] == hypothesis_revision - ] - current_signatures = { - ( - item["decision_ref"], - item["outcome"], - item["reason"], - item["decided_by"], - ) - for item in current - } - conflict = len(current_signatures) > 1 - decision = current[-1] if len(current_signatures) == 1 else None - stale = bool(history) and not decision and not conflict - prior_assertion = any( - item["outcome"] == "promoted" - or ( - item["outcome"] == "retired" - and item["reason"] == "contradicted" - ) - for item in history - if item["hypothesis_evidence_revision"] != hypothesis_revision - ) - review = ( - _review_resolution( - reviews=reviews, - decision_ref=decision["decision_ref"], - hypothesis_revision=hypothesis_revision, - registered_agent_ids=registered_agent_ids, - ) - if decision - else { - "state": "not_applicable", - "independent": False, - "verdict": None, - "reviews": [], - } - ) - finding_status = None - if conflict or stale: - finding_status = "tentative" - elif not decision and str(node.get("status") or "") == "supported": - finding_status = "tentative" - elif decision and decision["outcome"] == "promoted": - finding_status = ( - "confirmed" - if review["state"] == "independent_reviewed" - and review["verdict"] == "approve" - else "tentative" - ) - elif ( - decision - and decision["outcome"] == "retired" - and decision["reason"] == "contradicted" - ): - finding_status = ( - "refuted" - if review["state"] == "independent_reviewed" - and review["verdict"] == "approve" - else "tentative" - ) - elif decision and decision["outcome"] == "retired" and prior_assertion: - finding_status = "tentative" - return { - "hypothesis_id": hypothesis_id, - "parent_hypothesis_id": str(node.get("parent_hypothesis_id") or ""), - "todo_id": str(node.get("todo_id") or ""), - "producer_agent_id": str(node.get("claimed_by") or ""), - "mechanism_family": str(node.get("mechanism_family") or ""), - "hypothesis": str(node.get("hypothesis") or hypothesis_id), - "research_status": str(node.get("status") or ""), - "evidence_graph_revision": graph_revision, - "hypothesis_evidence_revision": hypothesis_revision, - "decision_state": ( - "conflict" - if conflict - else "current" - if decision - else "stale" - if stale - else "pending" - ), - "terminal_decision": decision, - "peer_review": review, - "finding_status": finding_status, - "decision_history": history, - } - - -def build_terminal_result_query( - *, - evidence_graph: Mapping[str, Any], - rollout_events: Sequence[Mapping[str, Any]], - registered_agent_ids: Sequence[str], - hypothesis_id: str | None = None, - include_history: bool = False, -) -> dict[str, Any]: - graph_revision = research_evidence_graph_revision(evidence_graph) - decisions = _terminal_decision_events(rollout_events) - reviews = _peer_review_events(rollout_events) - requested = ( - _compact_public_token(hypothesis_id, field="hypothesis_id") - if hypothesis_id - else None - ) - records = [ - _result_record( - evidence_graph=evidence_graph, - node=node, - graph_revision=graph_revision, - decisions=decisions, - reviews=reviews, - registered_agent_ids=registered_agent_ids, - ) - for node in evidence_graph.get("nodes") or [] - if isinstance(node, Mapping) - and (requested is None or str(node.get("hypothesis_id") or "") == requested) - ] - if requested and not records: - raise ValueError(f"unknown hypothesis_id: {requested}") - if not include_history: - records = [ - { - key: value - for key, value in record.items() - if key != "decision_history" - } - for record in records - ] - return { - "ok": True, - "schema_version": AUTO_RESEARCH_TERMINAL_RESULT_QUERY_SCHEMA_VERSION, - "goal_id": str(evidence_graph.get("goal_id") or ""), - "evidence_graph_revision": graph_revision, - "hypothesis_id": requested, - "include_history": include_history, - "result_count": len(records), - "results": records, - "boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - }, - } diff --git a/demo/auto_research/terminal_results.py b/demo/auto_research/terminal_results.py deleted file mode 100644 index 411ab4fe75..0000000000 --- a/demo/auto_research/terminal_results.py +++ /dev/null @@ -1,349 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from typing import Any, Mapping, Sequence - -from .research_state import build_research_evidence_graph_from_rollout_events -from .terminal_result_contract import ( - AUTO_RESEARCH_PEER_REVIEW_SCHEMA_VERSION, - AUTO_RESEARCH_TERMINAL_DECISION_SCHEMA_VERSION, - _decision_view, - _event_decision_ref, - _review_view, - build_peer_review_event, - build_terminal_decision_event, -) -from .terminal_result_projection import ( - build_terminal_result_explore_events, - explore_result_ids, -) -from .terminal_result_query import build_terminal_result_query -from loopx.capabilities.explore.result_log import ( - append_explore_result_events, - build_explore_result_projection, - explore_result_log_path, - load_explore_result_events_strict, -) -from loopx.agent_registry import ( - registered_agent_ids_from_registry, - require_registered_agent_id, -) -from loopx.history import load_registry -from loopx.paths import resolve_runtime_root -from loopx.rollout_event_log import ( - append_rollout_event, - load_rollout_events, - rollout_event_log_path, -) - - -AUTO_RESEARCH_TERMINAL_RESULT_PROJECTION_SCHEMA_VERSION = ( - "auto_research_terminal_result_projection_v0" -) - - -def _append_validation_event_once( - *, - log_path: Path, - event: Mapping[str, Any], - rollout_events: Sequence[Mapping[str, Any]], - dry_run: bool, -) -> dict[str, Any]: - decision_ref = _event_decision_ref(event) - matching = [ - dict(existing) - for existing in rollout_events - if _event_decision_ref(existing) == decision_ref - ] - if len(matching) > 1: - raise ValueError(f"duplicate validation decision ref: {decision_ref}") - if matching: - existing = matching[0] - comparable_existing = _validation_event_semantics(existing) - comparable_new = _validation_event_semantics(event) - if comparable_existing != comparable_new: - raise ValueError(f"conflicting validation decision ref: {decision_ref}") - return {"appended": False, "reused": True, "event": existing} - if not dry_run: - append_rollout_event(log_path, event) - return { - "appended": not dry_run, - "reused": False, - "event": dict(event), - } - - -def _validation_event_semantics(event: Mapping[str, Any]) -> dict[str, Any]: - comparable = { - key: value - for key, value in event.items() - if key not in {"event_id", "recorded_at"} - } - details = comparable.get("details") - if isinstance(details, Mapping): - comparable["details"] = { - key: value - for key, value in details.items() - if key != "evidence_graph_revision" - } - return comparable - - -def _load_runtime_state( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, -) -> tuple[Path, list[dict[str, Any]], dict[str, Any]]: - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - rollout_path = rollout_event_log_path(runtime_root, goal_id) - rollout_events = load_rollout_events(rollout_path) - evidence_graph = build_research_evidence_graph_from_rollout_events( - goal_id=goal_id, - rollout_events=rollout_events, - ) - return runtime_root, rollout_events, evidence_graph - - -def decide_terminal_result( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - hypothesis_id: str, - outcome: str, - reason: str, - decided_by: str, - evidence_refs: Sequence[str] | None, - dry_run: bool, -) -> dict[str, Any]: - runtime_root, rollout_events, evidence_graph = _load_runtime_state( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - ) - decider = require_registered_agent_id( - registry_path=registry_path, - goal_id=goal_id, - agent_id=decided_by, - field="decided_by", - ) - event = build_terminal_decision_event( - evidence_graph=evidence_graph, - hypothesis_id=hypothesis_id, - outcome=outcome, - reason=reason, - decided_by=decider, - evidence_refs=evidence_refs, - ) - write = _append_validation_event_once( - log_path=rollout_event_log_path(runtime_root, goal_id), - event=event, - rollout_events=rollout_events, - dry_run=dry_run, - ) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_TERMINAL_DECISION_SCHEMA_VERSION, - "goal_id": goal_id, - "dry_run": dry_run, - "decision": _decision_view(write["event"]), - "appended": write["appended"], - "reused": write["reused"], - } - - -def review_terminal_result( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - hypothesis_id: str, - reviewer_agent_id: str, - verdict: str, - evidence_refs: Sequence[str] | None, - require_independent: bool, - dry_run: bool, -) -> dict[str, Any]: - runtime_root, rollout_events, evidence_graph = _load_runtime_state( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - ) - reviewer = require_registered_agent_id( - registry_path=registry_path, - goal_id=goal_id, - agent_id=reviewer_agent_id, - field="reviewer_agent_id", - ) - event = build_peer_review_event( - evidence_graph=evidence_graph, - rollout_events=rollout_events, - hypothesis_id=hypothesis_id, - reviewer_agent_id=reviewer, - verdict=verdict, - evidence_refs=evidence_refs, - require_independent=require_independent, - registered_agent_ids=registered_agent_ids_from_registry( - registry_path, - goal_id, - ), - ) - write = _append_validation_event_once( - log_path=rollout_event_log_path(runtime_root, goal_id), - event=event, - rollout_events=rollout_events, - dry_run=dry_run, - ) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_PEER_REVIEW_SCHEMA_VERSION, - "goal_id": goal_id, - "dry_run": dry_run, - "review": _review_view( - write["event"], - registered_agent_ids=registered_agent_ids_from_registry( - registry_path, - goal_id, - ), - ), - "appended": write["appended"], - "reused": write["reused"], - } - - -def query_terminal_results( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - hypothesis_id: str | None, - include_history: bool, -) -> dict[str, Any]: - runtime_root, rollout_events, evidence_graph = _load_runtime_state( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - ) - query = build_terminal_result_query( - evidence_graph=evidence_graph, - rollout_events=rollout_events, - registered_agent_ids=registered_agent_ids_from_registry( - registry_path, - goal_id, - ), - hypothesis_id=hypothesis_id, - include_history=include_history, - ) - explore_events = load_explore_result_events_strict( - explore_result_log_path(runtime_root, goal_id), - goal_id=goal_id, - ) - projection = build_explore_result_projection( - explore_events, - goal_id=goal_id, - finding_limit=max(1, len(explore_events)), - ) - expected_nodes = { - explore_result_ids(str(record.get("hypothesis_id") or ""))[0] - for record in query["results"] - if record.get("finding_status") - or record.get("decision_state") != "pending" - } - projected_nodes = { - str(node.get("node_id") or "") - for node in projection.get("nodes") or [] - } - expected_findings = { - explore_result_ids(str(record.get("hypothesis_id") or ""))[1]: str( - record.get("finding_status") or "" - ) - for record in query["results"] - if record.get("finding_status") - } - projected_findings = { - str(finding.get("finding_id") or ""): str(finding.get("status") or "") - for finding in projection.get("findings") or [] - } - node_current = expected_nodes.issubset(projected_nodes) - finding_current = all( - projected_findings.get(finding_id) == status - for finding_id, status in expected_findings.items() - ) - query["explore_projection"] = { - "state": ( - "current" - if (expected_nodes or expected_findings) and node_current and finding_current - else "not_required" - if not expected_nodes and not expected_findings - else "missing_or_stale" - ), - "expected_node_ids": sorted(expected_nodes), - "projected_node_ids": sorted(expected_nodes & projected_nodes), - "expected_findings": expected_findings, - "projected_findings": { - finding_id: projected_findings.get(finding_id) - for finding_id in sorted(expected_findings) - if finding_id in projected_findings - }, - "counts": projection.get("counts"), - } - return query - - -def project_terminal_results( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - dry_run: bool, -) -> dict[str, Any]: - runtime_root, rollout_events, evidence_graph = _load_runtime_state( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - ) - query = build_terminal_result_query( - evidence_graph=evidence_graph, - rollout_events=rollout_events, - registered_agent_ids=registered_agent_ids_from_registry( - registry_path, - goal_id, - ), - include_history=True, - ) - events = build_terminal_result_explore_events(query=query) - write = ( - { - "requested_event_count": len(events), - "appended_event_count": 0, - "reused_event_count": 0, - } - if dry_run - else append_explore_result_events( - explore_result_log_path(runtime_root, goal_id), - events, - expected_goal_id=goal_id, - ) - ) - projection = build_explore_result_projection( - events, - goal_id=goal_id, - finding_limit=max(1, len(events)), - ) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_TERMINAL_RESULT_PROJECTION_SCHEMA_VERSION, - "goal_id": goal_id, - "dry_run": dry_run, - "evidence_graph_revision": query["evidence_graph_revision"], - "result_count": query["result_count"], - "event_count": len(events), - "would_append_count": len(events), - "appended_count": write["appended_event_count"], - "reused_count": write["reused_event_count"], - "results": query["results"], - "explore_projection": projection, - "boundary": query["boundary"], - } diff --git a/demo/auto_research/user_contract.py b/demo/auto_research/user_contract.py deleted file mode 100644 index 9340191916..0000000000 --- a/demo/auto_research/user_contract.py +++ /dev/null @@ -1,190 +0,0 @@ -from __future__ import annotations - -import shlex - -from .defaults import ( - AUTO_RESEARCH_KNN_DEMO_CONTEXT, - AUTO_RESEARCH_KNN_DEMO_PRESET_ID, - AUTO_RESEARCH_SUPPORTED_PRESET_IDS, -) - -AUTO_RESEARCH_USER_CONTRACT_SCHEMA_VERSION = "auto_research_user_contract_v0" - - -def normalize_auto_research_preset_id(preset_id: str | None) -> str | None: - text = str(preset_id or "").strip() - if not text: - return None - if text not in AUTO_RESEARCH_SUPPORTED_PRESET_IDS: - supported = ", ".join(AUTO_RESEARCH_SUPPORTED_PRESET_IDS) - raise ValueError(f"unsupported auto-research preset {text!r}; supported: {supported}") - return text - - -def build_auto_research_preset_context(preset_id: str | None) -> dict[str, object] | None: - normalized = normalize_auto_research_preset_id(preset_id) - if normalized == AUTO_RESEARCH_KNN_DEMO_PRESET_ID: - return dict(AUTO_RESEARCH_KNN_DEMO_CONTEXT) - return None - - -def infer_auto_research_output_language( - open_question: str, - *, - output_language: str = "auto", -) -> str: - requested = str(output_language or "auto").strip().lower() - if requested in {"zh", "en"}: - return requested - question = str(open_question or "") - return "zh" if any("\u4e00" <= char <= "\u9fff" for char in question) else "en" - - -def build_auto_research_user_contract( - open_question: str, - *, - max_todos: int = 5, - output_language: str = "auto", - preset_id: str | None = None, -) -> dict[str, object]: - question = " ".join(str(open_question or "").strip().split()) - if not question: - raise ValueError('auto-research requires an open question, e.g. loopx auto-research "..."') - preset_context = build_auto_research_preset_context(preset_id) - normalized_preset_id = ( - str(preset_context.get("preset_id")) if preset_context else None - ) - todo_limit = min(max(1, int(max_todos)), 5) - resolved_language = infer_auto_research_output_language( - question, - output_language=output_language, - ) - action_plan = [ - {"priority": priority, "todo": todo, "owner_layer": owner} - for priority, todo, owner in [ - ("P0", "Define the research brief and claim boundary before making factual claims.", "auto_research_preset"), - ("P0", "Collect evidence refs from allowed code, docs, benchmarks, issues, and PRs.", "generic_kernel_agents"), - ("P1", "Run a decentralized multi-agent pass: research-curator frames the boundary, hypothesis-proposer grows claims, research-executor gathers evidence, and evaluator-promoter checks promotion readiness.", "generic_kernel_runner"), - ("P1", "Produce the next executable step and say whether it can run automatically.", "auto_research_preset"), - ("P2", "Summarize gates and unresolved evidence gaps without expanding the user contract.", "auto_research_preset"), - ][:todo_limit] - ] - preset_flag = ( - f" --preset {shlex.quote(normalized_preset_id)}" - if normalized_preset_id - else "" - ) - language_flag = f" --language {shlex.quote(resolved_language)}" if resolved_language != "en" else "" - start_command = ( - f"loopx auto-research start {shlex.quote(question)}{preset_flag}{language_flag} --execute" - ) - takeover_command = f"{start_command} --attach" - command_template = ( - f'loopx auto-research start ""{preset_flag} --execute' - ) - preview_template = f'loopx auto-research start ""{preset_flag}' - return { - "ok": True, - "schema_version": AUTO_RESEARCH_USER_CONTRACT_SCHEMA_VERSION, - "mode": "user_contract", - "product_id": "auto-research", - "open_question": question, - "output_language": { - "requested": str(output_language or "auto"), - "resolved": resolved_language, - "human_process_language": "Chinese" if resolved_language == "zh" else "English", - "machine_schema_language": "English", - }, - "layering": { - "user_layer": "one open question", - "auto_research_layer": "fixed output contract only", - "kernel_layer": "multi-agent runner, Codex TUI panes, pane-local tick, todo/evidence/status protocol", - }, - "command_contract": { - "canonical_invocation": 'loopx auto-research ""', - "explicit_invocation": 'loopx auto-research contract ""', - "one_click_start_invocation": command_template, - "user_required_inputs": ["open_question"], - "user_optional_inputs": ["output_language", "preset"], - "auto_research_required_outputs": [ - "research_brief", - "action_plan", - "evidence_refs", - "next_executable_step", - "gate", - ], - "max_action_plan_todos": 5, - }, - "one_click_start": { - "schema_version": "auto_research_one_click_start_v0", - "command_template": command_template, - "command": start_command, - "operator_takeover_command_template": ( - f"{command_template} --attach" - ), - "operator_takeover_command": takeover_command, - "preview_command_template": preview_template, - "preview_command": ( - f"loopx auto-research start {shlex.quote(question)}{preset_flag}{language_flag}" - ), - "starts": "visible_codex_tui_lanes", - "attach_semantics": ( - "--attach means operator takeover first; it skips the default visible role wake." - ), - "uses_generic_kernel": True, - "coordination_model": "decentralized_state_a2a", - "user_does_not_choose": [ - "agent_ids", - "tmux_layout", - "pane_tick_command", - "evidence_packet_schema", - "frontier_protocol", - ], - }, - **({"preset_context": preset_context} if preset_context else {}), - "research_brief": { - "read": [], - "not_read": [ - "source code", - "docs", - "benchmarks", - "issues", - "pull requests", - ], - "claim_boundary": ( - "Initial contract only: no factual claim is accepted until it is backed " - "by an evidence ref and checked by the verifier lane." - ), - }, - "action_plan": action_plan, - "evidence_refs": { - "code": [], - "docs": [], - "benchmarks": [], - "issues": [], - "pull_requests": [], - }, - "next_executable_step": { - "can_run_automatically": True, - "step": ( - "Start with read-only evidence discovery and claim-boundary drafting. " - "Escalate to the user only when private material, credentials, production " - "actions, or destructive git operations are required." - ), - }, - "gate": { - "user_judgment_needed": [ - "Which private or restricted sources, if any, are allowed.", - "Whether production actions, credentialed connectors, or destructive git are allowed.", - "Whether unresolved evidence gaps are acceptable for the final claim.", - ], - "default_without_user_gate": "public/read-only discovery plus local evidence draft", - }, - "public_boundary": { - "raw_logs": False, - "private_material": False, - "credentials": False, - "destructive_git": False, - "production_actions": False, - }, - } diff --git a/demo/auto_research/worker_loop.py b/demo/auto_research/worker_loop.py deleted file mode 100644 index 89ed598e50..0000000000 --- a/demo/auto_research/worker_loop.py +++ /dev/null @@ -1,163 +0,0 @@ -from __future__ import annotations - -from collections.abc import Callable, Sequence -from pathlib import Path - -from .worker_runtime import run_auto_research_worker_turn - - -AUTO_RESEARCH_WORKER_LOOP_SCHEMA_VERSION = "auto_research_worker_loop_v0" - -AppendEvidence = Callable[[str], dict[str, object]] - - -def _compact_turn(turn: dict[str, object], *, round_index: int) -> dict[str, object]: - completion = turn.get("completion") if isinstance(turn.get("completion"), dict) else {} - append = turn.get("append") if isinstance(turn.get("append"), dict) else {} - live_evidence = turn.get("live_evidence") if isinstance(turn.get("live_evidence"), dict) else {} - evaluation_summary = ( - turn.get("evaluation_summary") if isinstance(turn.get("evaluation_summary"), dict) else {} - ) - successor_todos = ( - turn.get("successor_todos") if isinstance(turn.get("successor_todos"), dict) else {} - ) - successors = ( - successor_todos.get("successors") - if isinstance(successor_todos.get("successors"), list) - else [] - ) - return { - "round": round_index, - "agent_id": turn.get("agent_id"), - "role_id": turn.get("role_id"), - "mode": turn.get("mode"), - "executed": bool(turn.get("executed")), - "selected_todo_id": turn.get("selected_todo_id"), - "selected_action": turn.get("selected_action"), - "completion_status": completion.get("status"), - "appended_count": append.get("appended_count"), - "dev_metric": turn.get("dev_metric"), - "holdout_metric": turn.get("holdout_metric"), - "claim_allowed": evaluation_summary.get("claim_allowed"), - "best_holdout_metric": evaluation_summary.get("best_holdout_metric"), - "holdout_metric_sequence": evaluation_summary.get("holdout_metric_sequence"), - "holdout_improvement_count": evaluation_summary.get("holdout_improvement_count"), - "live_evidence_written": bool(live_evidence.get("written")), - "successor_todo_count": len(successors), - "successor_todos": [ - { - "todo_id": successor.get("todo_id"), - "target_agent_id": successor.get("target_agent_id") - or successor.get("claimed_by"), - "target_role_id": successor.get("target_role_id"), - "source_todo_id": successor.get("unblocks_todo_id"), - "action_kind": successor.get("action_kind"), - } - for successor in successors - if isinstance(successor, dict) - ], - } - - -def run_auto_research_worker_loop( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - agent_ids: Sequence[str], - objective: str, - workspace: Path, - output_dir: str = "auto_research_lightweight_kernel", - evidence_dir: str = ".local/auto-research-worker", - execute: bool = False, - append_evidence: AppendEvidence | None = None, - lane_count: int | None = None, - visible_lanes_accepted: bool = False, - live_evidence_output: str = "live-codex-e2e-evidence.public.json", - complete_selected_todo: bool = False, - max_rounds: int = 4, -) -> dict[str, object]: - """Run repeated LoopX-selected worker turns for a fixed visible lane set. - - The loop is intentionally thin: it does not pick research work itself. Each - pass delegates to worker-turn, so every lane re-reads quota/frontier before - writing any evidence. - """ - - clean_agents = [agent.strip() for agent in agent_ids if agent.strip()] - if not clean_agents: - raise ValueError("worker-loop requires at least one --agent-id") - if max_rounds < 1: - raise ValueError("worker-loop requires --max-rounds >= 1") - if execute and append_evidence is None: - raise ValueError("worker-loop --execute requires an append_evidence callback") - - workspace = workspace.resolve() - effective_lane_count = lane_count or len(clean_agents) - stop_marker = workspace / ".loopx-auto-research-stop" - turns: list[dict[str, object]] = [] - stop_reason = "max_rounds" - for round_index in range(1, max_rounds + 1): - if stop_marker.exists(): - stop_reason = "operator_stop_requested" - break - round_turns: list[dict[str, object]] = [] - for agent_id in clean_agents: - turn = run_auto_research_worker_turn( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - agent_id=agent_id, - objective=objective, - workspace=workspace, - output_dir=output_dir, - evidence_dir=evidence_dir, - execute=execute, - append_evidence=append_evidence if execute else None, - lane_count=effective_lane_count, - visible_lanes_accepted=visible_lanes_accepted, - live_evidence_output=live_evidence_output, - complete_selected_todo=complete_selected_todo, - ) - compact = _compact_turn(turn, round_index=round_index) - turns.append(compact) - round_turns.append(compact) - if not any(turn.get("mode") != "no_action" for turn in round_turns): - stop_reason = "no_runnable_frontier" - break - if execute and not any(turn.get("executed") for turn in round_turns): - if any(turn.get("mode") == "paused_by_quota" for turn in round_turns): - stop_reason = "quota_paused" - else: - stop_reason = "no_executed_turns" - break - - executed_turns = [turn for turn in turns if turn.get("executed")] - completed_turns = [turn for turn in turns if turn.get("completion_status") == "done"] - selected_actions = [ - str(turn.get("selected_action")) - for turn in turns - if turn.get("selected_action") - ] - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_LOOP_SCHEMA_VERSION, - "mode": "execute" if execute else "dry_run", - "goal_id": goal_id, - "agent_ids": clean_agents, - "round_count": max((int(turn["round"]) for turn in turns), default=0), - "max_rounds": max_rounds, - "stop_reason": stop_reason, - "turn_count": len(turns), - "executed_turn_count": len(executed_turns), - "completed_turn_count": len(completed_turns), - "selected_actions": selected_actions, - "turns": turns, - "public_boundary": { - "source": "loopx_quota_frontier_worker_turns", - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } diff --git a/demo/auto_research/worker_runtime.py b/demo/auto_research/worker_runtime.py deleted file mode 100644 index 7c1c2b9f92..0000000000 --- a/demo/auto_research/worker_runtime.py +++ /dev/null @@ -1,853 +0,0 @@ -from __future__ import annotations - -import json -import re -from collections.abc import Callable, Iterator -from contextlib import contextmanager -from pathlib import Path -from uuid import uuid4 - -from .defaults import ( - AUTO_RESEARCH_DEFAULT_GOAL_ID, - AUTO_RESEARCH_DEFAULT_OBJECTIVE, -) -from .live_evidence import ( - LIVE_CODEX_E2E_DEFAULT_OUTPUT, -) -from .preset import ( - auto_research_role_id_for_action, - auto_research_successor_specs_for_action, -) -from .research_state import ( - build_live_auto_research_projection, - build_research_decision_candidates, - build_research_evidence_graph_from_rollout_events, - build_research_failure_continuation_resolution, - normalize_auto_research_action, -) -from ..multi_agent.role_successor import ( - apply_role_successor_todos, - first_successor_followup, -) -from loopx.control_plane.todos.contract import ( - normalize_supported_todo_resume_when, - normalize_todo_id, -) -from loopx.control_plane.todos.handoff_mode import HANDOFF_MODE_HARD_LEASE -from loopx.control_plane.coordination.local_authority import ( - LocalCoordinationAuthorityUnavailable, read_canonical_todos_if_promoted, -) -from loopx.control_plane.work_items.task_lease import ( - TaskLeaseError, - acquire_task_lease, - inspect_task_lease, - release_task_lease, - runtime_root_from_registry, -) -from loopx.history import load_registry -from loopx.paths import resolve_runtime_root -from loopx.quota import build_quota_should_run -from loopx.rollout_event_log import load_rollout_events, rollout_event_log_path -from loopx.status import collect_status -from loopx.todos import complete_goal_todo - - -AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION = "auto_research_worker_turn_v0" -AUTO_RESEARCH_WORKER_FRONTIER_SCHEMA_VERSION = "auto_research_worker_frontier_v0" -SUPPORTED_WORKER_ACTIONS = { - "write_research_contract", - "review_research_contract", - "propose_hypothesis", - "review_hypothesis_frontier", - "run_dev_eval", - "run_holdout_eval", - "remediate_data_measurement", - "propose_failure_successor", - "write_evidence", - "classify_evidence", - "summarize_evidence", - "write_evaluation_summary", - "review_promotion_readiness", - "record_terminal_decision", - "review_terminal_decision", -} -AppendEvidence = Callable[[str], dict[str, object]] -AUTO_RESEARCH_STATE_SUMMARY_MODE = "rollout_evidence_summary" -AUTO_RESEARCH_MANUAL_RESEARCH_REQUIRED_MODE = "manual_research_required" -MANUAL_RESEARCH_REQUIRED_ACTIONS = { - "write_research_contract", - "propose_hypothesis", - "propose_failure_successor", - "run_dev_eval", - "run_holdout_eval", - "remediate_data_measurement", - "write_evidence", - "record_terminal_decision", - "review_terminal_decision", -} -SUMMARY_ACTIONS = { - "classify_evidence", - "summarize_evidence", - "write_evaluation_summary", - "review_research_contract", - "review_hypothesis_frontier", - "review_promotion_readiness", -} - - -def _scored_rollout_metric_events( - rollout_events: list[dict[str, object]], - *, - split: str | None = None, -) -> list[dict[str, object]]: - events: list[dict[str, object]] = [] - for event in rollout_events: - if str(event.get("event_kind") or "") != "research_evidence": - continue - details = event.get("details") if isinstance(event.get("details"), dict) else {} - if split is not None and str(details.get("split") or "") != split: - continue - if details.get("metric_value") is None: - continue - events.append(event) - return events - - -def _holdout_metric_sequence(rollout_events: list[dict[str, object]]) -> list[float]: - sequence: list[float] = [] - for event in _scored_rollout_metric_events(rollout_events, split="holdout"): - details = event.get("details") if isinstance(event.get("details"), dict) else {} - value = details.get("metric_value") - if isinstance(value, bool): - continue - if isinstance(value, (int, float)): - sequence.append(float(value)) - return sequence - - -def _holdout_improvement_count( - holdout_metrics: list[float], - *, - baseline: float | None, - direction: str, -) -> int: - if baseline is None: - return 0 - count = 0 - previous = baseline - for metric in holdout_metrics: - improved = metric < previous if direction == "minimize" else metric > previous - if improved: - count += 1 - previous = metric - return count - - -def _has_rollout_research_evidence( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, -) -> bool: - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - return any( - str(event.get("event_kind") or "") == "research_evidence" - for event in load_rollout_events(rollout_event_log_path(runtime_root, goal_id)) - ) - - -def _slug(value: object, *, default: str = "item") -> str: - text = re.sub(r"[^A-Za-z0-9._-]+", "-", str(value or "")).strip("-._") - return text[:80] or default - - -def _write_json(path: Path, payload: dict[str, object]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") - - -def _artifact_summary(kind: str, *, filename: str) -> dict[str, object]: - return { - "kind": kind, - "filename": filename, - "paths_are_local_only": True, - } - - -def _write_evaluation_summary_artifact( - *, - registry_path: Path, - runtime_root_arg: str | None, - output_path: Path, - goal_id: str, - todo_id: str, - lineage_todo_id: str | None, - require_selected_failure_match: bool, - agent_id: str, -) -> dict[str, object]: - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - rollout_events = load_rollout_events(rollout_event_log_path(runtime_root, goal_id)) - graph = build_research_evidence_graph_from_rollout_events( - goal_id=goal_id, - rollout_events=rollout_events, - ) - decisions = build_research_decision_candidates(graph) - failure_resolution = build_research_failure_continuation_resolution( - decisions, - selected_todo_id=todo_id, - selected_lineage_todo_id=lineage_todo_id, - require_selected_failure_match=require_selected_failure_match, - ) - failure_continuation = failure_resolution["failure_continuation"] - failure_continuation_unresolved = bool(failure_resolution["unresolved"]) - holdout_metrics = _holdout_metric_sequence(rollout_events) - metric = graph.get("metric") if isinstance(graph.get("metric"), dict) else {} - baseline = graph.get("baseline_metric") - holdout_improvements = _holdout_improvement_count( - holdout_metrics, - baseline=float(baseline) if isinstance(baseline, (int, float)) else None, - direction=str(metric.get("direction") or "maximize"), - ) - dev_pending_holdout_count = len(decisions.get("dev_promotion_candidates") or []) - artifact = { - "ok": True, - "schema_version": "auto_research_worker_evaluation_summary_v0", - "goal_id": goal_id, - "todo_id": todo_id, - "agent_id": agent_id, - "evidence_graph_summary": { - "event_count": graph.get("event_count"), - "hypothesis_count": graph.get("hypothesis_count"), - "best_dev_metric": graph.get("best_dev_metric"), - "best_holdout_metric": graph.get("best_holdout_metric"), - "holdout_improved": graph.get("holdout_improved"), - "holdout_metric_sequence": holdout_metrics, - "holdout_improvement_count": holdout_improvements, - "negative_evidence_count": graph.get("negative_evidence_count"), - }, - "decision_summary": { - "dev_promotion_candidate_count": len(decisions.get("dev_promotion_candidates") or []), - "dev_candidate_pending_holdout_count": dev_pending_holdout_count, - "validated_promotion_candidate_count": len(decisions.get("validated_promotion_candidates") or []), - "promotion_candidate_count": len(decisions.get("promotion_candidates") or []), - "retirement_candidate_count": len(decisions.get("retirement_candidates") or []), - "failure_continuation": failure_continuation, - "failure_continuation_unresolved": failure_continuation_unresolved, - "holdout_metric_sequence": holdout_metrics, - "holdout_improvement_count": holdout_improvements, - }, - "summary": { - "status": "evaluation_summary_written", - "claim_allowed": bool(decisions.get("validated_promotion_candidates")), - "promotion_decision_made": False, - }, - "public_boundary": { - "source": "loopx_rollout_event_log", - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - }, - } - _write_json(output_path, artifact) - return artifact - - -def _selected_failure_lineage_todo_id(selected: dict[str, object]) -> str | None: - unblocks_todo_id = normalize_todo_id(selected.get("unblocks_todo_id")) - if unblocks_todo_id: - return unblocks_todo_id - resume_when = normalize_supported_todo_resume_when(selected.get("resume_when")) - if not resume_when: - return None - kind, _separator, target = resume_when.partition(":") - return normalize_todo_id(target) if kind == "todo_done" else None - - -def _failure_lineage_unresolved_result( - *, - goal_id: str, - agent_id: str, - todo_id: str, - action: str, - artifact: dict[str, object], - frontier_packet: dict[str, object], - complete_selected_todo: bool, -) -> dict[str, object]: - decision_summary = artifact["decision_summary"] - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "execute", - "goal_id": goal_id, - "agent_id": agent_id, - "selected_todo_id": todo_id, - "selected_action": action, - "executed": True, - "summary_mode": AUTO_RESEARCH_STATE_SUMMARY_MODE, - "artifact": _artifact_summary("evaluation_summary", filename="evaluation-summary.public.json"), - "artifact_status": artifact["summary"]["status"], - "claim_allowed": artifact["summary"]["claim_allowed"], - "promotion_decision_made": artifact["summary"]["promotion_decision_made"], - "role_id": auto_research_role_id_for_action(action), - "evaluation_summary": { - "claim_allowed": artifact["summary"]["claim_allowed"], - "best_dev_metric": artifact["evidence_graph_summary"]["best_dev_metric"], - "best_holdout_metric": artifact["evidence_graph_summary"]["best_holdout_metric"], - "holdout_metric_sequence": artifact["evidence_graph_summary"][ - "holdout_metric_sequence" - ], - "holdout_improvement_count": decision_summary["holdout_improvement_count"], - "validated_promotion_candidate_count": decision_summary[ - "validated_promotion_candidate_count" - ], - "failure_continuation": None, - }, - "successor_todos": { - "schema_version": "multi_agent_role_successor_todos_v0", - "source": "role_profile_todo_command_template", - "needed": False, - "executed": False, - "role_id": auto_research_role_id_for_action(action), - "action": action, - "successors": [], - "skipped": [], - "reason": "failure_successor_lineage_unresolved", - }, - "followup": None, - "completion": { - "requested": complete_selected_todo, - "executed": False, - "reason": "failure_successor_lineage_unresolved", - }, - "frontier": frontier_packet, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - - -def _manual_research_required_result( - *, - goal_id: str, - agent_id: str, - todo_id: str, - action: str, - complete_selected_todo: bool, - frontier_packet: dict[str, object], -) -> dict[str, object]: - role_id = auto_research_role_id_for_action(action) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": AUTO_RESEARCH_MANUAL_RESEARCH_REQUIRED_MODE, - "goal_id": goal_id, - "agent_id": agent_id, - "selected_todo_id": todo_id, - "selected_action": action, - "role_id": role_id, - "executed": False, - "manual_research_required": True, - "reason": "auto_research_visible_roles_must_author_real_evidence", - "next_steps": [ - "read the role profile and selected frontier", - "perform the role's research work in the visible Codex TUI", - "write public-safe contract, hypothesis, evidence, or todo handoff through LoopX state", - "do not claim dev/holdout uplift unless it comes from an explicit evidence packet or real evaluator output", - ], - "completion": { - "requested": complete_selected_todo, - "executed": False, - "reason": "manual_research_required_before_completion", - }, - "frontier": frontier_packet, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - "fake_metrics_recorded": False, - }, - } - - -def _maybe_add_role_successor_todos( - *, - registry_path: Path, - goal_id: str, - source_todo_id: str, - agent_id: str, - role_id: str, - action: str, - decision_summary: dict[str, object], - execute: bool, -) -> dict[str, object]: - successor_specs = auto_research_successor_specs_for_action( - role_id=role_id, - action=action, - ) - failure_continuation = decision_summary.get("failure_continuation") - text_source_todo_id = None - if isinstance(failure_continuation, dict): - next_outcome = str(failure_continuation.get("next_outcome") or "") - successor_specs = [spec for spec in successor_specs if spec.get("action_kind") == next_outcome] - source_todo_id = str(failure_continuation.get("source_todo_id") or "") or source_todo_id - text_source_todo_id = source_todo_id - def apply_binding(parent_id: str, *, revision: str | None = None) -> dict[str, object]: - return apply_role_successor_todos(registry_path=registry_path, goal_id=goal_id, - source_todo_id=parent_id, current_agent_id=agent_id, role_id=role_id, - action=action, successor_specs=successor_specs, decision_summary=decision_summary, - execute=execute, text_source_todo_id=text_source_todo_id, - expected_provider_revision=revision) - - try: - return apply_binding(source_todo_id) - except LocalCoordinationAuthorityUnavailable as error: - if error.code != 'todo_semantic_duplicate_conflict' or text_source_todo_id is None: - raise - # Native create owns normalized title/intent matching. Qualify only its - # exact conflicting record, without rewriting a historical parent. - snapshot = read_canonical_todos_if_promoted( - runtime_root=runtime_root_from_registry(registry_path, None), goal_id=goal_id, - ) - todos = {row['todo_id']: row for row in (snapshot or {}).get('todos', [])} - existing = todos.get(error.payload.get('todo_id')) - parent_id = existing.get('unblocks_todo_id') if existing else None - parent = todos.get(parent_id) - if not (parent and parent['role'] == 'agent' and parent.get('action_kind') in SUMMARY_ACTIONS - and parent.get('unblocks_todo_id') == text_source_todo_id): - raise - return apply_binding(parent_id, revision=snapshot['provider_revision']) - - - -def _executed_successor_todo_ids(successor_todos: dict[str, object]) -> list[str]: - successors = successor_todos.get("successors") - if not isinstance(successors, list): - return [] - todo_ids: list[str] = [] - for successor in successors: - if not isinstance(successor, dict): - continue - todo_id = successor.get("todo_id") - if isinstance(todo_id, str) and todo_id: - todo_ids.append(todo_id) - return todo_ids - - -def _complete_selected_todo( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - todo_id: str, - agent_id: str, - action: str, - execute: bool, - successor_todo_ids: list[str] | None = None, - lease_proof: dict[str, object] | None = None, -) -> dict[str, object]: - if not execute: - return {"requested": True, "executed": False} - linked_successors = successor_todo_ids or [] - result = complete_goal_todo( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - todo_id=todo_id, - role="agent", - claimed_by=agent_id, - agent_id=agent_id, - note=f"auto-research state summary completed {action}", - evidence=( - f"state-summary agent={agent_id} action={action} wrote public-safe local artifact " - "and obeyed quota/frontier before completion" - ), - no_followup=not linked_successors, - successor_todo_ids=linked_successors or None, - self_merged=True, - dry_run=False, - **(lease_proof or {}), - ) - return { - "requested": True, - "executed": True, - "ok": bool(result.get("ok")), - "changed": bool(result.get("changed")), - "todo_id": result.get("todo_id"), - "status": "done" if result.get("completed") else None, - "self_merged": bool(result.get("self_merged")), - "successor_todo_ids": result.get("successor_todo_ids") or linked_successors, - } - - -@contextmanager -def _worker_execution_lease( - *, registry_path: Path, runtime_root_arg: str | None, - goal_id: str, todo_id: str, agent_id: str, -) -> Iterator[dict[str, object]]: - """Use the typed lease owner without borrowing another execution's fence.""" - runtime_root = runtime_root_from_registry(registry_path, runtime_root_arg) - current = inspect_task_lease( - registry_path=registry_path, runtime_root=runtime_root, - goal_id=goal_id, todo_id=todo_id, - ) - if current.get("ok") is not True: - raise TaskLeaseError("Worker lease inspection failed", code=str(current.get("error_code"))) - if current.get("handoff_mode") != HANDOFF_MODE_HARD_LEASE: - yield {} - return - key = "auto-research-worker-" + uuid4().hex - acquired = acquire_task_lease( - registry_path=registry_path, runtime_root=runtime_root, - goal_id=goal_id, todo_id=todo_id, owner=agent_id, - idempotency_key=key, - ) - version = acquired["lease"]["version"] - try: - yield {"task_lease_idempotency_key": key, "task_lease_expected_version": version} - finally: - # Completion may already have retired this execution. Cleanup must never - # release a newer lease acquired by another invocation. - current = inspect_task_lease( - registry_path=registry_path, runtime_root=runtime_root, - goal_id=goal_id, todo_id=todo_id, - ) - if current.get("ok") is not True: - raise TaskLeaseError("Worker lease cleanup readback failed", code=str(current.get("error_code"))) - lease = current.get("lease") or {} - if (current.get("active") is True and lease.get("owner") == agent_id - and lease.get("idempotency_key") == key and lease.get("version") == version): - release_task_lease( - registry_path=registry_path, runtime_root=runtime_root, - goal_id=goal_id, todo_id=todo_id, owner=agent_id, - idempotency_key=key, expected_version=version, - ) - - -def load_auto_research_worker_frontier( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str, - agent_id: str, - workspace: Path, -) -> dict[str, object]: - """Read the same quota/frontier surfaces a visible worker must obey.""" - - status_payload = collect_status( - registry_path=registry_path, - runtime_root_override=runtime_root_arg, - limit=5, - goal_id=goal_id, - ) - quota_payload = build_quota_should_run( - status_payload, - goal_id=goal_id, - agent_id=agent_id, - ) - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - projection = build_live_auto_research_projection( - goal_id=goal_id, - agent_id=agent_id, - quota_payload=quota_payload, - rollout_events=load_rollout_events(rollout_event_log_path(runtime_root, goal_id)), - ) - frontier = projection["frontier"] - selected = frontier.get("selected") if isinstance(frontier, dict) else None - completion = frontier.get("completion") if isinstance(frontier, dict) else None - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_FRONTIER_SCHEMA_VERSION, - "goal_id": goal_id, - "agent_id": agent_id, - "quota": { - "ok": bool(quota_payload.get("ok")), - "should_run": bool(quota_payload.get("should_run")), - "state": quota_payload.get("state"), - "user_action_required": bool( - ((quota_payload.get("interaction_contract") or {}).get("user_channel") or {}).get( - "action_required" - ) - ), - }, - "frontier": { - "selected": selected, - "runnable_count": len(frontier.get("runnable") or []) if isinstance(frontier, dict) else 0, - "blocked_count": len(frontier.get("blocked") or []) if isinstance(frontier, dict) else 0, - "source_kind": frontier.get("source_kind") if isinstance(frontier, dict) else None, - "completion": completion if isinstance(completion, dict) else None, - }, - "public_boundary": { - "source": "loopx_quota_and_auto_research_frontier", - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - }, - } - - -def run_auto_research_worker_turn( - *, - registry_path: Path, - runtime_root_arg: str | None, - goal_id: str = AUTO_RESEARCH_DEFAULT_GOAL_ID, - agent_id: str, - objective: str = AUTO_RESEARCH_DEFAULT_OBJECTIVE, - workspace: Path, - output_dir: str = "auto_research_lightweight_kernel", - evidence_dir: str = ".local/auto-research-worker", - execute: bool = False, - append_evidence: AppendEvidence | None = None, - lane_count: int = 1, - visible_lanes_accepted: bool = False, - live_evidence_output: str = LIVE_CODEX_E2E_DEFAULT_OUTPUT, - complete_selected_todo: bool = False, -) -> dict[str, object]: - """Run one LoopX-selected visible worker action. - - The worker does not choose its own work. It reads quota/frontier, checks the - selected action, then performs one small public-safe research turn. - """ - - workspace = workspace.resolve() - frontier_packet = load_auto_research_worker_frontier( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - agent_id=agent_id, - workspace=workspace, - ) - selected = frontier_packet["frontier"].get("selected") if isinstance(frontier_packet["frontier"], dict) else None - raw_action = str((selected or {}).get("allowed_action") or "") - action = normalize_auto_research_action(raw_action) - todo_id = str((selected or {}).get("todo_id") or "") - lineage_todo_id = ( - _selected_failure_lineage_todo_id(selected) - if isinstance(selected, dict) - else None - ) - if not selected or not todo_id: - completion = ( - frontier_packet["frontier"].get("completion") - if isinstance(frontier_packet.get("frontier"), dict) - else None - ) - if isinstance(completion, dict) and completion.get("quiet_completion_allowed") is True: - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "quiet_completion", - "goal_id": goal_id, - "agent_id": agent_id, - "executed": False, - "completion": completion, - "frontier": frontier_packet, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "no_action", - "goal_id": goal_id, - "agent_id": agent_id, - "executed": False, - "frontier": frontier_packet, - } - quota_ok = bool(frontier_packet.get("quota", {}).get("ok")) - quota_should_run = bool(frontier_packet.get("quota", {}).get("should_run")) - if quota_ok and not quota_should_run: - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "paused_by_quota", - "goal_id": goal_id, - "agent_id": agent_id, - "selected_todo_id": todo_id, - "selected_action": action, - "executed": False, - "quota": frontier_packet.get("quota"), - "frontier": frontier_packet, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - if action not in SUPPORTED_WORKER_ACTIONS: - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "unsupported_action", - "goal_id": goal_id, - "agent_id": agent_id, - "selected_todo_id": todo_id, - "selected_action": action, - "supported_actions": sorted(SUPPORTED_WORKER_ACTIONS), - "executed": False, - "frontier": frontier_packet, - } - if not execute: - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "dry_run", - "goal_id": goal_id, - "agent_id": agent_id, - "selected_todo_id": todo_id, - "selected_action": action, - "would_execute": action, - "completion": {"requested": complete_selected_todo, "executed": False}, - "frontier": frontier_packet, - } - if action in MANUAL_RESEARCH_REQUIRED_ACTIONS: - return _manual_research_required_result( - goal_id=goal_id, - agent_id=agent_id, - todo_id=todo_id, - action=action, - complete_selected_todo=complete_selected_todo, - frontier_packet=frontier_packet, - ) - - run_dir = workspace / evidence_dir / _slug(agent_id, default="agent") / _slug(todo_id, default="todo") - evaluation_summary_path = run_dir / "evaluation-summary.public.json" - - if action in SUMMARY_ACTIONS: - if not _has_rollout_research_evidence( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - ): - return _manual_research_required_result( - goal_id=goal_id, - agent_id=agent_id, - todo_id=todo_id, - action=action, - complete_selected_todo=complete_selected_todo, - frontier_packet=frontier_packet, - ) - frontier_completion = ( - frontier_packet["frontier"].get("completion") - if isinstance(frontier_packet.get("frontier"), dict) - else None - ) - require_selected_failure_match = bool( - isinstance(frontier_completion, dict) - and frontier_completion.get("status") == "failure_successor_required" - ) - with _worker_execution_lease( - registry_path=registry_path, runtime_root_arg=runtime_root_arg, - goal_id=goal_id, todo_id=todo_id, agent_id=agent_id, - ) as lease_proof: - artifact = _write_evaluation_summary_artifact( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - output_path=evaluation_summary_path, - goal_id=goal_id, - todo_id=todo_id, - lineage_todo_id=lineage_todo_id, - require_selected_failure_match=require_selected_failure_match, - agent_id=agent_id, - ) - if artifact["decision_summary"]["failure_continuation_unresolved"]: - return _failure_lineage_unresolved_result( - goal_id=goal_id, - agent_id=agent_id, - todo_id=todo_id, - action=action, - artifact=artifact, - frontier_packet=frontier_packet, - complete_selected_todo=complete_selected_todo, - ) - role_id = auto_research_role_id_for_action(action) - successor_todos = _maybe_add_role_successor_todos( - registry_path=registry_path, - goal_id=goal_id, - source_todo_id=todo_id, - agent_id=agent_id, - role_id=role_id, - action=action, - decision_summary=artifact["decision_summary"], - execute=True, - ) - successor_todo_ids = _executed_successor_todo_ids(successor_todos) - followup = first_successor_followup(successor_todos) - completion = ( - _complete_selected_todo( - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - goal_id=goal_id, - todo_id=todo_id, - agent_id=agent_id, - action=action, - execute=True, - successor_todo_ids=successor_todo_ids, - lease_proof=lease_proof, - ) - if complete_selected_todo - else {"requested": False} - ) - return { - "ok": True, - "schema_version": AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, - "mode": "execute", - "goal_id": goal_id, - "agent_id": agent_id, - "selected_todo_id": todo_id, - "selected_action": action, - "executed": True, - "summary_mode": AUTO_RESEARCH_STATE_SUMMARY_MODE, - "artifact": _artifact_summary("evaluation_summary", filename="evaluation-summary.public.json"), - "artifact_status": artifact["summary"]["status"], - "claim_allowed": artifact["summary"]["claim_allowed"], - "promotion_decision_made": artifact["summary"]["promotion_decision_made"], - "role_id": role_id, - "evaluation_summary": { - "claim_allowed": artifact["summary"]["claim_allowed"], - "best_dev_metric": artifact["evidence_graph_summary"]["best_dev_metric"], - "best_holdout_metric": artifact["evidence_graph_summary"]["best_holdout_metric"], - "holdout_metric_sequence": artifact["evidence_graph_summary"][ - "holdout_metric_sequence" - ], - "holdout_improvement_count": artifact["decision_summary"][ - "holdout_improvement_count" - ], - "validated_promotion_candidate_count": artifact["decision_summary"][ - "validated_promotion_candidate_count" - ], - "failure_continuation": artifact["decision_summary"][ - "failure_continuation" - ], - }, - "successor_todos": successor_todos, - "followup": followup, - "completion": completion, - "frontier": frontier_packet, - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } - - return _manual_research_required_result( - goal_id=goal_id, - agent_id=agent_id, - todo_id=todo_id, - action=action, - complete_selected_todo=complete_selected_todo, - frontier_packet=frontier_packet, - ) diff --git a/demo/auto_research/worker_skill/SKILL.md b/demo/auto_research/worker_skill/SKILL.md deleted file mode 100644 index e742601ba5..0000000000 --- a/demo/auto_research/worker_skill/SKILL.md +++ /dev/null @@ -1,359 +0,0 @@ ---- -name: loopx-auto-research -description: Use when a LoopX worker is operating an auto-research lane, demo pane, frontier item, evidence packet, promotion/retirement decision, or visible tmux/Codex auto-research rehearsal. Identity must come from the LoopX role profile and quota/frontier packet; this skill only provides role-specific execution checklists, artifact contracts, and stop conditions. ---- - -# LoopX Auto Research - -This is a worker-local role playbook for auto-research panes. It is packaged -with the auto-research capability and should be injected or referenced by the -worker launcher; it is not a global LoopX skill for ordinary project agents. - -## Routing Boundary - -Use this skill after a LoopX auto-research worker has a role profile, -frontier item, launcher packet, or user-visible demo pane. The skill is the -role playbook. It is not the source of truth for identity, authority, current -frontier, or merge/publication permission. - -Identity comes from LoopX control-plane metadata: - -- `auto_research_role_profile_v0` in the launcher/frontier/bootstrap packet; -- `quota should-run --goal-id ... --agent-id ...`; -- todo claim, capability token, write scope, and protected scope; -- repository or workspace `AGENTS.md` rules, which can only make the boundary - stricter. - -No role owns the full graph. Do not infer role from pane title, branch name, -tmux window name, or the section of this skill that happens to be visible. - -## Pane Tick Contract - -The generic multi-agent kernel owns the default LoopX project/doc-registry -skills and the fixed A2A wake prompt. This skill should stay role-specific: -use it after the pane-local tick has resolved identity, quota, and frontier -from LoopX. - -Compact frontier command: `loopx --format json auto-research frontier --goal-id "$LOOPX_GOAL_ID" --agent-id "$LOOPX_AGENT_ID"`. Also honor `quota should-run`. - -If the launcher exported `LOOPX_ROLE_ID`, `LOOPX_ROLE_PROFILE_REF`, or a profile -JSON path, compare those values with the quota and frontier packets. Stop when -they disagree. Do not guess the intended role. - -If the role profile includes `successor_todos`, treat those declarations as the -only role-local way to create the next agent todo. A successor declaration must -name the target agent and include a `todo_command_template` such as -`loopx todo add ... --claimed-by {target_agent_id_shell}`. In visible -auto-research, the pane-local tick is a guard/frontier read, not a research -writer. Render and run a successor todo only after the visible role has authored -real public-safe evidence or notes that satisfy the declared condition. Do not -invent an extra continuation plan in prose, and do not ask a leader pane to pick -the next role. - -Before completing with no follow-up, compare the evidence summary with -`role_profile.continuation_policy`. When the target is still unmet and a -declared successor condition is satisfied, create or link that successor first. -No-follow-up is only valid after the target is reached, a projected blocker or -user gate stops the lane, or evidence-backed retirement closes the frontier. - -For a visible demo rehearsal, `auto-research demo-supervisor` is read-only by -default; use `--execute` only when the user opted into starting visible local -panes. The default rehearsal must not start Codex, write LoopX state, or spend -quota by itself. - -## Role Resolution - -Map the role profile to one of these sections: - -| Role id or lane | Skill section | Authority source | -| --- | --- | --- | -| `research_curator` | Research curator | role profile, quota packet, contract todo | -| `hypothesis_proposer` | Hypothesis proposer | role profile, frontier packet, hypothesis todo | -| `research_executor` | Research executor | role profile, selected frontier item, write scope | -| `evaluator_promoter` | Evaluator/promoter | role profile, evidence packet, promotion policy | -| `research-narrator`, `product_narrator` | Projection narrator | read-only projection packet and first-screen gate | -| `control-plane-guard` | Control-plane guard | quota/status/check packet and repository rules | - -The current demo may render fewer or differently named panes than the four -logical research roles. That is only a host layout choice. Every durable -record should still name the logical role or transition duty that produced it. - -## Shared Stop Conditions - -Stop and report the exact blocker when any of these are true: - -- quota says `should_run=false`, `delivery_allowed=false`, or a user/operator - gate is open; -- the selected todo is missing, claimed by another agent, or not compatible - with the profile's `capability_token`; -- the next edit touches `protected_scope`, credentials, private material, raw - logs, raw evaluator data, or unapproved publication surfaces; -- the profile, frontier, `AGENTS.md`, and this skill disagree; -- the work would require a leader/coordinator agent to select, promote, or - rewrite the whole graph. - -## Benchmark Workspace Hints - -When the role profile or workspace exposes a benchmark contract, use that -contract before writing research claims. For KNN-style demos this means: - -- read `research_contract.public.json`, `README.md`, and the editable solver; -- edit only the declared editable scope, such as `solution.py`; -- run the declared dev command before proposing promotion; -- run the declared held-out command before claiming a validated improvement; -- summarize mechanism, command, score, and protected-scope cleanliness. -- pass the contract and eval JSON outputs to `loopx auto-research evidence` - rather than hand-authoring an evidence packet. - -The pane-local tick can point at a todo; it cannot count as benchmark evidence. - -## Research Curator - -Use when the role owns the original research wish, objective, required -artifacts, acceptance criteria, metric, editable scope, protected scope, -budget, and gates. - -Allowed actions: - -- create or refresh `auto_research_delivery_contract_v0`, embedding the - existing `research_contract_v0`; -- make protected boundaries explicit; -- write user/operator gate todos when promotion or publication needs judgment; -- request read-only projections from existing evidence. - -Useful command: - -```bash -loopx --format json auto-research frontier \ - --goal-id "$LOOPX_GOAL_ID" \ - --agent-id "$LOOPX_AGENT_ID" -``` - -Artifact contract: - -- objective is public-safe and bounded; -- the original wish, assumptions, non-goals, and contract reference are - public-safe and explicit; -- every required artifact and acceptance criterion has a stable public-safe id; -- metric direction and protected evaluator are explicit; -- write scope and protected scope are named; -- promotion policy says what evidence is sufficient. -- failure policy names fallback artifacts and the conditions for re-entering - research when the current contract cannot be fulfilled. - -Must not: - -- pick winners; -- run experiments; -- present unsupported metrics as product value. - -## Hypothesis Proposer - -Use when the role turns ideas into todo-backed hypotheses, refinements, -successors, or retirements. - -Allowed actions: - -- create `research_hypothesis_v0` records with `todo_id`, `claimed_by`, - mechanism family, parent link, and grounding refs or no-grounding rationale; -- retire duplicates, exhausted retries, or contradicted directions while - keeping negative evidence visible; -- add the next bounded agent todo. - -Before writing: - -- confirm the idea is not claiming novelty from the same source used to ideate; -- confirm the hypothesis can be attempted inside allowed write scope; -- keep todo order and rationale in LoopX state, not only in chat. - -Must not: - -- delete failures; -- select a winner; -- hide contradictory evidence by replacing a hypothesis with a cleaner story. - -## Research Executor - -Use when the role runs exactly one selected hypothesis in an isolated -workspace/worktree and records attempt evidence. - -Allowed actions: - -- claim the current frontier item selected for this agent; -- edit only allowed solution or experiment scope; -- run dev or holdout evaluation only when the contract permits it; -- build an `auto_research_evidence_packet_v0` or equivalent public-safe event; -- create only the role-declared successor todo, such as a holdout validation - todo or post-holdout verifier summary todo, when the profile's - `successor_todos.condition` is satisfied. - -Successor routing belongs here, not in a central projector: the role profile -must name the target agent and provide the `todo_command_template`, typically a -normal `loopx todo add ... --claimed-by {target_agent_id_shell}` command. The -kernel only validates the target agent and executes the normal LoopX todo -writer. - -Evidence writeback should use an explicit lane-authored evidence packet or -normal LoopX todo/evidence commands exposed by the current state. Append only -after reviewing packet boundary, then capture compact live evidence from the -lane-authored packet when visible lanes are accepted. Do not use worker-turn to -manufacture dev or holdout metrics. - -After a real append/capture succeeds for the selected frontier todo, close out -that selected todo with compact public-safe evidence. Dependent evaluator or -successor lanes usually resume from `todo_done:`; leaving the -executor todo open after supported evidence strands the next round. - -Must not: - -- edit protected evaluator/data scope; -- promote results; -- omit failed, inconclusive, or guardrail-failed attempts. - -## Evaluator/Promoter - -Use when the role reads evidence and classifies it as supported, -contradicted, retry-needed, promotion-ready, or retirement-ready. - -Allowed actions: - -- run held-out validation only when the selected frontier action is - `run_holdout_eval` and the contract permits that split; -- apply the contract's metric and promotion policy to scored or unscored - evidence; -- request retry with a bounded reason and resumable ref; -- create promotion, retirement, or gate candidates; -- write compact validation notes for the next worker; -- add only the role-declared successor todo when evidence needs another bounded - split, using the profile's `todo_command_template`. -- do not close with no-follow-up while `continuation_policy` still reports an - unmet target and a role-declared successor condition is satisfied. - -Verification checklist: - -- split label and metric direction are explicit; -- dev evidence is not represented as held-out proof; -- boundary says protected scope stayed clean; -- negative evidence remains queryable. - -When evidence reaches a terminal boundary: - -- use `loopx auto-research decide` to record `promoted` or `retired`; a - promotion candidate is not a terminal result; -- bind the decision to the current evidence graph revision and keep the - decision evidence refs public-safe; -- use `loopx auto-research review --require-independent` only from a different - registered peer than both the hypothesis producer and decision agent; -- treat self-review as visible review evidence, never as independent review; -- when the curator supplied `auto_research_delivery_contract_v0`, run - `loopx auto-research artifact-receipt --contract ` after the - terminal decision and required review; -- return every non-verified receipt to the user with its failure kinds, - verified boundary, fallback artifacts, and reentry conditions; do not turn - one failed attempt into a terminal impossibility claim; -- use `loopx auto-research project-results` after decisions and reviews so - exact `loopx auto-research results` queries can verify Explore readback. - -Example terminal path: - -```bash -loopx auto-research decide \ - --goal-id "$LOOPX_GOAL_ID" \ - --hypothesis-id "" \ - --outcome promoted \ - --reason holdout_validated \ - --agent-id "$LOOPX_AGENT_ID" \ - --execute - -loopx auto-research review \ - --goal-id "$LOOPX_GOAL_ID" \ - --hypothesis-id "" \ - --reviewer-agent-id "$LOOPX_AGENT_ID" \ - --verdict approve \ - --require-independent \ - --execute - -loopx auto-research artifact-receipt \ - --contract "" -``` - -Must not: - -- bypass an owner/operator gate; -- certify a showcase claim; -- rewrite the hypothesis graph to make the result look cleaner. -- label the same producer or decision agent as an independent reviewer. - -## Projection Narrator - -Use when the role is read-only product narration over accepted projections. This -is a transition duty in v0 and may become a separate role later. - -Allowed actions: - -- render `research_evidence_graph_v0` from promoted, retired, and retry - evidence; -- update public-safe docs or Frontstage surfaces only from projection refs; -- preserve failed and retired directions as useful learning. - -Useful command: - -```bash -loopx --format json auto-research project-results \ - --goal-id "$LOOPX_GOAL_ID" \ - --execute - -loopx --format json auto-research results \ - --goal-id "$LOOPX_GOAL_ID" \ - --include-history -``` - -Must stop before: - -- inventing metrics; -- reading private source bodies; -- changing first viewport, hero, primary CTA, or opening nav without the - first-screen review gate. - -## Control-Plane Guard - -Use when the role checks whether a visible demo, frontier, evidence append, -merge, or publication action is safe and interruptible. - -Allowed actions: - -- run quota/status/check packets; -- validate public/private boundary; -- confirm attach/stop/takeover controls are visible; -- write blockers or repair todos when projection is contradictory. - -Useful command: - -```bash -loopx --format json auto-research demo-supervisor \ - --goal-id "$LOOPX_GOAL_ID" \ - --workspace "$LOOPX_PROJECT" -``` - -Must not: - -- act as a leader agent; -- select experiments for other roles; -- approve its own gate. - -## Writeback - -After a validated step, write back only the smallest durable artifact allowed -by the role: - -- `research_contract_v0`; -- `research_hypothesis_v0`; -- `auto_research_evidence_packet_v0`; -- promotion/retirement/gate candidate; -- `research_evidence_graph_v0`; -- LoopX todo completion plus next todo/rationale; -- `loopx refresh-state` and one quota spend only after validation when the - quota contract permits it. - -If the step is blocked, write the blocker as a todo/rationale and do not spend -quota merely for discovering an unchanged gate. diff --git a/demo/multi_agent/__init__.py b/demo/multi_agent/__init__.py deleted file mode 100644 index 0f1eff109d..0000000000 --- a/demo/multi_agent/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Generic LoopX multi-agent runner capability.""" diff --git a/demo/multi_agent/codex_executable.py b/demo/multi_agent/codex_executable.py deleted file mode 100644 index 325ada3ba3..0000000000 --- a/demo/multi_agent/codex_executable.py +++ /dev/null @@ -1,207 +0,0 @@ -from __future__ import annotations - -import os -import re -import shlex -import shutil -import subprocess -from collections.abc import Mapping -from pathlib import Path - - -CODEX_EXECUTABLE_RESOLUTION_SCHEMA_VERSION = "visible_codex_executable_resolution_v0" -_VERSION_PATTERN = re.compile( - r"(?\d+)\.(?P\d+)\.(?P\d+)" - r"(?:-(?P[0-9A-Za-z.-]+))?" -) - - -def _candidate_version(path: Path) -> tuple[str | None, tuple[object, ...] | None]: - try: - completed = subprocess.run( - [str(path), "--version"], - check=False, - capture_output=True, - text=True, - timeout=5.0, - ) - except (OSError, subprocess.TimeoutExpired): - return None, None - output = "\n".join(part for part in (completed.stdout, completed.stderr) if part) - match = _VERSION_PATTERN.search(output) - if not match: - return None, None - version = match.group(0) - prerelease = match.group("prerelease") - prerelease_key = tuple( - (0, int(part)) if part.isdigit() else (1, part) - for part in (prerelease or "").split(".") - if part - ) - key: tuple[object, ...] = ( - int(match.group("major")), - int(match.group("minor")), - int(match.group("patch")), - 1 if prerelease is None else 0, - prerelease_key, - ) - return version, key - - -def _append_candidate( - candidates: list[dict[str, object]], - seen: set[str], - *, - path: Path, - source: str, - public_ref: str, -) -> None: - expanded = path.expanduser() - if not expanded.is_file() or not os.access(expanded, os.X_OK): - return - identity = str(expanded.resolve()) - if identity in seen: - return - seen.add(identity) - version, version_key = _candidate_version(expanded) - candidates.append( - { - "path": str(expanded.absolute()), - "source": source, - "public_ref": public_ref, - "version": version, - "version_key": version_key, - } - ) - - -def _default_codex_candidates(env: Mapping[str, str]) -> list[dict[str, object]]: - candidates: list[dict[str, object]] = [] - seen: set[str] = set() - for index, directory in enumerate(str(env.get("PATH") or "").split(os.pathsep), start=1): - if directory: - _append_candidate( - candidates, - seen, - path=Path(directory) / "codex", - source="path", - public_ref=f"PATH candidate #{index}", - ) - - home = Path(env.get("HOME") or str(Path.home())) - _append_candidate( - candidates, - seen, - path=home / ".npm-global" / "bin" / "codex", - source="user_npm_global", - public_ref="~/.npm-global/bin/codex", - ) - npm_prefix = str(env.get("NPM_CONFIG_PREFIX") or "").strip() - if npm_prefix: - _append_candidate( - candidates, - seen, - path=Path(npm_prefix) / "bin" / "codex", - source="npm_config_prefix", - public_ref="$NPM_CONFIG_PREFIX/bin/codex", - ) - npm = shutil.which("npm", path=env.get("PATH")) - if npm: - try: - prefix = subprocess.run( - [npm, "prefix", "-g"], - check=False, - capture_output=True, - text=True, - timeout=5.0, - env=dict(env), - ).stdout.strip() - except (OSError, subprocess.TimeoutExpired): - prefix = "" - if prefix: - _append_candidate( - candidates, - seen, - path=Path(prefix) / "bin" / "codex", - source="npm_global_prefix", - public_ref="$(npm prefix -g)/bin/codex", - ) - - _append_candidate( - candidates, - seen, - path=Path("/Applications/ChatGPT.app/Contents/Resources/codex"), - source="chatgpt_app_bundle", - public_ref="ChatGPT.app bundled codex", - ) - return candidates - - -def resolve_codex_executable( - requested: str, - *, - env: Mapping[str, str] | None = None, -) -> tuple[str, dict[str, object]]: - effective_env = dict(os.environ if env is None else env) - requested_value = str(requested or "codex").strip() or "codex" - if requested_value != "codex": - resolved = shutil.which(requested_value, path=effective_env.get("PATH")) - if not resolved: - raise ValueError(f"codex_bin executable not found: {requested_value}") - path = Path(resolved).absolute() - version, _version_key = _candidate_version(path) - return str(path), { - "schema_version": CODEX_EXECUTABLE_RESOLUTION_SCHEMA_VERSION, - "requested": "explicit", - "selection_policy": "explicit_authoritative", - "selected_source": "explicit_codex_bin", - "selected_public_ref": "explicit codex_bin", - "selected_version": version, - "path_frozen": True, - "candidate_count": 1, - } - - candidates = _default_codex_candidates(effective_env) - if not candidates: - raise ValueError("codex_bin executable not found in PATH or known host locations") - versioned = [item for item in candidates if item.get("version_key") is not None] - selected = ( - max(versioned, key=lambda item: item["version_key"]) - if versioned - else candidates[0] - ) - path_default = next( - (item for item in candidates if item.get("source") == "path"), None - ) - return str(selected["path"]), { - "schema_version": CODEX_EXECUTABLE_RESOLUTION_SCHEMA_VERSION, - "requested": "default_codex", - "selection_policy": "newest_version_across_host_candidates", - "selected_source": selected["source"], - "selected_public_ref": selected["public_ref"], - "selected_version": selected["version"], - "path_default_version": path_default.get("version") if path_default else None, - "path_default_bypassed": bool( - path_default and selected["path"] != path_default["path"] - ), - "path_frozen": True, - "candidate_count": len(candidates), - "candidate_versions": [ - { - "source": item["source"], - "version": item["version"], - } - for item in candidates - ], - } - - -def write_codex_compatibility_shim(*, directory: Path, executable: str) -> Path: - directory.mkdir(parents=True, exist_ok=True) - shim = directory / "codex" - shim.write_text( - "#!/bin/sh\n" f"exec {shlex.quote(executable)} \"$@\"\n", - encoding="utf-8", - ) - shim.chmod(0o700) - return shim diff --git a/demo/multi_agent/collective_round_ledger.py b/demo/multi_agent/collective_round_ledger.py deleted file mode 100644 index 4e9e56bb3f..0000000000 --- a/demo/multi_agent/collective_round_ledger.py +++ /dev/null @@ -1,387 +0,0 @@ -from __future__ import annotations - -from collections.abc import Iterable, Mapping -from typing import Any - - -MULTI_AGENT_COLLECTIVE_ROUND_LEDGER_SCHEMA_VERSION = ( - "multi_agent_collective_round_ledger_v0" -) - - -def _string(value: object, *, default: str = "") -> str: - text = str(value or "").strip() - return text or default - - -def _bool_or_none(value: object) -> bool | None: - return value if isinstance(value, bool) else None - - -def _int_or_none(value: object) -> int | None: - if isinstance(value, bool): - return None - return value if isinstance(value, int) else None - - -def _number_or_none(value: object) -> float | None: - if isinstance(value, bool): - return None - if isinstance(value, (int, float)): - return float(value) - return None - - -def _number_list(values: object) -> list[float]: - if not isinstance(values, list): - return [] - numbers: list[float] = [] - for value in values: - number = _number_or_none(value) - if number is not None: - numbers.append(number) - return numbers - - -def _improvement_count(metrics: list[float], *, baseline: float) -> int: - previous = baseline - count = 0 - for metric in metrics: - if metric > previous: - count += 1 - previous = metric - return count - - -def _dicts(values: Iterable[object] | None) -> list[dict[str, Any]]: - return [dict(item) for item in values or [] if isinstance(item, Mapping)] - - -def _lane_key(lane: Mapping[str, object]) -> tuple[str, str, str]: - return ( - _string(lane.get("agent_id")), - _string(lane.get("lane_id")), - _string(lane.get("role_id")), - ) - - -def _normalize_lane(lane: Mapping[str, object]) -> dict[str, object]: - return { - "agent_id": _string(lane.get("agent_id")), - "lane_id": _string(lane.get("lane_id")), - "role_id": _string(lane.get("role_id")), - } - - -def _normalize_outcome(outcome: Mapping[str, object]) -> dict[str, object]: - completion_status = _string(outcome.get("completion_status")) - executed = _bool_or_none(outcome.get("executed")) - round_index = _int_or_none(outcome.get("round")) - if round_index is None: - round_index = _int_or_none(outcome.get("round_index")) - selected_todo_id = _string(outcome.get("selected_todo_id")) - selected_action = _string(outcome.get("selected_action")) - productive_turn = bool(selected_todo_id or selected_action) - return { - "round": round_index, - "agent_id": _string(outcome.get("agent_id")), - "lane_id": _string(outcome.get("lane_id")), - "role_id": _string(outcome.get("role_id")), - "selected_todo_id": selected_todo_id, - "selected_action": selected_action, - "executed": executed, - "completion_status": completion_status or None, - "completed": productive_turn and (executed is True or completion_status == "done"), - "dev_metric": _number_or_none(outcome.get("dev_metric")), - "holdout_metric": _number_or_none(outcome.get("holdout_metric")), - "appended_count": _int_or_none(outcome.get("appended_count")), - "successor_todo_declared": bool(outcome.get("successor_todo_declared")), - } - - -def _derive_lanes(outcomes: list[dict[str, object]]) -> list[dict[str, object]]: - lanes: list[dict[str, object]] = [] - seen: set[tuple[str, str, str]] = set() - for outcome in outcomes: - lane = _normalize_lane(outcome) - key = _lane_key(lane) - if key == ("", "", "") or key in seen: - continue - seen.add(key) - lanes.append(lane) - return lanes - - -def _normalize_successor(todo: Mapping[str, object]) -> dict[str, object]: - return { - "todo_id": _string(todo.get("todo_id")), - "target_agent_id": _string(todo.get("target_agent_id") or todo.get("claimed_by")), - "target_role_id": _string(todo.get("target_role_id")), - "source_todo_id": _string(todo.get("source_todo_id") or todo.get("unblocks_todo_id")), - "action_kind": _string(todo.get("action_kind")), - } - - -def _sorted_count_map(counts: Mapping[str, int]) -> dict[str, int]: - return {key: int(counts.get(key, 0)) for key in sorted(counts)} - - -def _full_participation_count_basis( - *, - synchronous_count: int, - asynchronous_count: int, - effective_count: int, -) -> str: - if effective_count == synchronous_count == asynchronous_count: - return "synchronous_and_asynchronous" - if effective_count == asynchronous_count: - return "asynchronous_role_cycle" - return "synchronous_round_index" - - -def build_multi_agent_collective_round_ledger( - *, - source: str, - expected_lanes: Iterable[object] | None = None, - lane_outcomes: Iterable[object] | None = None, - integrated_evidence: Mapping[str, object] | None = None, - role_declared_successor_todos: Iterable[object] | None = None, - baseline_metric: float | None = None, - required_full_participation_round_count: int | None = None, - required_holdout_improvement_count: int | None = None, -) -> dict[str, object]: - """Build a domain-neutral ledger for decentralized multi-agent rounds. - - A collective round is only an observation over normal LoopX state: each lane - gets a chance to read its own quota/frontier and run one bounded turn. This - helper deliberately does not select work, wake panes, write todos, or decide - promotion. Product presets may wrap the ledger with domain-specific metrics. - """ - - outcomes = [_normalize_outcome(item) for item in _dicts(lane_outcomes)] - lanes = [_normalize_lane(item) for item in _dicts(expected_lanes)] - if not lanes: - lanes = _derive_lanes(outcomes) - evidence = dict(integrated_evidence or {}) - successors = [ - _normalize_successor(item) for item in _dicts(role_declared_successor_todos) - ] - round_indexes = sorted( - { - int(outcome["round"]) - for outcome in outcomes - if isinstance(outcome.get("round"), int) and outcome.get("executed") is True - } - ) - completed_outcomes = [ - outcome for outcome in outcomes if outcome.get("completed") is True - ] - expected_agent_ids = { - str(lane.get("agent_id") or "") - for lane in lanes - if str(lane.get("agent_id") or "").strip() - } - expected_agent_id_list = sorted(expected_agent_ids) - completed_agents_by_round: dict[int, set[str]] = {} - completed_turn_count_by_agent: dict[str, int] = {} - for outcome in completed_outcomes: - round_index = outcome.get("round") - agent_id = str(outcome.get("agent_id") or "").strip() - if agent_id: - completed_turn_count_by_agent[agent_id] = ( - completed_turn_count_by_agent.get(agent_id, 0) + 1 - ) - if isinstance(round_index, int) and agent_id: - completed_agents_by_round.setdefault(round_index, set()).add(agent_id) - full_participation_round_indexes = [ - round_index - for round_index in round_indexes - if expected_agent_ids - and expected_agent_ids <= completed_agents_by_round.get(round_index, set()) - ] - asynchronous_full_participation_round_count = ( - min(completed_turn_count_by_agent.get(agent_id, 0) for agent_id in expected_agent_ids) - if expected_agent_ids - else 0 - ) - synchronous_full_participation_round_count = len(full_participation_round_indexes) - effective_full_participation_round_count = max( - synchronous_full_participation_round_count, - asynchronous_full_participation_round_count, - ) - count_basis = _full_participation_count_basis( - synchronous_count=synchronous_full_participation_round_count, - asynchronous_count=asynchronous_full_participation_round_count, - effective_count=effective_full_participation_round_count, - ) - evidence_event_count = _int_or_none(evidence.get("evidence_event_count")) - if evidence_event_count is None: - evidence_events = evidence.get("events") - evidence_event_count = len(evidence_events) if isinstance(evidence_events, list) else 0 - dev_metric = _number_or_none(evidence.get("dev_metric")) - holdout_metric = _number_or_none(evidence.get("holdout_metric")) - dev_metric_sequence = _number_list(evidence.get("dev_metric_sequence")) - holdout_metric_sequence = _number_list(evidence.get("holdout_metric_sequence")) - if not dev_metric_sequence: - dev_metric_sequence = [ - metric - for outcome in outcomes - for metric in [_number_or_none(outcome.get("dev_metric"))] - if metric is not None - ] - if not holdout_metric_sequence: - holdout_metric_sequence = [ - metric - for outcome in outcomes - for metric in [_number_or_none(outcome.get("holdout_metric"))] - if metric is not None - ] - if dev_metric is None and dev_metric_sequence: - dev_metric = dev_metric_sequence[-1] - if holdout_metric is None and holdout_metric_sequence: - holdout_metric = holdout_metric_sequence[-1] - if not dev_metric_sequence and dev_metric is not None: - dev_metric_sequence = [dev_metric] - if not holdout_metric_sequence and holdout_metric is not None: - holdout_metric_sequence = [holdout_metric] - holdout_improvement_count = _int_or_none(evidence.get("holdout_improvement_count")) - baseline = _number_or_none(baseline_metric) - if holdout_improvement_count is None and baseline is not None: - holdout_improvement_count = _improvement_count( - holdout_metric_sequence, - baseline=baseline, - ) - full_rounds_required = _int_or_none(required_full_participation_round_count) - holdout_improvements_required = _int_or_none(required_holdout_improvement_count) - full_round_requirement_met = ( - None - if full_rounds_required is None - else effective_full_participation_round_count >= full_rounds_required - ) - completed_turn_counts = _sorted_count_map( - { - agent_id: completed_turn_count_by_agent.get(agent_id, 0) - for agent_id in expected_agent_id_list - } - ) - full_round_shortfall_by_agent = ( - {} - if full_rounds_required is None - else { - agent_id: full_rounds_required - count - for agent_id, count in completed_turn_counts.items() - if count < full_rounds_required - } - ) - full_participation_requirement_gap = { - "schema_version": "multi_agent_full_participation_gap_v0", - "required_count": full_rounds_required, - "count_basis": count_basis, - "completed_turn_count_by_agent": completed_turn_counts, - "shortfall_by_agent": full_round_shortfall_by_agent, - "missing_agent_count": len(full_round_shortfall_by_agent), - "met": full_round_requirement_met, - } - holdout_improvement_requirement_met = ( - None - if holdout_improvements_required is None - else (holdout_improvement_count or 0) >= holdout_improvements_required - ) - verification_checks = [ - check - for check in ( - full_round_requirement_met, - holdout_improvement_requirement_met, - ) - if check is not None - ] - collective_research_verified = bool(verification_checks) and all(verification_checks) - return { - "schema_version": MULTI_AGENT_COLLECTIVE_ROUND_LEDGER_SCHEMA_VERSION, - "source": _string(source, default="unknown"), - "coordination_model": "decentralized_state_a2a", - "round_unit": "collective_agent_pass", - "definition": ( - "one collective round means each expected lane has one normal " - "LoopX quota/frontier/turn opportunity; pane-local tick loops are " - "evidence, not a central workflow" - ), - "expected_lanes": lanes, - "expected_lane_count": len(lanes), - "expected_agent_ids": expected_agent_id_list, - "lane_outcomes": outcomes, - "lane_outcome_count": len(outcomes), - "completed_lane_turn_count": len(completed_outcomes), - "completed_turn_count_by_agent": completed_turn_counts, - "collective_round_indexes": round_indexes, - "collective_round_count": len(round_indexes), - "full_participation_round_indexes": full_participation_round_indexes, - "synchronous_full_participation_round_count": synchronous_full_participation_round_count, - "asynchronous_full_participation_round_count": ( - asynchronous_full_participation_round_count - ), - "full_participation_round_count": effective_full_participation_round_count, - "full_participation_count_basis": count_basis, - "full_participation_requirement_gap": full_participation_requirement_gap, - "full_participation_verified": ( - bool(round_indexes) - and len(full_participation_round_indexes) == len(round_indexes) - ), - "multi_round_interaction_verified": ( - len(round_indexes) >= 2 or effective_full_participation_round_count >= 2 - ), - "integrated_evidence": { - "loaded": bool(evidence), - "evidence_event_count": evidence_event_count, - "dev_metric": dev_metric, - "holdout_metric": holdout_metric, - "dev_metric_sequence": dev_metric_sequence, - "holdout_metric_sequence": holdout_metric_sequence, - "holdout_improvement_count": holdout_improvement_count, - "result_status": _string(evidence.get("result_status")), - "protected_scope_clean": _bool_or_none(evidence.get("protected_scope_clean")), - }, - "collective_research_verification": { - "schema_version": "multi_agent_collective_research_verification_v0", - "baseline_metric": baseline, - "required_full_participation_round_count": full_rounds_required, - "required_holdout_improvement_count": holdout_improvements_required, - "full_participation_round_count": effective_full_participation_round_count, - "synchronous_full_participation_round_count": len(full_participation_round_indexes), - "asynchronous_full_participation_round_count": ( - asynchronous_full_participation_round_count - ), - "full_participation_count_basis": count_basis, - "completed_turn_count_by_agent": completed_turn_counts, - "full_participation_requirement_gap": full_participation_requirement_gap, - "full_participation_requirement_met": full_round_requirement_met, - "dev_metric_over_baseline": ( - None - if baseline is None or dev_metric is None - else dev_metric > baseline - ), - "holdout_metric_over_baseline": ( - None - if baseline is None or holdout_metric is None - else holdout_metric > baseline - ), - "holdout_improvement_count": holdout_improvement_count, - "holdout_improvement_requirement_met": holdout_improvement_requirement_met, - "verified": collective_research_verified, - }, - "role_declared_successor_todos": successors, - "successor_todo_count": len(successors), - "source_of_truth": [ - "loopx_quota_should_run", - "agent_scoped_frontier", - "todo_claims", - "public_safe_evidence", - ], - "owner_layer": "generic_multi_agent_kernel", - "public_boundary": { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, - } diff --git a/demo/multi_agent/contract.py b/demo/multi_agent/contract.py deleted file mode 100644 index 76c649b0f9..0000000000 --- a/demo/multi_agent/contract.py +++ /dev/null @@ -1,474 +0,0 @@ -from __future__ import annotations - -from collections.abc import Iterable -from pathlib import Path - - -TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION = "tui_multi_agent_runner_contract_v0" -INTERACTIVE_TUI_CONTRACT_SCHEMA_VERSION = "multi_agent_visible_interactive_tui_contract_v0" -VISIBLE_LAUNCHER_ACCEPTANCE_CONTRACT_SCHEMA_VERSION = ( - "multi_agent_visible_launcher_acceptance_contract_v0" -) -DECENTRALIZED_A2A_DRIVER_CONTRACT_SCHEMA_VERSION = ( - "multi_agent_decentralized_a2a_driver_contract_v1" -) -GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION = "generic_multi_agent_role_profile_v0" -GENERIC_MULTI_AGENT_COMPACT_STATUS_SCHEMA_VERSION = "generic_multi_agent_compact_status_v0" -THREE_LAYER_MINIMALITY_CONTRACT_SCHEMA_VERSION = ( - "multi_agent_three_layer_minimality_contract_v0" -) -GENERIC_MULTI_AGENT_DEFAULT_KERNEL_SKILLS = ("loopx-project", "loopx-doc-registry") - -PANE_LOCAL_A2A_WAKEUP_PROMPT = ( - "LoopX targeted wake. First run $LOOPX_PANE_A2A_TICK once; do not inspect skills " - "or role artifacts before this gate. The tick reads only your own LOOPX_GOAL_ID and " - "LOOPX_AGENT_ID state. If it reports runnable work, read " - "$LOOPX_CODEX_TUI_PROMPT_ARTIFACT as needed and continue your role. If not, stay " - "quiet with a brief no-action note. This wake is not research evidence or completion. " - "LoopX state, not the scheduler, decides the work." -) - -GENERIC_MULTI_AGENT_KERNEL_MECHANICS = ( - "multi_agent_runner", - "real_codex_tui_panes", - "workspace_and_trust_safe_launch", - "decentralized_a2a_driver", - "pane_local_a2a_status_check", - "todo_evidence_status_protocol", - "compact_human_status", - "default_role_prompt_scaffolding", -) - - -def positive_int_value(value: object, *, default: int) -> int: - try: - parsed = int(str(value).strip()) - except Exception: - return default - return parsed if parsed > 0 else default - - -def as_string_list(value: object) -> list[str]: - if value is None: - return [] - if isinstance(value, str): - return [value.strip()] if value.strip() else [] - if isinstance(value, Iterable): - return [str(item).strip() for item in value if str(item).strip()] - return [str(value).strip()] if str(value).strip() else [] - - -def role_skill_profile(skill: object) -> dict[str, str]: - if not skill: - return {} - if isinstance(skill, str): - source = skill.strip() - if not source: - return {} - name = Path(source).parent.name or Path(source).stem or "worker-skill" - return {"required_skill": name, "worker_skill_source": source} - if not isinstance(skill, dict): - return {} - name = str(skill.get("name") or skill.get("skill_name") or "").strip() - source = str(skill.get("source") or skill.get("path") or "").strip() - if not name and source: - name = Path(source).parent.name or Path(source).stem - if not name or not source: - return {} - return {"required_skill": name, "worker_skill_source": source} - - -def build_generic_role_profile( - *, - role_id: str, - agent_id: str, - scope: str, - responsibility: str, - handoff_hints: list[str], - skill_profile: dict[str, str], - extra_profile: object = None, -) -> dict[str, object]: - role_profile: dict[str, object] = { - "schema_version": GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, - "role_id": role_id, - "agent_id": agent_id, - "agent_scope": scope, - "responsibility": responsibility, - "handoff_hints": handoff_hints, - "default_kernel_skills": list(GENERIC_MULTI_AGENT_DEFAULT_KERNEL_SKILLS), - } - if isinstance(extra_profile, dict): - role_profile.update(extra_profile) - role_profile.update(skill_profile) - return role_profile - - -def generic_role_prompt( - *, - goal_id: str, - agent_id: str, - role_id: str, - scope: str, - handoff_hints: list[str], - skill_name: str | None, - output_language: str | None = None, -) -> str: - lines = [ - "LoopX multi-agent role", - f"Goal: {goal_id}", - f"Agent: {agent_id}", - f"Role: {role_id}", - ] - if scope: - lines.extend(["", "Scope:", scope]) - if skill_name: - lines.extend(["", "Local skill:", f"Use ${skill_name} when it applies to this role."]) - if output_language: - language_label = "Chinese" if output_language == "zh" else "English" - lines.extend( - [ - "", - "Human output language:", - ( - f"- Use {language_label} for human-readable progress, summaries, " - "and blockers in this pane." - ), - "- Keep LoopX commands, schema keys, todo ids, and artifact filenames unchanged.", - ] - ) - lines.extend( - [ - "", - "Default LoopX skills:", - "- Use $loopx-project for LoopX state, quota/frontier, todo, rationale, and repo-goal coordination.", - "- Use $loopx-doc-registry when durable project material, wiki/doc authority, or source registration matters.", - "- Keep worker-local skills focused on role semantics; generic LoopX mechanics belong to the kernel prompt and state.", - ] - ) - lines.extend( - [ - "", - "How to work:", - "- Treat LoopX state as the shared A2A surface.", - "- When this pane receives an initial or targeted wake prompt, run `$LOOPX_PANE_A2A_TICK` once to refresh guard/frontier state before deciding what to do.", - "- Treat `$LOOPX_PANE_TICK_SUMMARY` as previous pane-local evidence; it is not a gate that cancels later targeted retries.", - "- On each wake, read your own LoopX quota/frontier and run the bounded `$LOOPX_PANE_A2A_TICK` once when runnable work remains or the user asks for another round.", - "- If no runnable frontier remains, stay quiet with a brief no-action note.", - "- The tick reads your quota/frontier; it runs a role worker only when the preset explicitly configures one.", - "- If the tick says no worker is configured or manual research is required, do the role's research work visibly and write public-safe evidence/todos yourself.", - "- When a real research action completes, summarize what changed, what remains blocked, and stay interactive for user takeover.", - "- For machine reads, run `$LOOPX_PANE_LOOPX_JSON` as the command and redirect output into `$LOOPX_PANE_ARTIFACT_DIR/.public.json`; it defaults to `--format json`.", - "- Never write output to `$LOOPX_PANE_LOOPX_JSON`; it is the executable wrapper, not an artifact path.", - "- Write compact public-safe evidence before completing or handing off a todo.", - "- The user may type into this pane; respond like a normal Codex CLI agent.", - ] - ) - if handoff_hints: - lines.append("") - lines.append("Handoff hints:") - lines.extend(f"- {hint}" for hint in handoff_hints) - return "\n".join(lines) - - -def build_decentralized_a2a_driver_contract( - *, - wake_command: str = "loopx multi-agent wake --session-name ", -) -> dict[str, object]: - """Describe the reusable state-aware wake driver for live Codex TUI agents.""" - - return { - "schema_version": DECENTRALIZED_A2A_DRIVER_CONTRACT_SCHEMA_VERSION, - "owner_layer": "generic_multi_agent_kernel", - "driver_model": "todo_readiness_edge_plus_fixed_retry", - "coordination_pattern": "decentralized_state_a2a", - "prompt": { - "owner_layer": "generic_multi_agent_kernel", - "trigger": PANE_LOCAL_A2A_WAKEUP_PROMPT, - "instruction_only": True, - "frontier_embedded": False, - "todo_embedded": False, - "target_role_embedded": False, - "target_role_artifact_ref": "$LOOPX_CODEX_TUI_PROMPT_ARTIFACT", - "tick_summary_ref": "$LOOPX_PANE_TICK_SUMMARY", - "wake_round": "fresh_agent_scoped_quota_frontier_check", - "tick_summary_semantics": "prior_pane_status_not_research_evidence", - }, - "broadcaster": { - "command": wake_command, - "model": "todo_readiness_targeted_wake", - "reads_frontier": False, - "reads_todo_readiness": True, - "selects_todo": False, - "runs_worker_turn": False, - "writes_loopx_state": False, - "spends_quota": False, - "decides_work": False, - }, - "pane": { - "decision_owner": "codex_tui_agent_via_loopx_state", - "tick_command": "$LOOPX_PANE_A2A_TICK", - "first_action": "codex_tui_first_turn_status_check", - "cadence_action": "targeted_wakeup_then_own_quota_frontier_check_when_runnable", - "reads": [ - "own_LOOPX_GOAL_ID", - "own_LOOPX_AGENT_ID", - "own_quota_should_run", - "own_agent_scoped_frontier", - "own_prior_status_summary", - ], - "may_run": ["LOOPX_PANE_WORKER_TURN", "LOOPX_PANE_WORKER_LOOP"], - "writes": ["public_safe_evidence", "todo_completion_when_worker_turn_configured"], - "status_artifact": "$LOOPX_PANE_ARTIFACT_DIR/pane-a2a-status.public.json", - "counts_as_research_round": False, - }, - "layer_budget": { - "user_layer": ["goal_id", "roles", "optional_overrides"], - "preset_layer": ["domain_roles", "handoff_hints", "optional_worker_hook"], - "kernel_layer": [ - "tmux_codex_tui_lifecycle", - "todo_readiness_edge_wakeup", - "fixed_retry_wakeup", - "pane_local_status_check_runtime", - "loopx_state_protocol", - ], - }, - "acceptance": { - "broadcaster_is_not_workflow": True, - "no_leader_frontier_scan": True, - "each_pane_decides_from_state": True, - "tick_summary_does_not_gate_wake_tick": True, - "tui_first_turn_owns_tick": True, - "user_and_preset_do_not_own_tick_driver": True, - }, - } - - -def build_tui_multi_agent_runner_contract( - *, - session_name: str, - lane_count: int, - attach_command: str, - stop_command: str, - retry_command: str, - all_lane_workspace_isolation: bool, -) -> dict[str, object]: - """Describe the reusable visible-TUI runner without domain-specific steps.""" - - driver_contract = build_decentralized_a2a_driver_contract() - return { - "schema_version": TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - "runner_surface": "tmux_codex_cli_tui", - "coordination_model": { - "leader_required": False, - "state_bus": "loopx_registry_runtime_todo_quota_frontier", - "selection": "agent_scoped_quota_should_run_frontier", - "handoff": "todo_and_public_safe_evidence", - }, - "tmux_lifecycle": { - "session_name": session_name, - "lane_count": lane_count, - "one_window_per_role": False, - "single_window_tiled_role_panes": True, - "remain_on_exit": True, - "replace_existing": "execute_flag_only", - "attach_command": attach_command, - "stop_command": stop_command, - "retry_command": retry_command, - }, - "lane_runtime_env": { - "shared": [ - "LOOPX_PROJECT", - "LOOPX_REGISTRY", - "LOOPX_RUNTIME_ROOT", - "LOOPX_VISIBLE_SESSION", - "LOOPX_VISIBLE_ARTIFACT_DIR", - ], - "role_identity": [ - "LOOPX_ROLE_ID", - "LOOPX_ROLE_PROFILE_REF", - "LOOPX_ROLE_PROFILE_ARTIFACT", - "LOOPX_GOAL_ID", - "LOOPX_AGENT_ID", - ], - "pane_tools": [ - "LOOPX_PANE_LOOPX", - "LOOPX_PANE_LOOPX_JSON", - "LOOPX_PANE_ARTIFACT_DIR", - "LOOPX_PANE_A2A_TICK", - "LOOPX_PANE_BOOTSTRAP_PROMPT", - "LOOPX_PANE_TICK_SUMMARY", - "LOOPX_PANE_TICK_OUTPUT_ARTIFACT", - "LOOPX_PANE_WORKER_TURN", - "LOOPX_PANE_WORKER_LOOP", - ], - "workspace_policy": ( - "shared_goal_surface_by_default; mutating lanes may opt into lane workspaces" - ), - "all_lane_workspace_isolation": all_lane_workspace_isolation, - }, - "role_prompt_and_skill": { - "bootstrap_prompt": "written_to_public_artifact_for_targeted_wake_context", - "role_profile": "role-local public json artifact", - "default_kernel_skills": list(GENERIC_MULTI_AGENT_DEFAULT_KERNEL_SKILLS), - "default_kernel_skill_policy": { - "owner_layer": "generic_multi_agent_kernel", - "injected_via": "generic_role_prompt", - "preset_should_not_repeat_skill_playbooks": True, - "worker_skill_scope": "role_specific_semantics_and_successor_declarations", - }, - "wake_prompt_owner": "generic_multi_agent_kernel", - "skill_materialization": ".codex/skills//SKILL.md", - "worker_local_skill_scope": "role_specific_semantics_only", - }, - "decentralized_a2a_driver": driver_contract, - "pane_local_a2a": { - "tick_command": driver_contract["pane"]["tick_command"], - "first_action": ( - "Codex TUI first turn runs $LOOPX_PANE_A2A_TICK as a status check; " - "later live wakes check own quota/frontier when runnable" - ), - "cadence_wakeup_command": driver_contract["broadcaster"]["command"], - "cadence_wakeup_model": driver_contract["broadcaster"]["model"], - "cadence_broadcaster_decides_work": driver_contract["broadcaster"]["decides_work"], - "reads": ["own prior status summary", "own quota should-run", "own agent-scoped frontier"], - "runs": driver_contract["pane"]["may_run"], - "status_artifact": driver_contract["pane"]["status_artifact"], - "counts_as_research_round": driver_contract["pane"]["counts_as_research_round"], - "tick_summary": "$LOOPX_PANE_TICK_SUMMARY", - "tick_output": "$LOOPX_PANE_TICK_OUTPUT_ARTIFACT", - "machine_json_policy": "file_or_explicit_machine_channel_only", - "machine_json_destination": "$LOOPX_PANE_ARTIFACT_DIR/*.public.json", - "human_default": "markdown_status_inside_codex_tui", - }, - "user_takeover": { - "interactive_codex_tui": True, - "may_type_into_each_role": True, - "attach": attach_command, - "stop": stop_command, - }, - "debug_artifacts": { - "machine_json": "redirected_public_artifacts_only", - "launcher_scripts": "runtime_local_debug_files", - "raw_transcripts": False, - "credentials": False, - }, - "boundaries": { - "domain_specific_research_logic": False, - "runs_agent_processes_in_dry_run": False, - "writes_loopx_state_in_dry_run": False, - "spends_quota_in_dry_run": False, - }, - } - - -def build_three_layer_minimality_contract( - *, - product_id: str, - preset_id: str, - user_intent_fields: Iterable[object] | None = None, - preset_responsibilities: Iterable[object] | None = None, - preset_forbidden_mechanics: Iterable[object] | None = None, - kernel_mechanics: Iterable[object] | None = None, - extension_points: Iterable[object] | None = None, -) -> dict[str, object]: - """Describe how a product preset stays thin on the generic kernel. - - The contract is intentionally product-agnostic: callers supply the preset's - domain responsibilities, while this module records the shared mechanics that - must remain in the reusable multi-agent kernel. - """ - - intent_fields = as_string_list(user_intent_fields) or ["objective"] - preset_owns = as_string_list(preset_responsibilities) - kernel_owns = as_string_list(kernel_mechanics) or list(GENERIC_MULTI_AGENT_KERNEL_MECHANICS) - forbidden = as_string_list(preset_forbidden_mechanics) or kernel_owns - extensions = as_string_list(extension_points) - return { - "schema_version": THREE_LAYER_MINIMALITY_CONTRACT_SCHEMA_VERSION, - "product_id": product_id, - "preset_id": preset_id, - "principle": "user_and_preset_stay_thin_kernel_owns_reusable_mechanics", - "line_budget_goal": { - "user_layer": "few_field_intent_or_config", - "preset_layer": "small_domain_defaults_and_handoff_contract", - "kernel_layer": "shared_multi_agent_runtime_and_state_protocol", - }, - "user_layer": { - "owns": "intent", - "fields": intent_fields, - "forbidden": [ - "tmux_or_process_lifecycle", - "quota_frontier_protocol_details", - "pane_local_status_check_commands", - "raw_worker_or_status_plumbing", - ], - }, - "preset_layer": { - "owns": preset_owns, - "forbidden": forbidden, - "role_skill_limit": "role_specific_semantics_and_successor_declarations_only", - "must_remain_reusable": True, - }, - "kernel_layer": { - "owns": kernel_owns, - "default_skills": list(GENERIC_MULTI_AGENT_DEFAULT_KERNEL_SKILLS), - "fixed_wake_prompt": "PANE_LOCAL_A2A_WAKEUP_PROMPT", - "cross_product_reuse_required": True, - }, - "extension_points": extensions, - "acceptance": { - "user_can_copy_small_recipe": True, - "preset_has_no_runner_process_logic": True, - "kernel_contract_is_domain_agnostic": True, - "other_multi_agent_products_can_reuse_kernel": True, - }, - } - - -def build_compact_human_status(payload: dict[str, object]) -> dict[str, object]: - """Return the small status shape a product surface can show before attach.""" - - lanes = [lane for lane in payload.get("lanes", []) if isinstance(lane, dict)] - commands = payload.get("commands") if isinstance(payload.get("commands"), dict) else {} - runner = payload.get("runner_contract") if isinstance(payload.get("runner_contract"), dict) else {} - driver = ( - runner.get("decentralized_a2a_driver") - if isinstance(runner.get("decentralized_a2a_driver"), dict) - else {} - ) - session = str(payload.get("session_name") or "") - role_summaries = [] - for lane in lanes: - pane = lane.get("pane_local_a2a") if isinstance(lane.get("pane_local_a2a"), dict) else {} - role_summaries.append( - { - "lane_id": lane.get("lane_id"), - "role_id": lane.get("role_id"), - "agent_id": lane.get("agent_id"), - "scope": lane.get("agent_scope"), - "tick": pane.get("tick_command") or "$LOOPX_PANE_A2A_TICK", - "worker_turn_configured": bool(pane.get("worker_turn_configured")), - "worker_loop_configured": bool(pane.get("worker_loop_configured")), - } - ) - return { - "schema_version": GENERIC_MULTI_AGENT_COMPACT_STATUS_SCHEMA_VERSION, - "summary": ( - f"{len(role_summaries)} interactive Codex TUI role(s)" - + (f" in tmux session {session}" if session else "") - ), - "goal_id": payload.get("goal_id"), - "mode": payload.get("mode"), - "session_name": session, - "role_count": len(role_summaries), - "roles": role_summaries, - "attach": commands.get("attach"), - "stop": commands.get("stop"), - "first_action": "$LOOPX_PANE_A2A_TICK", - "driver_model": driver.get("driver_model") - or "todo_readiness_edge_plus_fixed_retry", - "coordination_pattern": driver.get("coordination_pattern") - or "decentralized_state_a2a", - "machine_json_policy": "artifact_only_in_visible_panes", - "user_takeover": "attach to the session and type into any role pane", - } diff --git a/demo/multi_agent/recipe.py b/demo/multi_agent/recipe.py deleted file mode 100644 index 7c3c5a6c3f..0000000000 --- a/demo/multi_agent/recipe.py +++ /dev/null @@ -1,110 +0,0 @@ -from __future__ import annotations - -from collections.abc import Callable, Iterable, Mapping - - -GENERIC_MULTI_AGENT_MINIMAL_RECIPE_SCHEMA_VERSION = ( - "generic_multi_agent_minimal_recipe_v0" -) - -DEFAULT_DECENTRALIZED_A2A_PROOF_CONTRACT: dict[str, object] = { - "broadcaster_selects_todo": False, - "broadcaster_runs_worker_turn": False, - "each_pane_reads_own_quota_frontier": True, - "successor_todos_declared_by_role_profile": True, - "leader_agent_required": False, -} - -DEFAULT_MULTI_AGENT_KERNEL_REUSE = ( - "generic visible multi-agent runner", - "fixed prompt broadcast", - "pane-local A2A tick", - "LoopX todo/evidence/status protocol", -) - - -def _recipe_lines(lines: Iterable[object] | None) -> list[str]: - return [str(line).strip() for line in lines or [] if str(line).strip()] - - -def build_minimal_decentralized_a2a_recipe( - *, - user_recipe_lines: Iterable[object], - preset_recipe_lines: Iterable[object], - product_id: str, - claim: str, - claim_boundary: str, - schema_version: str = GENERIC_MULTI_AGENT_MINIMAL_RECIPE_SCHEMA_VERSION, - line_unit: str = "declarative_recipe_line", - coordination_model: str = "decentralized_state_a2a", - a2a_proof_contract: Mapping[str, object] | None = None, - kernel_reuse: Iterable[object] | None = None, - shared_kernel_counted_as_recipe_lines: bool = False, -) -> dict[str, object]: - """Return the generic line-count contract for a product preset. - - Product presets provide only user and preset recipe lines. The generic - kernel owns decentralized A2A proof fields, process launch, pane-local - ticks, and LoopX state handoff mechanics. - """ - - user_lines = _recipe_lines(user_recipe_lines) - preset_lines = _recipe_lines(preset_recipe_lines) - proof = dict(a2a_proof_contract or DEFAULT_DECENTRALIZED_A2A_PROOF_CONTRACT) - kernel_lines = _recipe_lines(kernel_reuse or DEFAULT_MULTI_AGENT_KERNEL_REUSE) - return { - "schema_version": schema_version, - "product_id": str(product_id or "").strip() or "multi-agent-product", - "claim": claim, - "line_unit": line_unit, - "user_line_count": len(user_lines), - "preset_role_spec_line_count": len(preset_lines), - "user_plus_preset_line_count": len(user_lines) + len(preset_lines), - "shared_kernel_counted_as_recipe_lines": shared_kernel_counted_as_recipe_lines, - "claim_boundary": claim_boundary, - "user_recipe_lines": user_lines, - "preset_recipe_lines": preset_lines, - "coordination_model": coordination_model, - "a2a_proof_contract": proof, - "kernel_reuse": kernel_lines, - "owner_layers": { - "user_layer": "intent_lines_only", - "preset_layer": "role_defaults_and_domain_handoff_only", - "kernel_layer": "decentralized_a2a_runtime_and_state_protocol", - }, - } - - -def parse_multi_agent_role_spec_lines( - *, - agent_specs: Iterable[object] | None, - default_agent_specs: Iterable[object], - resolve_role_id: Callable[[str, int], str], - default_scope_by_lane: Mapping[str, object] | None = None, -) -> list[dict[str, str]]: - """Parse `agent_id[:lane_id[:role_id[:scope]]]` recipe lines.""" - - parsed_specs = _recipe_lines(agent_specs) or _recipe_lines(default_agent_specs) - default_scope = { - str(lane_id): str(scope) - for lane_id, scope in (default_scope_by_lane or {}).items() - if str(lane_id).strip() - } - lanes: list[dict[str, str]] = [] - for index, raw in enumerate(parsed_specs, start=1): - parts = [part.strip() for part in raw.split(":")] - if len(parts) not in {1, 2, 3, 4} or not parts[0]: - raise ValueError("agent specs must be agent_id[:lane_id[:role_id[:scope]]]") - agent_id = parts[0] - lane_id = parts[1] if len(parts) >= 2 and parts[1] else f"lane-{index}" - role_id = resolve_role_id(parts[2] if len(parts) >= 3 else "", index) - scope = parts[3] if len(parts) >= 4 and parts[3] else default_scope.get(lane_id, role_id) - lanes.append( - { - "agent_id": agent_id, - "lane_id": lane_id, - "role_id": role_id, - "scope": scope, - } - ) - return lanes diff --git a/demo/multi_agent/role_successor.py b/demo/multi_agent/role_successor.py deleted file mode 100644 index 4a054582a8..0000000000 --- a/demo/multi_agent/role_successor.py +++ /dev/null @@ -1,298 +0,0 @@ -from __future__ import annotations - -import shlex -from collections.abc import Iterable, Mapping -from pathlib import Path - -from loopx.agent_registry import registered_agent_ids_from_registry -from loopx.todos import add_goal_todo - - -MULTI_AGENT_ROLE_SUCCESSOR_TODOS_SCHEMA_VERSION = "multi_agent_role_successor_todos_v0" -ROLE_SUCCESSOR_SOURCE = "role_profile_todo_command_template" - - -def _value_at_path(payload: Mapping[str, object], path: str) -> object: - current: object = payload - for part in [part for part in path.split(".") if part]: - if not isinstance(current, Mapping) or part not in current: - return None - current = current[part] - return current - - -def _compare_condition_value(actual: object, *, op: str, expected: object) -> bool: - if op in {"truthy", "exists"}: - return bool(actual) - if op in {"falsy", "missing"}: - return not bool(actual) - if op in {"eq", "=="}: - return actual == expected - if op in {"ne", "!="}: - return actual != expected - if op in {"gt", "gte", "lt", "lte", ">", ">=", "<", "<="}: - try: - actual_number = float(actual) # type: ignore[arg-type] - expected_number = float(expected) # type: ignore[arg-type] - except (TypeError, ValueError): - return False - return { - "gt": actual_number > expected_number, - ">": actual_number > expected_number, - "gte": actual_number >= expected_number, - ">=": actual_number >= expected_number, - "lt": actual_number < expected_number, - "<": actual_number < expected_number, - "lte": actual_number <= expected_number, - "<=": actual_number <= expected_number, - }[op] - return False - - -def _successor_condition_met( - condition: object, - *, - decision_summary: Mapping[str, object], -) -> tuple[bool, str]: - if not condition: - return True, "always" - if condition == "always": - return True, "always" - if not isinstance(condition, Mapping): - return False, "unsupported_successor_condition_shape" - - context: dict[str, object] = {"decision_summary": dict(decision_summary)} - predicates = condition.get("all") - if not isinstance(predicates, list): - return False, "unsupported_successor_condition_shape" - if not predicates: - return True, "all" - - for raw_predicate in predicates: - if not isinstance(raw_predicate, Mapping): - return False, "unsupported_successor_predicate_shape" - path = str(raw_predicate.get("path") or "") - op = str(raw_predicate.get("op") or "truthy") - expected = raw_predicate.get("value") - actual = _value_at_path(context, path) - if not _compare_condition_value(actual, op=op, expected=expected): - return False, str(raw_predicate.get("fail_reason") or f"condition_failed:{path}") - return True, "condition_met" - - -def _resolve_successor_target_agent( - *, - registry_path: Path, - goal_id: str, - current_agent_id: str, - spec: Mapping[str, object], -) -> str: - registered = registered_agent_ids_from_registry(registry_path, goal_id) - target = str(spec.get("target_agent_id") or "").strip() - if target == "$current_agent": - return current_agent_id - if target and target in registered: - return target - if target: - raise ValueError(f"successor target_agent_id {target!r} is not registered for goal {goal_id!r}") - target_role_id = str(spec.get("target_role_id") or "").strip() - if target_role_id: - raise ValueError( - f"successor target_role_id {target_role_id!r} requires an explicit registered target_agent_id" - ) - for agent_id in registered: - if agent_id != current_agent_id: - return agent_id - return current_agent_id - - -class _SafeFormatDict(dict[str, object]): - def __missing__(self, key: str) -> str: - return "{" + key + "}" - - -def _render_successor_command_template( - *, - template: object, - goal_id: str, - source_todo_id: str, - target_agent_id: str, - task_class: str, - action_kind: str, - text: str, -) -> str | None: - if not template: - return None - if not isinstance(template, str): - raise ValueError("successor todo_command_template must be a string") - values: dict[str, object] = { - "goal_id": goal_id, - "goal_id_shell": shlex.quote(goal_id), - "source_todo_id": source_todo_id, - "source_todo_id_shell": shlex.quote(source_todo_id), - "target_agent_id": target_agent_id, - "target_agent_id_shell": shlex.quote(target_agent_id), - "task_class": task_class, - "task_class_shell": shlex.quote(task_class), - "action_kind": action_kind, - "action_kind_shell": shlex.quote(action_kind), - "text": text, - "text_shell": shlex.quote(text), - } - return template.format_map(_SafeFormatDict(values)) - - -def _render_successor_text( - *, - text: str, - goal_id: str, - source_todo_id: str, - target_agent_id: str, - task_class: str, - action_kind: str, -) -> str: - values: dict[str, object] = { - "goal_id": goal_id, - "source_todo_id": source_todo_id, - "target_agent_id": target_agent_id, - "task_class": task_class, - "action_kind": action_kind, - } - return text.format_map(_SafeFormatDict(values)) - - -def apply_role_successor_todos( - *, - registry_path: Path, - goal_id: str, - source_todo_id: str, - current_agent_id: str, - role_id: str, - action: str, - successor_specs: Iterable[object], - decision_summary: Mapping[str, object], - execute: bool, - text_source_todo_id: str | None = None, - expected_provider_revision: str | None = None, -) -> dict[str, object]: - """Create successors, retaining qualified historical source bindings. - - Normally one source supplies both title and lineage. A caller recovering - an existing successor may preserve its original parent while rendering - the same title from the independently verified evidence lineage. - """ - - specs = [dict(spec) for spec in successor_specs if isinstance(spec, Mapping)] - if not specs: - return { - "schema_version": MULTI_AGENT_ROLE_SUCCESSOR_TODOS_SCHEMA_VERSION, - "source": ROLE_SUCCESSOR_SOURCE, - "needed": False, - "reason": "no_role_successor_spec", - "role_id": role_id, - "action": action, - "successors": [], - } - - successors: list[dict[str, object]] = [] - skipped: list[dict[str, object]] = [] - for index, spec in enumerate(specs): - condition = spec.get("condition") or spec.get("when") or "always" - condition_met, reason = _successor_condition_met( - condition, - decision_summary=decision_summary, - ) - action_kind = str(spec.get("action_kind") or "advance_todo") - task_class = str(spec.get("task_class") or "advancement_task") - raw_text = str(spec.get("text") or f"Run {action_kind}.") - claimed_by = _resolve_successor_target_agent( - registry_path=registry_path, - goal_id=goal_id, - current_agent_id=current_agent_id, - spec=spec, - ) - text = _render_successor_text( - text=raw_text, - goal_id=goal_id, - source_todo_id=text_source_todo_id or source_todo_id, - target_agent_id=claimed_by, - task_class=task_class, - action_kind=action_kind, - ) - todo_command = _render_successor_command_template( - template=spec.get("todo_command_template"), - goal_id=goal_id, - source_todo_id=source_todo_id, - target_agent_id=claimed_by, - task_class=task_class, - action_kind=action_kind, - text=text, - ) - successor_preview = { - "index": index, - "needed": condition_met, - "executed": False, - "condition": condition, - "reason": reason, - "target_agent_id": claimed_by, - "target_role_id": spec.get("target_role_id"), - "action_kind": action_kind, - "task_class": task_class, - "unblocks_todo_id": source_todo_id, - "todo_command": todo_command, - "source": ROLE_SUCCESSOR_SOURCE, - } - if not condition_met: - skipped.append(successor_preview) - continue - if not execute: - successors.append(successor_preview) - continue - result = add_goal_todo( - registry_path=registry_path, - goal_id=goal_id, - role="agent", - text=text, - task_class=task_class, - action_kind=action_kind, - claimed_by=claimed_by, - unblocks_todo_id=source_todo_id, - expected_provider_revision=expected_provider_revision, - dry_run=False, - ) - successors.append( - successor_preview - | { - "executed": True, - "added": bool(result.get("added")), - "already_exists": bool(result.get("already_exists")), - "metadata_updated": bool(result.get("metadata_updated")), - "todo_id": result.get("todo_id"), - "claimed_by": result.get("claimed_by") or claimed_by, - "action_kind": result.get("action_kind") or action_kind, - "unblocks_todo_id": result.get("unblocks_todo_id") or source_todo_id, - } - ) - return { - "schema_version": MULTI_AGENT_ROLE_SUCCESSOR_TODOS_SCHEMA_VERSION, - "source": ROLE_SUCCESSOR_SOURCE, - "needed": bool(successors), - "executed": bool(execute and successors), - "role_id": role_id, - "action": action, - "successors": successors, - "skipped": skipped, - "reason": None if successors else (skipped[0]["reason"] if skipped else "no_role_successor_spec"), - } - - -def first_successor_followup(successor_todos: Mapping[str, object]) -> dict[str, object]: - successors = successor_todos.get("successors") - if isinstance(successors, list) and successors: - first = dict(successors[0]) - first["needed"] = True - return first - return { - "needed": False, - "reason": successor_todos.get("reason") or "no_successor", - } diff --git a/demo/multi_agent/runtime_scripts.py b/demo/multi_agent/runtime_scripts.py deleted file mode 100644 index 106dfbe624..0000000000 --- a/demo/multi_agent/runtime_scripts.py +++ /dev/null @@ -1,426 +0,0 @@ -from __future__ import annotations - - -SCOPED_LOOPX_WRAPPER_PY = r""" -import os -from pathlib import Path - -project = Path(os.environ["LOOPX_PROJECT"]) -real = os.environ["LOOPX_REAL_CLI"] -registry = os.environ["LOOPX_REGISTRY"] -runtime = os.environ["LOOPX_RUNTIME_ROOT"] -bin_dir = project / ".local" / "bin" -bin_dir.mkdir(parents=True, exist_ok=True) - -json_target = bin_dir / "loopx-json" -json_target.write_text( - "#!/usr/bin/env python3\n" - "import os, stat, sys\n" - f"real = {real!r}\n" - f"registry = {registry!r}\n" - f"runtime = {runtime!r}\n" - "explicit = os.environ.get('LOOPX_MACHINE_JSON') == '1' or os.environ.get('LOOPX_ALLOW_TTY_JSON') == '1' or os.environ.get('LOOPX_ALLOW_VISIBLE_JSON') == '1'\n" - "stdout_mode = os.fstat(sys.stdout.fileno()).st_mode\n" - "stdout_is_file = stat.S_ISREG(stdout_mode)\n" - "if not explicit and not stdout_is_file:\n" - " print('\\n[LoopX machine JSON hidden]\\nraw JSON is not printed in visible panes.\\n')\n" - " print('Use $LOOPX_PANE_LOOPX for human-readable output, or redirect machine JSON:')\n" - " print(' mkdir -p \"$LOOPX_PANE_ARTIFACT_DIR\"')\n" - " print(' $LOOPX_PANE_LOOPX_JSON > \"$LOOPX_PANE_ARTIFACT_DIR/.public.json\"')\n" - " print('$LOOPX_PANE_LOOPX_JSON is a command path, not an output file.')\n" - " print('It injects --format json unless you pass an explicit --format.')\n" - " print('Internal launcher pipes must set LOOPX_MACHINE_JSON=1 explicitly.')\n" - " raise SystemExit(2)\n" - "args = sys.argv[1:]\n" - "has_format = any(arg == '--format' or arg.startswith('--format=') for arg in args)\n" - "if not has_format:\n" - " args = ['--format', 'json'] + args\n" - "os.execv(real, [real, '--registry', registry, '--runtime-root', runtime] + args)\n", - encoding="utf-8", -) -json_target.chmod(0o700) - -human_target = bin_dir / "loopx" -human_target.write_text( - "#!/usr/bin/env python3\n" - "import os, sys\n" - f"real = {real!r}\n" - f"registry = {registry!r}\n" - f"runtime = {runtime!r}\n" - "args = sys.argv[1:]\n" - "force = os.environ.get('LOOPX_VISIBLE_FORCE_MARKDOWN', '1') != '0'\n" - "machine_json = os.environ.get('LOOPX_MACHINE_JSON') == '1'\n" - "changed = False\n" - "if force and not machine_json:\n" - " rewritten = []\n" - " index = 0\n" - " while index < len(args):\n" - " arg = args[index]\n" - " if arg == '--format' and index + 1 < len(args):\n" - " rewritten.append(arg)\n" - " value = args[index + 1]\n" - " if value == 'json':\n" - " value = 'markdown'\n" - " changed = True\n" - " rewritten.append(value)\n" - " index += 2\n" - " continue\n" - " if arg == '--format=json':\n" - " arg = '--format=markdown'\n" - " changed = True\n" - " rewritten.append(arg)\n" - " index += 1\n" - " args = rewritten\n" - "if changed:\n" - " print('\\n[LoopX human view]\\nformat=markdown; machine_json_command=$LOOPX_PANE_LOOPX_JSON artifact_dir=$LOOPX_PANE_ARTIFACT_DIR\\n', flush=True)\n" - "os.execv(real, [real, '--registry', registry, '--runtime-root', runtime] + args)\n", - encoding="utf-8", -) -human_target.chmod(0o700) - -prompt_target = bin_dir / "loopx-build-codex-bootstrap-prompt" -prompt_target.write_text( - "#!/usr/bin/env python3\n" - "import json, os, sys\n" - "from pathlib import Path\n" - "def load_json_path(value):\n" - " if not value:\n" - " return {}\n" - " try:\n" - " return json.loads(Path(value).read_text(encoding='utf-8'))\n" - " except Exception:\n" - " return {}\n" - "def compact(value, *, default=''):\n" - " text = ' '.join(str(value or '').split())\n" - " return text[:240] if text else default\n" - "def compact_list(value):\n" - " if isinstance(value, list):\n" - " return ', '.join(compact(item) for item in value if compact(item))\n" - " return compact(value)\n" - "def latest_selected_round(summary):\n" - " rounds = summary.get('rounds') if isinstance(summary, dict) else []\n" - " if not isinstance(rounds, list):\n" - " return {}\n" - " for item in reversed(rounds):\n" - " if isinstance(item, dict) and (item.get('selected_todo_id') or item.get('selected_action')):\n" - " return item\n" - " return {}\n" - "artifact = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(os.environ.get('LOOPX_CODEX_TUI_PROMPT_ARTIFACT', ''))\n" - "visible_artifact = Path(sys.argv[2]) if len(sys.argv) > 2 and sys.argv[2] else None\n" - "if not str(artifact):\n" - " raise SystemExit(2)\n" - "base = artifact.read_text(encoding='utf-8') if artifact.exists() else ''\n" - "profile = load_json_path(os.environ.get('LOOPX_ROLE_PROFILE_ARTIFACT'))\n" - "summary = load_json_path(os.environ.get('LOOPX_PANE_TICK_SUMMARY'))\n" - "round_item = latest_selected_round(summary)\n" - "language = compact(profile.get('output_language') or os.environ.get('LOOPX_AUTO_RESEARCH_OUTPUT_LANGUAGE'))\n" - "open_question = compact(profile.get('open_question'))\n" - "preset_context = profile.get('preset_context') if isinstance(profile.get('preset_context'), dict) else {}\n" - "benchmark_contract = compact(preset_context.get('benchmark_contract_file'))\n" - "editable_scope = compact_list(preset_context.get('editable_scope'))\n" - "protected_scope = compact_list(preset_context.get('protected_scope'))\n" - "dev_eval_command = compact(preset_context.get('dev_eval_command'))\n" - "holdout_eval_command = compact(preset_context.get('holdout_eval_command'))\n" - "language_note = ''\n" - "if language == 'zh':\n" - " language_note = 'use Chinese for human-readable progress; keep machine keys unchanged'\n" - "elif language:\n" - " language_note = 'use this language for human-readable progress; keep machine keys unchanged'\n" - "lines = ['', '## LoopX Agent Bootstrap Context']\n" - "lines.append('- audience: role-local Codex agent; keep user-facing summaries compact.')\n" - "lines.append(f\"- agent_id: {compact(profile.get('agent_id') or os.environ.get('LOOPX_AGENT_ID'), default='unknown')}\")\n" - "lines.append(f\"- role_id: {compact(profile.get('role_id') or os.environ.get('LOOPX_ROLE_ID'), default='unknown')}\")\n" - "if open_question:\n" - " lines.append(f'- open_question: {open_question}')\n" - "if preset_context:\n" - " lines.append(f\"- preset_context: preset={compact(preset_context.get('preset_id'), default='none')} metric={compact(preset_context.get('metric_name'), default='none')} baseline={compact(preset_context.get('baseline_metric'), default='none')}\")\n" - " claim_boundary = compact(preset_context.get('claim_boundary'))\n" - " if claim_boundary:\n" - " lines.append(f'- claim_boundary: {claim_boundary}')\n" - " if benchmark_contract or editable_scope or protected_scope or dev_eval_command or holdout_eval_command:\n" - " lines.extend(['', '### Benchmark Workspace'])\n" - " if benchmark_contract:\n" - " lines.append(f'- benchmark_contract: {benchmark_contract}')\n" - " if editable_scope:\n" - " lines.append(f'- editable_scope: {editable_scope}')\n" - " if protected_scope:\n" - " lines.append(f'- protected_scope: {protected_scope}')\n" - " if dev_eval_command:\n" - " lines.append(f'- dev_eval_command: `{dev_eval_command}`')\n" - " if holdout_eval_command:\n" - " lines.append(f'- holdout_eval_command: `{holdout_eval_command}`')\n" - "responsibility = compact(profile.get('responsibility') or profile.get('agent_scope'))\n" - "if responsibility:\n" - " lines.append(f'- responsibility: {responsibility}')\n" - "if language_note:\n" - " lines.append(f'- human_output_language: {language}; {language_note}.')\n" - "visible_first_steps = profile.get('visible_first_steps') if isinstance(profile.get('visible_first_steps'), list) else []\n" - "visible_first_steps = [compact(item) for item in visible_first_steps if compact(item)]\n" - "if visible_first_steps:\n" - " lines.extend(['', '### Role First Move'])\n" - " for item in visible_first_steps[:6]:\n" - " lines.append(f'- {item}')\n" - "continuation_policy = profile.get('continuation_policy') if isinstance(profile.get('continuation_policy'), dict) else {}\n" - "continuation_line = ''\n" - "if continuation_policy:\n" - " required_holdouts = compact(continuation_policy.get('required_holdout_improvement_count'), default='')\n" - " successor_source = compact(continuation_policy.get('successor_source'), default='role_profile.successor_todos')\n" - " if required_holdouts:\n" - " continuation_line = f'target_holdout_improvements={required_holdouts}; if unmet, use {successor_source} before no-follow-up'\n" - " else:\n" - " continuation_line = f'if acceptance target is unmet, use {successor_source} before no-follow-up'\n" - "if continuation_line:\n" - " lines.extend(['', '### Continuation Target', f'- {continuation_line}'])\n" - "lines.extend(['', '### Current Frontier'])\n" - "if round_item:\n" - " lines.append(f\"- selected_todo_id: {compact(round_item.get('selected_todo_id'), default='unknown')}\")\n" - " lines.append(f\"- selected_action: {compact(round_item.get('selected_action'), default='unknown')}\")\n" - " lines.append(f\"- worker_status: {compact(round_item.get('worker_status'), default='unknown')}\")\n" - "else:\n" - " lines.append('- selected_frontier: not read yet; this Codex TUI owns the first pane-local tick.')\n" - "lines.extend(['', '### Safety Contract'])\n" - "lines.append('- protected_scope_rule: do not edit protected evaluator/data, credentials, private material, or raw logs.')\n" - "lines.append('- claim_allowed_rule: false until role-authored public-safe evidence validates; do not promote from dev-only evidence.')\n" - "lines.append('- successor_rule: create only role-declared successor todos through normal LoopX todo commands.')\n" - "lines.extend(['', '### Visible Research Rule'])\n" - "lines.append('- tick_rule: pane-local tick reads quota/frontier/status only; it must not manufacture research conclusions.')\n" - "lines.append('- evidence_rule: write only lane-authored public-safe evidence or successor todos after doing visible role work.')\n" - "full_prompt = base.rstrip() + '\\n\\n' + '\\n'.join(lines).strip() + '\\n'\n" - "artifact.write_text(full_prompt, encoding='utf-8')\n" - "selected_todo = compact(round_item.get('selected_todo_id'), default='none') if round_item else 'none'\n" - "selected_action = compact(round_item.get('selected_action'), default='none') if round_item else 'none'\n" - "frontier_text_zh = f'{selected_todo} / {selected_action}' if round_item else 'tick 后读取；若仍为空，用 todo list/status 查本角色 open todo'\n" - "frontier_text_en = f'{selected_todo} / {selected_action}' if round_item else 'read after tick; if still empty, use todo list/status to find this role open todo'\n" - "benchmark_lines_zh = []\n" - "benchmark_lines_en = []\n" - "if benchmark_contract:\n" - " benchmark_lines_zh.append(f'- benchmark 合同: {benchmark_contract}')\n" - " benchmark_lines_en.append(f'- benchmark contract: {benchmark_contract}')\n" - "if editable_scope:\n" - " benchmark_lines_zh.append(f'- 可改文件: {editable_scope}')\n" - " benchmark_lines_en.append(f'- editable files: {editable_scope}')\n" - "if protected_scope:\n" - " benchmark_lines_zh.append(f'- 保护文件: {protected_scope}')\n" - " benchmark_lines_en.append(f'- protected files: {protected_scope}')\n" - "if dev_eval_command:\n" - " benchmark_lines_zh.append(f'- dev 验证: `{dev_eval_command}`')\n" - " benchmark_lines_en.append(f'- dev eval: `{dev_eval_command}`')\n" - "if holdout_eval_command:\n" - " benchmark_lines_zh.append(f'- held-out 验证: `{holdout_eval_command}`')\n" - " benchmark_lines_en.append(f'- held-out eval: `{holdout_eval_command}`')\n" - "continuation_lines_zh = []\n" - "continuation_lines_en = []\n" - "if continuation_line:\n" - " continuation_lines_zh.append(f'- 续轮规则: {continuation_line}')\n" - " continuation_lines_en.append(f'- continuation rule: {continuation_line}')\n" - "if visible_artifact:\n" - " first_step_lines_zh = [f'- 首轮动作: {item}' for item in visible_first_steps[:6]]\n" - " first_step_lines_en = [f'- first move: {item}' for item in visible_first_steps[:6]]\n" - " if language == 'zh':\n" - " visible_lines = [\n" - " '请用中文继续这个 LoopX auto-research pane。',\n" - " f'- 角色: {compact(profile.get(\"role_id\") or os.environ.get(\"LOOPX_ROLE_ID\"), default=\"unknown\")}',\n" - " f'- 研究问题: {open_question or \"未提供\"}',\n" - " f'- Preset: {compact(preset_context.get(\"preset_id\"), default=\"none\")} / metric={compact(preset_context.get(\"metric_name\"), default=\"none\")} / baseline={compact(preset_context.get(\"baseline_metric\"), default=\"none\")}',\n" - " f'- 当前 frontier: {frontier_text_zh}',\n" - " *benchmark_lines_zh,\n" - " *first_step_lines_zh,\n" - " *continuation_lines_zh,\n" - " '- 先运行 `$LOOPX_PANE_A2A_TICK` 一次读取本 pane 的 quota/frontier；tick 是 guard/status，不会替你生成研究结论。',\n" - " '- 若 tick 没有选中 todo，不要关闭 todo 或自行记录 gap；先用 `$LOOPX_PANE_LOOPX todo list --goal-id \"$LOOPX_GOAL_ID\"` 确认本角色是否在等待 resume gate。只有发现未 gate 的本角色 open todo 时，才继续执行上面的首轮动作；不要一上来 diagnose。',\n" - " '- tick 只读取 LoopX state；不要把 tick 产物当作研究证明。',\n" - " '- 如果 tick 显示 no_worker_command_configured 或 manual_research_required，请按本角色亲自研究并写 public-safe evidence/todo，不要把 tick 完成当作研究完成。',\n" - " '- 如果你完成了当前 selected todo 的真实动作，请用 LoopX todo/evidence 命令写回并完成该 todo；下游 resume gate 依赖 todo_done 状态进入下一轮。',\n" - " '- 完成真实研究动作后用 3-5 行中文说明：产出了什么证据、下一步能否自动做、是否需要用户 gate。',\n" - " '- 需要细节时再读取 $LOOPX_CODEX_FULL_BOOTSTRAP_ARTIFACT 和 $LOOPX_PANE_TICK_SUMMARY；不要把完整 bootstrap 合约当作用户首屏复述。',\n" - " ]\n" - " else:\n" - " visible_lines = [\n" - " 'Continue this LoopX auto-research pane.',\n" - " f'- role: {compact(profile.get(\"role_id\") or os.environ.get(\"LOOPX_ROLE_ID\"), default=\"unknown\")}',\n" - " f'- open question: {open_question or \"not provided\"}',\n" - " f'- preset: {compact(preset_context.get(\"preset_id\"), default=\"none\")} / metric={compact(preset_context.get(\"metric_name\"), default=\"none\")} / baseline={compact(preset_context.get(\"baseline_metric\"), default=\"none\")}',\n" - " f'- current frontier: {frontier_text_en}',\n" - " *benchmark_lines_en,\n" - " *first_step_lines_en,\n" - " *continuation_lines_en,\n" - " '- Run `$LOOPX_PANE_A2A_TICK` once to read this pane quota/frontier; the tick is guard/status and does not generate research conclusions for you.',\n" - " '- If the tick has no selected todo, do not close a todo or record a gap by yourself; first use `$LOOPX_PANE_LOOPX todo list --goal-id \"$LOOPX_GOAL_ID\"` to check whether this role is waiting on a resume gate. Continue with the first move above only when you find an ungated open todo for this role, and do not start with diagnose.',\n" - " '- The tick only reads LoopX state; do not treat tick output as research proof.',\n" - " '- If the tick reports no_worker_command_configured or manual_research_required, perform this role research yourself and write public-safe evidence/todos; do not treat tick completion as research completion.',\n" - " '- If you complete the real work for the current selected todo, write back through LoopX todo/evidence commands and complete that todo; downstream resume gates rely on todo_done to enter the next round.',\n" - " '- After real research work, reply with a compact 3-5 line progress/status summary: evidence produced, what can run next, and whether a user gate is needed.',\n" - " '- Read $LOOPX_CODEX_FULL_BOOTSTRAP_ARTIFACT and $LOOPX_PANE_TICK_SUMMARY only when details are needed; do not replay the full bootstrap contract as the first screen.',\n" - " ]\n" - " visible_artifact.write_text('\\n'.join(visible_lines).strip() + '\\n', encoding='utf-8')\n", - encoding="utf-8", -) -prompt_target.chmod(0o700) - -tick_target = bin_dir / "loopx-pane-a2a-tick" -tick_target.write_text( - "#!/usr/bin/env python3\n" - "import json, os, shlex, subprocess, sys\n" - "from pathlib import Path\n" - "loopx = os.environ.get('LOOPX_PANE_LOOPX') or str(Path(os.environ.get('LOOPX_PROJECT', '.')) / '.local' / 'bin' / 'loopx')\n" - "goal = os.environ.get('LOOPX_GOAL_ID', '').strip()\n" - "agent = os.environ.get('LOOPX_AGENT_ID', '').strip()\n" - "role = os.environ.get('LOOPX_ROLE_ID', '').strip() or os.environ.get('LOOPX_LANE_ID', '').strip() or agent\n" - "artifact_dir = Path(os.environ.get('LOOPX_PANE_ARTIFACT_DIR') or Path(os.environ.get('LOOPX_PROJECT', '.')) / '.local' / 'pane-artifacts' / (role or agent or 'lane'))\n" - "artifact_dir.mkdir(parents=True, exist_ok=True)\n" - "status_artifact = artifact_dir / 'pane-a2a-status.public.json'\n" - "check_records = []\n" - "def write_status_summary(*, ok, status, worker_label=None, worker_configured=False):\n" - " payload = {\n" - " 'ok': bool(ok),\n" - " 'schema_version': 'pane_local_a2a_status_check_v0',\n" - " 'source': 'pane_local_a2a_status_check',\n" - " 'goal_id': goal,\n" - " 'agent_id': agent,\n" - " 'role_id': role,\n" - " 'coordination_model': 'decentralized_state_a2a',\n" - " 'workflow_driver': False,\n" - " 'status': status,\n" - " 'status_check_count': len(check_records),\n" - " 'worker_label': worker_label,\n" - " 'worker_configured': bool(worker_configured),\n" - " 'counts_as_research_round': False,\n" - " 'checks': check_records,\n" - " 'public_boundary': {\n" - " 'raw_logs_recorded': False,\n" - " 'private_artifacts_recorded': False,\n" - " 'absolute_paths_recorded': False,\n" - " 'credentials_recorded': False,\n" - " 'local_workspace_path_redacted': True,\n" - " },\n" - " }\n" - " status_artifact.write_text(json.dumps(payload, indent=2, sort_keys=True) + '\\n', encoding='utf-8')\n" - "if not goal or not agent:\n" - " print('\\n[LoopX pane A2A]\\nmissing LOOPX_GOAL_ID or LOOPX_AGENT_ID; cannot read role-local frontier.\\n', flush=True)\n" - " write_status_summary(ok=False, status='missing_goal_or_agent')\n" - " raise SystemExit(2)\n" - "def run(args):\n" - " print('\\n[LoopX pane A2A] ' + ' '.join(shlex.quote(str(arg)) for arg in args), flush=True)\n" - " return subprocess.call([str(arg) for arg in args])\n" - "def parse_worker_payload(text):\n" - " if not text:\n" - " return {}\n" - " try:\n" - " payload = json.loads(text)\n" - " return payload if isinstance(payload, dict) else {}\n" - " except Exception:\n" - " pass\n" - " start = text.find('{')\n" - " end = text.rfind('}')\n" - " if start >= 0 and end > start:\n" - " try:\n" - " payload = json.loads(text[start:end + 1])\n" - " return payload if isinstance(payload, dict) else {}\n" - " except Exception:\n" - " return {}\n" - " return {}\n" - "def attach_worker_payload(check_record, payload):\n" - " if not payload:\n" - " return\n" - " evaluation = payload.get('evaluation_summary') if isinstance(payload.get('evaluation_summary'), dict) else {}\n" - " live = payload.get('live_evidence') if isinstance(payload.get('live_evidence'), dict) else {}\n" - " for key in ('selected_todo_id', 'selected_action', 'role_id', 'dev_metric', 'holdout_metric'):\n" - " if payload.get(key) is not None:\n" - " check_record[key] = payload.get(key)\n" - " append = payload.get('append') if isinstance(payload.get('append'), dict) else {}\n" - " if append.get('appended_count') is not None:\n" - " check_record['appended_count'] = append.get('appended_count')\n" - " if payload.get('successor_todos'):\n" - " check_record['successor_todo_declared'] = True\n" - " if evaluation.get('claim_allowed') is not None:\n" - " check_record['claim_allowed'] = bool(evaluation.get('claim_allowed'))\n" - " if evaluation.get('holdout_improvement_count') is not None:\n" - " check_record['holdout_improvement_count'] = evaluation.get('holdout_improvement_count')\n" - " if live.get('written') is not None:\n" - " check_record['live_evidence_written'] = bool(live.get('written'))\n" - "print(f'\\n[LoopX pane A2A] role={role} agent={agent}\\n', flush=True)\n" - "turn = os.environ.get('LOOPX_PANE_WORKER_TURN', '').strip()\n" - "loop = os.environ.get('LOOPX_PANE_WORKER_LOOP', '').strip()\n" - "command = turn or loop\n" - "if not command:\n" - " status = run([loopx, '--format', 'markdown', 'quota', 'should-run', '--goal-id', goal, '--agent-id', agent])\n" - " frontier_status = None\n" - " if status == 0:\n" - " frontier_status = run([loopx, '--format', 'markdown', 'auto-research', 'frontier', '--goal-id', goal, '--agent-id', agent])\n" - " check_records.append({'check_index': 1, 'quota_status': status, 'frontier_status': frontier_status, 'worker_configured': False, 'worker_executed': False, 'worker_status': None})\n" - " ok = status == 0 and frontier_status == 0\n" - " write_status_summary(ok=ok, status='no_worker_command_configured' if ok else 'frontier_failed' if status == 0 else 'quota_failed')\n" - " if status != 0:\n" - " raise SystemExit(status)\n" - " if frontier_status != 0:\n" - " raise SystemExit(frontier_status)\n" - " print('\\n[LoopX pane A2A]\\nNo automated worker-turn is configured. Use the quota/frontier above and your role profile to do real visible research, then write public-safe evidence/todos with $LOOPX_PANE_LOOPX. Do not treat this tick as research completion.\\n', flush=True)\n" - " raise SystemExit(0)\n" - "label = 'worker-turn' if turn else 'worker-loop'\n" - "status = run([loopx, '--format', 'markdown', 'quota', 'should-run', '--goal-id', goal, '--agent-id', agent])\n" - "check_record = {'check_index': 1, 'quota_status': status, 'worker_label': label, 'worker_configured': True, 'worker_executed': False, 'worker_status': None}\n" - "check_records.append(check_record)\n" - "if status != 0:\n" - " write_status_summary(ok=False, status='quota_failed', worker_label=label, worker_configured=True)\n" - " raise SystemExit(status)\n" - "print(f'\\n[LoopX pane A2A {label}]\\n{command}\\n', flush=True)\n" - "worker_env = os.environ.copy()\n" - "worker_env['LOOPX_MACHINE_JSON'] = '1'\n" - "worker = subprocess.run(command, shell=True, executable=os.environ.get('SHELL') or '/bin/bash', capture_output=True, text=True, env=worker_env)\n" - "if worker.stdout:\n" - " print(worker.stdout, end='' if worker.stdout.endswith('\\n') else '\\n', flush=True)\n" - "if worker.stderr:\n" - " print(worker.stderr, end='' if worker.stderr.endswith('\\n') else '\\n', file=sys.stderr, flush=True)\n" - "result = worker.returncode\n" - "check_record['worker_executed'] = True\n" - "check_record['worker_status'] = result\n" - "attach_worker_payload(check_record, parse_worker_payload(worker.stdout))\n" - "write_status_summary(ok=result == 0, status='completed' if result == 0 else 'worker_failed', worker_label=label, worker_configured=True)\n" - "if result != 0:\n" - " raise SystemExit(result)\n" - "raise SystemExit(0)\n", - encoding="utf-8", -) -tick_target.chmod(0o700) -""" - - -CODEX_TUI_EXEC_PY = r""" -import json -import os -import subprocess -from pathlib import Path - -codex = os.environ["LOOPX_CODEX_BIN"] -project = os.environ["LOOPX_PROJECT"] -reasoning_effort = os.environ["LOOPX_CODEX_REASONING_EFFORT"] - -args = [codex] -if os.environ.get("LOOPX_CODEX_TRUST_WORKSPACE") == "1": - candidates = [project, os.path.realpath(project)] - git_root = subprocess.run( - ["git", "-C", project, "rev-parse", "--show-toplevel"], - check=False, - capture_output=True, - text=True, - ).stdout.strip() - if git_root: - candidates.extend([git_root, os.path.realpath(git_root)]) - seen = set() - for path in candidates: - if not path or path in seen: - continue - seen.add(path) - args.extend(["-c", f"projects.{json.dumps(path)}.trust_level=\"trusted\""]) - -args.extend(["-c", f"model_reasoning_effort={reasoning_effort}", "-C", project]) -prompt_path = os.environ.get("LOOPX_CODEX_TUI_PROMPT_ARTIFACT") -if prompt_path and os.environ.get("LOOPX_CODEX_INITIAL_PROMPT_ENABLED", "1") != "0": - try: - prompt = Path(prompt_path).read_text(encoding="utf-8").strip() - except Exception: - prompt = "" - if prompt: - args.append(prompt) -os.execvp(codex, args) -""" diff --git a/demo/multi_agent/visible_launch_policy.py b/demo/multi_agent/visible_launch_policy.py deleted file mode 100644 index f7ebd2e943..0000000000 --- a/demo/multi_agent/visible_launch_policy.py +++ /dev/null @@ -1,149 +0,0 @@ -from __future__ import annotations - -import argparse -from collections.abc import Callable, Iterable -from dataclasses import dataclass -from pathlib import Path -from typing import Optional, Tuple - -from ..visible_multi_agent_launcher import wake_visible_multi_agent_panes - - -@dataclass(frozen=True) -class VisibleLaunchPolicy: - """Generic visible-launch decisions shared by product presets.""" - - launch_visible: bool - wake_visible_after_launch: bool - attach: bool - - -VisibleLauncherCallback = Callable[[dict[str, object], Path, Optional[str], Path], dict[str, object]] -VisibleLaunchExecutor = Callable[..., dict[str, object]] -VisibleWorkspacePolicy = Callable[[Path], Tuple[Optional[str], bool, bool]] - - -def resolve_visible_launch_policy( - args: argparse.Namespace, - *, - launch_visible: bool, - default_wake_allowed: bool, - default_attach_allowed: bool, - attach_wake_conflict_message: str | None = None, -) -> VisibleLaunchPolicy: - """Resolve attach/wake policy without product-specific runner logic.""" - - no_attach = bool(getattr(args, "no_attach", False)) - attach_requested = bool(getattr(args, "attach", False)) - if no_attach and attach_requested: - raise ValueError("--attach cannot be combined with --no-attach") - - wake_setting = getattr(args, "wake_visible_after_launch", None) - if wake_setting is True and attach_requested: - raise ValueError( - attach_wake_conflict_message - or "--attach cannot be combined with --wake-visible-after-launch" - ) - - wake = False - if launch_visible: - if wake_setting is True: - wake = True - elif default_wake_allowed: - wake = bool(wake_setting is not False and not attach_requested) - - attach = bool( - attach_requested - or ( - launch_visible - and default_attach_allowed - and not no_attach - and not wake - ) - ) - return VisibleLaunchPolicy( - launch_visible=bool(launch_visible), - wake_visible_after_launch=wake, - attach=attach, - ) - - -def resolve_codex_trust_workspace( - args: argparse.Namespace, - *, - launch_visible: bool, - default: bool, -) -> bool: - """Resolve the optional Codex workspace trust default for visible launches.""" - - if not launch_visible: - return False - trust_setting = getattr(args, "codex_trust_workspace", None) - if trust_setting is None: - return bool(default) - return bool(trust_setting) - - -def make_visible_launcher_callback( - *, - launch_visible: bool, - launch_executor: VisibleLaunchExecutor, - launcher: str, - tmux_bin: str, - cli_bin: str, - codex_bin: str, - attach: bool, - replace_existing: bool, - workspace_policy: VisibleWorkspacePolicy, - auto_wake: bool = False, - auto_wake_interval_seconds: float = 45.0, -) -> VisibleLauncherCallback | None: - """Create the generic callback shape expected by visible demo runners.""" - - if not launch_visible: - return None - - def visible_launcher( - payload: dict[str, object], - registry_path: Path, - runtime_root_arg: str | None, - default_workspace: Path, - ) -> dict[str, object]: - workspace, create_workspace, codex_trust_workspace = workspace_policy( - default_workspace - ) - return launch_executor( - payload, - registry_path=registry_path, - runtime_root_arg=runtime_root_arg, - launcher=launcher, - tmux_bin=tmux_bin, - cli_bin=cli_bin, - codex_bin=codex_bin, - attach=attach, - replace_existing=replace_existing, - workspace=workspace, - create_workspace=create_workspace, - codex_trust_workspace=codex_trust_workspace, - auto_wake=auto_wake, - auto_wake_interval_seconds=auto_wake_interval_seconds, - ) - - return visible_launcher - - -def make_visible_wake_callback( - *, - tmux_bin: str, -) -> Callable[[str, list[str]], dict[str, object]]: - """Return a pane-local A2A wake callback for visible launcher integrations.""" - - def visible_wake(session: str, lanes: Iterable[str]) -> dict[str, object]: - return wake_visible_multi_agent_panes( - session_name=session, - tmux_bin=tmux_bin, - lanes=lanes, - execute=True, - ) - - return visible_wake diff --git a/demo/multi_agent/visible_wake_scheduler.py b/demo/multi_agent/visible_wake_scheduler.py deleted file mode 100644 index 2c0bef25ab..0000000000 --- a/demo/multi_agent/visible_wake_scheduler.py +++ /dev/null @@ -1,392 +0,0 @@ -from __future__ import annotations - -import argparse -import json -import subprocess -import time -from collections.abc import Iterable, Mapping -from datetime import datetime, timezone -from pathlib import Path -from typing import Any - - -STATE_AWARE_WAKE_MODEL = "todo_readiness_edge_plus_fixed_retry" -DEFAULT_READINESS_POLL_SECONDS = 2.0 - - -def lane_agent_map(lanes: Iterable[Mapping[str, object]]) -> dict[str, str]: - result: dict[str, str] = {} - for lane in lanes: - lane_id = str(lane.get("lane_id") or "").strip() - agent_id = str(lane.get("agent_id") or lane_id).strip() - if lane_id: - result[lane_id] = agent_id - return result - - -def todo_readiness_projection( - payload: Mapping[str, object], - *, - lane_agents: Mapping[str, str], -) -> dict[str, object]: - todos = payload.get("todos") - if not isinstance(todos, list): - return { - "recognized": False, - "reason": "todo_projection_missing", - "mapped_todo_count": 0, - "runnable_lane_tokens": {}, - } - - lanes_by_agent: dict[str, list[str]] = {} - for lane_id, agent_id in lane_agents.items(): - lanes_by_agent.setdefault(agent_id, []).append(lane_id) - - goal_todo_count = sum(1 for item in todos if isinstance(item, Mapping)) - mapped_count = 0 - runnable: dict[str, list[str]] = {} - for raw_item in todos: - if not isinstance(raw_item, Mapping): - continue - claimed_by = str(raw_item.get("claimed_by") or "").strip() - raw_excluded = raw_item.get("excluded_agents") or [] - excluded_values = ( - raw_excluded - if isinstance(raw_excluded, (list, tuple, set)) - else [raw_excluded] - ) - excluded = { - str(value).strip() for value in excluded_values if str(value).strip() - } - target_lanes = lanes_by_agent.get(claimed_by, []) - if target_lanes: - mapped_count += 1 - elif not claimed_by and str( - raw_item.get("task_class") or "advancement_task" - ).strip() == "advancement_task": - target_lanes = [ - lane_id - for lane_id, agent_id in lane_agents.items() - if agent_id not in excluded - ] - else: - continue - if ( - raw_item.get("done") is True - or str(raw_item.get("status") or "open").strip().lower() != "open" - or raw_item.get("resume_ready") is False - or claimed_by in excluded - ): - continue - todo_id = str(raw_item.get("todo_id") or raw_item.get("index") or "").strip() - for lane_id in target_lanes: - runnable.setdefault(lane_id, []).append(todo_id or "open") - - tokens = { - lane_id: tuple(sorted(todo_ids)) - for lane_id, todo_ids in runnable.items() - if todo_ids - } - return { - "recognized": goal_todo_count > 0, - "reason": "goal_todo_projection" if goal_todo_count else "no_goal_todos", - "goal_todo_count": goal_todo_count, - "mapped_todo_count": mapped_count, - "runnable_lane_tokens": tokens, - } - - -def read_todo_readiness( - *, - cli_bin: str, - registry: Path, - runtime_root: Path, - goal_id: str, - lane_agents: Mapping[str, str], - timeout_seconds: float = 10.0, -) -> dict[str, object]: - command = [ - cli_bin, - "--format", - "json", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "todo", - "list", - "--goal-id", - goal_id, - "--role", - "agent", - ] - try: - completed = subprocess.run( - command, - check=False, - capture_output=True, - text=True, - timeout=timeout_seconds, - ) - except (OSError, subprocess.TimeoutExpired) as exc: - return { - "recognized": False, - "reason": "todo_projection_command_failed", - "error": str(exc), - "mapped_todo_count": 0, - "runnable_lane_tokens": {}, - } - if completed.returncode != 0: - return { - "recognized": False, - "reason": "todo_projection_command_failed", - "error": (completed.stderr or completed.stdout).strip()[:500], - "mapped_todo_count": 0, - "runnable_lane_tokens": {}, - } - try: - payload = json.loads(completed.stdout) - except json.JSONDecodeError as exc: - return { - "recognized": False, - "reason": "todo_projection_invalid_json", - "error": str(exc), - "mapped_todo_count": 0, - "runnable_lane_tokens": {}, - } - if not isinstance(payload, Mapping): - return { - "recognized": False, - "reason": "todo_projection_invalid_payload", - "mapped_todo_count": 0, - "runnable_lane_tokens": {}, - } - return todo_readiness_projection(payload, lane_agents=lane_agents) - - -def resolve_initial_wake_plan( - *, - lanes: Iterable[Mapping[str, object]], - cli_bin: str, - registry: Path, - runtime_root: Path, - goal_id: str, -) -> dict[str, object]: - agents = lane_agent_map(lanes) - projection = ( - read_todo_readiness( - cli_bin=cli_bin, - registry=registry, - runtime_root=runtime_root, - goal_id=goal_id, - lane_agents=agents, - ) - if goal_id and agents - else { - "recognized": False, - "reason": "goal_or_lane_agent_mapping_missing", - "mapped_todo_count": 0, - "runnable_lane_tokens": {}, - } - ) - tokens = projection.get("runnable_lane_tokens", {}) - return { - "lane_agents": agents, - "projection": projection, - "state_aware": bool(projection.get("recognized")), - "target_lanes": list(tokens) if isinstance(tokens, Mapping) else [], - } - - -def _jsonable_tokens(tokens: Mapping[str, tuple[str, ...]]) -> dict[str, list[str]]: - return {lane_id: list(value) for lane_id, value in tokens.items()} - - -def _append_event(path: Path, payload: Mapping[str, object]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - event = { - "recorded_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), - **payload, - } - with path.open("a", encoding="utf-8") as handle: - handle.write(json.dumps(event, ensure_ascii=False, sort_keys=True) + "\n") - - -def run_state_aware_wake_loop( - *, - cli_bin: str, - tmux_bin: str, - registry: Path, - runtime_root: Path, - goal_id: str, - session: str, - lane_agents: Mapping[str, str], - initial_projection: Mapping[str, object], - artifact: Path, - retry_interval_seconds: float, - poll_interval_seconds: float = DEFAULT_READINESS_POLL_SECONDS, -) -> None: - all_lanes = list(lane_agents) - state_aware = bool(initial_projection.get("recognized")) - previous_tokens = { - str(lane_id): tuple(str(item) for item in token) - for lane_id, token in ( - initial_projection.get("runnable_lane_tokens", {}) - if isinstance(initial_projection.get("runnable_lane_tokens"), Mapping) - else {} - ).items() - if isinstance(token, (list, tuple)) - } - now = time.monotonic() - last_wake = {lane_id: now for lane_id in previous_tokens} - projection_error_active = False - _append_event( - artifact, - { - "event": "scheduler_started", - "goal_id": goal_id, - "mode": STATE_AWARE_WAKE_MODEL if state_aware else "fixed_retry_fallback", - "runnable_lane_tokens": _jsonable_tokens(previous_tokens), - }, - ) - - retry_interval = max(5.0, float(retry_interval_seconds or 0)) - poll_interval = max(0.5, float(poll_interval_seconds or 0)) - while subprocess.run( - [tmux_bin, "has-session", "-t", session], - check=False, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ).returncode == 0: - time.sleep(poll_interval) - projection = read_todo_readiness( - cli_bin=cli_bin, - registry=registry, - runtime_root=runtime_root, - goal_id=goal_id, - lane_agents=lane_agents, - ) - if projection.get("recognized"): - state_aware = True - projection_error_active = False - raw_tokens = projection.get("runnable_lane_tokens", {}) - if not isinstance(raw_tokens, Mapping): - raw_tokens = {} - current_tokens = { - str(lane_id): tuple(str(item) for item in token) - for lane_id, token in raw_tokens.items() - if isinstance(token, (list, tuple)) - } - elif state_aware: - if not projection_error_active: - _append_event( - artifact, - { - "event": "projection_read_failed", - "reason": projection.get("reason"), - "error": projection.get("error"), - }, - ) - projection_error_active = True - continue - else: - current_tokens = {lane_id: ("fallback",) for lane_id in all_lanes} - - now = time.monotonic() - edge_lanes = [ - lane_id - for lane_id, token in current_tokens.items() - if token != previous_tokens.get(lane_id) - ] - retry_lanes = [ - lane_id - for lane_id in current_tokens - if lane_id not in edge_lanes - and now - last_wake.get(lane_id, 0.0) >= retry_interval - ] - targets = [*edge_lanes, *retry_lanes] - previous_tokens = current_tokens - if not targets: - continue - - command = [ - cli_bin, - "--format", - "json", - "multi-agent", - "wake", - "--session-name", - session, - "--tmux-bin", - tmux_bin, - ] - for lane_id in targets: - command.extend(["--lane", lane_id]) - command.append("--execute") - completed = subprocess.run(command, check=False, capture_output=True, text=True) - try: - wake_payload: Any = json.loads(completed.stdout) - except json.JSONDecodeError: - wake_payload = { - "ok": False, - "returncode": completed.returncode, - "error": (completed.stderr or completed.stdout).strip()[:500], - } - _append_event( - artifact, - { - "event": "wake", - "trigger": "readiness_edge" if edge_lanes else "fixed_retry", - "target_lanes": targets, - "runnable_lane_tokens": _jsonable_tokens(current_tokens), - "wake": wake_payload, - }, - ) - for lane_id in targets: - last_wake[lane_id] = now - if isinstance(wake_payload, Mapping) and wake_payload.get( - "auto_wake_backoff_recommended" - ) is True: - break - - -def main() -> int: - parser = argparse.ArgumentParser(add_help=False) - parser.add_argument("--cli-bin", required=True) - parser.add_argument("--tmux-bin", required=True) - parser.add_argument("--registry", type=Path, required=True) - parser.add_argument("--runtime-root", type=Path, required=True) - parser.add_argument("--goal-id", required=True) - parser.add_argument("--session", required=True) - parser.add_argument("--lane-agents-json", required=True) - parser.add_argument("--initial-projection-json", required=True) - parser.add_argument("--artifact", type=Path, required=True) - parser.add_argument("--retry-interval-seconds", type=float, required=True) - parser.add_argument( - "--poll-interval-seconds", - type=float, - default=DEFAULT_READINESS_POLL_SECONDS, - ) - args = parser.parse_args() - lane_agents = json.loads(args.lane_agents_json) - initial_projection = json.loads(args.initial_projection_json) - if not isinstance(lane_agents, dict) or not isinstance(initial_projection, dict): - raise ValueError("state-aware wake scheduler requires object projections") - run_state_aware_wake_loop( - cli_bin=args.cli_bin, - tmux_bin=args.tmux_bin, - registry=args.registry, - runtime_root=args.runtime_root, - goal_id=args.goal_id, - session=args.session, - lane_agents=lane_agents, - initial_projection=initial_projection, - artifact=args.artifact, - retry_interval_seconds=args.retry_interval_seconds, - poll_interval_seconds=args.poll_interval_seconds, - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/demo/multi_agent_cli.py b/demo/multi_agent_cli.py deleted file mode 100644 index 84bfea6dd9..0000000000 --- a/demo/multi_agent_cli.py +++ /dev/null @@ -1,213 +0,0 @@ -from __future__ import annotations - -import argparse -import json -from collections.abc import Callable -from pathlib import Path - -from loopx.history import load_registry -from loopx.paths import resolve_runtime_root -from .visible_multi_agent_launcher import ( - build_visible_multi_agent_payload_from_spec, - execute_visible_multi_agent_launcher, -) -from .visible_multi_agent_tmux import ( - wake_visible_multi_agent_panes, -) - - -PrintPayload = Callable[ - [dict[str, object], str, Callable[[dict[str, object]], str]], - None, -] -FormatSelector = Callable[..., str] -AddFormat = Callable[[argparse.ArgumentParser], None] - - -def register_multi_agent_commands( - subparsers: argparse._SubParsersAction, - add_subcommand_format: AddFormat, -) -> None: - parser = subparsers.add_parser( - "multi-agent", - help="Launch visible Codex TUI agents from a small role spec.", - ) - multi_agent_sub = parser.add_subparsers(dest="multi_agent_command", required=True) - launch = multi_agent_sub.add_parser( - "launch", - help="Preview or launch a tmux session with one interactive Codex TUI per role.", - ) - add_subcommand_format(launch) - launch.add_argument("--spec", required=True, help="Path to a generic_multi_agent_launch_spec_v0 JSON file.") - launch.add_argument("--execute", action="store_true", help="Start the visible TUI session. Omit for preview.") - launch.add_argument("--workspace", help="Workspace root for all lanes. Defaults to the current directory.") - launch.add_argument("--create-workspace", action="store_true", help="Create --workspace if it is missing.") - launch.add_argument("--session-name", help="Override spec.session_name.") - launch.add_argument("--reasoning-effort", help="Override spec.default_reasoning_effort.") - launch.add_argument("--launcher", choices=["auto", "tmux"], default="tmux") - launch.add_argument("--tmux-bin", default="tmux") - launch.add_argument("--cli-bin", default="loopx") - launch.add_argument("--codex-bin", default="codex") - launch.add_argument("--attach", action="store_true", help="Attach to the session after launch.") - launch.add_argument("--replace-existing", action="store_true", help="Replace an existing session of the same name.") - launch.add_argument( - "--auto-wake", - action=argparse.BooleanOptionalAction, - default=False, - help=( - "Run a session-scoped background wake loop. The loop only broadcasts " - "the fixed pane-local A2A prompt; each pane still reads its own state." - ), - ) - launch.add_argument( - "--auto-wake-interval-seconds", - type=float, - default=45.0, - help=argparse.SUPPRESS, - ) - launch.add_argument( - "--codex-trust-workspace", - action=argparse.BooleanOptionalAction, - default=False, - help="Pass a trusted project config to each Codex TUI lane.", - ) - wake = multi_agent_sub.add_parser( - "wake", - help="Broadcast the fixed pane-local A2A prompt to live visible Codex TUI panes.", - ) - add_subcommand_format(wake) - wake.add_argument("--session-name", required=True, help="tmux session to wake.") - wake.add_argument( - "--lane", - action="append", - default=[], - help="Specific pane/window name to wake. Repeat to target multiple panes. Omit with --execute to list the session windows.", - ) - wake.add_argument("--tmux-bin", default="tmux") - wake.add_argument("--execute", action="store_true", help="Send the fixed prompt to tmux panes.") - - -def _load_spec(path: Path) -> dict[str, object]: - with path.open(encoding="utf-8") as handle: - value = json.load(handle) - if not isinstance(value, dict): - raise ValueError("multi-agent spec must be a JSON object") - return value - - -def _render_multi_agent_launch_markdown(payload: dict[str, object]) -> str: - if not payload.get("ok"): - return f"LoopX multi-agent launch failed: {payload.get('error')}" - lanes = [lane for lane in payload.get("lanes", []) if isinstance(lane, dict)] - lines = [ - "# LoopX Multi-Agent Launch", - f"- mode: {payload.get('mode')}", - f"- goal: {payload.get('goal_id')}", - f"- session: {payload.get('session_name')}", - f"- roles: {len(lanes)}", - ] - for lane in lanes: - lines.append( - f"- {lane.get('lane_id')}: {lane.get('role_id')} " - f"({lane.get('agent_id')})" - ) - commands = payload.get("commands") if isinstance(payload.get("commands"), dict) else {} - if commands.get("attach"): - lines.append("") - lines.append(f"attach: `{commands.get('attach')}`") - launch_result = payload.get("launch_result") - if isinstance(launch_result, dict): - accepted = ( - launch_result.get("visible_acceptance", {}) - if isinstance(launch_result.get("visible_acceptance"), dict) - else {} - ) - lines.append("") - lines.append(f"visible accepted: {accepted.get('accepted')}") - lines.append(f"started lanes: {', '.join(str(item) for item in launch_result.get('started_lanes', []))}") - return "\n".join(lines) - - -def _render_multi_agent_wake_markdown(payload: dict[str, object]) -> str: - if not payload.get("ok"): - return f"LoopX multi-agent wake failed: {payload.get('error')}" - lines = [ - "# LoopX Multi-Agent Wake", - f"- mode: {payload.get('mode')}", - f"- session: {payload.get('session_name')}", - f"- wakeup_model: {payload.get('wakeup_model')}", - f"- coordination_model: {payload.get('coordination_model')}", - f"- workflow_driver: {payload.get('workflow_driver')}", - f"- target_lanes: {', '.join(str(item) for item in payload.get('target_lanes') or [])}", - ] - return "\n".join(lines) - - -def handle_multi_agent_command( - args: argparse.Namespace, - *, - registry_path: Path, - runtime_root_arg: str | None, - output_format: FormatSelector, - print_payload: PrintPayload, -) -> int | None: - if args.command != "multi-agent": - return None - try: - if args.multi_agent_command == "wake": - payload = wake_visible_multi_agent_panes( - session_name=args.session_name, - tmux_bin=args.tmux_bin, - lanes=args.lane, - execute=bool(args.execute), - ) - print_payload(payload, output_format(args), _render_multi_agent_wake_markdown) - return 0 - if args.multi_agent_command != "launch": - raise ValueError("multi-agent requires the `launch` or `wake` subcommand") - spec_path = Path(args.spec).expanduser() - spec = _load_spec(spec_path) - if args.session_name: - spec = {**spec, "session_name": args.session_name} - if args.reasoning_effort: - spec = {**spec, "default_reasoning_effort": args.reasoning_effort} - payload = build_visible_multi_agent_payload_from_spec( - spec, - tmux_bin=args.tmux_bin, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - ) - if args.execute: - registry = load_registry(registry_path) - runtime_root = resolve_runtime_root(registry, runtime_root_arg) - launch_result, chosen, workspace_mode = execute_visible_multi_agent_launcher( - payload=payload, - registry=registry_path, - runtime_root=runtime_root, - requested_launcher=args.launcher, - tmux_bin=args.tmux_bin, - cli_bin=args.cli_bin, - codex_bin=args.codex_bin, - attach=bool(args.attach), - replace_existing=bool(args.replace_existing), - workspace=args.workspace, - create_workspace=bool(args.create_workspace), - cwd=Path.cwd(), - codex_trust_workspace=bool(args.codex_trust_workspace), - source_root=spec_path.parent, - auto_wake=bool(args.auto_wake), - auto_wake_interval_seconds=args.auto_wake_interval_seconds, - ) - payload["mode"] = "execute" - payload["chosen_launcher"] = chosen - payload["workspace_mode"] = workspace_mode - payload["launch_result"] = launch_result - payload["boundary"] = { - **payload["boundary"], - "starts_visible_processes": True, - "runs_agent_processes": True, - } - except Exception as exc: - payload = {"ok": False, "mode": "multi-agent", "error": str(exc)} - print_payload(payload, output_format(args), _render_multi_agent_launch_markdown) - return 0 if payload.get("ok") else 1 diff --git a/demo/visible_multi_agent_launcher.py b/demo/visible_multi_agent_launcher.py deleted file mode 100644 index ca516b2101..0000000000 --- a/demo/visible_multi_agent_launcher.py +++ /dev/null @@ -1,1253 +0,0 @@ -from __future__ import annotations - -import json -import os -import re -import shlex -import shutil -import subprocess -import time -from collections.abc import Iterable -from pathlib import Path - -from .multi_agent.contract import ( - GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, - INTERACTIVE_TUI_CONTRACT_SCHEMA_VERSION, - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - VISIBLE_LAUNCHER_ACCEPTANCE_CONTRACT_SCHEMA_VERSION, - as_string_list as _as_string_list, - build_compact_human_status, - build_generic_role_profile, - build_tui_multi_agent_runner_contract, - generic_role_prompt as _generic_role_prompt, - role_skill_profile as _role_skill_profile, -) -from .multi_agent.codex_executable import ( - resolve_codex_executable, - write_codex_compatibility_shim, -) -from .multi_agent.runtime_scripts import ( - CODEX_TUI_EXEC_PY as _CODEX_TUI_EXEC_PY, - SCOPED_LOOPX_WRAPPER_PY as _SCOPED_LOOPX_WRAPPER_PY, -) -from .multi_agent.visible_wake_scheduler import ( - DEFAULT_READINESS_POLL_SECONDS, - STATE_AWARE_WAKE_MODEL, - resolve_initial_wake_plan, -) -from .visible_multi_agent_tmux import ( - PANE_A2A_INPUT_READY_TIMEOUT_SECONDS, # noqa: F401 - PANE_A2A_WAKEUP_PROMPT, # noqa: F401 - PANE_A2A_WAKEUP_SCHEMA_VERSION, # noqa: F401 - TMUX_LANE_ID_OPTION, - build_pane_a2a_wakeup_prompt, # noqa: F401 - wake_visible_multi_agent_panes, # noqa: F401 -) - - -def _q(value: object) -> str: - return shlex.quote(str(value)) - - -_WORKER_COMMAND_RE = re.compile(r"^[A-Za-z0-9_./:+-]+(?: [A-Za-z0-9_./:+-]+)*$") - - -def validate_worker_command(command: str, *, field: str) -> str: - """Return a worker command that is safe to pass through a shell env var. - - The command is executed by the pane worker through a shell, so it must be - a plain executable plus space-separated safe arguments: no quotes, command - substitution, pipes, redirects, or other shell metacharacters. - """ - - if command is None: - return "" - text = str(command or "").strip() - if not text: - return "" - if not _WORKER_COMMAND_RE.match(text): - raise ValueError( - f"{field} contains unsafe shell metacharacters: {text!r}" - ) - return text - - -AUTO_WAKE_LOOP_SCHEMA_VERSION = "multi_agent_auto_wake_loop_v0" - - -def require_executable(command: str, *, field: str) -> str: - path = shutil.which(command) - if not path: - raise ValueError(f"{field} executable not found on PATH: {command}") - return path - - -def _codex_config_path() -> Path: - return Path(os.environ.get("CODEX_HOME") or (Path.home() / ".codex")) / "config.toml" - - -def _toml_project_header(path: str) -> str: - return f"[projects.{json.dumps(path)}]" - - -def _trust_project_in_config_text(text: str, path: str) -> str: - header = _toml_project_header(path) - lines = text.splitlines() - for index, line in enumerate(lines): - if line.strip() != header: - continue - end = index + 1 - while end < len(lines) and not lines[end].lstrip().startswith("["): - end += 1 - for trust_index in range(index + 1, end): - if lines[trust_index].strip().startswith("trust_level"): - lines[trust_index] = 'trust_level = "trusted"' - break - else: - lines.insert(index + 1, 'trust_level = "trusted"') - return "\n".join(lines) + "\n" - suffix = "" if not text or text.endswith("\n") else "\n" - return text + suffix + f"\n{header}\ntrust_level = \"trusted\"\n" - - -def _codex_trust_candidate_paths(project: Path) -> list[str]: - candidates = [str(project), os.path.realpath(project)] - git_root = subprocess.run( - ["git", "-C", str(project), "rev-parse", "--show-toplevel"], - check=False, - capture_output=True, - text=True, - ).stdout.strip() - if git_root: - candidates.extend([git_root, os.path.realpath(git_root)]) - trusted: list[str] = [] - seen = set() - for path in candidates: - if path and path not in seen: - seen.add(path) - trusted.append(path) - return trusted - - -def _persist_codex_workspace_trust(paths: Iterable[str]) -> dict[str, object]: - config = _codex_config_path() - config.parent.mkdir(parents=True, exist_ok=True) - before = config.read_text(encoding="utf-8") if config.exists() else "" - after = before - trusted_paths = [] - for path in paths: - trusted_paths.append(path) - after = _trust_project_in_config_text(after, path) - if after != before: - config.write_text(after, encoding="utf-8") - return { - "persisted": after != before, - "trusted_path_count": len(trusted_paths), - } - - -def runtime_shell_command( - command: str, - *, - project: Path, - registry: Path, - runtime_root: Path, - visible_session: str | None = None, - errexit: bool = True, -) -> str: - parts = [ - "set -euo pipefail" if errexit else "set -uo pipefail", - f"export LOOPX_PROJECT={_q(project)}", - f"export LOOPX_REGISTRY={_q(registry)}", - f"export LOOPX_RUNTIME_ROOT={_q(runtime_root)}", - ] - if visible_session is not None: - parts.append(f"export LOOPX_VISIBLE_SESSION={_q(visible_session)}") - parts.extend( - [ - 'export LOOPX_VISIBLE_ARTIFACT_DIR="${LOOPX_VISIBLE_ARTIFACT_DIR:-$LOOPX_RUNTIME_ROOT/visible-launcher-artifacts/${LOOPX_VISIBLE_SESSION:-default}}"', - 'mkdir -p "$LOOPX_VISIBLE_ARTIFACT_DIR"', - command, - ] - ) - return "; ".join(parts) - - -def resolve_visible_workspace( - workspace: str | None, - *, - create: bool, - cwd: Path, -) -> tuple[Path, str]: - if not workspace: - return cwd.resolve(), "current_directory" - path = Path(workspace).expanduser() - if not path.is_absolute(): - path = cwd / path - if not path.exists(): - if not create: - raise ValueError("workspace does not exist; pass --create-workspace to create it") - path.mkdir(parents=True, exist_ok=True) - if not path.is_dir(): - raise ValueError("workspace must be a directory") - return path.resolve(), "explicit_workspace" - - -def resolve_visible_launcher(*, requested: str, tmux_bin: str) -> str: - if requested not in {"auto", "tmux"}: - raise ValueError("only the tmux visible launcher is supported") - require_executable(tmux_bin, field="tmux_bin") - return "tmux" - - -def build_visible_lane_command( - *, - role_id: str, - role_profile_ref: str, - role_profile_command: str, - bootstrap_command: str, - cli_bin: str, - codex_bin: str, - reasoning_effort: str, - goal_id: str | None = None, - agent_id: str | None = None, - worker_turn_command: str | None = None, - worker_loop_command: str | None = None, - visible_lane_count: int | None = None, -) -> str: - codex_exec_env = ( - 'export LOOPX_CODEX_BIN="${LOOPX_RESOLVED_CODEX_BIN:-}"; ' - 'if [ -z "$LOOPX_CODEX_BIN" ]; then ' - f"export LOOPX_CODEX_BIN={_q(codex_bin)}; " - "fi; " - f"export LOOPX_CODEX_REASONING_EFFORT={_q(reasoning_effort)}; " - ) - scoped_loopx_wrapper = ( - f"LOOPX_REAL_CLI=\"$(command -v {_q(cli_bin)})\"; " - "export LOOPX_REAL_CLI; " - 'mkdir -p "$LOOPX_PROJECT/.local/bin"; ' - f"python3 -c {_q(_SCOPED_LOOPX_WRAPPER_PY)}; " - 'chmod +x "$LOOPX_PROJECT/.local/bin/loopx"; ' - 'chmod +x "$LOOPX_PROJECT/.local/bin/loopx-json"; ' - 'chmod +x "$LOOPX_PROJECT/.local/bin/loopx-pane-a2a-tick"; ' - 'chmod +x "$LOOPX_PROJECT/.local/bin/loopx-build-codex-bootstrap-prompt"; ' - 'export LOOPX_PANE_LOOPX="$LOOPX_PROJECT/.local/bin/loopx"; ' - 'export LOOPX_PANE_LOOPX_JSON="$LOOPX_PROJECT/.local/bin/loopx-json"; ' - 'export LOOPX_PANE_A2A_TICK="$LOOPX_PROJECT/.local/bin/loopx-pane-a2a-tick"; ' - 'export LOOPX_PANE_BOOTSTRAP_PROMPT="$LOOPX_PROJECT/.local/bin/loopx-build-codex-bootstrap-prompt"; ' - 'export LOOPX_VISIBLE_FORCE_MARKDOWN="${LOOPX_VISIBLE_FORCE_MARKDOWN:-1}"; ' - 'export PATH="$LOOPX_PROJECT/.local/bin:$PATH"; ' - ) - pane_a2a_env = "" - if goal_id: - pane_a2a_env += f"export LOOPX_GOAL_ID={_q(goal_id)}; " - if agent_id: - pane_a2a_env += f"export LOOPX_AGENT_ID={_q(agent_id)}; " - if worker_turn_command: - pane_a2a_env += f"export LOOPX_PANE_WORKER_TURN={_q(worker_turn_command)}; " - if worker_loop_command: - pane_a2a_env += f"export LOOPX_PANE_WORKER_LOOP={_q(worker_loop_command)}; " - if visible_lane_count and visible_lane_count > 0: - pane_a2a_env += f"export LOOPX_VISIBLE_LANE_COUNT={_q(visible_lane_count)}; " - return ( - "set -uo pipefail; " - "export LOOPX_VISIBLE_TUI_SILENT_BOOTSTRAP=1; " - f"export LOOPX_ROLE_ID={_q(role_id)}; " - f"export LOOPX_ROLE_PROFILE_REF={_q(role_profile_ref)}; " - 'export LOOPX_PANE_ARTIFACT_DIR="${LOOPX_PANE_ARTIFACT_DIR:-$LOOPX_VISIBLE_ARTIFACT_DIR/${LOOPX_ROLE_ID:-lane}}"; ' - 'mkdir -p "$LOOPX_PANE_ARTIFACT_DIR"; ' - 'cd "$LOOPX_PROJECT"; ' - f"{scoped_loopx_wrapper}" - f"{pane_a2a_env}" - f"{role_profile_command}" - 'VISIBLE_ARTIFACT_PREFIX="${LOOPX_LANE_ID:-${LOOPX_ROLE_ID:-lane}}"; ' - f"BOOTSTRAP_PROMPT=\"$({bootstrap_command} 2>&1)\"; " - "BOOTSTRAP_STATUS=$?; " - "export BOOTSTRAP_PROMPT; " - 'BOOTSTRAP_ARTIFACT="$LOOPX_VISIBLE_ARTIFACT_DIR/$VISIBLE_ARTIFACT_PREFIX.bootstrap-prompt.public.txt"; ' - 'printf "%s\\n" "$BOOTSTRAP_PROMPT" > "$BOOTSTRAP_ARTIFACT"; ' - "if [ \"$BOOTSTRAP_STATUS\" -ne 0 ]; then " - "printf '\\n[LoopX blocked reason]\\n'; " - "printf 'bootstrap_command_failed exit=%s\\n' \"$BOOTSTRAP_STATUS\"; " - "exec /bin/sh -i; " - "fi; " - 'export LOOPX_PANE_TICK_SUMMARY="$LOOPX_PANE_ARTIFACT_DIR/pane-a2a-status.public.json"; ' - 'export LOOPX_PANE_TICK_OUTPUT_ARTIFACT="$LOOPX_PANE_ARTIFACT_DIR/pane-a2a-tick.output.txt"; ' - 'VISIBLE_PROMPT_ARTIFACT="$LOOPX_PANE_ARTIFACT_DIR/codex-visible-first-prompt.public.txt"; ' - 'export LOOPX_CODEX_FULL_BOOTSTRAP_ARTIFACT="$BOOTSTRAP_ARTIFACT"; ' - '"$LOOPX_PANE_BOOTSTRAP_PROMPT" "$BOOTSTRAP_ARTIFACT" "$VISIBLE_PROMPT_ARTIFACT"; ' - "export LOOPX_CODEX_TUI_MODE=interactive; " - "export LOOPX_CODEX_TUI_PROMPT_ARTIFACT=\"$VISIBLE_PROMPT_ARTIFACT\"; " - f"{codex_exec_env}" - f"exec python3 -c {_q(_CODEX_TUI_EXEC_PY)}" - ) - - -def build_visible_multi_agent_payload( - *, - goal_id: str, - session_name: str, - lanes: Iterable[dict[str, object]], - tmux_bin: str = "tmux", - schema_version: str = "multi_agent_visible_launcher_v0", -) -> dict[str, object]: - lane_list = [lane for lane in lanes if isinstance(lane, dict)] - if not lane_list: - raise ValueError("visible multi-agent launcher has no lanes") - session = str(session_name or "loopx-visible-agents") - attach_command = f"{_q(tmux_bin)} attach -t {_q(session)}" - stop_command = f"{_q(tmux_bin)} kill-session -t {_q(session)}" - window_name = "four-up" if len(lane_list) == 4 else "roles" - retry_command = ( - "rerun the same visible launcher packet after refreshing quota, " - "frontier, and bootstrap" - ) - all_lane_workspace_isolation = all( - bool(lane.get("workspace") or lane.get("project")) for lane in lane_list - ) - runner_contract = build_tui_multi_agent_runner_contract( - session_name=session, - lane_count=len(lane_list), - attach_command=attach_command, - stop_command=stop_command, - retry_command=retry_command, - all_lane_workspace_isolation=all_lane_workspace_isolation, - ) - start_script = [ - "set -uo pipefail", - ": ${LOOPX_PROJECT:?set LOOPX_PROJECT to the repo root before running}", - ": ${LOOPX_REGISTRY:?set LOOPX_REGISTRY to the LoopX registry path before running}", - ": ${LOOPX_RUNTIME_ROOT:?set LOOPX_RUNTIME_ROOT to the LoopX runtime root before running}", - f"export LOOPX_VISIBLE_SESSION={_q(session)}", - 'export LOOPX_VISIBLE_ARTIFACT_DIR="${LOOPX_VISIBLE_ARTIFACT_DIR:-$LOOPX_RUNTIME_ROOT/visible-launcher-artifacts/$LOOPX_VISIBLE_SESSION}"', - 'mkdir -p "$LOOPX_VISIBLE_ARTIFACT_DIR"', - ] - for index, lane in enumerate(lane_list): - lane_id = str(lane.get("lane_id") or "agent-lane") - launch_command = str(lane.get("visible_launch_command") or "") - if not launch_command: - raise ValueError(f"lane {lane_id} is missing visible_launch_command") - if index == 0: - start_script.append( - f"{_q(tmux_bin)} new-session -d -s {_q(session)} " - f"-n {_q(window_name)} bash -lc {_q(launch_command)}" - ) - start_script.append( - f"{_q(tmux_bin)} select-pane -t {_q(session + ':' + window_name)} " - f"-T {_q(lane_id)}" - ) - start_script.append( - f"{_q(tmux_bin)} set-option -p -t {_q(session + ':' + window_name)} " - f"{_q(TMUX_LANE_ID_OPTION)} {_q(lane_id)}" - ) - start_script.append(f"{_q(tmux_bin)} set-option -t {_q(session)} remain-on-exit on") - else: - pane_var = f"LOOPX_TMUX_PANE_{index}" - start_script.append( - f"{pane_var}=\"$({_q(tmux_bin)} split-window -d -P " - f"-F '#{{pane_id}}' -t {_q(session + ':' + window_name)} " - f"bash -lc {_q(launch_command)})\"" - ) - start_script.append( - f"{_q(tmux_bin)} select-pane -t \"${pane_var}\" -T {_q(lane_id)}" - ) - start_script.append( - f"{_q(tmux_bin)} set-option -p -t \"${pane_var}\" " - f"{_q(TMUX_LANE_ID_OPTION)} {_q(lane_id)}" - ) - start_script.append(f"{_q(tmux_bin)} select-layout -t {_q(session + ':' + window_name)} tiled") - start_script.append( - f"{_q(tmux_bin)} display-message -t {_q(session)} " - f"{_q('LoopX visible multi-agent Codex TUI session started; each window is interactive')}" - ) - return { - "ok": True, - "schema_version": schema_version, - "mode": "dry_run", - "goal_id": str(goal_id), - "session_name": session, - "lanes": lane_list, - "runner_contract": runner_contract, - "interactive_tui_contract": { - "schema_version": INTERACTIVE_TUI_CONTRACT_SCHEMA_VERSION, - "runner_contract": TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - "human_pane": [ - "codex_cli_tui", - "role_prompt_inside_codex", - "normal_user_typing", - "normal_codex_tool_output", - "takeover_controls", - ], - "machine_artifacts": [ - "quota.public.json", - "frontier.public.json", - "bootstrap-prompt.public.txt", - "role_local_public_artifacts", - ], - "machine_json_policy": "file_or_explicit_machine_channel_only", - "visible_json_policy": "not_printed_before_tui", - "codex_surface": "interactive_cli_tui", - "forbidden_visible_content": [ - "raw_quota_json", - "raw_frontier_json", - "raw_role_profile_json", - "pre_codex_character_stream", - "credentials", - "raw_private_logs", - "absolute_local_artifact_paths", - ], - }, - "commands": { - "start_script": start_script, - "attach": attach_command, - "stop": stop_command, - "retry": retry_command, - }, - "acceptance": { - "schema_version": VISIBLE_LAUNCHER_ACCEPTANCE_CONTRACT_SCHEMA_VERSION, - "required_runtime_shape": [ - "single_tmux_window_with_role_panes", - "recording_friendly_tiled_layout", - "each_role_pane_execs_codex_cli_tui", - "no_frontier_or_json_status_window", - "no_pre_codex_character_stream", - ], - "interactive_tui_contract": INTERACTIVE_TUI_CONTRACT_SCHEMA_VERSION, - "runner_contract": TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - "machine_json_file_bound": True, - "codex_tui_interactive": True, - }, - "boundary": { - "starts_visible_processes": False, - "runs_agent_processes": False, - "writes_loopx_state": False, - "spends_loopx_quota": False, - "reads_raw_transcripts": False, - "reads_session_files": False, - "reads_credentials": False, - "hidden_prompt_injection": False, - "shared_goal_surface": True, - "all_lane_workspace_isolation": all_lane_workspace_isolation, - "public_safe_redaction": True, - }, - } - - -def build_visible_multi_agent_payload_from_spec( - spec: dict[str, object], - *, - tmux_bin: str = "tmux", - cli_bin: str = "loopx", - codex_bin: str = "codex", -) -> dict[str, object]: - """Build a visible launcher packet from a small user-facing role spec.""" - - if not isinstance(spec, dict): - raise ValueError("multi-agent launch spec must be an object") - goal_id = str(spec.get("goal_id") or "").strip() - if not goal_id: - raise ValueError("multi-agent launch spec requires goal_id") - roles = spec.get("roles") or spec.get("agents") or spec.get("lanes") - if not isinstance(roles, list) or not roles: - raise ValueError("multi-agent launch spec requires a non-empty roles list") - session_name = str(spec.get("session_name") or f"loopx-{_script_slug(goal_id)}-agents") - default_reasoning_effort = str( - spec.get("default_reasoning_effort") or spec.get("reasoning_effort") or "high" - ) - lanes: list[dict[str, object]] = [] - for index, raw_role in enumerate(roles, start=1): - if not isinstance(raw_role, dict): - raise ValueError(f"role #{index} must be an object") - agent_id = str(raw_role.get("agent_id") or "").strip() - if not agent_id: - raise ValueError(f"role #{index} requires agent_id") - lane_id = str(raw_role.get("lane_id") or raw_role.get("role_id") or agent_id).strip() - role_id = str(raw_role.get("role_id") or lane_id).strip() - scope = str( - raw_role.get("scope") - or raw_role.get("agent_scope") - or raw_role.get("responsibility") - or "" - ).strip() - responsibility = str(raw_role.get("responsibility") or scope or role_id).strip() - role_profile_ref = str( - raw_role.get("role_profile_ref") - or f"generic_multi_agent_launch_spec_v0:{role_id}" - ).strip() - handoff_hints = _as_string_list( - raw_role.get("handoff") - or raw_role.get("handoff_hints") - or raw_role.get("interaction_hints") - ) - skill_profile = _role_skill_profile(raw_role.get("skill")) - reasoning_effort = str(raw_role.get("reasoning_effort") or default_reasoning_effort) - worker_turn_command = validate_worker_command( - raw_role.get("worker_turn_command"), - field="worker_turn_command", - ) - worker_loop_command = validate_worker_command( - raw_role.get("worker_loop_command"), - field="worker_loop_command", - ) - output_language = str( - raw_role.get("output_language") - or ( - raw_role.get("role_profile", {}).get("output_language") - if isinstance(raw_role.get("role_profile"), dict) - else "" - ) - or "" - ).strip() - role_profile = build_generic_role_profile( - role_id=role_id, - agent_id=agent_id, - scope=scope, - responsibility=responsibility, - handoff_hints=handoff_hints, - skill_profile=skill_profile, - extra_profile=raw_role.get("role_profile"), - ) - lane_slug = _script_slug(lane_id) - role_profile_json = json.dumps(role_profile, ensure_ascii=False, sort_keys=True) - role_profile_command = ( - "mkdir -p \"$LOOPX_VISIBLE_ARTIFACT_DIR\"; " - f"export LOOPX_ROLE_PROFILE_ARTIFACT=\"$LOOPX_VISIBLE_ARTIFACT_DIR/{lane_slug}.role-profile.public.json\"; " - f"printf '%s\\n' {_q(role_profile_json)} " - "> \"$LOOPX_ROLE_PROFILE_ARTIFACT\"; " - ) - prompt = _generic_role_prompt( - goal_id=goal_id, - agent_id=agent_id, - role_id=role_id, - scope=scope, - handoff_hints=handoff_hints, - skill_name=skill_profile.get("required_skill"), - output_language=output_language or None, - ) - bootstrap_command = f"printf '%s\\n' {_q(prompt)}" - lane = { - "lane_id": lane_id, - "agent_id": agent_id, - "role_id": role_id, - "responsibility": responsibility, - "agent_scope": scope, - "handoff_hints": handoff_hints, - "role_profile": role_profile, - "role_profile_ref": role_profile_ref, - "output_language": output_language or None, - "quota_guard": ( - "mkdir -p \"$LOOPX_PANE_ARTIFACT_DIR\" && " - "$LOOPX_PANE_LOOPX_JSON quota should-run " - f"--goal-id {_q(goal_id)} --agent-id {_q(agent_id)} " - "--runtime-profile codex_cli " - "> \"$LOOPX_PANE_ARTIFACT_DIR/quota.public.json\"" - ), - "frontier": "agent-scoped LoopX todo/quota/frontier projection", - "pane_local_a2a": { - "tick_command": "$LOOPX_PANE_A2A_TICK", - "worker_turn_configured": bool(worker_turn_command), - "worker_loop_configured": bool(worker_loop_command), - "auto_start": True, - "auto_start_owner": "codex_tui_first_turn_prompt", - "status_check_only": True, - "counts_as_research_round": False, - }, - "bootstrap_message": "role_prompt_public_artifact_for_first_turn_and_fixed_wake", - "visible_launch_command": build_visible_lane_command( - role_id=role_id, - role_profile_ref=role_profile_ref, - role_profile_command=role_profile_command, - bootstrap_command=bootstrap_command, - cli_bin=cli_bin, - codex_bin=codex_bin, - reasoning_effort=reasoning_effort, - goal_id=goal_id, - agent_id=agent_id, - worker_turn_command=worker_turn_command, - worker_loop_command=worker_loop_command, - visible_lane_count=len(roles), - ), - "reasoning_effort": reasoning_effort, - "lane_timeline": [ - "role_profile", - "codex_tui", - "tui_first_turn_quota_frontier_status_check", - "frontier", - ], - } - if raw_role.get("workspace") or raw_role.get("project"): - lane["workspace"] = str(raw_role.get("workspace") or raw_role.get("project")) - lanes.append(lane) - - packet = build_visible_multi_agent_payload( - goal_id=goal_id, - session_name=session_name, - lanes=lanes, - tmux_bin=tmux_bin, - ) - packet.update( - { - "product_spec": { - "schema_version": "generic_multi_agent_launch_spec_v0", - "input_shape": ["goal_id", "session_name", "roles"], - "role_fields": [ - "agent_id", - "role_id", - "scope", - "skill", - "handoff_hints", - "output_language", - "reasoning_effort", - "worker_turn_command", - "worker_loop_command", - ], - "role_count": len(lanes), - "role_profile_schema_version": GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, - "uses_generic_runner": True, - "domain_specific": False, - }, - "reasoning_contract": { - "default_reasoning_effort": default_reasoning_effort, - "codex_cli_config_key": "model_reasoning_effort", - }, - "shared_goal_surface": { - "shared_goal_id": goal_id, - "shared_state_route": "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", - "shared_frontier": True, - "lane_identity_source": "role_profile_plus_agent_scoped_quota", - "all_lane_workspace_isolation": any("workspace" in lane for lane in lanes), - "mutation_isolation_policy": ( - "mutating attempts require agent-scoped todo/frontier and a claimed " - "worktree or equivalent execution boundary" - ), - }, - "cli_contract": { - "cli_bin": cli_bin, - "codex_bin": codex_bin, - "tmux_bin": tmux_bin, - "machine_json_policy": "artifact_only_in_visible_panes", - }, - } - ) - packet["compact_human_status"] = build_compact_human_status(packet) - return packet - - -def _materialize_worker_skills( - *, - payload: dict[str, object], - project: Path, - source_root: Path, -) -> list[dict[str, object]]: - results: list[dict[str, object]] = [] - lanes = [item for item in payload.get("lanes", []) if isinstance(item, dict)] - for lane in lanes: - profile = lane.get("role_profile") - if not isinstance(profile, dict): - continue - skill_name = str(profile.get("required_skill") or "").strip() - source_name = str(profile.get("worker_skill_source") or "").strip() - if not skill_name or not source_name: - continue - source, source_resolution = _resolve_worker_skill_source( - source_name, - source_root=source_root, - ) - workspace_values = [project] - lane_workspace = _lane_workspace(lane, default_project=project) - if lane_workspace != project: - workspace_values.append(lane_workspace) - item = { - "skill": skill_name, - "source": source_name, - "destination": f".codex/skills/{skill_name}/SKILL.md", - "materialized": False, - "workspace_count": len(workspace_values), - "source_resolution": source_resolution, - } - if source.is_file(): - for workspace in workspace_values: - destination = workspace / ".codex" / "skills" / skill_name / "SKILL.md" - destination.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(source, destination) - item["materialized"] = True - else: - item["missing_source"] = True - results.append(item) - return results - - -def _resolve_worker_skill_source(source_name: str, *, source_root: Path) -> tuple[Path, str]: - source = Path(source_name) - if source.is_absolute(): - return source, "absolute" - - package_root = Path(__file__).resolve().parents[1] - candidates = [ - ("source_root", source_root / source), - ("package_root", package_root / source), - ("module_root", Path(__file__).resolve().parent / source), - ] - seen: set[Path] = set() - for label, candidate in candidates: - resolved = candidate.resolve() - if resolved in seen: - continue - seen.add(resolved) - if resolved.is_file(): - return resolved, label - return (source_root / source), "missing" - - -def _worker_skill_materialization_errors(items: list[dict[str, object]]) -> list[str]: - errors: list[str] = [] - for item in items: - if item.get("missing_source"): - errors.append(f"{item.get('skill')}: missing {item.get('source')}") - elif item and not item.get("materialized"): - errors.append(f"{item.get('skill')}: not materialized") - return errors - - -def _lane_workspace(lane: dict[str, object], *, default_project: Path) -> Path: - raw = lane.get("workspace") or lane.get("project") - if not raw: - return default_project - path = Path(str(raw)).expanduser() - if not path.is_absolute(): - path = default_project / path - return path.resolve() - - -def _script_slug(value: str) -> str: - slug = "".join(ch if ch.isalnum() or ch in {"-", "_"} else "-" for ch in value) - slug = "-".join(part for part in slug.split("-") if part) - return (slug or "lane")[:80] - - -def _write_tmux_script(*, script_dir: Path, name: str, command: str) -> Path: - script_dir.mkdir(parents=True, exist_ok=True) - script = script_dir / f"{_script_slug(name)}.sh" - script.write_text(f"#!/usr/bin/env bash\n{command}\n", encoding="utf-8") - script.chmod(0o700) - return script - - -def _start_auto_wake_loop( - *, - script_dir: Path, - project: Path, - registry: Path, - runtime_root: Path, - session: str, - lane_agents: dict[str, str], - goal_id: str, - initial_projection: dict[str, object], - tmux_bin: str, - cli_bin: str, - interval_seconds: float, - env: dict[str, str], -) -> dict[str, object]: - interval = max(5.0, float(interval_seconds or 0)) - artifact = ( - runtime_root - / "visible-launcher-artifacts" - / _script_slug(session) - / "auto-wake.public.jsonl" - ) - wake_command = ( - "exec python3 -m demo.multi_agent.visible_wake_scheduler " - f"--cli-bin {_q(cli_bin)} --tmux-bin {_q(tmux_bin)} " - f"--registry {_q(registry)} --runtime-root {_q(runtime_root)} " - f"--goal-id {_q(goal_id)} --session {_q(session)} " - f"--lane-agents-json {_q(json.dumps(lane_agents, sort_keys=True))} " - f"--initial-projection-json {_q(json.dumps(initial_projection, sort_keys=True))} " - f"--artifact {_q(artifact)} " - f"--retry-interval-seconds {_q(f'{interval:g}')} " - f"--poll-interval-seconds {_q(f'{DEFAULT_READINESS_POLL_SECONDS:g}')}" - ) - script = _write_tmux_script( - script_dir=script_dir, - name="auto-wake-loop", - command=runtime_shell_command( - wake_command, - project=project, - registry=registry, - runtime_root=runtime_root, - visible_session=session, - errexit=False, - ), - ) - subprocess.Popen( - ["bash", str(script)], - env=env, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - start_new_session=True, - ) - return { - "schema_version": AUTO_WAKE_LOOP_SCHEMA_VERSION, - "enabled": True, - "mode": "background_process", - "interval_seconds": interval, - "target_lanes": list(lane_agents), - "wakeup_model": ( - STATE_AWARE_WAKE_MODEL - if initial_projection.get("recognized") - else "fixed_prompt_broadcast_fallback" - ), - "workflow_driver": False, - "broadcaster_reads_frontier": False, - "broadcaster_reads_todo_readiness": True, - "broadcaster_selects_todo": False, - "pane_decision_owner": "codex_tui_agent_via_loopx_state", - "artifact": "auto-wake.public.jsonl", - "artifact_path": str(artifact), - "readiness_poll_seconds": DEFAULT_READINESS_POLL_SECONDS, - "initial_projection": initial_projection, - "process_started": True, - "pid_recorded": False, - "boundary": { - "writes_loopx_state": False, - "spends_loopx_quota": False, - "reads_raw_transcripts": False, - "reads_credentials": False, - "runs_worker_turn_directly": False, - }, - } - - -def execute_visible_multi_agent_launcher( - *, - payload: dict[str, object], - registry: Path, - runtime_root: Path, - requested_launcher: str, - tmux_bin: str, - cli_bin: str, - codex_bin: str, - attach: bool, - replace_existing: bool, - workspace: str | None, - create_workspace: bool, - cwd: Path, - codex_trust_workspace: bool = False, - source_root: Path | None = None, - auto_wake: bool = False, - auto_wake_interval_seconds: float = 45.0, - launch_result_schema: str = "multi_agent_visible_launch_result_v0", - acceptance_schema: str = "multi_agent_visible_launch_acceptance_v0", - lane_default: str = "agent-lane", -) -> tuple[dict[str, object], str, str]: - require_executable(cli_bin, field="cli_bin") - resolved_codex_bin, codex_resolution = resolve_codex_executable(codex_bin) - chosen = resolve_visible_launcher(requested=requested_launcher, tmux_bin=tmux_bin) - project, workspace_mode = resolve_visible_workspace(workspace, create=create_workspace, cwd=cwd) - worker_skills = _materialize_worker_skills( - payload=payload, - project=project, - source_root=source_root or cwd, - ) - worker_skill_errors = _worker_skill_materialization_errors(worker_skills) - if worker_skill_errors: - raise ValueError( - "worker-local skill materialization failed: " - + "; ".join(worker_skill_errors) - ) - result = _launch_with_tmux( - payload=payload, - project=project, - workspace_mode=workspace_mode, - registry=registry, - runtime_root=runtime_root, - tmux_bin=tmux_bin, - cli_bin=cli_bin, - codex_bin=resolved_codex_bin, - attach=attach, - replace_existing=replace_existing, - launch_result_schema=launch_result_schema, - acceptance_schema=acceptance_schema, - lane_default=lane_default, - codex_trust_workspace=codex_trust_workspace, - auto_wake=auto_wake, - auto_wake_interval_seconds=auto_wake_interval_seconds, - ) - result["worker_skill_materialization"] = worker_skills - result["codex_executable_resolution"] = codex_resolution - return result, chosen, workspace_mode - - -def _launch_with_tmux( - *, - payload: dict[str, object], - project: Path, - workspace_mode: str, - registry: Path, - runtime_root: Path, - tmux_bin: str, - cli_bin: str, - codex_bin: str, - attach: bool, - replace_existing: bool, - launch_result_schema: str, - acceptance_schema: str, - lane_default: str, - codex_trust_workspace: bool, - auto_wake: bool, - auto_wake_interval_seconds: float, -) -> dict[str, object]: - session = str(payload.get("session_name") or "loopx-visible-agents") - lanes = [item for item in payload.get("lanes", []) if isinstance(item, dict)] - if not lanes: - raise ValueError("visible multi-agent launcher has no lanes to launch") - codex_trust_config: dict[str, object] = { - "persisted": False, - "trusted_path_count": 0, - } - if codex_trust_workspace: - trust_paths: list[str] = [] - for lane in lanes: - trust_paths.extend( - _codex_trust_candidate_paths(_lane_workspace(lane, default_project=project)) - ) - codex_trust_config = _persist_codex_workspace_trust(trust_paths) - - env = os.environ.copy() - env.update( - { - "LOOPX_PROJECT": str(project), - "LOOPX_REGISTRY": str(registry), - "LOOPX_RUNTIME_ROOT": str(runtime_root), - "LOOPX_VISIBLE_SESSION": session, - "LOOPX_CODEX_TRUST_WORKSPACE": "1" if codex_trust_workspace else "0", - } - ) - exists = subprocess.run( - [tmux_bin, "has-session", "-t", session], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - check=False, - env=env, - ) - if exists.returncode == 0: - if not replace_existing: - raise ValueError( - f"tmux session already exists: {session}; use --replace-existing or attach manually" - ) - subprocess.run([tmux_bin, "kill-session", "-t", session], check=True, env=env) - - script_dir = runtime_root / "visible-launcher" / _script_slug(session) - codex_shim = write_codex_compatibility_shim( - directory=script_dir / "resolved-codex-bin", - executable=codex_bin, - ) - goal_id = str(payload.get("goal_id") or "").strip() - initial_wake_plan = resolve_initial_wake_plan( - lanes=lanes, - cli_bin=cli_bin, - registry=registry, - runtime_root=runtime_root, - goal_id=goal_id, - ) - lane_agents = dict(initial_wake_plan["lane_agents"]) - initial_projection = dict(initial_wake_plan["projection"]) - initial_runnable_lanes = set(initial_wake_plan["target_lanes"]) - state_aware_initial_prompt = bool(initial_wake_plan["state_aware"]) - started_lanes = [] - lane_targets: dict[str, str] = {} - launcher_scripts: dict[str, str] = {} - window_name = "four-up" if len(lanes) == 4 else "roles" - for index, lane in enumerate(lanes): - lane_id = str(lane.get("lane_id") or lane_default) - launch_command = str(lane.get("visible_launch_command") or "") - if not launch_command: - raise ValueError(f"lane {lane_id} is missing visible_launch_command") - lane_project = _lane_workspace(lane, default_project=project) - if not lane_project.is_dir(): - raise ValueError(f"lane {lane_id} workspace does not exist") - lane_script = _write_tmux_script( - script_dir=script_dir, - name=lane_id, - command=runtime_shell_command( - f"export LOOPX_CODEX_TRUST_WORKSPACE={_q('1' if codex_trust_workspace else '0')}; " - f"export LOOPX_RESOLVED_CODEX_BIN={_q(codex_bin)}; " - f"export PATH={_q(str(codex_shim.parent))}:\"$PATH\"; " - "export LOOPX_CODEX_INITIAL_PROMPT_ENABLED=" - f"{_q('1' if not state_aware_initial_prompt or lane_id in initial_runnable_lanes else '0')}; " - f"export LOOPX_VISIBLE_LANE_COUNT={_q(len(lanes))}; " - f"{launch_command}", - project=lane_project, - registry=registry, - runtime_root=runtime_root, - visible_session=session, - errexit=False, - ), - ) - if index == 0: - subprocess.run( - [ - tmux_bin, - "new-session", - "-d", - "-s", - session, - "-n", - window_name, - "bash", - str(lane_script), - ], - check=True, - env=env, - ) - subprocess.run( - [tmux_bin, "set-option", "-t", session, "remain-on-exit", "on"], - check=False, - env=env, - ) - pane_id = subprocess.run( - [tmux_bin, "display-message", "-p", "-t", f"{session}:{window_name}", "#{pane_id}"], - check=False, - capture_output=True, - text=True, - env=env, - ).stdout.strip() or f"{session}:{window_name}" - else: - pane_id = subprocess.run( - [ - tmux_bin, - "split-window", - "-d", - "-P", - "-F", - "#{pane_id}", - "-t", - f"{session}:{window_name}", - "bash", - str(lane_script), - ], - check=True, - capture_output=True, - text=True, - env=env, - ).stdout.strip() or f"{session}:{window_name}" - subprocess.run( - [tmux_bin, "select-pane", "-t", pane_id, "-T", lane_id], - check=False, - env=env, - ) - subprocess.run( - [tmux_bin, "set-option", "-p", "-t", pane_id, TMUX_LANE_ID_OPTION, lane_id], - check=False, - env=env, - ) - subprocess.run( - [tmux_bin, "select-layout", "-t", f"{session}:{window_name}", "tiled"], - check=False, - env=env, - ) - started_lanes.append(lane_id) - lane_targets[lane_id] = pane_id - launcher_scripts[lane_id] = str(lane_script) - auto_wake_result: dict[str, object] = { - "schema_version": AUTO_WAKE_LOOP_SCHEMA_VERSION, - "enabled": False, - } - if auto_wake: - auto_wake_result = _start_auto_wake_loop( - script_dir=script_dir, - project=project, - registry=registry, - runtime_root=runtime_root, - session=session, - lane_agents={ - lane_id: lane_agents[lane_id] - for lane_id in started_lanes - if lane_id in lane_agents - }, - goal_id=goal_id, - initial_projection=initial_projection, - tmux_bin=tmux_bin, - cli_bin=cli_bin, - interval_seconds=auto_wake_interval_seconds, - env=env, - ) - if attach: - subprocess.run([tmux_bin, "attach", "-t", session], check=True, env=env) - acceptance = _tmux_acceptance( - tmux_bin=tmux_bin, - session=session, - expected_lanes=started_lanes, - lane_targets=lane_targets, - env=env, - schema_version=acceptance_schema, - lane_scripts=launcher_scripts, - codex_bin=codex_bin, - ) - return { - "schema_version": launch_result_schema, - "executed": True, - "launcher": "tmux", - "session_name": session, - "started_lane_count": len(started_lanes), - "started_lanes": started_lanes, - "started_lane_targets": lane_targets, - "surviving_lane_count": len(acceptance["surviving_lanes"]), - "surviving_lanes": acceptance["surviving_lanes"], - "attach_command": f"{tmux_bin} attach -t {session}", - "stop_command": f"{tmux_bin} kill-session -t {session}", - "workspace_mode": workspace_mode, - "codex_trust_workspace": codex_trust_workspace, - "codex_trust_config": codex_trust_config, - "codex_trust_scope": ( - "persisted_selected_workspace_and_git_root" - if codex_trust_workspace - else "native_codex_trust_prompt" - ), - "script_mode": "runtime_local_files", - "launcher_script_count": len(launcher_scripts), - "attach_requested": attach, - "operator_takeover": "attach to the tmux session, interrupt any lane, or kill the session", - "auto_wake": auto_wake_result, - "visible_acceptance": acceptance, - "initial_prompt": { - "mode": ( - "todo_readiness_targeted" - if state_aware_initial_prompt - else "all_lanes_compatibility_fallback" - ), - "target_lanes": [ - lane_id for lane_id in started_lanes if lane_id in initial_runnable_lanes - ] - if state_aware_initial_prompt - else started_lanes, - "deferred_lanes": [ - lane_id for lane_id in started_lanes if lane_id not in initial_runnable_lanes - ] - if state_aware_initial_prompt - else [], - "projection": initial_projection, - }, - "tmux_layout": { - "window_name": window_name, - "single_window": True, - "layout": "tiled", - "recording_friendly": True, - }, - } - - -def _tmux_acceptance( - *, - tmux_bin: str, - session: str, - expected_lanes: list[str], - lane_targets: dict[str, str], - env: dict[str, str], - schema_version: str, - lane_scripts: dict[str, str], - codex_bin: str, -) -> dict[str, object]: - codex_name = Path(codex_bin).name or "codex" - last_payload: dict[str, object] | None = None - min_stable_acceptance_attempts = 12 - for attempt in range(32): - time.sleep(0.25) - list_result = subprocess.run( - [tmux_bin, "list-panes", "-s", "-t", session, "-F", "#{pane_id}\t#{pane_title}\t#{window_name}"], - check=False, - capture_output=True, - text=True, - env=env, - ) - observed_targets: set[str] = set() - observed_titles: set[str] = set() - for line in list_result.stdout.splitlines(): - pane_id, pane_title, _window_name = (line.split("\t") + ["", "", ""])[:3] - if pane_id.strip(): - observed_targets.add(pane_id.strip()) - if pane_title.strip(): - observed_titles.add(pane_title.strip()) - surviving = [ - lane - for lane in expected_lanes - if lane_targets.get(lane) in observed_targets or lane in observed_titles - ] - pane_checks = [] - for lane in expected_lanes: - target = lane_targets.get(lane) or f"{session}:{lane}" - capture = subprocess.run( - [tmux_bin, "capture-pane", "-pt", target, "-S", "-200"], - check=False, - capture_output=True, - text=True, - env=env, - ).stdout - current_command = subprocess.run( - [tmux_bin, "display-message", "-p", "-t", target, "#{pane_current_command}"], - check=False, - capture_output=True, - text=True, - env=env, - ).stdout.strip() - script_path = Path(lane_scripts.get(lane, "")) - script_text = script_path.read_text(encoding="utf-8") if script_path.is_file() else "" - failure_markers = [ - "stopped_before_bootstrap", - "stopped_before_codex", - "quota_wait_timeout", - "bootstrap_command_failed", - ] - trust_prompt_markers = [ - "Do you trust the contents of this directory?", - "Working with untrusted contents", - "Trusting will apply to the repository root:", - "Press enter to continue", - ] - blocked_before_bootstrap = any(marker in capture for marker in failure_markers) - trust_prompt_blocked = any(marker in capture for marker in trust_prompt_markers) - script_words = script_text.replace("\n", " ").replace(";", " ").split() - uses_headless_codex_subcommand = any( - Path(word).name == codex_name - and index + 1 < len(script_words) - and script_words[index + 1] == "exec" - for index, word in enumerate(script_words) - ) - script_execs_codex_tui = ( - "exec " in script_text - and codex_name in script_text - and "| python3" not in script_text - and not uses_headless_codex_subcommand - ) - ok = ( - lane in surviving - and not blocked_before_bootstrap - and not trust_prompt_blocked - and script_execs_codex_tui - ) - pane_checks.append( - { - "lane_id": lane, - "pane_target": target, - "accepted": ok, - "blocked_before_bootstrap": blocked_before_bootstrap, - "trust_prompt_blocked": trust_prompt_blocked, - "interactive_codex_tui_script": script_execs_codex_tui, - "pane_current_command": current_command, - } - ) - accepted = list_result.returncode == 0 and len(surviving) == len(expected_lanes) and all( - item["accepted"] for item in pane_checks - ) - last_payload = { - "schema_version": schema_version, - "accepted": accepted, - "surviving_lanes": surviving, - "missing_lanes": [lane for lane in expected_lanes if lane not in surviving], - "pane_checks": pane_checks, - } - if any(item["trust_prompt_blocked"] for item in pane_checks): - return last_payload - if accepted and attempt + 1 >= min_stable_acceptance_attempts: - return last_payload - assert last_payload is not None - return last_payload diff --git a/demo/visible_multi_agent_tmux.py b/demo/visible_multi_agent_tmux.py deleted file mode 100644 index 64c244e125..0000000000 --- a/demo/visible_multi_agent_tmux.py +++ /dev/null @@ -1,429 +0,0 @@ -from __future__ import annotations - -import os -import shutil -import subprocess -import time -from collections.abc import Iterable -from hashlib import sha256 - -from .multi_agent.contract import ( - PANE_LOCAL_A2A_WAKEUP_PROMPT, - build_decentralized_a2a_driver_contract, -) - - -PANE_A2A_WAKEUP_SCHEMA_VERSION = "multi_agent_pane_a2a_wakeup_v0" -PANE_A2A_WAKEUP_PROMPT = PANE_LOCAL_A2A_WAKEUP_PROMPT -PANE_A2A_INPUT_READY_TIMEOUT_SECONDS = 5.0 -TMUX_LANE_ID_OPTION = "@loopx_lane_id" -_CODEX_TUI_USAGE_LIMIT_MARKERS = ( - "you've hit your usage limit", - "you have hit your usage limit", - "purchase more credits or try again", - "rate limit exceeded", - "429 too many requests", -) -_CODEX_TUI_BACKOFF_REASONS = frozenset({"codex_tui_usage_or_rate_limited"}) - - -def _require_executable(command: str, *, field: str) -> str: - path = shutil.which(command) - if not path: - raise ValueError(f"{field} executable not found on PATH: {command}") - return path - - -def build_pane_a2a_wakeup_prompt() -> str: - """Return the fixed prompt broadcast to live Codex TUI panes.""" - - return PANE_A2A_WAKEUP_PROMPT - - -def _codex_tui_input_ready(capture: str) -> bool: - if not capture: - return False - if _codex_tui_block_reason(capture): - return False - prompt_index = max(capture.rfind("\n› "), capture.rfind("\r\n› ")) - if prompt_index < 0: - return False - model_lines = [line for line in capture.splitlines() if "model:" in line] - if model_lines and "loading" in model_lines[-1]: - return False - has_codex_surface = "OpenAI Codex" in capture or "gpt-" in capture[prompt_index:] - if not has_codex_surface: - return False - # Codex keeps a queueable input prompt visible while a turn is still running. - # Busy markers render immediately above that prompt, so checking only the - # text after the final prompt incorrectly treats an active turn as idle. - if "Queued follow-up inputs" in capture or "Working (" in capture: - return False - current_view = capture[prompt_index:] - if "Starting MCP servers" in current_view: - return False - return True - - -def _codex_tui_block_reason(capture: str) -> str | None: - lowered = capture.lower() - if any(marker in lowered for marker in _CODEX_TUI_USAGE_LIMIT_MARKERS): - return "codex_tui_usage_or_rate_limited" - return None - - -def _wait_for_tmux_pane_input_ready( - *, - tmux_bin: str, - session: str, - lane: str, - target: str | None = None, - env: dict[str, str], - timeout_seconds: float = PANE_A2A_INPUT_READY_TIMEOUT_SECONDS, -) -> dict[str, object]: - deadline = time.monotonic() + max(0.25, timeout_seconds) - attempts = 0 - tmux_target = target or f"{session}:{lane}" - while True: - attempts += 1 - capture = subprocess.run( - [tmux_bin, "capture-pane", "-pt", tmux_target, "-S", "-80"], - check=False, - capture_output=True, - text=True, - env=env, - ) - block_reason = _codex_tui_block_reason(capture.stdout) - if block_reason: - return { - "lane": lane, - "target": tmux_target, - "ready": False, - "attempt_count": attempts, - "capture_ok": capture.returncode == 0, - "ready_marker": None, - "not_ready_reason": block_reason, - "backoff_recommended": True, - } - ready = capture.returncode == 0 and _codex_tui_input_ready(capture.stdout) - if ready: - return { - "lane": lane, - "target": tmux_target, - "ready": True, - "attempt_count": attempts, - "capture_ok": True, - "ready_marker": "codex_tui_first_turn_ready", - } - if time.monotonic() >= deadline: - return { - "lane": lane, - "target": tmux_target, - "ready": False, - "attempt_count": attempts, - "capture_ok": capture.returncode == 0, - "ready_marker": None, - "not_ready_reason": "codex_tui_busy_or_not_ready", - "backoff_recommended": False, - } - time.sleep(0.25) - - -def _list_tmux_lane_targets( - *, - tmux_bin: str, - session: str, - env: dict[str, str], -) -> list[dict[str, str]]: - listed = subprocess.run( - [ - tmux_bin, - "list-panes", - "-s", - "-t", - session, - "-F", - f"#{{pane_id}}\t#{{{TMUX_LANE_ID_OPTION}}}\t#{{pane_title}}\t#{{window_name}}\t#{{pane_index}}", - ], - check=False, - capture_output=True, - text=True, - env=env, - ) - panes: list[dict[str, str]] = [] - if listed.returncode == 0: - for line in listed.stdout.splitlines(): - pane_id, lane_id, pane_title, window_name, pane_index = ( - line.split("\t") + ["", "", "", "", ""] - )[:5] - pane_id = pane_id.strip() - label = lane_id.strip() or pane_id - if pane_id: - panes.append( - { - "lane": label, - "target": pane_id, - "pane_title": pane_title.strip(), - "window_name": window_name.strip(), - "pane_index": pane_index.strip(), - } - ) - return panes - - -def _resolve_tmux_lane_targets( - *, - tmux_bin: str, - session: str, - lanes: Iterable[str], - env: dict[str, str], -) -> list[dict[str, str]]: - target_lanes = [str(lane).strip() for lane in lanes if str(lane).strip()] - pane_targets = _list_tmux_lane_targets(tmux_bin=tmux_bin, session=session, env=env) - by_label = {item["lane"]: item["target"] for item in pane_targets} - by_target = {item["target"]: item["target"] for item in pane_targets} - if not target_lanes: - if pane_targets: - return pane_targets - listed = subprocess.run( - [tmux_bin, "list-windows", "-t", session, "-F", "#{window_name}"], - check=True, - capture_output=True, - text=True, - env=env, - ) - target_lanes = [line.strip() for line in listed.stdout.splitlines() if line.strip()] - resolved: list[dict[str, str]] = [] - for lane in target_lanes: - target = by_label.get(lane) or by_target.get(lane) - if not target: - target = lane if lane.startswith("%") else f"{session}:{lane}" - resolved.append({"lane": lane, "target": target}) - return resolved - - -def _prompt_still_waiting_for_submit(capture: str, prompt: str) -> bool: - if not capture or not prompt: - return False - prompt_index = max(capture.rfind("\n› "), capture.rfind("\r\n› ")) - current_view = capture[prompt_index:] if prompt_index >= 0 else capture - if "Working (" in current_view or "Queued follow-up inputs" in current_view: - return False - if "[Pasted Content" in current_view: - return True - words = [part for part in prompt.split() if part] - head = words[:4] - tail = words[-4:] - return (bool(head) and all(word in current_view for word in head)) or ( - bool(tail) and all(word in current_view for word in tail) - ) - - -def _paste_and_submit_tmux_prompt( - *, - tmux_bin: str, - target: str, - buffer_name: str, - prompt: str, - env: dict[str, str], -) -> dict[str, object]: - subprocess.run( - [tmux_bin, "paste-buffer", "-b", buffer_name, "-t", target], - check=True, - env=env, - ) - subprocess.run( - [tmux_bin, "send-keys", "-t", target, "Enter"], - check=True, - env=env, - ) - retry = False - capture_ok = False - observation_count = 0 - for _ in range(20): - time.sleep(0.25) - observation_count += 1 - capture = subprocess.run( - [tmux_bin, "capture-pane", "-pt", target, "-S", "-80"], - check=False, - capture_output=True, - text=True, - env=env, - ) - capture_ok = capture.returncode == 0 - if not capture_ok: - continue - if _prompt_still_waiting_for_submit(capture.stdout, prompt): - retry = True - break - if "Working (" in capture.stdout or "Queued follow-up inputs" in capture.stdout: - break - if retry: - subprocess.run( - [tmux_bin, "send-keys", "-t", target, "Enter"], - check=True, - env=env, - ) - return { - "target": target, - "initial_submit_key": "Enter", - "retry_submit_key": "Enter" if retry else None, - "retry_count": 1 if retry else 0, - "capture_ok": capture_ok, - "observation_count": observation_count, - } - - -def wake_visible_multi_agent_panes( - *, - session_name: str, - tmux_bin: str = "tmux", - lanes: Iterable[str] | None = None, - execute: bool = False, - prompt: str | None = None, - input_ready_timeout_seconds: float | None = None, -) -> dict[str, object]: - """Broadcast the fixed A2A prompt; each pane still decides via LoopX state.""" - - session = str(session_name or "").strip() - if not session: - raise ValueError("multi-agent wake requires --session-name") - prompt_text = str(prompt or build_pane_a2a_wakeup_prompt()).strip() - if not prompt_text: - raise ValueError("multi-agent wake prompt must not be empty") - prompt_hash = sha256(prompt_text.encode("utf-8")).hexdigest()[:16] - target_lanes = [str(lane).strip() for lane in lanes or [] if str(lane).strip()] - driver_contract = build_decentralized_a2a_driver_contract() - input_ready_checks: list[dict[str, object]] = [] - prompt_submit_checks: list[dict[str, object]] = [] - - if execute: - _require_executable(tmux_bin, field="tmux_bin") - env = os.environ.copy() - target_specs = _resolve_tmux_lane_targets( - tmux_bin=tmux_bin, - session=session, - lanes=target_lanes, - env=env, - ) - if not target_specs: - raise ValueError("multi-agent wake found no target panes") - resolved_target_lanes = [str(spec["lane"]) for spec in target_specs] - input_ready_checks = [ - _wait_for_tmux_pane_input_ready( - tmux_bin=tmux_bin, - session=session, - lane=spec["lane"], - target=spec["target"], - env=env, - timeout_seconds=( - PANE_A2A_INPUT_READY_TIMEOUT_SECONDS - if input_ready_timeout_seconds is None - else input_ready_timeout_seconds - ), - ) - for spec in target_specs - ] - ready_lanes = [ - str(check.get("lane")) - for check in input_ready_checks - if check.get("ready") is True - ] - if ready_lanes: - buffer_name = f"loopx-pane-a2a-wakeup-{prompt_hash}" - subprocess.run( - [tmux_bin, "set-buffer", "-b", buffer_name, "--", prompt_text], - check=True, - env=env, - ) - ready_targets = { - str(spec["lane"]): str(spec["target"]) - for spec in target_specs - if str(spec["lane"]) in ready_lanes - } - for lane in ready_lanes: - target = ready_targets[lane] - prompt_submit_checks.append( - _paste_and_submit_tmux_prompt( - tmux_bin=tmux_bin, - target=target, - buffer_name=buffer_name, - prompt=prompt_text, - env=env, - ) - ) - not_ready = [ - str(check.get("lane")) - for check in input_ready_checks - if check.get("ready") is not True - ] - else: - ready_lanes = [] - not_ready = [] - resolved_target_lanes = target_lanes - - auto_wake_backoff_recommended = ( - execute - and bool(input_ready_checks) - and not ready_lanes - and all( - str(check.get("not_ready_reason") or "") in _CODEX_TUI_BACKOFF_REASONS - for check in input_ready_checks - ) - ) - if not execute: - prompt_delivery = "dry_run" - elif auto_wake_backoff_recommended: - prompt_delivery = "skipped_terminal_pane_backoff" - elif not prompt_submit_checks: - prompt_delivery = "skipped_no_input_ready_panes" - elif not_ready: - prompt_delivery = "tmux_paste_buffer_after_ready_subset" - else: - prompt_delivery = "tmux_paste_buffer_after_codex_tui_first_turn_ready" - - return { - "ok": True, - "schema_version": PANE_A2A_WAKEUP_SCHEMA_VERSION, - "mode": "execute" if execute else "dry_run", - "session_name": session, - "target_lanes": resolved_target_lanes if resolved_target_lanes else [""], - "prompt": prompt_text, - "prompt_hash": prompt_hash, - "coordination_model": "decentralized_state_a2a", - "wakeup_model": "fixed_prompt_broadcast", - "driver_contract": driver_contract, - "workflow_driver": False, - "broadcaster_reads_frontier": driver_contract["broadcaster"]["reads_frontier"], - "broadcaster_reads_todo_readiness": driver_contract["broadcaster"].get( - "reads_todo_readiness", False - ), - "broadcaster_selects_todo": driver_contract["broadcaster"]["selects_todo"], - "pane_decision_owner": driver_contract["pane"]["decision_owner"], - "pane_input_ready_verified": ( - bool(input_ready_checks) - and all(check.get("ready") is True for check in input_ready_checks) - ) - if execute - else False, - "pane_input_ready_checks": input_ready_checks, - "pane_input_ready_timeout_seconds": ( - PANE_A2A_INPUT_READY_TIMEOUT_SECONDS - if input_ready_timeout_seconds is None - else input_ready_timeout_seconds - ) - if execute - else None, - "ready_lanes": ready_lanes, - "not_ready_lanes": not_ready, - "prompt_submit_checks": prompt_submit_checks, - "prompt_delivery": prompt_delivery, - "auto_wake_backoff_recommended": auto_wake_backoff_recommended, - "boundary": { - "writes_loopx_state": False, - "spends_loopx_quota": False, - "reads_raw_transcripts": False, - "reads_credentials": False, - "runs_worker_turn_directly": False, - }, - } diff --git a/docs/architecture/rfcs/harness-selection-dsh-pi-v0.md b/docs/architecture/rfcs/harness-selection-dsh-pi-v0.md index a8184bb7a8..0305e23301 100644 --- a/docs/architecture/rfcs/harness-selection-dsh-pi-v0.md +++ b/docs/architecture/rfcs/harness-selection-dsh-pi-v0.md @@ -776,13 +776,17 @@ entry must derive caller identity from a verified binding. Per-Agent readback uses `loopx shared-goal-alignment --goal-id --agent-id `, within its Stage 1/2 source-facts boundary. The plan receipt does not -project that entire state. The [three-layer contract](../../reference/protocols/multi-agent-three-layer-minimality-v0.md) -and [visible launcher](../../reference/protocols/multi-agent-visible-launcher-v0.md) +project that entire state. The [three-layer contract](https://github.com/loopx-project/loopx/blob/910ff3bca5c8ff5ee52bd691ae924cc37dbaf05f/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md) +and [visible launcher](https://github.com/loopx-project/loopx/blob/910ff3bca5c8ff5ee52bd691ae924cc37dbaf05f/docs/reference/protocols/multi-agent-visible-launcher-v0.md) remain independent owners: user intent, preset procedure and kernel declarations join by Goal/Agent/Todo identity, not by creating another runner, pane owner, vision budget or evidence loop. Selecting an executor or storing credentials grants none of these effects. +> The tmux multi-agent and auto-research demos are retired. Demo-contract links +> above are pinned to the pre-retirement revision for historical design context; +> they do not describe current entrypoints or shared runtime ownership. + ## Steward Channel Readiness by Milestone (2026-09-16) The steward channel consumes both this document's host selection and the manager diff --git a/docs/architecture/rfcs/harness-selection-dsh-pi-v0.zh-CN.md b/docs/architecture/rfcs/harness-selection-dsh-pi-v0.zh-CN.md index 4517444e7d..073fc9d275 100644 --- a/docs/architecture/rfcs/harness-selection-dsh-pi-v0.zh-CN.md +++ b/docs/architecture/rfcs/harness-selection-dsh-pi-v0.zh-CN.md @@ -556,7 +556,10 @@ binding。本阶段交付分配边界及可用性 fallback，不从对话 prose [Peer directory](../../reference/protocols/peer-agent-directory-and-observation-v0.md) 供 manager 与 peer 共用。`loopx agent-directory --goal-id [--agent-id ]` 复用 management projection，本地最多 24 行并报告省略数；没有分页、presence provider 或 lease epoch。传入未注册 caller 得到 scope gap。本地 CLI membership 检查不认证远端 caller；未来远端入口必须从可验证 binding 派生身份。 -每 lane 通过 `loopx shared-goal-alignment --goal-id --agent-id ` 回读，仍限 Stage 1/2 source-facts 语义，计划 receipt 不投影完整状态。[三层合同](../../reference/protocols/multi-agent-three-layer-minimality-v0.md) 与[可见 launcher](../../reference/protocols/multi-agent-visible-launcher-v0.md) 保留独立归属：用户意图、preset 流程、kernel 声明按 Goal/Agent/Todo 身份连接，不再建 runner、pane owner、vision budget 或 evidence loop。选择执行器和保存凭据都不授予这些效果。 +每 lane 通过 `loopx shared-goal-alignment --goal-id --agent-id ` 回读，仍限 Stage 1/2 source-facts 语义，计划 receipt 不投影完整状态。[三层合同](https://github.com/loopx-project/loopx/blob/910ff3bca5c8ff5ee52bd691ae924cc37dbaf05f/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md) 与[可见 launcher](https://github.com/loopx-project/loopx/blob/910ff3bca5c8ff5ee52bd691ae924cc37dbaf05f/docs/reference/protocols/multi-agent-visible-launcher-v0.md) 保留独立归属：用户意图、preset 流程、kernel 声明按 Goal/Agent/Todo 身份连接，不再建 runner、pane owner、vision budget 或 evidence loop。选择执行器和保存凭据都不授予这些效果。 + +> 旧 tmux multi-agent 与 auto-research 演示已退役；上述演示合同链接固定到退役前版本， +> 仅用于解释历史设计，不是当前执行入口或共享运行时的归属。 ## 按里程碑看管家通道的就绪度（2026-09-16） diff --git a/docs/architecture/rfcs/monorepo-distribution-split-v0.md b/docs/architecture/rfcs/monorepo-distribution-split-v0.md index df8c5cf6bf..c8e7ce79f9 100644 --- a/docs/architecture/rfcs/monorepo-distribution-split-v0.md +++ b/docs/architecture/rfcs/monorepo-distribution-split-v0.md @@ -215,11 +215,15 @@ Runtime composition audit on `9cae3b28939f72fa4aadab5cfe49e19bcbdab372`: - [Benchmark toolkit](../../../loopx/capabilities/benchmark_toolkit/README.md) owns reusable permissions, concurrency and integrity. [Benchmark policy](../../../benchmark/README.md) keeps native execution/evaluation with the benchmark; no second launcher is restored. -- The [three-layer protocol](../../reference/protocols/multi-agent-three-layer-minimality-v0.md) +- The [three-layer protocol](https://github.com/loopx-project/loopx/blob/910ff3bca5c8ff5ee52bd691ae924cc37dbaf05f/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md) uses “kernel” for a broader reusable layer. Here, governance kernel names the authority boundary and boot closure names a selected entrypoint's required code. A reusable runner or presentation facility need not enter every boot closure. +> The tmux multi-agent and auto-research demos are retired. Demo-contract links +> above are pinned to the pre-retirement revision for historical design context; +> they do not describe current entrypoints or shared runtime ownership. + ## 5. Proposed architecture ### Ownership and authority diff --git a/docs/architecture/rfcs/monorepo-distribution-split-v0.zh-CN.md b/docs/architecture/rfcs/monorepo-distribution-split-v0.zh-CN.md index a4bfc31e48..5136da5181 100644 --- a/docs/architecture/rfcs/monorepo-distribution-split-v0.zh-CN.md +++ b/docs/architecture/rfcs/monorepo-distribution-split-v0.zh-CN.md @@ -137,10 +137,13 @@ Minimal 必须执行并恢复真实的受治理任务；standard 围绕同一内 [DSH/Pi 选择](harness-selection-dsh-pi-v0.zh-CN.md) 负责宿主资格。装配复用这些契约。 - [Benchmark toolkit](../../../loopx/capabilities/benchmark_toolkit/README.md) 负责通用权限、并发与完整性。 [Benchmark 政策](../../../benchmark/README.md) 把原生执行/评估留在 benchmark，不恢复第二个 launcher。 -- [三层协议](../../reference/protocols/multi-agent-three-layer-minimality-v0.md) 的 “kernel” 指更宽的复用层。 +- [三层协议](https://github.com/loopx-project/loopx/blob/910ff3bca5c8ff5ee52bd691ae924cc37dbaf05f/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md) 的 “kernel” 指更宽的复用层。 本文治理内核指 authority 边界，启动依赖闭包指所选入口必需代码。 可复用 runner 或展示设施无须进入每个启动闭包。 +> 旧 tmux multi-agent 与 auto-research 演示已退役；上述演示合同链接固定到退役前版本， +> 仅用于解释历史设计，不是当前执行入口或共享运行时的归属。 + ## 5. 提议的架构 ### 归属与权威 diff --git a/docs/archive/incidents/auto-research-replan-multiround-badcase-20260705.md b/docs/archive/incidents/auto-research-replan-multiround-badcase-20260705.md index c6875b7fc2..e9a96c8233 100644 --- a/docs/archive/incidents/auto-research-replan-multiround-badcase-20260705.md +++ b/docs/archive/incidents/auto-research-replan-multiround-badcase-20260705.md @@ -1,5 +1,8 @@ # Auto-Research Replan And Multi-Round Demo Bad Case +> Historical incident: the tmux auto-research demo has been retired. Commands and +> code references below describe the original incident, not current entrypoints. + Date: 2026-07-05 Audience: LoopX auto-research owners, quota/status maintainers, todo/replan diff --git a/docs/development/control-plane-course/02-goal-control-plane-architecture.md b/docs/development/control-plane-course/02-goal-control-plane-architecture.md index e1e3eea0bd..05405bf61b 100644 --- a/docs/development/control-plane-course/02-goal-control-plane-architecture.md +++ b/docs/development/control-plane-course/02-goal-control-plane-architecture.md @@ -241,83 +241,16 @@ issue body、raw check log、凭据或本地路径。GitHub 仍是权威来源 完整案例见 [Issue-Fix 能力](../../../loopx/capabilities/issue_fix/README.zh-CN.md) 和 [State Kernel × Domain State 案例](../../../loopx/capabilities/issue_fix/docs/state-kernel-domain-state-case-study.zh-CN.md)。 -## Showcase B：Multi-Agent Auto Research +## Showcase B：Multi-Agent 研究团队 -Auto Research 的产品承诺是：多个研究 Agent 围绕同一 research contract 持续提出假设、 -执行实验、评价证据并形成 promotion、retirement 或 retry 候选。研究树可以被投影出来， -但不需要一个拥有整棵树的 coordinator agent。 +研究协作复用 Goal、Todo、claim、lease、证据与独立验收。角色只声明自己的研究工作与 +写入边界；创建任务或发送消息不等于启动 worker，也不授予实验、推广或发布权限。 -```mermaid -flowchart LR - C["Research contract
metric · baseline · protected scope"] --> H["Hypothesis todos"] - H --> Q["Per-agent quota + frontier"] - Q --> E["Isolated executor turn"] - E --> P["Typed evidence packet
dev · holdout · boundary"] - P --> G["Evidence graph"] - G --> D["promotion / retirement / retry"] - D --> S["role-scoped successor todos"] - S --> Q -``` - -默认角色不是层级，而是写入不同 typed record 的 equal peers： - -| Role | 主要产物 | 不能做什么 | -| --- | --- | --- | -| Curator | research contract、metric、protected scope | 选择赢家或修改受保护 evaluator | -| Hypothesis proposer | 有 grounding 和 todo lineage 的 hypothesis | 用同一材料同时声称独立 novelty | -| Executor | 隔离实验、dev/holdout result、evidence packet | 自行 promotion 或隐藏失败尝试 | -| Evaluator / promoter | promotion、retirement、retry candidate | 把 dev-only lift 当成最终证据 | -| Product narrator | public-safe evidence graph 与案例叙事 | 发明指标或改写 source state | - -### 没有中央研究经理，研究怎样继续 - -`run_auto_research_worker_loop()` 只轮询一组可见 lane。每个 worker turn 都重新读取当前 -agent 的 quota 和 research frontier，再执行一个被选中的 todo： - -```python -for agent_id in agent_ids: - turn = run_auto_research_worker_turn( - goal_id=goal_id, - agent_id=agent_id, - ..., - ) -if no_lane_has_action: - stop_reason = "no_runnable_frontier" -``` - -真正的选择发生在 `load_auto_research_worker_frontier()`：它先调用 -`build_quota_should_run(..., agent_id=agent_id)`，再把 rollout evidence 投影成该 agent -可见的 research frontier。worker 不拥有全局 executor queue，也不能因为看见一个假设就 -越过 claim 和 quota。 - -另一端，`build_research_decision_candidates()` 根据 evidence graph 产生有限结果： - -| 证据状态 | 下一步 | -| --- | --- | -| dev 改善但缺少 holdout | 创建或暴露 holdout successor | -| holdout 改善且 boundary clean | 进入 promotion review 或满足目标策略 | -| negative / guardrail evidence | 形成 retirement candidate | -| 尝试未计分但可恢复 | 保留 retry candidate 与 artifact ref | -| 无 runnable frontier 且完成条件满足 | quiet completion | - -低层调用路径可以这样读： - -```text -build_auto_research_preset_summary - -> role profiles + initial todos - -> run_auto_research_worker_loop - -> run_auto_research_worker_turn - -> build_auto_research_evidence_packet + rollout append - -> build_research_evidence_graph_from_rollout_events - -> build_research_decision_candidates - -> build_auto_research_completion_status - -> role-scoped successor / promotion gate / quiet completion -``` - -Auto Research 是建立在通用 multi-agent kernel 上的 thin preset：preset 提供角色、领域默认值 -和 successor hints；goal、todo、claim、quota、evidence、handoff 与完成语义仍由 Kernel -拥有。更完整的角色合同见 -[Auto Research Lane Contract](../../reference/protocols/auto-research-lane-contract-v1.md)。 +当前可运行示例见 +[managed research team](../../../examples/managed-research-team/README.md)， +通用交付示例见 +[collaboration delivery](../../../examples/collaboration-delivery/README.md)。 +旧 tmux Auto Research 演示已退役，其源码与演示合同保留在 Git 历史中。 ## Showcase C：Single-Agent Auto ML @@ -588,9 +521,6 @@ runtime event、cancel 和 retry。接入 LoopX 时，只需对齐四个面： | ML 候选怎样形成默认关闭的 advisory packet | `build_ml_experiment_advisory_packet` | `examples/ml-experiment-domain-pack-smoke.py` | | Explore evidence 与资源怎样形成只读组合建议 | `build_explore_worker_branch_plan`、`resolve_explore_harness_gate` | `examples/explore-worker-plan-gate-smoke.py` | | Explore finding 怎样形成可重建拓扑 | `append_explore_result_events`、`build_explore_graph_view` | `examples/explore-result-layer-smoke.py` | -| Auto Research role 怎样声明 | `build_auto_research_preset_summary` | `examples/auto-research-dev-thin-preset-smoke.py` | -| 每个研究 lane 怎样重新进入 Kernel | `load_auto_research_worker_frontier`、`run_auto_research_worker_turn` | `examples/auto-research-worker-turn-smoke.py` | -| 研究证据怎样形成决策 | `build_research_decision_candidates`、`build_auto_research_completion_status` | `examples/auto-research-layered-e2e-acceptance-smoke.py` | | 通用 Kernel 怎样选择本轮动作 | `build_quota_should_run`、`build_interaction_contract` | 第 6 讲的 quota smokes | | 一轮结果怎样写回并恢复 | `run_loopx_turn_once`、`refresh-state` | 第 7、8 讲的 transaction / refresh smokes | diff --git a/docs/development/control-plane-course/11-extension-layer.md b/docs/development/control-plane-course/11-extension-layer.md index edfd3eac22..5e7333f909 100644 --- a/docs/development/control-plane-course/11-extension-layer.md +++ b/docs/development/control-plane-course/11-extension-layer.md @@ -1,11 +1,11 @@ -# 第 11 讲：扩展层、Explore 与 Multi-Agent 产品 +# 第 11 讲：扩展层、Explore 与 Agent 协作 > **本讲结论：** Extension 交付可选 provider、domain facts、capability 或 presentation； > 它复用同一份 goal、todo、quota、scheduler、evidence 与 handoff contract，不创建第二个 > kernel。 建议时长：150 分钟。扩展地图与 governed external execution 50 分钟、Explore 30 分钟、 -Single-Agent Auto ML 25 分钟、Multi-agent/Auto Research 30 分钟、实验 15 分钟。 +Single-Agent Auto ML 25 分钟、Agent 协作 30 分钟、实验 15 分钟。 一小时专题[长程任务如何收敛](topic-long-horizon-convergence.md)从“如何退出局部循环”解释 Graph、Harness 与 Kernel 的关系；本讲继续展开它们的扩展边界和产品组合方式。 @@ -17,31 +17,31 @@ Graph、Harness 与 Kernel 的关系；本讲继续展开它们的扩展边界 1. 使用 `configure-goal` 预览和开启 default-off feature。 2. 准确区分 Explore Graph 和 Explore Harness。 3. 解释 Harness 为什么只是 analysis-only planner，而不是执行授权。 -4. 解释 user、preset、kernel 三层 multi-agent minimality。 +4. 解释领域配置与通用协作控制面的边界。 5. 判断 supervisor、multi-subagent、reward memory、connector 各自扩展哪个边界。 6. 区分 Provider 的运行责任与 Extension 的交付生命周期。 7. 用 Single-Agent Auto ML 解释 Explore Graph、Harness、Domain Pack 与 Kernel 怎样组合。 8. 用 governed external execution 解释外部 effect 怎样安全 start、reconcile、writeback 和 spend。 本讲采用分层阅读：所有开发者先读 Extension Contract、分层原则和 Feature Catalog； -做探索产品再读 Graph/Harness 与 Single-Agent Auto ML；做 multi-agent 产品再读 Generic -Kernel 与 Auto Research；只有接入对应 surface 时，才需要继续读 Supervisor、Reward +做探索产品再读 Graph/Harness 与 Single-Agent Auto ML；做 Agent 协作再读 +Multi-Subagent；只有接入对应 surface 时，才需要继续读 Supervisor、Reward Memory 或 Lark Event Inbox。 -## 从 Auto Research 反推 Extension Contract +## 从研究协作反推 Extension Contract -第 2 讲把 Auto Research 当作产品 Showcase；本讲把它拆成一个可复用 extension： +第 2 讲用研究团队说明协作边界；本讲按可复用 extension 拆分责任： -| Surface | Auto Research 提供 | 继续由通用层拥有 | +| Surface | 研究场景提供 | 继续由通用层拥有 | | --- | --- | --- | | User entry | open question、少量 preset 选项 | goal identity、preview/execute boundary | | Provider | evaluator、artifact source 与可选 sink 的 observation/readback | transition 与 goal lifecycle | | Capability Pack | role defaults、evidence adapter、decision candidates | todo/gate/quota/handoff/terminal semantics | | Domain State | hypothesis、experiment、dev/holdout evidence | claim、permission、scheduler、spend | -| Host integration | visible worker panes、isolated executor turn | session lifecycle、workspace guard、effect receipt | +| Host integration | local/cloud workers、isolated executor turn | session lifecycle、workspace guard、effect receipt | | Projection | evidence graph、research frontier、showcase | canonical state 与 transition authority | -如果一个新 extension 能按这五行回答，通常不需要复制 runner。若它必须自己维护 agent +如果一个新 extension 能按这些边界回答，通常不需要复制 runner。若它必须自己维护 agent 身份、runnable queue、retry、gate、completion 和 cadence，它已经不是薄 capability，而是 第二个 control plane。 @@ -708,83 +708,6 @@ loopx ml-experiment preview --format json \ analysis-only，确认 planner boundary 中 `writes_state`、`claims_todos`、`acquires_leases`、 `starts_agents` 和 `changes_quota` 均为 false。这个实验验证组合边界，不需要真实训练任务。 -## Generic Multi-Agent Kernel - -LoopX 的 multi-agent 产品采用三层 minimality： - -| 层 | Owns | Must not own | -| --- | --- | --- | -| User | objective 和少量产品选项 | pane、tick、quota/frontier 细节 | -| Preset | domain roles、handoff hints、metric/evidence adapter、defaults | runner lifecycle、通用 replan、TUI、claim/quota protocol | -| Kernel | runner、真实 Codex TUI panes、workspace-safe launch、pane-local tick、todo/evidence/status、vision/replan | domain-specific research/support/sales 语义 | - -一个新产品不应 copy Auto Research runner。它只写自己的薄 preset，然后复用 `demo/multi_agent/`。 - -Kernel 的关键 invariant： - -```text -leader_agent_required = false -broadcaster_selects_todo = false -each_pane_reads_own_quota_frontier = true -todos_and_evidence_are_handoff_authority = true -``` - -## Auto Research 是 Preset，不是第二内核 - -用户入口只有一个开放问题： - -```bash -loopx auto-research "" -``` - -它先输出固定 contract，不启动 pane： - -- research brief； -- P0/P1/P2 action plan； -- evidence refs； -- next executable step； -- exact gate。 - -预览一键启动： - -```bash -loopx --format json auto-research start \ - "How should we evaluate autonomous research agents?" -``` - -执行 visible lanes： - -```bash -loopx auto-research start \ - "How should we evaluate autonomous research agents?" \ - --execute -``` - -默认角色： - -| Role | Owns | -| --- | --- | -| research-curator | contract、boundary、metric、stop/gate | -| hypothesis-proposer | todo-backed hypotheses、successor、retirement rationale | -| research-executor | isolated attempts、scored/unscored evidence | -| evaluator-promoter | holdout/verification、claim classification、promotion gate | - -Launcher 打开真实 Codex TUI panes，但不选择 todo、不执行 worker turn、不写研究结果。当前默认由每个可见 pane 运行自己的第一次 quota/frontier tick；需要显式 post-launch 广播时才使用 `--wake-visible-after-launch`。无论哪种唤醒方式，broadcaster 都不能替 pane 决定 todo。 - -### KNN Demo - -```bash -loopx --format json auto-research start \ - "How can the KNN solver improve exact-neighbor speedup?" \ - --preset knn-demo \ - --language zh \ - --execute -``` - -Preset 可以定义 benchmark workspace、editable/protected files、metric 和 role hints，但不能添加产品专属 coordinator。 - -可见 pane 启动只证明 runner positive path，不证明 lane 已产生研究成果。研究 evidence 必须由实际 role 工作后写入。 - ## Multi-Subagent `multi_subagent` 允许某个 executor 在 bounded todo 内调用 child agents，适合并行只读调查或隔离子问题。 @@ -915,7 +838,7 @@ loopx --format json explore worker-branch-plan \ 验证 disabled packet 是否说明 `required_contract`，且没有 claim/lease/launch side effect。 -## 核心代码领读：Explore、Auto ML 与 Auto Research 怎样复用 kernel +## 核心代码领读：Explore 与 Auto ML 怎样复用 kernel 扩展层最重要的判断不是“功能多不多”，而是它有没有重新发明 todo、quota、lease、scheduler 或 evidence。下面沿真实配置与执行边界读。 @@ -1072,47 +995,7 @@ production_actions_enabled = false 静默获得训练或生产 effect。项目 provider 要增加 delivery authority，仍须经过 registry goal boundary、quota、preflight、effect receipt 与 writeback。 -### 6. Auto Research 是薄 preset，不是第二个 kernel - -`demo/auto_research/preset.py` 公开地限定了 line-count claim： - -```python -def build_auto_research_minimal_a2a_recipe(...): - user_line = ( - "loopx auto-research start " - f"{quoted_open_question}{language_flag} --execute" - ) - return build_minimal_decentralized_a2a_recipe( - product_id="auto-research", - user_recipe_lines=[user_line], - preset_recipe_lines=default_auto_research_agent_specs(), - claim_boundary=( - "line count covers user intent and auto-research preset defaults only; " - "the reusable kernel owns visible process launch, fixed wake prompt, " - "pane-local quota/frontier tick, todo/evidence/status protocol, " - "and public artifact routing" - ), - ) -``` - -每个 role 只声明 domain-specific profile： - -```python -return { - "agent_id": lane["agent_id"], - "lane_id": lane["lane_id"], - "role_id": role_id, - "scope": lane["scope"], - "role_profile": role_profile, - "skill": {"name": "loopx-auto-research-worker", ...}, - "handoff_hints": role_profile.get("handoff") or [], - "reasoning_effort": reasoning_effort, -} -``` - -Preset 拥有 research roles、handoff hints、metric hints 和 domain defaults；共享 multi-agent kernel 拥有 process launch、fixed wake prompt、pane-local quota tick、todo/evidence/status 与 artifact routing。 - -### 7. 两个 Explore 功能怎样开启 +### 6. 两个 Explore 功能怎样开启 Explore Graph 与 Explore Harness 可独立开启。推荐先 preview： @@ -1134,7 +1017,7 @@ explore_harness.enabled=true 否则它仍会停在 `analysis_only`。开启 Graph 不会自动开启 Harness，开启 Harness 也不会自动授权 external sink 或 worker launch。 -### 8. 一条 extension 的最小审查路径 +### 7. 一条 extension 的最小审查路径 ```text configure preview @@ -1158,11 +1041,10 @@ terminal effect receipt、durable writeback 和 quota settlement。 - `build_explore_worker_branch_plan:871`：gate 如何限制 width 与 commands； - `append_explore_result_events:517`：幂等与冲突； - `build_ml_experiment_advisory_packet:655`：default-off advisory 与 effect authority 的边界； -- `build_auto_research_preset_role:99`：preset domain 与 kernel mechanics 的边界。 读完应能回答：Graph 和 Harness 为什么独立、analysis-only 如何帮助单 Agent 管理昂贵 -实验、ML pack 为什么不带 launch authority、Auto Research 的“四行配置”没有计算哪些 -kernel 代码、supervisor 为什么不能借扩展层获得 durable leader authority。 +实验、ML pack 为什么不带 launch authority、supervisor 为什么不能借扩展层获得 durable +leader authority。 ## 代码阅读路线 @@ -1171,15 +1053,11 @@ kernel 代码、supervisor 为什么不能借扩展层获得 durable leader auth 3. `loopx/capabilities/explore/` 4. `docs/product/domain-capability-packs.md` 5. `loopx/domain_packs/ml_experiment.py` -6. `docs/reference/protocols/multi-agent-three-layer-minimality-v0.md` -7. `demo/multi_agent/` -8. `demo/auto_research/README.md` -9. `demo/auto_research/preset.py` -10. `docs/reference/protocols/peer-supervisor-v0.md` -11. `loopx/extensions/governed_capability_execution.py` -12. `loopx/control_plane/governed_capability.ts` -13. `loopx/control_plane/turn_driver/settlement.py` -14. `docs/reference/extensions.md` +6. `docs/reference/protocols/peer-supervisor-v0.md` +7. `loopx/extensions/governed_capability_execution.py` +8. `loopx/control_plane/governed_capability.ts` +9. `loopx/control_plane/turn_driver/settlement.py` +10. `docs/reference/extensions.md` ## 代表性 Smoke @@ -1187,7 +1065,6 @@ kernel 代码、supervisor 为什么不能借扩展层获得 durable leader auth - `examples/explore-configure-goal-smoke.py` - `examples/explore-worker-plan-gate-smoke.py` - `examples/ml-experiment-domain-pack-smoke.py` -- `examples/auto-research-layered-e2e-acceptance-smoke.py` - `examples/control_plane/peer-supervisor-smoke.py` - `examples/showcase-catalog-smoke.py` - `tests/extensions/test_governed_capability_execution.py` diff --git a/docs/development/control-plane-course/topic-long-horizon-convergence.md b/docs/development/control-plane-course/topic-long-horizon-convergence.md index 37406a480e..cd50811025 100644 --- a/docs/development/control-plane-course/topic-long-horizon-convergence.md +++ b/docs/development/control-plane-course/topic-long-horizon-convergence.md @@ -1552,8 +1552,6 @@ LoopX 已经提供的通用机制包括： | Issue lifecycle | `loopx/capabilities/issue_fix/pr_lifecycle.py::build_issue_fix_pr_lifecycle_monitor_packet` | 外部 PR observation 怎样变成有限 proposal | | Explore result log | `loopx/capabilities/explore/result_log.py::append_explore_result_events` | finding/edge 怎样幂等保存且不获得执行权 | | Explore planning | `loopx/capabilities/explore/worker_branch_plan.py::build_explore_worker_branch_plan` | analysis-only 输出怎样受 scope/capacity/gate 限制 | -| Research decision | `demo/auto_research/research_state.py::build_research_decision_candidates` | dev/holdout/negative evidence 怎样形成 promotion、retirement、retry | -| Research completion | `demo/auto_research/research_state.py::build_auto_research_completion_status` | no runnable frontier 为什么不自动等于完成 | ## 代表性验证 @@ -1565,8 +1563,7 @@ LoopX 已经提供的通用机制包括： 6. `examples/issue-fix-pr-lifecycle-smoke.py` 7. `examples/explore-result-layer-smoke.py` 8. `examples/explore-worker-plan-gate-smoke.py` -9. `examples/auto-research-layered-e2e-acceptance-smoke.py` -10. `tests/control_plane/test_goal_frontier_replan_rules.py` +9. `tests/control_plane/test_goal_frontier_replan_rules.py` 这些验证分别证明局部机制。要证明“长程收敛”这一组合性质，还需要把方向、evidence、 frontier、cadence、authority 和 terminal 放在同一条 case trace 中推演，不能只看一个 diff --git a/docs/guides/README.md b/docs/guides/README.md index 2ab66e5931..3c4a7022f4 100644 --- a/docs/guides/README.md +++ b/docs/guides/README.md @@ -15,9 +15,6 @@ Guides are task-oriented paths for people starting or operating LoopX. - [Custom Agent runner integration (中文)](custom-agent-runner-integration.zh-CN.md) - [KunlunCode adapter guide (中文 Markdown)](kunluncode-adapter.zh-CN.md) - [KunlunCode adapter guide (中文 HTML)](kunluncode-adapter.zh-CN.html) -- [Auto-research command path](../../demo/auto_research/README.md) -- [Auto Research stop, takeover, and state-aware wake](auto-research-stop-takeover-wake-walkthrough.md) -- [Multi-agent product recipe](multi-agent-product-recipe.md) - [Experimental planner-worker mode](planner-worker-experimental.md) - [Codex App multi-provider routing extension](../../packages/loopx-codex-provider-routing/README.md) - [Codex 多 App 隔离与运维最佳实践（中文）](codex-multi-app-best-practices.zh-CN.md) diff --git a/docs/guides/auto-research-stop-takeover-wake-walkthrough.md b/docs/guides/auto-research-stop-takeover-wake-walkthrough.md deleted file mode 100644 index 9b16a202f0..0000000000 --- a/docs/guides/auto-research-stop-takeover-wake-walkthrough.md +++ /dev/null @@ -1,228 +0,0 @@ -# Auto Research Stop, Takeover, And State-Aware Wake - -Contributor walkthrough for the shipped Auto Research control transitions. -Reuse the existing one-command path. Do not add a second launcher, and do not -change the README first screen without maintainer preview. - -Runtime control transitions landed in -[#2786](https://github.com/huangruiteng/loopx/pull/2786) (related -[#2783](https://github.com/huangruiteng/loopx/issues/2783)). This guide is the -operator-facing proof path for GH-C43. - -Canonical command path: -[Auto-research command path](../../demo/auto_research/README.md). - -## What This Walkthrough Covers - -| Transition | Operator signal | Result | -| --- | --- | --- | -| External stop | Place `workspace/.loopx-auto-research-stop` | Worker-loop exits with `stop_reason = operator_stop_requested` before the next round | -| Operator takeover | `auto-research start … --execute --attach` | Attach to the visible tmux session; skip default background wake so the operator acts first | -| State-aware wake | `--wake-visible-after-launch` with `--no-attach` | Only lanes with a selected runnable todo and `quota.should_run` receive the fixed A2A wake | -| Quota pause | Live `quota.ok && !quota.should_run` | Worker turn returns `mode = paused_by_quota`; the loop reports `stop_reason = quota_paused` | - -Individual `worker-turn` calls are not gated by the stop marker. That preserves -manual lane advance while the multi-round loop is paused. - -## Boundaries - -- Stay on the shipped `loopx auto-research start` / `worker-loop` / `worker-turn` - commands. Do not invent a parallel demo launcher. -- Use synthetic or redacted public-safe evidence only. No credentials, raw - Codex transcripts, private paths, or live model claims. -- `--attach` and `--wake-visible-after-launch` conflict by design: choose - operator takeover or background wake, not both. -- Stopping the worker-loop is not the same as killing the tmux session. Remove - the stop marker to resume the loop; use `tmux kill-session` only when the - visible rehearsal itself should end. - -## 1. Start From The Existing Command Path - -Export one explicit goal and workspace identity first, and reuse it in every -later command. Without `--goal-id`, `start` creates a fresh timestamped goal -that the stop/resume commands below could not address: - -```bash -export GOAL_ID="loopx-auto-research-demo" -export WORKSPACE="$HOME/loopx-auto-research-demo" - -export LOOPX_REGISTRY="$HOME/.loopx/registry.global.json" -export LOOPX_RUNTIME_ROOT="$HOME/.loopx" - -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" \ - --goal-id "$GOAL_ID" \ - --workspace "$WORKSPACE" \ - --create-workspace - -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" \ - --goal-id "$GOAL_ID" \ - --workspace "$WORKSPACE" \ - --create-workspace \ - --execute \ - --replace-existing -``` - -For JSON automation that records wake evidence without attaching: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research start \ - "How should we evaluate autonomous research agents?" \ - --goal-id "$GOAL_ID" \ - --workspace "$WORKSPACE" \ - --create-workspace \ - --execute \ - --no-attach \ - --wake-visible-after-launch -``` - -## 2. Stop The Worker Loop Without Killing State - -While a multi-round worker-loop is running against the `$WORKSPACE` research -workspace, place the stop marker there. `worker-loop` reads the marker from -its own working directory, so run it from `$WORKSPACE`: - -```bash -touch "$WORKSPACE/.loopx-auto-research-stop" -``` - -The next round check exits with: - -```json -{ - "ok": true, - "stop_reason": "operator_stop_requested", - "turn_count": 0 -} -``` - -when the marker is present before round 1, or with prior turns retained when the -marker appears between rounds. The marker is checked at the top of every round, -not only at process entry. - -Resume by removing the marker and calling `worker-loop` again from the same -workspace with the same goal: - -```bash -rm -f "$WORKSPACE/.loopx-auto-research-stop" - -cd "$WORKSPACE" - -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - --format json auto-research worker-loop \ - --goal-id "$GOAL_ID" \ - --agent-id research-curator \ - --agent-id hypothesis-proposer \ - --agent-id research-executor \ - --agent-id evaluator-promoter \ - --max-rounds 2 \ - --execute \ - --complete-selected-todo -``` - -`operator_stop_requested` is distinct from `quota_paused`, `no_executed_turns`, -`no_runnable_frontier`, and `max_rounds`. - -## 3. Take Over A Visible Lane - -Immediate operator takeover uses the same start command with `--attach`, -targeting the same explicit goal and workspace. This skips the default -visible-role wake so the operator enters the tmux session first: - -```bash -loopx --registry "$LOOPX_REGISTRY" \ - --runtime-root "$LOOPX_RUNTIME_ROOT" \ - auto-research start "How should we evaluate autonomous research agents?" \ - --goal-id "$GOAL_ID" \ - --workspace "$WORKSPACE" \ - --create-workspace \ - --execute \ - --attach -``` - -Inside the session, interrupt a pane, run a single-lane worker-turn or advance -todos manually, then continue. Attach later without relaunching: - -```bash -tmux attach -t loopx-auto-research -``` - -Stop only the visible rehearsal (not the LoopX goal state) with: - -```bash -tmux kill-session -t loopx-auto-research -``` - -## 4. State-Aware Wake Filter - -When wake is requested after launch, Auto Research loads each lane frontier -through `load_auto_research_worker_frontier()` and skips lanes that should not -receive the fixed A2A prompt: - -| Skip reason | Meaning | -| --- | --- | -| `quiet_completion_allowed` | Goal already allows quiet completion | -| `no_selected_todo` | Frontier has no selected runnable todo | -| `quota_should_run_false` | Scheduler blocked the lane | -| `no_agent_mapping` | Started lane has no agent id mapping | -| `frontier_load_failed` | Frontier load failed; receipt uses `error_code`, never a raw exception string | - -If every lane is filtered, the wake receipt is a public-safe no-op: - -```json -{ - "ok": true, - "schema_version": "multi_agent_pane_a2a_wakeup_v0", - "mode": "no_op_all_filtered", - "target_lanes": [], - "prompt_delivery": "skipped_no_ready_lanes", - "wakeup_model": "state_aware_filter_no_ready_lanes" -} -``` - -An empty ready set must not call the underlying wake helper with `[]`, because -legacy wake semantics treat an empty list as “all lanes”. - -Filtered lanes appear under `state_aware_filter` for auditability. The -broadcaster still does not select todos or write LoopX research truth. - -## 5. Quota Pause Versus Operator Stop - -When a turn sees `quota.ok` and `quota.should_run == false`, it returns -`mode = paused_by_quota` before execution. If every turn in a round is paused -that way, the worker-loop stops with `stop_reason = quota_paused`. - -Do not treat resource pressure as operator intent: - -- `quota_paused` — scheduler said not to spend; -- `operator_stop_requested` — operator placed the stop marker. - -## Reproducible Validation - -No live model is required. Run the synthetic smokes that pin each transition, -then the optional cycle smoke: - -```bash -python3 examples/auto-research-stop-marker-smoke.py -python3 examples/auto-research-state-aware-wake-smoke.py -python3 examples/auto-research-quota-pause-smoke.py -python3 examples/auto-research-stop-takeover-walkthrough-smoke.py -python3 examples/auto-research-demo-e2e-worker-loop-smoke.py -python3 examples/auto-research-visible-worker-hook-smoke.py -python3 examples/showcase-catalog-smoke.py -loopx check --scan-path docs/showcases --scan-path docs/guides -``` - -## Evidence Boundary - -This walkthrough documents shipped public contracts with synthetic fixtures. -It does not claim that a research finding is production-ready, does not record -raw logs or credentials, and does not promote Auto Research as a second core -scheduler. Promotion still requires rollout-backed evidence and normal LoopX -gate or writeback rules. diff --git a/docs/guides/multi-agent-product-recipe.md b/docs/guides/multi-agent-product-recipe.md deleted file mode 100644 index 9d0b51c659..0000000000 --- a/docs/guides/multi-agent-product-recipe.md +++ /dev/null @@ -1,235 +0,0 @@ -# Multi-Agent Product Recipe - -This guide is the product-author path for building a LoopX multi-agent -composition without copying auto-research internals. The target shape is: - -1. **User layer:** a few intent fields. -2. **Product preset:** a small domain recipe. -3. **Multi-agent kernel:** every reusable runner and state mechanic. - -The public promise of "launch multi-agent auto-research in the fewest lines" is -a forcing function for this split. It is not enough to make the user command -short if the auto-research preset becomes a second runner. Both the user layer -and the product preset must stay thin, so another product can reuse the same -kernel with a different role list, skill snippet, and evidence loop. - -Use this with: - -- [Multi-agent visible launcher v0](../reference/protocols/multi-agent-visible-launcher-v0.md) -- [Three-layer minimality contract](../reference/protocols/multi-agent-three-layer-minimality-v0.md) -- [Auto-research command path](../../demo/auto_research/README.md) - -## Layer Rule - -| Layer | Keep It Small By Owning | Must Not Own | -| --- | --- | --- | -| User | Topic, objective, rounds, optional role overrides, and optional data or eval entrypoint. | Tmux, Codex TUI launch flags, pane-local tick commands, quota/frontier plumbing, machine JSON routing, or role bootstrap scripts. | -| Product preset | Domain role list, agent scope, handoff/todo hints, worker skill snippet, default seed todos, and domain evidence or metric adapter. | Generic multi-agent runner, real Codex TUI panes, workspace/trust-safe launch, pane-local A2A tick, todo/evidence/status protocol, attach/stop/retry, or JSON visibility policy. | -| Kernel | Multi-agent runner, true interactive Codex panes, pane-local A2A tick, workspace/trust-safe launch, todo/evidence/status protocol, compact human status, role prompt scaffolding, and host lifecycle controls. | Research, support, benchmark, sales, or other product-specific semantics. | - -If a second product would need the same code, move that code into the kernel. -If the logic names a domain result, domain role, metric, artifact type, or -handoff meaning, keep it in the preset. - -## Product Preset Shape - -A preset should be close to data plus one or two small adapters. It should not -construct tmux commands, Codex invocation strings, or pane wrapper scripts. - -The minimum preset fields are: - -- `product_id`: stable product namespace, such as `auto-research`; -- `objective_fields`: user-facing intent fields the product accepts; -- `roles`: role list with `agent_id`, `role_id`, `lane_id`, and agent scope; -- `skill`: worker-local skill snippet or source path for each role; -- `handoff_hints`: how a role creates or completes LoopX todos for the next - role; -- `seed_todos`: first todo titles or action kinds for a fresh run; -- `evidence_adapter`: tiny domain writeback or metric loop, if the product - has scored evidence. - -Example product recipe: - -```json -{ - "schema_version": "generic_multi_agent_launch_spec_v0", - "goal_id": "loopx-demo-goal", - "session_name": "loopx-product-team", - "default_reasoning_effort": "high", - "roles": [ - { - "lane_id": "planner", - "agent_id": "codex-main-control", - "role_id": "planner", - "scope": "Turn the objective into one bounded todo and a review handoff.", - "skill": { - "name": "product-planner", - "source": "skills/planner/SKILL.md" - }, - "handoff_hints": [ - "Create a LoopX todo for builder when the plan is ready." - ] - }, - { - "lane_id": "builder", - "agent_id": "codex-side-bypass", - "role_id": "builder", - "scope": "Claim the selected todo, execute the bounded work, and write evidence.", - "skill": { - "name": "product-builder", - "source": "skills/builder/SKILL.md" - }, - "handoff_hints": [ - "Complete the todo or create a focused review todo with compact evidence." - ] - } - ] -} -``` - -The kernel expands that spec into a visible launcher packet with role profiles, -scoped LoopX wrappers, `$LOOPX_PANE_A2A_TICK`, artifact-only machine JSON, and -tmux host controls. -Every pane also receives the generic `$loopx-project` and `$loopx-doc-registry` -skill hints from the kernel role prompt. Product worker skills should not copy -those generic project/doc-registry instructions; they should only say how this -role decides, writes evidence, and hands off todos. - -## Worker Skill Snippet - -Each role should receive a small worker-local skill. The skill explains domain -behavior, not runner mechanics. - -Good skill snippet: - -```markdown -# product-builder - -Use when this pane has a selected builder todo. - -1. Read your selected frontier through LoopX state. -2. Work only inside the selected todo and role scope. -3. Write public-safe evidence before completing the todo. -4. If another role should continue, create a focused LoopX todo and name the - target role in the handoff hint. -``` - -Avoid putting these in a product skill: - -- how to start tmux; -- how to run Codex; -- how to write pane wrapper scripts; -- how to hide or redirect machine JSON; -- how quota/frontier/status files are routed; -- how attach, stop, retry, or workspace trust is implemented. - -Those are kernel responsibilities. - -## Handoff And Todo Hints - -The product preset should describe handoff in LoopX terms, not as a hidden -workflow engine. A role can: - -- complete the selected todo after writing compact evidence; -- create a successor todo with `claimed_by` or target role metadata when the - next role is known; -- leave blocker rationale when no safe action is available; -- write a public-safe evidence pointer for the next role to read. - -The preset should not directly call another agent, inject text into another -pane, or keep private side-channel state. Agents coordinate by reading and -writing the shared LoopX state surface: registry, runtime root, todo projection, -quota/frontier, run history, and public-safe evidence. - -## One-Command Launch - -For a custom composition, the generic path is: - -```bash -loopx multi-agent launch \ - --spec ./multi-agent-spec.public.json \ - --workspace "$PWD" \ - --execute \ - --attach -``` - -For the auto-research preset, the product path remains: - -```bash -loopx auto-research start "" --execute -``` - -Both commands should open real interactive Codex CLI TUI panes. The first -screen should be the role TUI, not raw JSON, shell streams, a hidden executor, -or a generated worktree trust prompt. - -For auto-research, a validated visible proof promotes the existing preset as -the reference recipe only when the proof keeps that layer split intact. The -operator still gets one command, but fixed-prompt wake, pane-local A2A tick, -runner lifecycle, compact status, and public artifact routing stay in the -generic multi-agent kernel. The auto-research preset may provide roles, -handoff hints, seed todos, and evidence defaults; it must not grow a private -launcher or hidden workflow driver to make the demo look shorter. - -## Attach, Stop, Retry - -Every visible multi-agent product should expose the same host controls: - -```bash -tmux attach -t -tmux kill-session -t -``` - -Retry means rerun the product command after refreshing quota/frontier/bootstrap -state. It must not replay stale hidden prompts or assume a previous pane still -has authority. Use `--replace-existing` only when the operator intentionally -wants to replace a session with the same name. - -Inside any pane, the user may interrupt the role and type normally. The pane is -a real Codex CLI agent, not a passive log viewer. - -## Acceptance Checklist - -A new product preset is healthy when: - -- the user-facing entrypoint is intent plus a few options; -- the preset defines role list, agent scope, skill snippet, handoff/todo hints, - seed todos, and optional evidence adapter; -- the preset imports the generic multi-agent kernel instead of duplicating - runner, TUI, tick, workspace, status, or JSON policy code; -- every visible pane starts as an interactive Codex CLI TUI; -- each role's first action is the pane-local A2A tick; -- machine JSON is written to public artifacts or an explicit machine channel, - not dumped into the first visible screen; -- todos and evidence are the only handoff authority; -- attach, stop, and retry are available without product-specific host logic; -- a promoted auto-research proof demonstrates the same one-command visible - path with real Codex TUI panes, fixed-prompt wake, pane-local A2A ticks, and - compact public-safe live evidence when lane-authored evidence is claimed; -- no raw logs, credentials, private material, absolute local paths, or - generated transcripts enter committed docs or fixtures. - -## Auto-Research As The Reference Preset - -Auto-research should stay a reference preset, not the kernel. Its preset owns: - -- research roles such as curator, hypothesis mapper, evidence runner, and - verifier; -- role-specific allowed actions; -- research handoff hints; -- metric/evidence loop defaults; -- seed todo wording for the demo. - -It should not own: - -- multi-agent runner lifecycle; -- real Codex TUI pane startup; -- pane-local A2A tick implementation; -- workspace trust handling; -- generic todo/evidence/status protocol; -- attach, stop, retry, or machine JSON visibility rules. - -When auto-research needs a new generic capability, implement it in the -multi-agent kernel first, then let the auto-research preset consume it through a -small role or evidence adapter. That is the path toward a reusable, lightweight -kernel and a small promotional example. diff --git a/docs/product/use-cases/auto-research/README.md b/docs/product/use-cases/auto-research/README.md index 901ddd468d..ab65a716db 100644 --- a/docs/product/use-cases/auto-research/README.md +++ b/docs/product/use-cases/auto-research/README.md @@ -1,5 +1,3 @@ # Auto Research -- [Decentralized auto-research showcase blueprint](decentralized-auto-research-showcase.md) - [Auto-research product metrics](auto-research-product-metrics.md) -- [Auto-research command path](../../../../demo/auto_research/README.md) diff --git a/docs/product/use-cases/auto-research/decentralized-auto-research-showcase.md b/docs/product/use-cases/auto-research/decentralized-auto-research-showcase.md deleted file mode 100644 index 8b7cdeca52..0000000000 --- a/docs/product/use-cases/auto-research/decentralized-auto-research-showcase.md +++ /dev/null @@ -1,438 +0,0 @@ -# Decentralized Auto-Research Showcase Blueprint - -This note turns the Arbor review into a LoopX product path. Arbor's public -showcase is strong because it is concrete: a benchmark, a hypothesis tree, -dev/held-out scores, replayable events, and a final report. LoopX should aim -for the same clarity while keeping its own architecture: decentralized agents -over one shared control plane, not one leader Coordinator. - -## What Arbor Demonstrates - -Arbor's public materials show a repeatable autonomous research loop: - -- a Research Contract that names objective, editable files, protected harness, - metric, budget, and review mode; -- an Idea Tree where hypotheses record status, evidence, score, branch, retry - status, grounding, and related-work audit; -- isolated executor worktrees for each experiment; -- a dev metric for iteration and a held-out metric for promotion; -- replay/report/export surfaces that make the run inspectable; -- a benchmark zoo, including `algotune_knn`, that is small enough for a user to - watch end to end. - -The Arbor `algotune_knn` demo is especially useful for LoopX because it is -public-safe, deterministic, CPU-only, and easy to explain: optimize a k-nearest -neighbors solver without editing the protected evaluator. Arbor reports an -example six-cycle improvement from roughly baseline speed to multi-x speedup, -with held-out validation. - -## LoopX Adaptation - -LoopX should reproduce the product value, not the topology. - -| Arbor shape | LoopX version | -| --- | --- | -| Coordinator-mediated tree management. | Kernel-owned evidence graph plus per-agent frontier projections. | -| Executors receive ideas from Coordinator. | Agents claim todo-backed hypotheses through `quota should-run --agent-id`. | -| Idea Tree is the durable memory. | `research_hypothesis_v0` plus `research_evidence_event_v0` in the shared state graph. | -| Merge/prune decided by Coordinator. | Promotion policy, held-out evidence, operator gates, and todo lifecycle decide. | -| Dashboard shows one run. | Frontstage shows lanes, claims, evidence, promotion candidates, and blockers. | - -The important product phrase is: - -> LoopX lets multiple agents run an autonomous research search without a -> leader agent: hypotheses, evidence, retries, and promotion decisions live in -> the control plane, and each agent receives only the frontier it is allowed to -> attempt. - -The executable product contract is split into three peer artifacts: - -- `decentralized_auto_research_state_v0` defines the records and projections: - contracts, todo-linked hypotheses, evidence events, frontier, evidence graph, - and compact decision candidates. -- `auto_research_lane_contract_v1` defines decentralized lanes: curator, - hypothesis proposer, executor, evaluator/promoter, and product narrator. Each - lane contributes typed records through claims and gates; none owns the whole - graph. -- `auto_research_role_state_machine_v0` defines the always-on digital employee - role map, state vocabulary, transition evidence, gate handoff, and user - takeover implications. -- [Auto-research product metrics](auto-research-product-metrics.md) defines - which user-value metrics the product surface should show. It intentionally - favors scored attempts, held-out lift, negative-evidence reuse, retry - recovery, and human promotion decisions over implementation counters such as - file count, smoke count, or dashboard row count. - -## Showcase Candidate - -**Title:** Decentralized Auto Research: k-NN Speedup - -**Public task:** make a brute-force k-nearest-neighbors solver faster while -preserving exact output. - -**Inputs:** - -- editable: `solution.py`; -- protected: `eval.py`, `task.py`, generated dev/test data; -- metric: `speedup`, higher is better; -- dev command: `bash eval.sh dev`; -- held-out command: `bash eval.sh test`; -- starting result: baseline around `1.0x`; -- expected value metric: best held-out speedup, plus number of useful negative - directions retained as future priors. - -**LoopX surfaces to show:** - -1. Research Contract card: objective, editable/protected scopes, metric, budget. -2. Decentralized frontier: which agent claimed which hypothesis, which ones are - blocked or retired, and why. -3. Evidence timeline: attempts, dev score, held-out score, branch/ref, retry - status. -4. Promotion decision: what got promoted, which alternatives were retired, and - which evidence proves the boundary. -5. User gates and takeover controls: first-screen review, promotion approval, - protected-scope stop, and real local-session launch must stay visible before - agents can convert experimental evidence into public positioning or live - process startup. -6. Product metrics: time to first scored attempt, useful hypotheses per active - day, held-out lift, negative-evidence reuse, retry recovery, and human - promotion decisions required. -7. Report: concise public-safe final summary with commands and artifacts. - -## Candidate Hypothesis Graph - -This graph is a fixture target, not a claim that LoopX has already achieved the -numbers. It gives the showcase a concrete shape to reproduce. - -```mermaid -graph TD - ROOT["Research contract: exact k-NN speedup"] - H1["hyp_001: vectorize pairwise distances"] - H2["hyp_002: partial selection with argpartition"] - H3["hyp_003: GEMM distance expansion"] - H4["hyp_004: batch query path"] - H5["hyp_005: dtype / contiguous array tuning"] - H6["hyp_006: approximate search (retire if correctness fails)"] - - ROOT --> H1 - H1 --> H2 - H2 --> H3 - H3 --> H4 - H4 --> H5 - H4 --> H6 -``` - -The user-facing point is not the exact technique. The point is that LoopX -retains the failed or bounded directions as explicit negative evidence, so the -next agent does not rediscover them from scratch. - -## Minimal Reproduction Plan - -New users should not have to learn the full command matrix first. The default -path previews a fresh demo-local goal surface and the visible Codex TUI lanes -that will work it: - -```bash -loopx --format json auto-research demo-e2e \ - --agent-id auto-research-operator -``` - -When the preview is acceptable, launch the visible lanes: - -```bash -loopx --format json auto-research demo-e2e \ - --agent-id auto-research-operator \ - --execute -``` - -This starts the product path: visible Codex TUI roles read their own -quota/frontier through LoopX state, then author public-safe research evidence or -successor todos from inside the pane. The pane-local tick is a guard/frontier -read; it does not manufacture research contracts, hypotheses, dev scores, or -holdout scores. - -The public line-count claim is intentionally narrower than the kernel. The -copyable recipe is one user command plus the four default auto-research role -specs; the reusable kernel still owns the runner, visible Codex TUI panes, -fixed wake prompt, pane-local A2A tick, todo/evidence/status protocol, and -public artifact routing. - -```text -loopx auto-research start "" --execute -research-curator:research-curator:research_curator -hypothesis-proposer:hypothesis-proposer:hypothesis_proposer -research-executor:research-executor:research_executor -evaluator-promoter:evaluator-promoter:evaluator_promoter -``` - -That is the marketing-safe claim: a five-line declarative recipe can start -decentralized A2A communication because the generic LoopX kernel already knows -how to wake each pane and let each pane read its own quota/frontier. - -The fixture-backed projection remains the read-only showcase state slice: - -```bash -loopx --format json auto-research frontier \ - --fixture examples/fixtures/decentralized-auto-research-knn.public.json \ - --agent-id hypothesis-proposer -``` - -This renders `decentralized_research_frontier_v0`, -`research_evidence_graph_v0`, and compact decision candidates from a public -fixture. It does not launch experiments or claim uplift; it proves that the -state shape can present a per-agent frontier without one leader agent. - -Public-safe evaluator outputs can also be converted into evidence records. A -minimal contract/eval pair is enough; the kernel does not require a shipped -domain pack: - -```bash -loopx --format json auto-research evidence \ - --contract research-contract.public.json \ - --eval-result dev-result.public.json \ - --eval-result holdout-result.public.json \ - --hypothesis-id hyp_state_a2a_round \ - --todo-id todo_auto_research_demo_001 \ - --agent-id research-executor \ - --claimed-by research-executor \ - --mechanism-family state_a2a_iteration \ - --hypothesis "Use a small state-mediated handoff loop to improve the shared candidate." \ - --branch-ref codex/auto-research-evidence-writer -``` - -The command emits an `auto_research_evidence_packet_v0` containing one -`research_hypothesis_v0` and split-aware `research_evidence_event_v0` records. -It preserves `needs_retry`, negative evidence, protected-scope clean flags, and -branch/artifact refs while keeping raw logs, local paths, and private artifacts -out of the public payload. - -Append the packet into LoopX's existing rollout event log when the evidence is -ready to become durable source state: - -```bash -loopx --format json auto-research append-evidence \ - --packet auto-research-evidence-packet.public.json -``` - -The append step writes one `research_hypothesis` rollout event and one -`research_evidence` event per split. It skips existing event ids on retry, which -keeps heartbeat-driven lanes replayable. - -After rollout evidence exists, the frontier read path exposes the compact -`research_evidence_graph_v0` plus promotion, retirement, and retry candidates. -That graph is the only durable research read model in the kernel; product -surfaces may consume it later, but they are not part of the auto-research -control loop. - -```bash -loopx --format json auto-research frontier \ - --goal-id loopx-auto-research-demo \ - --agent-id hypothesis-proposer -``` - -For fixture rehearsals, use `--fixture` instead of `--goal-id`. - -## Local Demo Supervisor - -The short-term multi-agent demo should be inspectable before it launches -anything. The supervisor command therefore starts as a dry-run packet: it plans -a visible tmux layout for multiple Codex CLI lanes, but it does not start tmux, -launch Codex, read session files, write LoopX state, or spend quota. - -```bash -loopx --format json auto-research demo-supervisor \ - --goal-id loopx-auto-research-demo -``` - -The packet has two important product properties: - -- the supervisor is a host shell layout, not a leader agent; -- every lane receives its own `quota should-run` and `auto-research frontier` - command, so work routing still comes from LoopX state, todo claims, gates, - and evidence graph projections. -- every lane prints `auto_research_role_profile_v0` before quota/frontier/ - bootstrap, so the visible worker knows its role, skill section, write scope, - protected scope, and stop conditions before it starts. -- the default "one-click" path is a dry-run rehearsal script: it checks the - required environment variables and prints the tmux start, attach, and stop - commands without starting tmux, launching Codex, writing LoopX state, or - spending quota. -- the packet names user takeover controls up front: inspect the rehearsal - output, paste the real start script only when ready, attach to tmux before - accepting any Codex prompt, and use the stop command or terminal interrupt to - take over. - -Operators can pass explicit lanes when rehearsing a real local demo: - -```bash -loopx --format json auto-research demo-supervisor \ - --goal-id loopx-auto-research-demo \ - --agent research-curator:research-curator:research_curator \ - --agent hypothesis-proposer:hypothesis-proposer:hypothesis_proposer \ - --agent research-executor:research-executor:research_executor \ - --agent evaluator-promoter:evaluator-promoter:evaluator_promoter -``` - -The third segment is optional for compatibility; when omitted, LoopX infers the -role from the lane name or from the default role order. The v0 default uses -four registered research-role agents: research curator, hypothesis proposer, -research executor, and evaluator/promoter. The explicit `--agent` form remains -the escape hatch for rehearsing a custom four-role layout. - -When the dry-run packet is acceptable, the same command can launch visible -local Codex CLI TUIs. This is intentionally opt-in: - -```bash -loopx auto-research demo-supervisor \ - --goal-id loopx-auto-research-demo \ - --agent research-curator:research-curator:research_curator \ - --agent hypothesis-proposer:hypothesis-proposer:hypothesis_proposer \ - --agent research-executor:research-executor:research_executor \ - --agent evaluator-promoter:evaluator-promoter:evaluator_promoter \ - --execute -``` - -`--execute` launches a tmux session. Use `--launcher tmux` to make the choice -explicit. With tmux, `--attach` immediately joins the session and -`--replace-existing` replaces a stale session with the same name. Takeover is -the normal pane interrupt, shell prompt, or session kill command. - -The executed launcher still is not a leader. Each lane window first runs its -own role profile, then runs `quota should-run`, then renders its own -`auto-research frontier`, then prints the `codex-cli-bootstrap-message`, and -only then starts `codex` with that visible bootstrap prompt. The launcher -itself does not write LoopX state or spend LoopX quota; any writeback must -happen through the visible Codex lane's normal LoopX todo/evidence commands. -All lanes share the same LoopX goal surface: registry, runtime root, frontier, -todo projection, and evidence graph. Workspace isolation is not applied to -every pane by default; only mutating research-executor attempts need a claimed -git worktree or equivalent execution boundary. - -The live worker path stays separate from the supervisor. The supervisor makes -lanes visible; `worker-loop` is the small state-mediated executor that polls -quota/frontier/todos and writes evidence through LoopX: - -```bash -loopx --format json auto-research worker-loop \ - --goal-id loopx-auto-research-demo \ - --agent-id research-curator \ - --agent-id hypothesis-proposer \ - --agent-id research-executor \ - --agent-id evaluator-promoter \ - --max-rounds 4 -``` - -When the dry-run selects safe runnable work, add `--execute` and -`--complete-selected-todo`. This keeps auto-research as a decentralized -state loop instead of growing a demo-specific leader workflow. - -### Visible Operator Rehearsal Path - -The first user-facing demo step is a single inspection command, not a hidden -launcher: - -```bash -loopx --format json auto-research demo-supervisor \ - --goal-id loopx-auto-research-demo -``` - -The user should see four concrete things in the packet: - -- `mode: dry_run`, plus a boundary block showing `starts_tmux`, - `runs_codex`, `writes_loopx_state`, and `spends_loopx_quota` are all false; -- one pane plan per default digital worker lane, with that lane's own - `auto_research_role_profile_v0`, `quota should-run`, `auto-research - frontier`, and `codex-cli-bootstrap-message` commands; -- a shared goal-surface contract showing that all panes use the same LoopX - registry/runtime/frontier/evidence graph, while mutation isolation is reserved - for research-executor attempts; -- a `start_script` array that can be copied into the user's shell only after - the user sets `LOOPX_PROJECT`, `LOOPX_REGISTRY`, and - `LOOPX_RUNTIME_ROOT`; -- a `lane_timeline` for each lane, making the visible sequence explicit: - role profile, quota guard, frontier projection, bootstrap prompt, then - visible Codex TUI; -- explicit takeover controls: `tmux attach -t loopx-auto-research` to inspect - every lane before accepting Codex prompts, and - `tmux kill-session -t loopx-auto-research` to stop the rehearsal. - -After `--execute`, the packet includes `launch_result` with the selected -launcher, started lanes, attach/stop command, and takeover instructions. For a -first live demo, prefer `--execute --attach` when tmux is installed. - -The safe demo acceptance bar is that the user can inspect the plan, attach to -the visible tmux session before any Codex prompt is accepted, interrupt any -lane manually, and confirm that each lane still routes through LoopX quota, -todo claims, frontier projection, and normal evidence writeback. The -supervisor never becomes a leader agent; it is only a shell layout that makes -the decentralized workers visible and interruptible. - -The launcher checklist is deliberately about observable behavior, not private -demo notes: each lane must show quota before frontier/bootstrap, attach and -stop controls must be visible, and dry-run boundary fields must show no tmux, -Codex, state, quota, credential, or session side effects. - -The generated dry-run shell plan uses environment placeholders such as -`LOOPX_PROJECT`, `LOOPX_REGISTRY`, and `LOOPX_RUNTIME_ROOT` instead of embedding -local absolute paths. The executed path resolves those values locally and -injects them into the launched visible terminals without recording the local -paths in the public packet. Same-session prompt injection remains blocked until -visible-attach and idle evidence pass. - -Next reproduction steps: - -1. Keep `research_contract_v0`, `research_hypothesis_v0`, and - `research_evidence_event_v0` as the public-safe record boundary. -2. Read `research_hypothesis` and `research_evidence` rollout events back into - the evidence graph instead of depending on fixture-only evidence. -3. Keep one local smoke that proves: - - protected files are not editable; - - each hypothesis is todo-linked; - - each evidence event names split and metric; - - no leader agent owns the graph; - - held-out promotion is required. -4. Build the showcase page from fixture evidence, then replace fixture numbers - with a real run when available. - -## Kernel/Capability Improvements - -P0 candidates: - -- **Research hypothesis ledger in core state.** Promote the existing - `hypothesis_ledger_v0` idea from the ML domain pack into a generic, - todo-linked research hypothesis shape. -- **Per-agent research frontier projection.** Extend status/quota projection so - a current agent sees only claim-compatible hypotheses and promotion - candidates, while other-agent claims remain visible context. -- **Retry semantics.** Add `needs_retry` as a reusable outcome for - incomplete/unscored research attempts, preserving branch/evidence refs. -- **Split-aware evidence.** Make dev/held-out split labels first-class in - evidence events and evidence graph projections. - -P1 candidates: - -- **Grounded ideation / novelty audit separation.** Add two explicit source - lanes so research input and novelty checking cannot contaminate each other. -- **Benchmark-zoo-style pack.** Add a `loopx research scaffold` path that turns - a small optimization task into a protected benchmark pack. -- **Replay/export surface.** Convert evidence graph events into a static HTML - replay for public-safe showcases. - -## Design Guardrails - -- The control plane may select a frontier; it must not become a hidden leader. -- Agents may propose and execute hypotheses within their claim/scope. -- Promotion requires evidence and gate policy, not a persuasive chat summary. -- A user can inspect every branch of the research graph from source refs. -- Public showcase pages must distinguish "fixture target" from "achieved run". -- Private docs, non-public links, raw logs, credentials, local paths, and raw - benchmark traces stay out of public artifacts. - -## Suggested Public Narrative - -LoopX does for autonomous research what it already does for long-running -engineering agents: it turns a noisy loop into a managed control plane. The -novel part is that research hypotheses become first-class work items with -claims, evidence, retry state, and promotion gates. The result can look like an -Arbor-style hypothesis tree to the user, while the implementation remains -LoopX-native and decentralized. diff --git a/docs/reference/protocols/README.md b/docs/reference/protocols/README.md index 60c59cd8d3..443dc4f8cd 100644 --- a/docs/reference/protocols/README.md +++ b/docs/reference/protocols/README.md @@ -47,8 +47,6 @@ until their documented migration is qualified. - [`long_horizon_agent_state_protocol_v0`](long-horizon-agent-state-protocol-v0.md): Long-horizon agent state protocol v0 - [`material_lifecycle_architecture_v0`](material-lifecycle-architecture-v0.md): Material lifecycle architecture v0 - [`material_lifecycle_architecture_v0`](material-lifecycle-architecture-v0.zh-CN.md): Material lifecycle architecture v0 (中文) -- [`multi_agent_three_layer_minimality_contract_v0`](multi-agent-three-layer-minimality-v0.md): Multi-agent three-layer minimality v0 -- [`multi_agent_visible_launcher_v0`](multi-agent-visible-launcher-v0.md): Multi-agent visible launcher v0 - [`peer_agent_directory_v0`](peer-agent-directory-and-observation-v0.md): Peer agent directory, bounded observation and delivery v0 - [`peer_agent_runtime_v1`](peer-agent-runtime-v1.md): Peer agent runtime v1 - [`peer_supervisor_v0`](peer-supervisor-v0.md): Peer supervisor v0 @@ -81,10 +79,6 @@ until their documented migration is qualified. ## Domain Capabilities -- [`auto_research_lane_contract_v1`](auto-research-lane-contract-v1.md): Auto-research lane contract v1 -- [`auto_research_role_profile_v0`](auto-research-role-profile-v0.md): Auto-research role profile v0 -- [`auto_research_role_state_machine_v0`](auto-research-role-state-machine-v0.md): Auto-research role state machine v0 -- [`decentralized_auto_research_state_v0`](decentralized-auto-research-state-v0.md): Decentralized auto-research state v0 - [`content_ops_surface_v0`](content-ops-surface-v0.md): Content operations surface v0 - [`cs_notes_explore_capability_map_v0`](cs-notes-explore-capability-map-v0.md): CS notes Explore capability map v0 - [`issue_fix_acceptance_loop_v0`](issue-fix-acceptance-loop-v0.md): Issue-fix acceptance loop v0 diff --git a/docs/reference/protocols/auto-research-lane-contract-v1.md b/docs/reference/protocols/auto-research-lane-contract-v1.md deleted file mode 100644 index a0ed8de53c..0000000000 --- a/docs/reference/protocols/auto-research-lane-contract-v1.md +++ /dev/null @@ -1,161 +0,0 @@ -# auto_research_lane_contract_v1 - -`auto_research_lane_contract_v1` defines how LoopX runs auto research with -several independent agent lanes over one shared control plane. It is a role and -capability contract, not a new coordinator service. - -The contract answers one product question: how can a user get an Arbor-style -research loop while LoopX stays decentralized? - -## Design Principles - -- The source of truth is the LoopX graph: todos, claims, rollout events, - evidence packets, gates, and read-only projections. -- Lanes contribute typed records. They do not own the full research tree. -- Selection is local to the requesting agent through - `quota should-run --agent-id ...`. -- Promotion is a policy decision over evidence, not a persuasive summary. -- Advisory lanes may shape hypotheses, but only evidence lanes can promote or - retire hypotheses. - -## Lane Roles - -| Role | Capability token | Primary contribution | Writes | Must not | -| --- | --- | --- | --- | --- | -| Curator | `research_curator` | Defines or refreshes the public-safe research contract, metric, protected scope, and novelty boundary. | `research_contract_v0`, grounding refs. | Select winners, edit protected evaluators, or own executor queues. | -| Hypothesis proposer | `hypothesis_proposer` | Proposes grounded, todo-backed hypotheses and parent-child refinements. | `research_hypothesis_v0`, agent todos, grounding refs. | Claim novelty from the same material used for ideation. | -| Executor | `research_executor` | Runs one claimed hypothesis in an isolated worktree and produces split-aware results. | branch refs, eval result projections, `auto_research_evidence_packet_v0`. | Mutate protected scope, promote results, or hide failed attempts. | -| Evaluator / promoter | `evaluator_promoter` | Converts scored attempts into promotion, retry, or retirement candidates, then records an explicit terminal decision when the contract permits it. | `research_evidence_event_v0`, promotion or retirement candidate projections, `auto_research_terminal_decision_v0`, gate todos. | Treat dev-only lift as promoted evidence, bypass owner gates, or label its own review independent. | -| Product narrator | `product_narrator` | Renders the public-safe case story from terminal decisions and projections for downstream product surfaces. | `research_evidence_graph_v0`, Explore result events, public docs, screenshots after first-screen review when needed. | Invent metrics, certify candidates as terminal, read private source bodies, or mutate research evidence. | - -No lane is privileged. A single Codex session may implement more than one role -when it has the matching todo claim and boundary, but the graph must still show -which capability produced which record. - -## Lane Claim Packet - -Each lane action should be representable as a compact claim packet. This packet -can live in a todo metadata projection, a rollout event detail, or a future -kernel API, but it must remain public-safe. - -```json -{ - "schema_version": "auto_research_lane_claim_v1", - "goal_id": "loopx-auto-research-demo", - "lane_role": "research_executor", - "agent_id": "research-executor", - "todo_id": "todo_auto_research_demo_001", - "hypothesis_id": "hyp_state_a2a_round", - "capability_token": "research_executor", - "allowed_actions": ["run_dev_attempt", "run_holdout_attempt", "write_evidence_packet"], - "write_scope": ["auto_research_evidence_packet_v0", "rollout_event_log"], - "blocked_by": [] -} -``` - -The claim packet is not a lock on the full graph. It only explains why this -agent may take the next bounded action. - -## Source And Projection Flow - -```mermaid -flowchart LR - Contract["research_contract_v0"] - Todo["todo_item_v0 claim"] - Hypothesis["research_hypothesis_v0"] - Evidence["research_evidence_event_v0"] - Graph["research_evidence_graph_v0"] - Frontier["decentralized_research_frontier_v0"] - Showcase["research_evidence_graph_v0"] - - Contract --> Hypothesis - Todo --> Hypothesis - Hypothesis --> Evidence - Evidence --> Graph - Graph --> Frontier - Graph --> Showcase -``` - -The graph can be rendered as a tree, but it is not owned by a tree manager. A -lane appends or updates the smallest source record it is authorized to touch; -projection builders derive frontiers and product views afterward. - -Successor work should stay equally small. A role profile may declare a -`successor_todos` rule such as "after `run_dev_eval`, if dev evidence is -supported and no holdout exists, add `run_holdout_eval` for -`research-executor`." The pane-local tick applies that declaration by writing a -normal LoopX todo with `claimed_by`, `action_kind`, and `unblocks_todo_id`. -The next agent still re-enters through its own `quota should-run` and frontier; -there is no separate continuation projector or central research manager. - -## Capability Rules - -1. A `research_curator` may create or amend `research_contract_v0` only inside - allowed docs/example scopes and with protected scope explicitly named. -2. A `hypothesis_proposer` may create a hypothesis only when it is backed by an - agent todo and has `claimed_by`, `todo_id`, `mechanism_family`, and - grounding refs or a clear no-grounding reason. -3. A `research_executor` may run only the hypothesis selected by current - agent-scoped quota or an explicitly claimed role-declared successor todo. -4. An `evaluator_promoter` may promote only when dev evidence, held-out - evidence, clean boundary, and required gates are present. -5. A registered peer may claim independent review only when it differs from - both the hypothesis producer and terminal-decision agent, binds the exact - evidence-graph revision, and records `approve`, `reject`, or - `needs_more_evidence`. Review does not mutate research truth. -6. A `product_narrator` may publish only from `research_evidence_graph_v0`, - an explicit terminal decision, and the current review projection; - independent approval is required for `confirmed` or `refuted` findings. - Without it, the finding remains `tentative`. Related evidence must stay - public-safe, and first-screen public surface changes still obey the - first-screen review gate. - -## Gates - -| Gate | Applies to | Required signal | -| --- | --- | --- | -| Protected boundary gate | Executor and promoter | `protected_scope_clean=true`, no protected file edits. | -| Held-out gate | Promoter | Held-out metric improves under the contract direction. | -| Novelty gate | Promoter or narrator | Independent novelty audit ref when claiming research novelty. | -| Owner gate | Promoter or public narrator | Explicit user/controller gate when merge, publication, or private boundary requires one. | -| First-screen gate | Product narrator | Preview before changing first viewport, hero, primary CTA, or opening nav. | - -## Promotion And Retirement - -Promotion and retirement are both useful outcomes: - -- promotion candidate: supported or promoted hypothesis with public-safe dev - evidence, required holdout/boundary checks, and a todo/branch/evidence ref; -- retirement candidate: contradicted or retired hypothesis, negative evidence, - guardrail failure, or repeated retry exhaustion; -- retry candidate: unscored but resumable attempt with a branch/ref and - `needs_retry` evidence. -- terminal decision: a separate evidence-revision-bound `promoted` or - `retired` record; candidate status alone never creates one; -- peer review: a separate receipt over the current terminal decision. A - same-agent review is visible but cannot produce a confirmed or refuted - finding. - -The product board should show all three. The user should see which result is -worth merging, which direction saved future search time by failing clearly, and -which attempt needs another bounded executor turn. - -## Acceptance Checks - -An implementation satisfies this lane contract when: - -- every lane role above is present as a named capability; -- every executable hypothesis has `todo_id`, `claimed_by`, and agent-scoped - quota selection; -- promotion and retirement candidates are derived from - `research_evidence_graph_v0`, not fixture-only prose; -- terminal results can be queried by exact hypothesis id, including history, - without loading raw transcripts; -- stale evidence revisions and conflicting terminal decisions fail closed to a - tentative or blocked projection; -- independent-review claims require a different registered peer and never - rewrite scored evidence or terminal decisions; -- grounded ideation and novelty audit remain separate lanes; -- no public surface needs a leader or coordinator agent to explain ownership; -- public docs and projections contain no raw logs, private paths, credentials, - internal documents, or raw protected evaluator material. diff --git a/docs/reference/protocols/auto-research-role-profile-v0.md b/docs/reference/protocols/auto-research-role-profile-v0.md deleted file mode 100644 index 38566eb70e..0000000000 --- a/docs/reference/protocols/auto-research-role-profile-v0.md +++ /dev/null @@ -1,221 +0,0 @@ -# auto_research_role_profile_v0 - -`auto_research_role_profile_v0` defines how a LoopX auto-research worker knows -who it is before it loads any role-specific playbook. It bridges three existing -surfaces: - -- the shared LoopX control plane, which grants identity and authority; -- worker-local role playbooks, which define how to act inside a phase; -- repository or workspace `AGENTS.md`, which defines long-lived local rules. - -The contract exists because Arbor and LoopX use skills differently. Arbor can -keep identity mostly inside a Coordinator / Executor topology, then load skills -as phase playbooks. LoopX is decentralized: several workers may run in separate -Codex sessions over one state graph, so identity must be explicit control-plane -state rather than something inferred from a skill name or pane title. - -## Ownership Split - -| Surface | Owns | Does not own | -| --- | --- | --- | -| LoopX control plane | `agent_id`, `role_id`, claim, capability token, phase, write boundary, gate state, and stop condition. | The detailed reasoning checklist for an implementation phase. | -| Worker-local role playbook | Commands, checklists, artifact schema reminders, review prompts, and phase-specific failure modes. | Authority to write, promote, merge, publish, or bypass gates. | -| `AGENTS.md` | Repository-local and workspace-local rules such as private boundary, PR hygiene, first-screen review, protected paths, and local launch policy. | Dynamic role assignment or current frontier selection. | -| Host launcher | Visible panes, environment variables, attach/stop controls, and takeover affordances. | Research truth, promotion decisions, or hidden scheduling authority. | - -This split keeps playbooks useful without letting them become a second source of -identity. A worker can load the same `loopx-auto-research` playbook in different -roles, but the profile tells it which section applies and which writes are -allowed. - -## Profile Shape - -The profile is public-safe and small enough to embed in a launcher packet, -frontier item, bootstrap prompt, or future kernel API. - -```json -{ - "schema_version": "auto_research_role_profile_v0", - "goal_id": "loopx-auto-research-demo", - "agent_id": "research-executor", - "role_id": "research_executor", - "display_name": "Research executor", - "phase": "attempt_running", - "capability_token": "research_executor", - "todo_id": "todo_auto_research_demo_001", - "hypothesis_id": "hyp_state_a2a_round", - "allowed_actions": ["claim_attempt", "edit_allowed_scope", "run_dev_eval", "write_evidence"], - "write_scope": ["solution.py", "experiments/**"], - "protected_scope": ["task.py", "eval.py", "data/**"], - "required_skill": "loopx-auto-research", - "skill_section": "Research executor", - "agents_overlay": ["workspace/AGENTS.md"], - "stop_conditions": [ - "quota should-run returns false", - "frontier is empty or claimed by another agent", - "protected scope would be edited", - "private material or credentials are required", - "operator gate is projected" - ], - "handoff_outputs": [ - "research_evidence_event_v0", - "branch_or_artifact_ref", - "retry_or_retirement_rationale" - ], - "successor_todos": [ - { - "after_action": "run_dev_eval", - "when": "dev_supported_without_holdout", - "target_agent_id": "research-executor", - "target_role_id": "research_executor", - "task_class": "advancement_task", - "action_kind": "run_holdout_eval", - "text": "Run held-out validation for the dev-supported hypothesis." - } - ], - "continuation_policy": { - "schema_version": "auto_research_continuation_policy_v0", - "successor_source": "role_profile.successor_todos", - "required_holdout_improvement_count": 2, - "unmet_target_rule": "when the target is unmet and a successor condition is satisfied, create or link that successor", - "no_followup_rule": "no-follow-up is valid only after target reached, projected blocker/gate, or evidence-backed retirement" - } -} -``` - -Required fields: - -- `goal_id`, `agent_id`, `role_id`, `phase`, and `todo_id` identify the current - worker and work item. -- `capability_token` must match a capability in - `auto_research_role_state_machine_v0`. -- `allowed_actions`, `write_scope`, `protected_scope`, and `stop_conditions` - bound what the worker may do. -- `required_skill` and `skill_section` route the worker to the correct how-to - instructions after identity is resolved. - -Optional fields such as `hypothesis_id`, `agents_overlay`, `handoff_outputs`, -`successor_todos`, and `continuation_policy` make the profile more ergonomic but do not grant -authority. A successor todo declaration is a small role-local handoff rule: when -the worker completes `after_action` and the `when` condition is satisfied, the -pane-local tick may write the named LoopX todo for `target_agent_id`. It is not a -graph-wide planner, and it must still pass quota, todo metadata, and -public/private boundary checks before another agent can run it. -`continuation_policy` is the compact acceptance guard for no-follow-up: if the -target is not met and a role-declared successor condition is satisfied, the -worker must create or link that successor instead of closing the lane. - -For failure continuation, the successor's title and `unblocks_todo_id` share -the failed evidence Todo from `research_failure_continuation_v0.source_todo_id`. -Several summaries of that lineage therefore create or reuse one identical -successor without rebinding its original parent. A historical successor may -retain its original summary parent when the complete canonical records prove -that summary belongs to the same failed evidence lineage; all other intent -fields still pass the native duplicate check. Each completed summary keeps -its own `successor_todo_ids` link to that successor. Other role handoffs still -bind the selected source Todo. These links grant no execution or claim rights; -the canonical owner continues to reject same-title requests with different -lineage or other intent. Historical-parent reuse binds create to the inspected -provider revision; a concurrent change refuses without rebinding and requires -a fresh read before retry. - -## Resolution Order - -Every visible auto-research worker should resolve its instructions in this -order: - -1. Read the role profile from the launcher packet, frontier item, or bootstrap - prompt. -2. Run `quota should-run --goal-id ... --agent-id ...` and confirm the selected - todo still matches the profile. -3. Read the current role/state-machine contract for allowed transitions. -4. Load the required skill and only the section named by `skill_section`. -5. Read applicable `AGENTS.md` overlays for repository and workspace rules. -6. Act within `allowed_actions` and `write_scope`, then write only the listed - handoff outputs and role-declared successor todos. - -If any source conflicts, the worker fails closed in this order: - -1. operator gate or private/security boundary; -2. control-plane quota, claim, capability, or write-scope mismatch; -3. repository `AGENTS.md` safety rule; -4. skill checklist disagreement; -5. launcher pane label or cosmetic prompt text. - -## Auto-Research Role Profiles - -The v0 demo should expose four logical always-on profiles. A host may render -fewer panes only when the merged duties are explicit in the profile and the -records still name the role that produced each transition. - -| Role id | Skill section | Primary phase | Writes | Must stop when | -| --- | --- | --- | --- | --- | -| `research_curator` | Research curator | `contract_ready`, `promotion_gate` | `research_contract_v0`, owner gate todos, protected-boundary notes. | The next step would select a winner, run an experiment, or publish unsupported evidence. | -| `hypothesis_proposer` | Hypothesis proposer | `hypothesis_proposed`, `retired` | `research_hypothesis_v0`, successor todos, no-follow-up rationale. | Novelty requires the same source that inspired the idea, or negative evidence would be hidden. | -| `research_executor` | Research executor | `frontier_selected`, `attempt_running` | Branch refs, dev/holdout eval evidence, retry packets, role-declared successor todos. | Protected scope changes, promotion decisions, or private/raw artifacts are needed. | -| `evaluator_promoter` | Evaluator/promoter | `evidence_recorded`, `evaluated`, `promotion_gate` | Held-out validation evidence, evaluation summary, promotion/retirement candidates, gate todos. | Evidence is dev-only but would be presented as promoted, or held-out data is missing when required. | - -Future roles such as gate steward, synthesis narrator, and frontier janitor are -split candidates, not required v0 panes. They should be introduced only when -evidence from the demo shows that a transition duty needs a separate owner. - -## Worker-Local Playbook Strategy - -Use one worker-local `loopx-auto-research` playbook that contains role sections: - -- `Research curator` -- `Hypothesis proposer` -- `Research executor` -- `Evaluator/promoter` -- `Visible takeover and stop controls` - -This keeps each visible worker aligned without exposing auto-research as a -global LoopX skill for ordinary project agents. Split into role-specific -worker playbooks only after a visible run shows repeated confusion that a single -role-routed playbook cannot prevent. The split decision should cite evidence -such as wrong section loading, unauthorized writes, hidden negative evidence, or -missed stop conditions. - -The playbook body should not invent current work. It should tell the worker to -read the role prompt/profile context and run role-local quota/frontier commands -through the pane-local LoopX wrapper inside the Codex TUI. This mirrors Arbor's -useful "load a checklist at the exact phase" behavior while preserving LoopX's -decentralized identity model. - -## Demo Launcher Implications - -A visible tmux or terminal launcher should silently materialize the profile and -start one fresh interactive Codex CLI TUI per role: - -```bash -export LOOPX_GOAL_ID=loopx-auto-research-demo -export LOOPX_AGENT_ID=research-executor -export LOOPX_ROLE_ID=research_executor -export LOOPX_ROLE_PROFILE_REF=auto_research_role_profile_v0 -exec codex -c model_reasoning_effort=high -C "$LOOPX_PROJECT" "$ROLE_PROMPT" -``` - -The pane title is cosmetic. The profile and quota/frontier projection are the -authority, but raw profile/frontier JSON should stay in local artifacts or an -explicit machine channel. The launcher must keep attach, interrupt, and stop -commands visible so the user can take over without reading hidden logs. - -## Acceptance Checks - -An implementation satisfies this contract when: - -- launcher and frontier packets expose `auto_research_role_profile_v0` for each - visible worker without printing raw profile JSON on the first screen; -- each profile names `agent_id`, `role_id`, `phase`, `capability_token`, - `allowed_actions`, `write_scope`, `protected_scope`, `required_skill`, - `skill_section`, and `stop_conditions`; -- the default demo uses the four logical v0 role identities, with owner takeover - as a visible control surface rather than a research role; -- the role-aware skill says identity comes from the profile and quota/frontier, - not from the skill itself; -- the profile and skill make no-follow-up depend on evidence against the - continuation target, not on one successful promotion candidate alone; -- `AGENTS.md` overlays can add stricter local rules but cannot expand a role's - authority beyond the control-plane profile; -- validation proves no leader/coordinator pane is required for the demo to be - understandable or interruptible. diff --git a/docs/reference/protocols/auto-research-role-state-machine-v0.md b/docs/reference/protocols/auto-research-role-state-machine-v0.md deleted file mode 100644 index 50d3683a32..0000000000 --- a/docs/reference/protocols/auto-research-role-state-machine-v0.md +++ /dev/null @@ -1,191 +0,0 @@ -# auto_research_role_state_machine_v0 - -`auto_research_role_state_machine_v0` defines the always-on digital employee -model for LoopX auto research. It maps Arbor-like research roles onto LoopX's -decentralized control plane without adding a leader agent, scheduler service, -or second source of truth. - -This contract answers a narrower question than the state and lane contracts: -which digital employee role may move a research item from one state to the -next, and what evidence must exist before that transition is visible to users? - -## Companion Contracts - -The auto-research kernel is intentionally split into three protocol surfaces: - -| Contract | Owns | Does not own | -| --- | --- | --- | -| `decentralized_auto_research_state_v0` | Record and projection shapes: contract, todo-linked hypothesis, evidence event, frontier, evidence graph, evidence graph projection. | Which role may write each transition. | -| `auto_research_lane_contract_v1` | Capability ownership for curator, proposer, executor, evaluator, and narrator lanes. | The ordered state machine and transition evidence. | -| `auto_research_role_state_machine_v0` | Digital employee role map, state transition rules, takeover gates, and no-leader invariants. | Runtime scheduling, model prompts, or hidden orchestration. | -| `auto_research_role_profile_v0` | Per-worker identity packet: `agent_id`, role, phase, capability, write scope, required skill section, AGENTS overlay, and stop conditions. | Research source records, phase checklist prose, or authority beyond quota/frontier. | - -All three contracts are peer artifacts under `docs/reference/protocols/`. -Together they are the public-safe control-plane map for auto research. - -## Digital Employee Role Map - -These roles can run continuously as separate LoopX agent todos, monitors, or -Codex sessions. None owns the full graph. The first version intentionally keeps -the always-on role set small; gate handling, public narration, and frontier -cleanup are transition duties rather than separate permanent workers. - -| Role | Capability token | Primary job | May write | Must not | -| --- | --- | --- | --- | --- | -| Research curator | `research_curator` | Keep the objective, editable scope, protected scope, metric, stop policy, and operator gates explicit. | `research_contract_v0`, protected-boundary notes, owner gate todos, public projection requests. | Pick winners, run experiments, or publish unsupported claims. | -| Hypothesis proposer | `hypothesis_proposer` | Turn research ideas into todo-backed hypotheses, parent links, mechanism families, and bounded retire/successor decisions. | `research_hypothesis_v0`, successor todos, grounding refs, no-follow-up rationale. | Claim novelty from the same source used to ideate or delete negative evidence. | -| Research executor | `research_executor` | Execute selected hypotheses in isolated worktrees and preserve dev or held-out attempt evidence. | branch refs, `research_evidence_event_v0`, retry packets. | Edit protected scope, hide failures, or promote results. | -| Evaluator/promoter | `evaluator_promoter` | Classify evidence as supported, contradicted, retry-needed, promotion-ready, or retirement-ready, and record a terminal decision only after the required gate. | evaluation summary, promotion candidate, retirement candidate, `auto_research_terminal_decision_v0`, gate todo, projection-ready evidence. | Treat dev-only lift as promoted, override missing held-out evidence, or self-certify an independent review. | - -The role names are product-facing labels. A single Codex session may perform -multiple roles only when it has the corresponding todo claim, capability, and -write boundary; the appended record must still name the role that produced it. - -Future versions may split gate stewardship, public narration, or frontier -janitor work into separate always-on roles after the demo proves those duties -need independent ownership. Until then, they remain transition duties owned by -the four-role v0 map above. - -## Future Role Splits - -These candidate roles are intentionally outside the v0 always-on role set. They -are recorded so the product roadmap can grow the digital-employee map without -reintroducing a leader or coordinator agent. - -| Future role | Current v0 duty | Split trigger | Still must not | -| --- | --- | --- | --- | -| Gate steward | Research curator plus evidence verifier handle `operator_gate` and `promotion_gate` transitions. | Gates become frequent enough that wait reasons, owner questions, and unblock evidence need independent monitoring. | Approve its own gate, bypass owner decisions, or select experiments. | -| Synthesis narrator | Read-only projection builder converts the evidence graph plus terminal decisions and reviews into queryable Explore results. | Users need a continuously updated report lane that summarizes evidence without slowing runners or verifiers. | Certify scores, hide negative evidence, or mutate source records. | -| Frontier janitor | Hypothesis mapper and evidence verifier retire duplicates, exhausted retries, and no-follow-up branches. | The frontier grows large enough that stale hypotheses crowd out active research. | Delete evidence, rewrite todo ownership, or prune without a public rationale. | - -Promoting any future role requires a smoke update that proves the role is a -bounded lane over the shared control plane, not a coordinator with authority -over the full graph. - -## State Vocabulary - -`auto_research_state_transition_v0` uses the following durable states: - -| State | Meaning | Typical next states | -| --- | --- | --- | -| `contract_ready` | The objective, editable/protected scope, metric, budget, and promotion policy are public-safe and explicit. | `hypothesis_proposed` | -| `hypothesis_proposed` | A todo-backed hypothesis exists, but no agent has started its attempt. | `frontier_selected`, `retired` | -| `frontier_selected` | `quota should-run --agent-id ...` selected the hypothesis for the current agent. | `attempt_running`, `operator_gate` | -| `attempt_running` | A claimed evidence runner is working in an isolated worktree. | `evidence_recorded`, `needs_retry` | -| `evidence_recorded` | Attempt evidence exists with split, metric, branch/artifact refs, and boundary facts. | `evaluated` | -| `evaluated` | A verifier classified the evidence under the research contract. | `supported`, `contradicted`, `needs_retry`, `promotion_gate`, `retired` | -| `supported` | Dev evidence supports the direction, but promotion is not complete. | `promotion_gate`, `attempt_running`, `retired` | -| `needs_retry` | Attempt is inconclusive but resumable from a ref or clearly bounded retry. | `frontier_selected`, `retired` | -| `contradicted` | Evidence shows regression, correctness failure, or guardrail failure. | `retired`, `hypothesis_proposed` | -| `promotion_gate` | Promotion requires held-out evidence, owner decision, merge gate, or public-boundary review. | `promoted`, `retired`, `operator_gate` | -| `promoted` | Promotion policy accepted the result into the current best artifact. | peer review, Explore result projection | -| `retired` | The direction is no longer active; negative evidence remains queryable. | peer review, Explore result projection | - -## State Machine - -```mermaid -flowchart TD - Contract["contract_ready"] - Proposed["hypothesis_proposed"] - Selected["frontier_selected"] - Running["attempt_running"] - Evidence["evidence_recorded"] - Evaluated["evaluated"] - Supported["supported"] - Retry["needs_retry"] - Contradicted["contradicted"] - Gate["promotion_gate"] - Promoted["promoted"] - Retired["retired"] - Showcase["research_evidence_graph_v0"] - - Contract --> Proposed - Proposed --> Selected - Proposed --> Retired - Selected --> Running - Selected --> Gate - Running --> Evidence - Running --> Retry - Evidence --> Evaluated - Evaluated --> Supported - Evaluated --> Retry - Evaluated --> Contradicted - Evaluated --> Gate - Evaluated --> Retired - Supported --> Gate - Supported --> Running - Retry --> Selected - Retry --> Retired - Contradicted --> Retired - Contradicted --> Proposed - Gate --> Promoted - Gate --> Retired - Promoted --> Showcase - Retired --> Showcase -``` - -## Transition Rules - -| Transition | Required role | Required evidence | -| --- | --- | --- | -| `contract_ready -> hypothesis_proposed` | Hypothesis mapper | `research_contract_v0`, `todo_id`, `claimed_by`, mechanism family, grounding refs or no-grounding reason. | -| `hypothesis_proposed -> frontier_selected` | LoopX quota projection | `quota should-run --agent-id ...` selected the todo and write boundary allows the attempt. | -| `frontier_selected -> attempt_running` | Evidence runner | agent claim, isolated worktree or equivalent execution boundary, protected scope reminder. | -| `attempt_running -> evidence_recorded` | Evidence runner | split label, metric status, branch/artifact ref, protected-scope clean flag, raw-private-artifact flags. | -| `evidence_recorded -> evaluated` | Evidence verifier | contract policy applied to scored or unscored evidence. | -| `evaluated -> supported` | Evidence verifier | dev evidence improves or otherwise satisfies the contract's support threshold. | -| `evaluated -> contradicted` | Evidence verifier | regression, correctness failure, boundary violation, or novelty failure. | -| `evaluated -> needs_retry` | Evidence verifier | unscored attempt with resumable ref or explicit bounded retry reason. | -| `evaluated -> promotion_gate` | Evidence verifier | holdout candidate, clean boundary, pending owner/merge/publication decision. | -| `promotion_gate -> promoted` | Research curator plus evidence verifier | held-out evidence when required, clean boundary, applicable operator gate accepted, and an explicit evidence-revision-bound terminal decision. | -| `supported|contradicted|needs_retry -> retired` | Hypothesis mapper plus evidence verifier | negative evidence, retry exhaustion, duplicate proof, or no-follow-up rationale plus an explicit retirement reason. | -| `promoted|retired -> auto_research_peer_review_v0` | A registered peer distinct from producer and decision agent when independence is claimed | exact terminal-decision ref, exact evidence-graph revision, public-safe evidence refs, and approve/reject/needs-more-evidence verdict. | -| `promoted|retired -> loopx_explore_result_event_v0` | Read-only projection builder | current evidence graph, terminal decision, and review projection refs only; no direct mutation of source records. | - -## No-Leader Invariants - -- No role owns the full graph or can rewrite global research truth. -- `quota should-run --agent-id ...` selects only the current agent frontier. -- Every executable hypothesis remains backed by a `todo_id` and `claimed_by`. -- Promotion is evidence plus gate policy, not a persuasive summary. -- A promotion or retirement candidate is not a terminal result; the terminal - decision is explicit and bound to one evidence-graph revision. -- Peer review is a receipt over the decision. It cannot rewrite the hypothesis - or evidence, and same-agent review is never labeled independent. -- A single promoted branch does not close a multi-round target by itself; when - the role profile's continuation target is unmet, the last role in the cycle - creates or links the next role-declared successor todo. -- Public narration reads `research_evidence_graph_v0`; it does not - certify scores or mutate source state. -- Gate handling is a transition duty; no role can bypass an operator gate. -- Failed, contradicted, and retry-exhausted attempts stay visible as negative - evidence unless a public/private boundary requires redaction. - -## Demo And Takeover Implications - -The visible auto-research demo can launch several digital employees only as a -user-visible rehearsal first. The default packet should remain `dry_run`: it -may show tmux panes, commands, and takeover controls, but it must not start -Codex, write LoopX state, or spend quota by itself. - -When a user opts into the real demo, each lane still runs its own: - -```bash -loopx --format json --registry "$LOOPX_REGISTRY" \ - quota should-run --goal-id "$LOOPX_GOAL_ID" --agent-id "$LOOPX_AGENT_ID" -``` - -and each lane reads its own auto-research frontier. The shell layout is only a -visibility and takeover surface, not a coordinator. - -## Acceptance Checks - -An implementation satisfies this role/state-machine contract when: - -- the digital employee role map is visible to users and docs; -- state transitions name the role and evidence required to move forward; -- the smoke suite checks `auto_research_role_state_machine_v0` next to the - state and lane contracts; -- generated demo packets expose user takeover controls before execution; -- no public artifact centralizes graph ownership in a leader, coordinator, or - supervisor role. diff --git a/docs/reference/protocols/auto-research-wish-to-artifact-v0.md b/docs/reference/protocols/auto-research-wish-to-artifact-v0.md deleted file mode 100644 index 59328459fa..0000000000 --- a/docs/reference/protocols/auto-research-wish-to-artifact-v0.md +++ /dev/null @@ -1,88 +0,0 @@ -# Auto Research Wish-to-Artifact v0 - -Auto Research accepts an open question today and can already produce -hypotheses, dev and held-out evidence, terminal decisions, independent review, -and Explore findings. This protocol closes the delivery contract around those -existing records. It does not add another scheduler, research store, or -capability. - -## Contract - -`auto_research_delivery_contract_v0` wraps the existing -`research_contract_v0` with: - -- one original public-safe wish and a stable `wish_id`; -- assumptions and non-goals; -- required artifact declarations; -- acceptance criteria bound to exact hypothesis ids; -- fallback artifacts and reentry conditions for unsuccessful outcomes. - -Normalization derives `contract_revision` from the complete normalized -contract. Recording the contract does not imply user acceptance. Evidence -produced from the delivery contract carries the atomic lineage tuple -(`wish_id`, `contract_ref`, `contract_revision`). All three fields must be -present and match exactly; partial or mismatched lineage is excluded from -verification. A compact contract event is -appended once per wish and revision. Legacy `research_contract_v0` packets -retain their prior event shape and remain supported. - -## Receipt - -```bash -loopx auto-research artifact-receipt \ - --contract delivery-contract.public.json \ - --format json -``` - -The command is read-only. It folds the current contract revision, Auto Research -evidence graph, terminal decisions, independent reviews, and artifact -references into `auto_research_artifact_receipt_v0`. - -The aggregate status is one of: - -| Status | Meaning | -| --- | --- | -| `verified` | Every required criterion has a current promoted decision, the required review, and every declared artifact. | -| `partial` | Some criterion or artifact is verified, but the complete delivery contract is not satisfied. | -| `inconclusive` | Evidence, a terminal decision, or a required independent review is still missing or conflicting. | -| `not_fulfilled` | Current evidence supports a terminal retired decision for at least one required criterion, with any required independent review. | -| `stale` | The recorded evidence or terminal decision belongs to an older contract revision. | - -`not_fulfilled` is intentionally stronger than an unsuccessful attempt. One -failed command, one regressed dev result, or an unresolved retry does not prove -that the wish cannot be fulfilled. - -## Failure Feedback - -Every non-verified receipt includes: - -- `failure_kinds`: typed reasons derived from current evidence, decision, and - review state; -- `unmet_criteria`: required contract criteria not satisfied; -- `verified_boundary`: criteria that remain verified; -- `missing_required_artifact_refs`: declared required artifacts not observed; -- `fallback_artifact_refs`: declared fallback artifacts that were actually - observed; -- `reentry_conditions`: owner-authored conditions plus precise criterion-level - repair hints. - -The receipt does not infer user acceptance. It also does not install or promote -a Skill, mutate an extension, spend quota, or write project state. A -`learning_disposition=candidate` only means that a verified or terminally -not-fulfilled outcome is eligible for a later governed experience review. - -## Staleness - -A terminal decision remains bound to the hypothesis evidence revision. The -hypothesis and evidence now also carry the delivery `contract_revision`. -Changing the wish assumptions, required artifacts, acceptance criteria, or -failure policy creates a new contract revision. A receipt for that new revision -must not reuse the old revision's evidence as current delivery proof. - -## Public Boundary - -The delivery contract and receipt use public-safe text and opaque or relative -artifact references. They do not contain raw logs, raw trajectories, -credentials, private material, or absolute local paths. Environment-specific -replay state remains adapter-owned; only bounded public evidence references may -enter this contract. diff --git a/docs/reference/protocols/decentralized-auto-research-state-v0.md b/docs/reference/protocols/decentralized-auto-research-state-v0.md deleted file mode 100644 index a397ff11a9..0000000000 --- a/docs/reference/protocols/decentralized-auto-research-state-v0.md +++ /dev/null @@ -1,316 +0,0 @@ -# decentralized_auto_research_state_v0 - -`decentralized_auto_research_state_v0` is a LoopX protocol for running -autonomous research without introducing a single leader agent. It borrows the -useful parts of Arbor's public design, especially durable hypothesis state, -worktree isolation, dev/held-out evaluation, and replayable evidence, but maps -them onto LoopX's shared control plane: todos, claims, quota, run history, -rollout events, gates, and read-only projections. - -The goal is to let multiple agents explore research hypotheses in parallel -while preserving provenance, ownership, and promotion rules. No agent owns the -whole research tree. The source of truth is the append-only state graph; each -agent sees a scoped frontier through `quota should-run --agent-id ...`. - -## Arbor Signals To Preserve - -Primary public Arbor surfaces reviewed: - -- `README.md`: Arbor frames the work as autonomous research through - hypothesis-tree refinement, with a Coordinator, Executors, worktrees, - held-out validation, reports, replay, and a benchmark zoo. -- `docs/how-it-works.md`: the six-step cycle is observe, ideate, select, - experiment, evaluate, backpropagate, then merge or prune. -- `src/coordinator/idea_tree.py`: each node records hypothesis, status, - insight, result, score split, test score, branch ref, grounding, - related-work audit, eval status, stop reason, and attempt count. -- `src/coordinator/tools/executor_run.py`: executor outcomes distinguish - scored success from `needs_retry` for timeout, max-turn, eval-crash, or - unparseable reports, while preserving branch/report/diff for continuation. -- `docs/search.md`: grounded ideation and novelty audit are separate lanes; - text fetched to inspire an idea is not reused to certify novelty. -- `arbor-zoo/algotune_knn`: the showcase benchmark packages an editable - solver, protected harness, dev/test split, provenance, and one `score:` line. - -LoopX should preserve these product lessons, not Arbor's centralized topology. - -## Non-Goals - -- Do not add a LoopX-wide "research director" agent that owns all planning. -- Do not replace todos, run history, or rollout events with a second research - database. -- Do not let a showcase dashboard mutate source state. -- Do not treat an advisory grounded-search result as proof of novelty. -- Do not merge or publish a research result without held-out evidence or an - explicit gate when the boundary requires one. - -## Source Protocol - -Source state is writable only through LoopX lifecycle commands, project-owned -state files, or future narrow kernel APIs. - -The companion -[`auto_research_lane_contract_v1`](auto-research-lane-contract-v1.md) defines -which decentralized agent lanes may create, execute, evaluate, promote, retire, -or narrate these records. The companion -[`auto_research_role_state_machine_v0`](auto-research-role-state-machine-v0.md) -defines the always-on digital employee role map and state transitions. This -file defines the record shapes and projections; the companion contracts define -capability ownership and transition evidence without introducing a leader -agent. - -| Source state | Existing or proposed anchor | Purpose | -| --- | --- | --- | -| `research_contract_v0` | proposed packet, registry goal metadata | Public-safe objective, editable scope, protected scope, metric direction, dev/held-out commands, budget, and stopping condition. | -| `todo_item_v0` | `loopx todo` | Formal executable work, user gate, blocker, or monitor. Research work remains todo-backed. | -| `research_hypothesis_v0` | proposed rollout event and optional domain pack | A hypothesis node linked to a todo, parent hypothesis, agent lane, mechanism family, and current status. | -| `research_evidence_event_v0` | proposed `loopx_rollout_event_v0` specialization | Append-only evidence for an attempt: score, split, command label, branch, artifact refs, eval status, and boundary facts. | -| `auto_research_terminal_decision_v0` | `validation` rollout event specialization | Explicit `promoted` or `retired` result bound to one evidence-graph revision; promotion and retirement candidates do not imply this record. | -| `auto_research_peer_review_v0` | `validation` rollout event specialization | Approve, reject, or needs-more-evidence review receipt bound to one terminal decision and evidence-graph revision. | -| `dataset_window_contract_v0` | `loopx/ml_experiment.py` | Train/dev/held-out window or split contract, including missing-window policy. | -| `agent_lane_next_action_v0` | `quota should-run --agent-id ...` | The current agent's selected frontier item; it does not replace the global state graph. | -| `operator_gate` | `loopx operator-gate`, review packet | Promotion, merge, private-material, or publication decision. | -| `human_reward` | `loopx reward` | Run-bound owner judgment, not general write permission. | - -### `research_contract_v0` - -```json -{ - "schema_version": "research_contract_v0", - "goal_id": "loopx-meta", - "research_objective": "optimize a benchmark task under a protected evaluator", - "editable_scope": ["solution.py"], - "protected_scope": ["eval.py", "task.py", "data/**"], - "metric": {"name": "speedup", "direction": "maximize"}, - "dev_eval": {"label": "eval_dev", "command_label": "bash eval.sh dev"}, - "holdout_eval": {"label": "eval_test", "command_label": "bash eval.sh test"}, - "promotion_policy": "requires_holdout_improvement_and_clean_boundary", - "budget": {"max_attempts": 8, "max_parallel_lanes": 3} -} -``` - -### `research_hypothesis_v0` - -```json -{ - "schema_version": "research_hypothesis_v0", - "goal_id": "loopx-meta", - "hypothesis_id": "hyp_0004", - "parent_hypothesis_id": "hyp_0002", - "todo_id": "todo_123", - "lane_id": "agent:research-executor", - "claimed_by": "research-executor", - "mechanism_family": "vectorized_distance_kernel", - "hypothesis": "Batch query points through a shared index to reduce per-query overhead.", - "status": "active", - "frontier_rank": 2, - "source_refs": ["research_contract:knn_speedup"], - "grounding_refs": [], - "novelty_audit_ref": null -} -``` - -Status vocabulary: - -| Status | Meaning | -| --- | --- | -| `proposed` | Drafted but not yet accepted into executable work. | -| `active` | Runnable frontier item, normally backed by an open agent todo. | -| `running` | Claimed attempt is in progress. | -| `needs_retry` | Attempt produced no trustworthy score but left resumable evidence. | -| `supported` | Dev evidence supports the hypothesis; not yet promoted. | -| `contradicted` | Evidence shows regression or guardrail failure. | -| `promoted` | Held-out evidence and promotion policy accepted it into the current best artifact. | -| `retired` | No longer worth exploring, but retained as negative evidence. | - -### `research_evidence_event_v0` - -```json -{ - "schema_version": "research_evidence_event_v0", - "goal_id": "loopx-meta", - "hypothesis_id": "hyp_0004", - "todo_id": "todo_123", - "agent_id": "research-executor", - "attempt": 1, - "split": "dev", - "metric": {"name": "speedup", "value": 11.4, "direction": "maximize"}, - "baseline_metric": 8.7, - "primary_metric_status": "improved", - "eval_status": "scored", - "code_ref": "codex/research-hyp-0004", - "artifact_refs": ["experiment:hyp_0004/report", "diff:hyp_0004"], - "protected_scope_clean": true, - "private_artifacts_recorded": false, - "raw_logs_recorded": false -} -``` - -`needs_retry` is first-class evidence, not a generic failure: - -```json -{ - "schema_version": "research_evidence_event_v0", - "hypothesis_id": "hyp_0005", - "eval_status": "failed_to_run", - "stop_reason": "max_turns", - "attempt": 1, - "code_ref": "codex/research-hyp-0005", - "resume_policy": "resume_from_code_ref_or_retire", - "primary_metric_status": "inconclusive" -} -``` - -## Projection Protocol - -Projection state is read-only. It may rank frontiers for display, but it must -carry source refs and remain recomputable from source state. - -| Projection | Purpose | -| --- | --- | -| `decentralized_research_frontier_v0` | Per-agent queue of runnable or blocked hypotheses after quota, claim, capability, and boundary checks. | -| `research_evidence_graph_v0` | Read-only graph joining hypotheses, todos, attempts, branches, metrics, gates, and promotion decisions. | -| `auto_research_terminal_result_projection_v0` | Deterministic projection of terminal decisions and review state into existing Explore nodes, findings, and lineage edges. | - -### `decentralized_research_frontier_v0` - -```json -{ - "schema_version": "decentralized_research_frontier_v0", - "goal_id": "loopx-meta", - "agent_id": "research-executor", - "selected": { - "hypothesis_id": "hyp_0004", - "todo_id": "todo_123", - "reason": "current-agent claim and dev evidence gap", - "allowed_action": "run_dev_attempt" - }, - "blocked": [ - { - "hypothesis_id": "hyp_0002", - "blocked_by": "claimed_by:research-curator", - "visible_as_context": true - } - ], - "promotion_candidates": [ - { - "hypothesis_id": "hyp_0004", - "requires": ["holdout_eval", "boundary_scan"] - } - ] -} -``` - -This projection intentionally replaces a centralized Coordinator decision. The -kernel selects only what the current agent may attempt; the agent still does -semantic implementation within its allowed boundary. - -## Decentralized Cycle - -Arbor's cycle maps to LoopX as a distributed control loop: - -| Arbor concept | LoopX decentralized equivalent | -| --- | --- | -| Observe | Agent reads `quota should-run`, active state, selected frontier, evidence graph, and relevant source docs. | -| Ideate | Any authorized lane may propose `research_hypothesis_v0` as a todo-backed candidate. | -| Select | `quota should-run --agent-id` filters by claim, gate, capability, and boundary. | -| Dispatch | The selected agent works in its own worktree and writes evidence. | -| Backpropagate | A deterministic projection summarizes supported/contradicted lessons into the evidence graph; no single agent rewrites global truth. | -| Decide | Promotion policy plus operator gates decide merge, retire, retry, or continue. | - -## Search Lanes - -LoopX should adopt Arbor's separation of search lanes: - -| Lane | When | Authority | Writes | -| --- | --- | --- | --- | -| `grounded_ideation` | Before or during hypothesis proposal | Advisory input to a candidate hypothesis. | `grounding_refs` on the hypothesis. | -| `novelty_audit` | After a hypothesis has real evidence | Advisory contribution/overlap check. | `novelty_audit_ref` and evidence note. | - -The two lanes must not share fetched text as proof. If a source shaped the idea, -it can be cited as grounding, but a later novelty audit must run an independent -source pass before claiming novelty. - -## Promotion Policy - -A research hypothesis is promotable only when: - -1. the editable/protected scope boundary is clean; -2. dev evidence is scored and public-safe; -3. held-out evidence satisfies the metric direction and margin; -4. required user/controller gates are resolved; -5. the promoted artifact has a branch or commit ref; -6. the evidence graph records negative evidence for close alternatives. - -Promotion is a state transition, not a chat conclusion. It should write a -todo completion, evidence event, and optional promotion gate/reward overlay. - -## Integration With Existing LoopX - -Already available: - -- `todo_item_v0` has claims, task classes, action kinds, blockers, user gates, - and resume conditions. -- `agent_lane_next_action_v0` scopes work per registered agent. -- `loopx/ml_experiment.py` already has `hypothesis_ledger_v0`, - `dataset_window_contract_v0`, and advisory result packets. -- `long_horizon_agent_state_protocol_v0` already defines source/projection - separation and concurrent lane views. -- `loopx auto-research frontier --fixture --agent-id ` - now renders a fixture-backed `decentralized_research_frontier_v0`, - and `research_evidence_graph_v0` without launching experiments or depending - on a leader agent. -- `loopx auto-research frontier --goal-id --agent-id ` now - renders a live todo/quota-backed frontier for the current agent from LoopX - status projection. -- The auto-research worker path now uses a built-in lightweight metric kernel - for smoke/demo evidence instead of a shipped domain-specific starter pack. -- `loopx auto-research evidence --contract - --eval-result ...` now builds an - `auto_research_evidence_packet_v0` containing a public-safe - `research_hypothesis_v0` and split-aware `research_evidence_event_v0` - records. It preserves `needs_retry`, negative evidence, - `protected_scope_clean`, and branch/artifact refs without recording raw logs - or private artifacts. -- `loopx auto-research append-evidence --packet ` appends that - packet into the existing `loopx_rollout_event_v0` log as one - `research_hypothesis` event plus one `research_evidence` event per split. - Re-running the same packet skips existing event ids, so heartbeat retries do - not duplicate evidence. -- `loopx auto-research frontier --goal-id --agent-id ` reads - `research_hypothesis` and `research_evidence` rollout events back into - `research_evidence_graph_v0` and derives promotion/retirement candidates for - the live frontier. -- `loopx auto-research decide` validates a candidate against current evidence, - then records an evidence-revision-bound terminal decision. -- `loopx auto-research review` records a review receipt; with - `--require-independent`, the reviewer must be a different registered peer - from both producer and decision agent. -- `loopx auto-research results` returns exact current and historical terminal - state by hypothesis id. `project-results` writes deterministic existing - Explore node/finding events, and repeated execution is idempotent. - -Needed next: - -- let pane-local workers consume the frontier directly from their Codex TUI; -- keep product narration and public launch claims outside the kernel until a - separate product surface intentionally consumes the compact evidence graph. - -## Acceptance Checks - -An implementation is acceptable when: - -- no single agent is required to own or mutate the full hypothesis graph; -- every executable hypothesis is linked to a todo and claim; -- per-agent frontier selection is derived from quota/status, not chat memory; -- `needs_retry` keeps resumable evidence instead of collapsing into `done`; -- grounded ideation and novelty audit are separated; -- held-out promotion is explicit; -- terminal decisions remain separate from promotion and retirement candidates; -- same-agent review remains visible but cannot confirm or refute a finding; -- stale evidence revisions and conflicting terminal decisions fail closed; -- user-facing research summaries derive from compact frontier and evidence - graph refs, not a bespoke kernel packet; -- public projections contain no raw logs, private paths, credentials, or raw - private documents; -- the public narration can be rendered from public-safe evidence refs. diff --git a/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md b/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md deleted file mode 100644 index 698e5a0a56..0000000000 --- a/docs/reference/protocols/multi-agent-three-layer-minimality-v0.md +++ /dev/null @@ -1,138 +0,0 @@ -# multi_agent_three_layer_minimality_contract_v0 - -`multi_agent_three_layer_minimality_contract_v0` defines the reusable layering -rule for LoopX multi-agent products: - -1. **User layer:** declares intent and a few product-level options. -2. **Preset layer:** supplies domain defaults, role semantics, handoff hints, - and product evidence or metric adapters. -3. **Kernel layer:** owns the reusable multi-agent mechanics. - -The goal is not only to minimize the user's snippet. The preset must also stay -thin so auto-research can be a reusable example for future multi-agent products -rather than a second product-specific runner. - -## Ownership - -| Layer | Owns | Must Not Own | -| --- | --- | --- | -| User | Objective, rounds, optional role overrides, and optional data/eval entrypoint. | Tmux/Codex TUI launch, pane-local tick commands, quota/frontier protocol details, worker plumbing, per-agent vision/replan state, or machine JSON routing. | -| Preset | Domain roles, handoff hints, metric/evidence loop, and domain defaults. | Generic runner lifecycle, real Codex TUI panes, workspace/trust-safe launch, pane-local A2A tick, todo/evidence/status protocol, per-agent vision budgets, replan state transitions, or compact human status. | -| Kernel | Multi-agent runner, real Codex TUI panes, workspace/trust-safe launch, pane-local A2A tick, todo/evidence/status protocol, CLI-enforced per-agent vision budgets, vision/replan state transitions, compact human status, and default role prompt scaffolding. | Domain-specific research, benchmark, support, or sales semantics. | - -## Contract Shape - -The reusable helper lives in `demo/multi_agent/contract.py`: - -```python -build_three_layer_minimality_contract( - product_id="customer-support", - preset_id="support_triage_preset", - user_intent_fields=["inbox", "rounds"], - preset_responsibilities=["triage_roles", "handoff_hints"], -) -``` - -It returns: - -```json -{ - "schema_version": "multi_agent_three_layer_minimality_contract_v0", - "principle": "user_and_preset_stay_thin_kernel_owns_reusable_mechanics", - "user_layer": { - "owns": "intent" - }, - "preset_layer": { - "must_remain_reusable": true - }, - "kernel_layer": { - "cross_product_reuse_required": true - } -} -``` - -## Auto-Research Preset - -Auto-research is one preset on top of the generic kernel. Its preset layer owns -research roles, handoff hints, the metric/evidence loop, and domain defaults. -It does not own the runner, TUI panes, workspace/trust-safe launch, -pane-local A2A tick, todo/evidence/status protocol, per-agent vision budgets, -or replan state transitions. - -This keeps the public promise honest: a small auto-research recipe should prove -that other products can also reuse the same kernel with their own thin preset. - -## Public Line-Count Claim - -The public "few lines of auto-research" claim counts declarative recipe lines, -not the shared kernel implementation. - -For the default auto-research demo, the bounded claim is: - -| Layer | Counted Lines | Meaning | -| --- | ---: | --- | -| User | 1 | `loopx auto-research start "" --execute` | -| Auto-research preset | 4 | default role specs: curator, mapper, runner, verifier | -| Generic kernel | 0 | shared runner, Codex TUI panes, fixed wake prompt, pane-local tick, todo/evidence/status protocol | - -So the honest slogan is: one user line plus a four-line preset can start a -decentralized A2A research loop on the shared LoopX kernel. The slogan must not -claim that tmux launch, Codex TUI bootstrap, quota/frontier, evidence routing, -or status projection are reimplemented inside the auto-research preset. - -## Developer Implementation Budget - -The same split applies to source code, not only to command examples. -`demo/auto_research/preset.py` should read like a small preset: -role defaults, role profiles, successor declarations, seed todo wording, and -thin wrappers around generic helpers. Reusable A2A proof fields such as -`broadcaster_selects_todo=false`, `each_pane_reads_own_quota_frontier=true`, -and `leader_agent_required=false` belong to -`demo/multi_agent/recipe.py`. - -This keeps the developer-facing promise honest: future products should be able -to copy the pattern by writing their own short preset, not by importing -auto-research internals. - -## Collective Round Ledger - -`multi_agent_collective_round_ledger_v0` is the kernel-owned proof surface for -multi-agent rounds. It records expected lanes, per-lane quota/frontier/turn -outcomes, integrated evidence, and role-declared successor todos. Product -presets may wrap the ledger with domain metrics, but they should not fork its -round definition or introduce a coordinator to decide work. - -## Canonical Auto-Research Recipe - -Auto-research should stay small enough that a developer can see the whole -product-specific recipe at a glance: - -```text -loopx auto-research start "" --execute -research-curator:research-curator:research_curator -hypothesis-proposer:hypothesis-proposer:hypothesis_proposer -research-executor:research-executor:research_executor -evaluator-promoter:evaluator-promoter:evaluator_promoter -``` - -Those five lines are the product recipe: one user question and four research -role identities. The fixed decentralized wake prompt, real Codex TUI panes, -pane-local quota/frontier tick, successor todo protocol, and -`multi_agent_collective_round_ledger_v0` proof remain generic kernel behavior. - -For the KNN demo, the auto-research preset may declare the research metric and -role successor hints, but it must not add a product-specific coordinator, -workflow runner, or metric aggregator. The required proof is: four collective -role rounds, at least two held-out metric improvements, public-safe evidence, -and a generic collective-round ledger saying which lanes participated. - -## Acceptance - -A change satisfies this contract only when: - -- the user recipe remains a few intent fields, not runner configuration; -- the preset has no host process lifecycle or pane-local tick implementation; -- the preset has no product-specific fork of per-agent vision/replan mechanics; -- the generic kernel contract stays domain-agnostic; -- another multi-agent product can reuse the same kernel without importing - auto-research code. diff --git a/docs/reference/protocols/multi-agent-visible-launcher-v0.md b/docs/reference/protocols/multi-agent-visible-launcher-v0.md deleted file mode 100644 index fbf58f8052..0000000000 --- a/docs/reference/protocols/multi-agent-visible-launcher-v0.md +++ /dev/null @@ -1,357 +0,0 @@ -# multi_agent_visible_launcher_v0 - -`multi_agent_visible_launcher_v0` is the generic LoopX contract for starting -several visible local agent panes for one shared goal. It is the reusable layer -under domain demos such as auto-research: LoopX owns the goal surface, lane -identity, quota/frontier/bootstrap guards, visible host controls, and public -acceptance; the domain capability owns only role semantics and evidence -writeback. - -This contract intentionally sits between two existing surfaces: - -- `local_agent_launch_plan_v1` previews what could be launched and remains - `mode=dry_run` only. -- Domain capabilities, such as auto-research, provide role profiles, frontier - commands, and evidence packets. - -The visible launcher may plan or start panes, but it must not become a leader -agent, hidden scheduler, promotion authority, or second source of truth. - -The launcher also follows -[`multi_agent_three_layer_minimality_contract_v0`](multi-agent-three-layer-minimality-v0.md): -both the user-facing recipe and the domain preset stay thin, while the generic -kernel owns runner/TUI/tick/todo/evidence/status mechanics. - -## Kernel Module - -The reusable control-plane kernel lives in `demo/multi_agent/`. -Domain capabilities should depend on that package for: - -- `tui_multi_agent_runner_contract_v0`; -- `generic_multi_agent_role_profile_v0`; -- `multi_agent_three_layer_minimality_contract_v0`; -- pane-local A2A prompt rules; -- artifact-only machine JSON policy; -- compact human status for launch previews. -- runtime scripts for scoped LoopX wrappers, pane-local A2A ticks, and Codex - TUI exec. - -The host launcher in `demo/visible_multi_agent_launcher.py` owns tmux process -execution and Codex TUI acceptance only. Auto-research, benchmark demos, and -future custom teams should not duplicate runner schema, role-profile -normalization, attach/stop semantics, pane-local tick scripts, or machine JSON -wrapper policy. - -## User-Facing Spec - -Most callers should not author `multi_agent_visible_launcher_v0` packets by -hand. They should provide a small `generic_multi_agent_launch_spec_v0` role -spec and let `loopx multi-agent launch --spec ` expand it into the -visible launcher packet. - -```json -{ - "schema_version": "generic_multi_agent_launch_spec_v0", - "goal_id": "loopx-meta", - "session_name": "loopx-custom-team", - "default_reasoning_effort": "high", - "roles": [ - { - "lane_id": "planner", - "agent_id": "codex-main-control", - "role_id": "planner", - "scope": "Plan the next bounded step from the shared goal surface.", - "skill": { - "name": "loopx-planner-worker", - "source": "worker/SKILL.md" - }, - "handoff_hints": [ - "Create or update a LoopX todo for critic when a plan needs review." - ] - }, - { - "lane_id": "critic", - "agent_id": "codex-side-bypass", - "role_id": "critic", - "scope": "Review the bounded step against the same todo and quota projection." - } - ] -} -``` - -The spec is intentionally small: - -- `goal_id` selects the shared LoopX state surface; -- each role names an `agent_id`, human role, and scope; -- `skill.source` is resolved relative to the spec file and materialized as a - worker-local skill for that role; -- `handoff_hints` describe how roles should use LoopX todos/evidence to pass - work, without creating a central workflow engine; -- `--execute` starts visible Codex TUI panes, while the default mode only - previews the packet. - -This keeps product integration focused on role design and state-mediated -handoff. The generated launcher packet still enforces artifact-only machine -JSON, interactive Codex TUI windows, attach/stop controls, and public boundary -fields. - -## Ownership Split - -| Surface | Owns | Does not own | -| --- | --- | --- | -| LoopX control plane | goal id, registered agents, todo claims, quota, gates, run history, and public/private boundary. | Domain-specific research, benchmark, or product semantics. | -| Multi-agent visible launcher | pane layout, environment projection, visible start commands, attach/stop/retry controls, acceptance checks. | Todo truth, promotion decisions, hidden session injection, or write permission. | -| Domain capability | lane roles, frontier command, role profile schema, domain artifact/evidence writeback. | Generic host process lifecycle, global quota, or cross-domain launcher policy. | -| Host shell or app | Actual tmux/terminal/window process control after explicit local execution. | Authority to bypass LoopX guards or hide work from the user. | - -The launcher is a host convenience surface. Each pane remains a normal LoopX -agent lane that must read the same goal state and pass its own guard. - -## Packet Shape - -```json -{ - "schema_version": "multi_agent_visible_launcher_v0", - "mode": "dry_run | execute", - "goal_id": "loopx-meta", - "session_name": "loopx-visible-goal", - "reasoning_contract": { - "default_reasoning_effort": "high", - "codex_cli_config_key": "model_reasoning_effort" - }, - "shared_goal_surface": { - "shared_goal_id": "loopx-meta", - "shared_state_route": "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", - "shared_frontier": true, - "lane_identity_source": "role_profile_plus_agent_scoped_quota", - "all_lane_workspace_isolation": false, - "mutation_isolation_policy": "only mutating attempts require a claimed worktree or equivalent execution boundary" - }, - "interactive_tui_contract": { - "schema_version": "multi_agent_visible_interactive_tui_contract_v0", - "human_pane": [ - "codex_cli_tui", - "role_prompt_inside_codex", - "normal_user_typing", - "normal_codex_tool_output", - "takeover_controls" - ], - "machine_artifacts": [ - "quota.public.json", - "frontier.public.json", - "bootstrap-prompt.public.txt", - "role_local_public_artifacts" - ], - "machine_json_policy": "file_or_explicit_machine_channel_only", - "visible_json_policy": "not_printed_before_tui", - "codex_surface": "interactive_cli_tui" - }, - "lanes": [], - "commands": { - "start_script": [], - "attach": "tmux attach -t loopx-visible-goal", - "stop": "tmux kill-session -t loopx-visible-goal", - "retry": "rerun the same packet after quota/frontier refresh" - }, - "acceptance": {}, - "boundary": {} -} -``` - -Required top-level fields: - -- `schema_version`: exactly `multi_agent_visible_launcher_v0`; -- `mode`: `dry_run` for inspect-only packets, or `execute` only after a local - host explicitly starts panes; -- `goal_id` and `session_name`; -- `reasoning_contract`; -- `shared_goal_surface`; -- `interactive_tui_contract`; -- `lanes[]`; -- `commands.attach`, `commands.stop`, and `commands.retry`; -- `acceptance`; -- `boundary`. - -## Shared Goal Surface - -Visible lanes share the target **goal surface**, not necessarily the same file -mutation area. The shared surface is: - -- the registry selected by `LOOPX_REGISTRY`; -- the runtime root selected by `LOOPX_RUNTIME_ROOT`; -- the `goal_id`; -- agent-scoped `quota should-run`; -- todo projection, frontier projection, and run history for that goal; -- public-safe evidence and rollout-event projections. - -`all_lane_workspace_isolation=false` is the normal LoopX multi-agent shape: -agents cooperate on one goal surface. Isolation is applied only to the lane or -attempt that mutates files, typically through an independent git worktree, -scratch directory, or equivalent execution boundary. The launcher must make that -policy explicit instead of silently moving every lane into unrelated state. -Visible Codex TUI panes must not default into a generated demo-local git -worktree or control-plane repository. By default they should start in the -caller-selected workspace, or in an explicit user-owned scratch workspace, so a -user sees the agent role TUI instead of a generated workspace trust prompt. - -## Lane Shape - -Each lane is small enough to inspect before the agent starts: - -```json -{ - "lane_id": "evidence-runner", - "agent_id": "codex-side-bypass", - "role_id": "evidence_runner", - "responsibility": "Run one bounded evidence attempt.", - "role_profile": { - "schema_version": "domain_role_profile_v0" - }, - "quota_guard": "loopx ... quota should-run --goal-id ... --agent-id ...", - "frontier": "loopx ... frontier --goal-id ... --agent-id ...", - "bootstrap_message": "loopx codex-cli-bootstrap-message ...", - "visible_launch_command": "...", - "reasoning_effort": "high", - "lane_timeline": [] -} -``` - -Required lane fields: - -- `lane_id`, `agent_id`, `role_id`, and `responsibility`; -- `role_profile` or `role_profile_ref`; -- `quota_guard`; -- `frontier`; -- `bootstrap_message`; -- `visible_launch_command` or host-equivalent command ref; -- `reasoning_effort`; -- `lane_timeline`. - -The pane title is cosmetic. The role profile plus agent-scoped quota/frontier -packet is the lane identity authority. - -## Start Order - -Every visible lane follows the same generic start order: - -1. Prepare the role profile, scoped LoopX wrappers, and bootstrap prompt as - local artifacts without printing raw JSON or shell marker streams. -2. Start one fresh interactive Codex CLI TUI per role window with - `codex -c model_reasoning_effort= -C "$LOOPX_PROJECT" "$PROMPT"`. -3. Let the Codex role read quota/frontier/todo state through the pane-local - LoopX wrapper inside the TUI. -4. Stop inside that Codex role if - `interaction_contract.user_channel.action_required=true`, delivery is - disallowed, quota is false, the frontier is contradictory, or the role would - bypass LoopX todo/evidence writeback. -5. Keep the pane interactive so the user can type, interrupt, close, or retry - manually like a normal Codex CLI session. - -The launcher must not inject prompts into an existing hidden session, print raw -control-plane JSON as the first screen, or continue a lane after a user gate is -projected. - -## Interactive TUI Contract - -The first screen of each visible pane is the Codex CLI TUI itself. Users should -feel that they opened several normal Codex agents, one per role, and can type -into any of them. Raw quota, frontier, role profile, bootstrap prompt, and -machine JSON belong in public-safe artifacts or an explicit machine channel, -not in the initial tmux viewport. - -Required runtime shape: - -- one tmux window per role; -- no extra frontier/status JSON window by default; -- every role window executes the interactive Codex CLI, not `codex exec`; -- no pre-Codex character stream, marker transcript, raw JSON dump, or copied - role profile appears before the TUI; -- no generated demo-local worktree trust prompt should be the first visible - screen; -- the role prompt instructs Codex to use the pane-local LoopX wrapper for - quota/frontier/todo/evidence work; -- users can interact with each role through ordinary Codex CLI controls. - -The TUI may later show human-readable LoopX summaries when the Codex role runs -commands. It should not dump raw JSON, credentials, private logs, raw -transcripts, or absolute local artifact paths into the visible pane. - -## Host Controls - -The packet must expose: - -- `attach`: how the user observes or takes over the whole session; -- `stop`: how the user kills the whole session; -- per-pane interrupt path using normal terminal controls; -- `retry`: the safe retry shape, which must recompute quota/frontier/bootstrap - first instead of replaying stale hidden state; -- visible acceptance proof that each pane was created as an interactive Codex - CLI TUI role and remains attachable or inspectable. - -Host commands are public-safe command shapes. They must not include credentials, -auth headers, raw transcript paths, private document ids, or local absolute -paths in committed docs or fixtures. - -## Boundary - -Required boundary fields: - -```json -{ - "starts_visible_processes": false, - "runs_agent_processes": false, - "writes_loopx_state": false, - "spends_loopx_quota": false, - "reads_raw_transcripts": false, - "reads_session_files": false, - "reads_credentials": false, - "hidden_prompt_injection": false, - "shared_goal_surface": true, - "all_lane_workspace_isolation": false, - "public_safe_redaction": true -} -``` - -For `mode=dry_run`, process and write fields must stay false. For -`mode=execute`, `starts_visible_processes` and `runs_agent_processes` may become -true only when the host actually starts visible panes; LoopX state write and -quota spend still happen through the normal agent lane after validated -writeback, not through the launcher packet itself. - -## Domain Adapter Responsibilities - -A domain capability that uses this contract supplies: - -- role profile schema and role-specific allowed actions; -- frontier command and blocked-reason shape; -- domain evidence or artifact writeback command; -- domain-specific acceptance criteria; -- public-safe demo fixture or deterministic positive seed when needed. - -The domain capability should not own generic attach/stop/retry, shared goal -surface, high-reasoning launch flags, or pane survival checks. - -## Acceptance Checks - -A public fixture or implementation satisfies the contract when: - -1. the packet or protocol names `multi_agent_visible_launcher_v0`; -2. it distinguishes `local_agent_launch_plan_v1` preview from visible launch; -3. it says the launcher is not a leader agent, scheduler, promotion authority, - or second source of truth; -4. it exposes shared goal surface through registry, runtime root, goal id, - agent-scoped quota, todo/frontier projection, run history, and evidence; -5. it requires per-lane role identity, quota guard, frontier, bootstrap message, - high reasoning, lane timeline, and visible launch command; -6. it requires attach, stop, retry, pane interrupt, and visible acceptance proof - for interactive Codex CLI TUI role windows; -7. it has an interactive TUI contract that keeps raw machine JSON in artifacts - or explicit machine channels instead of printing it before Codex starts; -8. dry-run mode starts no process, runs no agent, writes no LoopX state, and - spends no quota; -9. execute mode still writes state and spends quota only through normal LoopX - writeback after validation; -10. it keeps workspace isolation scoped to mutating attempts rather than - splitting the shared goal surface; and -11. public docs and fixtures contain no raw transcripts, credentials, private - links, internal project names, or local absolute paths. diff --git a/docs/showcases/README.md b/docs/showcases/README.md index 93da6ba745..5269e77dca 100644 --- a/docs/showcases/README.md +++ b/docs/showcases/README.md @@ -186,24 +186,6 @@ paths. CI Sweeper and Dependency Sweeper are visible because they are high-ROI maintainer workflows, but they stay opt-in and begin with a dry-run or policy report before any isolated worktree patch is attempted. -### Auto Research One-Click Start - -The auto-research path is the experimental one-command agent-team demo: - -```bash -loopx auto-research "How should we evaluate whether multi-agent auto research creates value?" -loopx auto-research start "How should we evaluate whether multi-agent auto research creates value?" --execute -``` - -The contract command previews the research brief, evidence boundary, and next -launch packet. The `start --execute` command opens visible Codex CLI lanes -through the generic multi-agent kernel; lane-authored evidence still has to be -written back through LoopX state before the demo can claim progress. See the -[auto-research command path](../../demo/auto_research/README.md). -For the shipped stop marker, `--attach` takeover, and state-aware wake cycle, -use the contributor -[stop/takeover/wake walkthrough](../guides/auto-research-stop-takeover-wake-walkthrough.md). - ### Review Agent Work Review Agent Work is also an experimental entry: it uses the read-first diff --git a/docs/superpowers/specs/2026-08-06-auto-research-failure-successors-design.md b/docs/superpowers/specs/2026-08-06-auto-research-failure-successors-design.md deleted file mode 100644 index cb63f61ab8..0000000000 --- a/docs/superpowers/specs/2026-08-06-auto-research-failure-successors-design.md +++ /dev/null @@ -1,366 +0,0 @@ -# Bounded Auto-Research Failure Successors Design - -**Status:** Accepted direction; pending written-spec review before implementation. - -## Change Contract - -**Goal:** Ensure a retired auto-research hypothesis produces one bounded next -control-plane outcome: a single data/measurement remediation, a new -hypothesis-proposer todo, an extension-evolution proposal, or an explicit -research-frontier exhaustion record. - -**Acceptance evidence:** - -- A negative-evidence fixture cannot finish as an implicit no-follow-up. -- A declared data/measurement gap creates at most one remediation todo for its - hypothesis lineage and never creates a monitor. -- A non-remediable retirement creates a proposer successor todo through the - existing role-successor lifecycle. -- A remediated lineage that still fails creates a proposer successor or an - explicit `frontier_exhausted` outcome. -- Every terminal research evaluation records one local evolution review before - completion, so an agent cannot silently stop after either a positive or - negative result. -- A reproducible product-contract violation can create one bounded extension - evolution proposal without waiting for a user to identify the gap. -- All resulting projections remain public-safe and use no network, scheduler, - service, or new package dependency. - -**Not in scope:** - -- Automatically inventing a research hypothesis, changing a strategy, or - relaxing thresholds. -- Waiting for future data through a `continuous_monitor` todo. -- Automatically editing, installing, enabling, upgrading, or publishing an - extension. -- Creating a coordinator, a background agent, or a second research database. - -**Expected implementation boundary:** - -- `demo/auto_research/` owns the completion and role-successor - semantics. -- `examples/auto-research-*.py` owns focused public regression proof. -- `docs/reference/protocols/auto-research-*.md` owns the public contract. -- A later extension-evolution proposal remains a normal LoopX todo, not a new - extension runtime. - -## Problem - -The current auto-research read model recognizes negative evidence and derives -`retirement_candidates`. Its completion status then becomes -`retirement_review_required`. The default evaluator/promoter role profile, -however, declares successors only for positive dev/holdout paths. Once a -retirement review is completed, no deterministic role successor is required. - -That lets an agent stop after a valid failed result even when the failure -contains a reusable research constraint or exposes a bounded next question. -The generic autonomous-replan mechanism detects repeated no-progress, but it -does not know whether a specific research failure should repair measurement, -change mechanism, or close the question. - -## Placement Rationale - -```text -capability_id: auto-research -provider_id: loopx-core -origin: builtin -placement: demo/auto_research/ -reason: The existing capability already owns research evidence, completion - status, worker roles, and role-declared successor todos. The new - behavior refines that lifecycle; it is not a provider-neutral - extension capability. -``` - -The later extension-evolution signal does not make a new extension. It is a -public-safe proposal for a human- or agent-owned product todo. The existing -extension runtime remains responsible only for provider lifecycle. - -## Design Principles - -1. **Every completed research result is an evolution input.** A terminal - evaluation must leave a durable constraint, successor, product-gap proposal, - or explicit exhaustion decision. -2. **Negative evidence is progress.** A retired hypothesis must leave a - durable constraint and a visible next decision. -3. **Data repair is exceptional and bounded.** A lineage gets at most one - declared data/measurement remediation. A weak metric does not by itself - qualify as a data gap. -4. **No disguised parameter search.** A remediation may repair a declared - source, unit, corporate-action, proxy, split, or measurement defect. It may - not change thresholds, horizons, assets, or selection criteria to rescue the - original conclusion. -5. **No future-data monitor.** If no current bounded action exists, the - research question closes explicitly. Future work can start from a new - contract when material is later available. -6. **No fake automation.** The control plane creates a todo and enforces the - permitted outcome. The hypothesis proposer still writes the actual - evidence-backed proposal; the kernel never invents research content. -7. **No external dependency.** All decisions derive from existing LoopX - rollout events, todo lineage, and local role-successor writes. - -## Completion Evolution Review - -Every terminal evaluator/promoter action (`supported`, `contradicted`, -`retired`, `promoted`, or retry exhaustion) must derive and append one -`research_evolution_review_v0` record before its source todo may complete. - -```json -{ - "schema_version": "research_evolution_review_v0", - "goal_id": "example-research", - "source_todo_id": "todo_distance_cache", - "hypothesis_id": "hyp_distance_cache", - "terminal_outcome": "contradicted", - "constraint_refs": ["research_constraint:distance_cache_regresses"], - "product_gap": { - "status": "none", - "fingerprint": null - }, - "next_outcome": "propose_failure_successor" -} -``` - -The review has exactly one `next_outcome`: - -| Next outcome | Meaning | -| --- | --- | -| `remediate_data_measurement` | The explicit repair budget remains and the defect is currently repairable. | -| `propose_failure_successor` | A proposer must write a new, mechanism-changing hypothesis or explicit exhaustion. | -| `extension_evolution_proposal` | Research exposed a bounded product or extension contract gap. | -| `constraint_recorded` | A supported or promoted result has no unresolved, safe in-scope successor; its method constraint is still durable knowledge. | -| `frontier_exhausted` | No current safe mechanism, material-preparation action, or product-gap proposal remains. | - -The system guarantee is deliberately about the **state transition**, not the -quality of generated research prose: it guarantees that a completed research -item cannot silently close without an evolution-review record and one of the -five visible outcomes. The hypothesis-proposer lane remains accountable for -the semantic content of a newly proposed hypothesis. - -## Failure Continuation Contract - -Introduce a compact projection and durable lifecycle record: -`research_failure_continuation_v0`. - -```json -{ - "schema_version": "research_failure_continuation_v0", - "goal_id": "example-research", - "hypothesis_id": "hyp_distance_cache", - "source_todo_id": "todo_distance_cache", - "failure_kind": "mechanism_contradicted", - "failure_evidence_refs": ["research_evidence:hyp_distance_cache"], - "remediation_attempt_count": 0, - "remediation_attempt_limit": 1, - "remediation_allowed": false, - "monitor_allowed": false, - "required_next_outcomes": [ - "propose_failure_successor", - "frontier_exhausted" - ] -} -``` - -The record is public-safe. It records compact ids and evidence aliases only; -it cannot contain raw evaluator logs, local paths, credentials, source bodies, -or provider payloads. - -### Failure Kinds - -| Failure kind | Remediation allowed | Required next outcome | -| --- | --- | --- | -| `mechanism_contradicted` | No | Proposer successor or frontier exhaustion. | -| `overfit_or_nonreplication` | No | Proposer successor with a changed mechanism, or frontier exhaustion. | -| `guardrail_or_protected_boundary` | No | Proposer successor only if a safe new mechanism exists; otherwise exhaustion. | -| `data_or_measurement_gap` | Yes, once | One remediation attempt, then re-evaluate. | -| `unrecoverable_data_gap` | No | Proposer successor with a different currently available information set, or exhaustion. | -| `retry_exhausted` | No | Proposer successor or exhaustion. | - -A data/measurement gap must be declared by the evaluator with a compact -failure-evidence ref and a specific measurement scope. The kernel must not -infer this classification merely from a poor score or a failing holdout. -Absent an explicit declaration, the fail-closed default is -`mechanism_contradicted` or the existing negative-evidence classification. - -### Repair Budget - -The repair count is computed from the parent hypothesis lineage and previous -`research_failure_continuation_v0` records, not from how many times an agent -retries a CLI command. - -- At count `0`, exactly one `remediate_data_measurement` successor is allowed - only for `data_or_measurement_gap`. -- At count `1`, `remediation_allowed=false` permanently for the lineage. -- A remediation retains the original research objective, protected scope, and - acceptance gates. It may correct only the declared measurement scope. -- Its result must return to evaluation. A second negative result cannot reopen - remediation; it must produce a proposer successor or explicit exhaustion. - -## State Transitions - -```text -evaluated - -> contradicted - -> failure_continuation_required - -> remediate_data_measurement (only explicit gap and count=0) - -> evaluated - -> propose_failure_successor (otherwise) - -> hypothesis_proposed | frontier_exhausted -``` - -`continuous_monitor` is not a legal edge in this state machine. - -### Completion Rule - -The evaluator/promoter may complete a failure-review todo only after one of -these machine-visible deltas exists: - -1. an exactly linked `remediate_data_measurement` successor with remaining - repair budget; -2. an exactly linked `propose_failure_successor` todo owned by the - hypothesis-proposer lane; or -3. a durable `frontier_exhausted` record with a compact no-follow-up rationale. - -If none exists, the worker reports `failure_successor_required` and leaves the -todo open. It must not use `no_followup=true`. - -### Proposer Successor Requirements - -`propose_failure_successor` is a manual research action, not an automatic -idea generator. The proposer must select exactly one outcome: - -- **New hypothesis:** preserve the parent hypothesis and failure refs, name the - changed mechanism or available information set, and state why the change is - not a parameter relaxation. -- **Frontier exhaustion:** record that no safe in-scope, currently executable - mechanism change exists. This closes the question without a monitor. - -The new hypothesis must not merely change a percentile cutoff, holding window, -asset, time slice, or ranking threshold while keeping the same failed -mechanism. - -## Local Implementation Shape - -### Read Model - -`research_state.py` derives a compact failure-continuation summary from -retirement candidates and existing rollout evidence: - -- failure candidate ids and evidence aliases; -- explicit data/measurement classification when recorded; -- lineage repair count and remaining budget; -- allowed successor actions; -- whether explicit frontier exhaustion is the only permitted closeout. - -`build_auto_research_completion_status()` changes the negative path from -`retirement_review_required` to `failure_successor_required` until a permitted -delta is visible. - -### Role Profiles and Worker Runtime - -The evaluator/promoter profile declares failure successors in addition to its -current positive-evidence successors: - -- create `remediate_data_measurement` only when the summary has - `remediation_allowed=true`; -- otherwise create `propose_failure_successor` for `hypothesis-proposer`. - -The worker runtime reuses `apply_role_successor_todos()`. It must create the -linked successor before calling the existing todo-completion helper. This keeps -the behavior local, idempotent, quota-visible, and compatible with the current -role-successor lifecycle. - -The hypothesis proposer profile adds `propose_failure_successor` to its -allowed actions and states the three allowed outcomes above. No new coordinator -or scheduler is introduced. - -### Durable Record - -The continuation decision is appended through the existing rollout-event -mechanism so a later worker can recompute the budget and closeout state. The -record must be idempotent for the same source todo and continuation outcome. - -## Extension Evolution Proposal (P1) - -P1 adds a separate, bounded product-improvement signal. It is not part of the -P0 failure-successor transition and cannot change an extension automatically. - -The completion evolution review classifies a possible gap into one of two -thresholds: - -| Gap class | Proposal threshold | -| --- | --- | -| `reproducible_product_contract_violation` | One research lineage is enough when a focused local fixture proves that a published or executable LoopX contract is violated. | -| `repeated_extension_or_provider_gap` | The same public-safe fingerprint must occur in at least two distinct research lineages. | - -Eligible fingerprints are: - -- a repeated data/measurement validation gap that the current extension - contract cannot express; -- a repeated provider boundary ambiguity that should fail closed earlier; or -- a repeated missing control-plane successor that forces manual user - intervention. - -Single-source failures remain research constraints unless they prove the -first-row product-contract violation. The proposal todo includes the compact -fingerprint, source evidence refs, affected owner, a bounded contract change, -explicit non-goals, and a focused validation target. It never installs, -enables, upgrades, modifies, or publishes an extension. - -This rule covers the current gap: the documented research state machine permits -`contradicted -> hypothesis_proposed`, but the executable evaluator completion -has no matching successor rule. A focused fixture can reproduce that mismatch, -so the evolution review may proactively create an auto-research product -improvement proposal from one completed research lineage. - -## Test Plan - -Tests are written before implementation and exercise real local functions and -todo lifecycle paths. - -1. **Negative evidence requires a delta.** A retired hypothesis produces - `failure_successor_required`; an evaluator completion without successor or - exhaustion is rejected. -2. **Every terminal result receives an evolution review.** Supported, - contradicted, promoted, and retry-exhausted fixtures each emit one - `research_evolution_review_v0` before their source todo can close. -3. **Default negative path creates proposer work.** A contradicted hypothesis - creates one linked `propose_failure_successor` todo for the registered - proposer lane and no monitor. -4. **One repair limit.** An explicitly declared data/measurement gap creates - one remediation successor at count zero. After that remediation is recorded, - a second failure creates a proposer successor rather than another repair. -5. **Explicit exhaustion is legal.** A durable `frontier_exhausted` record - permits no-follow-up without a monitor and remains visible in the frontier. -6. **Parameter-relaxation guard.** A successor proposal lacking a changed - mechanism or measurement scope is rejected. -7. **One-lineage contract violation can propose a product repair.** A fixture - showing a documented transition without a legal successor creates one - `extension_evolution_proposal`; a one-off data outage does not. -8. **Public-safe projection.** Failure continuation and any P1 proposal reject - paths, URLs, credentials, raw logs, and provider payloads. - -## Documentation Changes - -Update the existing auto-research lane, state, and role-state-machine protocols -to describe: - -- bounded remediation before failure successors; -- no-monitor failure closeout; -- proposer responsibility after retirement; and -- the distinction between a research constraint and a repeated-gap extension - proposal. - -No first-screen product surface changes are part of this work. - -## Delivery Plan - -1. **P0:** Add the failure-continuation read model, role successors, durable - record, and focused regression smoke. -2. **P1:** Add repeated-gap aggregation and normal extension-evolution proposal - todo generation without any provider mutation. -3. **P2:** Update public protocols and compact smoke coverage for the complete - lifecycle. - -Each batch remains reviewable on its own. P0 is the first implementation -segment because it directly prevents silent stopping after negative research -evidence. diff --git a/examples/auto-research-demo-e2e-worker-loop-smoke.py b/examples/auto-research-demo-e2e-worker-loop-smoke.py deleted file mode 100644 index 30a014ef3c..0000000000 --- a/examples/auto-research-demo-e2e-worker-loop-smoke.py +++ /dev/null @@ -1,213 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test demo-e2e no longer treats worker-loop as research output.""" - -from __future__ import annotations - -import json -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -AGENT_ID = "auto-research-operator" -GOAL_ID = "loopx-auto-research-demo-worker-loop-smoke" - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def main() -> int: - sys.path.insert(0, str(REPO_ROOT)) - from demo.auto_research.demo_e2e import ( - _build_collective_round_summary, - run_auto_research_demo_e2e, - ) - from demo.auto_research.human_view import render_auto_research_markdown - from demo.auto_research.preset import default_auto_research_agent_specs - - runtime_scripts_source = ( - REPO_ROOT / "demo/multi_agent/runtime_scripts.py" - ).read_text(encoding="utf-8") - preset_source = ( - REPO_ROOT / "demo/auto_research/preset.py" - ).read_text(encoding="utf-8") - assert "raw JSON is not printed in visible panes" in runtime_scripts_source - assert "No automated worker-turn is configured" in runtime_scripts_source - assert "Do not treat this tick as research completion" in runtime_scripts_source - assert "auto-research worker-turn" not in preset_source - assert "AUTO_RESEARCH_DEMO_STAGE_METRICS" not in preset_source - assert default_auto_research_agent_specs() == [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", - ] - - worker_markdown = render_auto_research_markdown( - { - "ok": True, - "schema_version": "auto_research_worker_turn_v0", - "mode": "manual_research_required", - "goal_id": GOAL_ID, - "agent_id": AGENT_ID, - "selected_todo_id": "todo_worker_smoke", - "selected_action": "run_dev_eval", - "executed": False, - "manual_research_required": True, - "completion": {"requested": True, "executed": False}, - "frontier": { - "quota": {"should_run": True, "state": "eligible"}, - "frontier": { - "selected": { - "todo_id": "todo_worker_smoke", - "allowed_action": "run_dev_eval", - } - }, - }, - } - ) - assert "# LoopX Auto Research Worker Turn" in worker_markdown - assert "- mode: `manual_research_required`" in worker_markdown - assert "- selected_action: `run_dev_eval`" in worker_markdown - - with tempfile.TemporaryDirectory(prefix="loopx-visible-worker-loop-smoke.") as tmp: - root = Path(tmp) - registry = root / "registry.json" - runtime_root = root / "runtime" - registry.write_text( - json.dumps({"common_runtime_root": str(runtime_root), "goals": []}), - encoding="utf-8", - ) - payload = run_auto_research_demo_e2e( - agent_id=AGENT_ID, - goal_id=GOAL_ID, - tracking_goal_id=None, - objective="Verify demo-e2e does not manufacture auto-research metrics.", - output_dir="auto_research_lightweight_kernel", - execute=True, - run_worker_loop=True, - worker_loop_rounds=1, - launch_visible=False, - keep_workspace=False, - registry_path=registry, - runtime_root_arg=str(runtime_root), - session_name="loopx-visible-worker-loop-smoke", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - output_language="en", - live_evidence_path=None, - append_evidence=lambda _packet: {}, - ) - - assert payload["ok"] is True, payload - assert payload["execution_kind"] == "loopx_worker_loop", payload - worker_loop = payload["worker_loop"] - assert worker_loop["turn_count"] == 4, worker_loop - assert worker_loop["executed_turn_count"] == 0, worker_loop - assert worker_loop["completed_turn_count"] == 0, worker_loop - assert worker_loop["stop_reason"] == "no_executed_turns", worker_loop - assert worker_loop["selected_actions"] == ["write_research_contract"], worker_loop - manual_turns = [ - turn for turn in worker_loop["turns"] if turn.get("mode") == "manual_research_required" - ] - assert [turn["agent_id"] for turn in manual_turns] == ["research-curator"], worker_loop - assert [turn["selected_action"] for turn in manual_turns] == [ - "write_research_contract" - ], worker_loop - no_action_turns = [turn for turn in worker_loop["turns"] if turn.get("mode") == "no_action"] - assert [turn["agent_id"] for turn in no_action_turns] == [ - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - ], worker_loop - assert all(turn.get("dev_metric") is None for turn in worker_loop["turns"]), worker_loop - assert all(turn.get("holdout_metric") is None for turn in worker_loop["turns"]), worker_loop - assert all("demo_iteration" not in turn for turn in worker_loop["turns"]), worker_loop - - collective = payload["collective_research_rounds"] - assert collective["multi_round_research_verified"] is False, collective - assert collective["claim_source"] == "worker_loop_collective_agent_passes", collective - assert collective["visible_role_participation_verified"] is False, collective - assert collective["visible_role_participation_basis"] == "headless_worker_loop_summary_only", collective - assert collective["holdout_improvement_count"] == 0, collective - assert collective["dev_metric_sequence"] == [], collective - assert collective["holdout_metric_sequence"] == [], collective - proof = payload["visible_worker_proof"] - assert proof["visible_role_participation_verified"] is False, proof - assert proof["decentralized_a2a_rounds_verified"] is False, proof - tonight = payload["tonight_experience"] - assert tonight["positive_result"] is False, tonight - assert tonight["ready"] is False, tonight - assert tonight["positive_result_basis"] == "requires_visible_lane_authored_evidence", tonight - assert tonight["dev_metric"] is None, tonight - assert tonight["holdout_metric"] is None, tonight - assert_public_safe(payload) - - headless_with_metrics = _build_collective_round_summary( - source="worker_loop_collective_agent_passes", - agent_count=4, - collective_round_count=4, - dev_metric=4.8, - holdout_metric=5.2, - dev_metric_sequence=[4.0, 4.8], - holdout_metric_sequence=[4.5, 5.2], - holdout_improvement_count=2, - expected_lanes=[ - {"agent_id": "research-curator"}, - {"agent_id": "hypothesis-proposer"}, - {"agent_id": "research-executor"}, - {"agent_id": "evaluator-promoter"}, - ], - lane_outcomes=[ - { - "round": round_index, - "agent_id": agent_id, - "selected_todo_id": f"todo_{round_index}_{agent_id}", - "selected_action": "summarize_evidence", - "executed": True, - "completion_status": "done", - } - for round_index in range(1, 5) - for agent_id in ( - "research-curator", - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - ) - ], - visible_role_participation_verified=False, - visible_role_participation_basis="headless_worker_loop_summary_only", - ) - assert headless_with_metrics["kernel_ledger"]["collective_research_verification"][ - "verified" - ] is True, headless_with_metrics - assert headless_with_metrics["multi_round_research_verified"] is False, ( - headless_with_metrics - ) - assert headless_with_metrics["claim_boundary"].startswith( - "worker-loop summaries are control-plane plumbing evidence only" - ), headless_with_metrics - - print("auto-research-demo-e2e-worker-loop-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-demo-goal-isolation-smoke.py b/examples/auto-research-demo-goal-isolation-smoke.py deleted file mode 100644 index dcf03ece54..0000000000 --- a/examples/auto-research-demo-goal-isolation-smoke.py +++ /dev/null @@ -1,223 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test isolated goal defaults for the one-command auto-research demo.""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -DEFAULT_GOAL_ID = "loopx-auto-research-demo" -FRESH_GOAL_ID = "loopx-auto-research-demo-smoke-001" -EXPLICIT_GOAL_ID = "loopx-auto-research-demo-explicit" -AGENT_ID = "auto-research-operator" - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def run_cli( - args: list[str], - *, - registry: Path, - runtime_root: Path, - cwd: Path, - check: bool = True, -) -> subprocess.CompletedProcess[str]: - env = os.environ.copy() - env["PYTHONDONTWRITEBYTECODE"] = "1" - env["PYTHONPATH"] = str(REPO_ROOT) - return subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "--format", - "json", - *args, - ], - cwd=cwd, - env=env, - check=check, - capture_output=True, - text=True, - ) - - -def read_json(result: subprocess.CompletedProcess[str]) -> dict[str, Any]: - payload = json.loads(result.stdout) - assert_public_safe(payload) - return payload - - -def assert_route( - payload: dict[str, Any], - *, - goal_id: str, - mode: str, - reuses_default: bool, -) -> None: - assert payload["ok"] is True, payload - assert payload["goal_id"] == goal_id, payload - route = payload["route_contract"] - assert route["frontier_goal_id"] == goal_id, route - assert route["visible_lanes_read_goal_id"] == goal_id, route - assert route["goal_surface_mode"] == mode, route - assert route["reuses_default_internal_goal"] is reuses_default, route - assert route["default_internal_goal_id"] == DEFAULT_GOAL_ID, route - assert route["dedicated_positive_demo_frontier"] is (not reuses_default), route - if mode == "fresh_demo_goal": - assert route["fresh_goal_default"] is True, route - assert route["inherits_default_goal"] is False, route - assert goal_id.startswith("loopx-auto-research-demo-"), route - if mode == "inherited_default_goal": - assert route["fresh_goal_default"] is False, route - assert route["inherits_default_goal"] is True, route - assert "--goal-id " + goal_id in payload["commands"]["one_command_worker_loop"], payload - - -def main() -> int: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - workspace = temp / "workspace" - workspace.mkdir() - registry = temp / "registry.json" - runtime_root = temp / "runtime" - registry.write_text( - json.dumps({"common_runtime_root": str(runtime_root), "goals": []}), - encoding="utf-8", - ) - - fresh = read_json( - run_cli( - [ - "auto-research", - "demo-e2e", - "--agent-id", - AGENT_ID, - "--demo-run-id", - "smoke-001", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - ) - assert_route( - fresh, - goal_id=FRESH_GOAL_ID, - mode="fresh_demo_goal", - reuses_default=False, - ) - - inherited = read_json( - run_cli( - [ - "auto-research", - "demo-e2e", - "--agent-id", - AGENT_ID, - "--inherit-default-goal", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - ) - assert_route( - inherited, - goal_id=DEFAULT_GOAL_ID, - mode="inherited_default_goal", - reuses_default=True, - ) - - explicit = read_json( - run_cli( - [ - "auto-research", - "demo-e2e", - "--agent-id", - AGENT_ID, - "--goal-id", - EXPLICIT_GOAL_ID, - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - ) - assert_route( - explicit, - goal_id=EXPLICIT_GOAL_ID, - mode="explicit_goal", - reuses_default=False, - ) - - supervisor = read_json( - run_cli( - [ - "auto-research", - "demo-supervisor", - "--demo-run-id", - "smoke-001", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - ) - assert supervisor["goal_id"] == FRESH_GOAL_ID, supervisor - surface = supervisor["goal_surface_route"] - assert surface["mode"] == "fresh_demo_goal", surface - assert surface["fresh_by_default"] is True, surface - assert surface["reuses_default_internal_goal"] is False, surface - - conflict = read_json( - run_cli( - [ - "auto-research", - "demo-e2e", - "--agent-id", - AGENT_ID, - "--goal-id", - EXPLICIT_GOAL_ID, - "--inherit-default-goal", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - check=False, - ) - ) - assert conflict["ok"] is False, conflict - assert "--inherit-default-goal cannot be combined with --goal-id" in conflict["error"], conflict - - print("auto-research demo goal isolation smoke passed") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-demo-supervisor-smoke.py b/examples/auto-research-demo-supervisor-smoke.py deleted file mode 100644 index c01989a455..0000000000 --- a/examples/auto-research-demo-supervisor-smoke.py +++ /dev/null @@ -1,443 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the thin auto-research visible TUI supervisor packet.""" - -from __future__ import annotations - -import json -import subprocess -import sys -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_supervisor import ( # noqa: E402 - AUTO_RESEARCH_DEMO_SUPERVISOR_SCHEMA_VERSION, - build_auto_research_demo_supervisor_plan, -) -from demo.auto_research.preset import ( # noqa: E402 - AUTO_RESEARCH_PRESET_SCHEMA_VERSION, -) -from demo.auto_research.user_contract import ( # noqa: E402 - build_auto_research_preset_context, -) - - -GOAL_ID = "loopx-auto-research-demo" -QUESTION = "如何提升 KNN 精确近邻检索速度？" -LANES = [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", -] - - -def assert_no_private_surface(payload: dict[str, Any]) -> None: - text = json.dumps(payload, sort_keys=True) - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "lark" + "office", - "byte" + "dance", - "http" + "://", - "https" + "://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def assert_supervisor_contract(payload: dict[str, Any]) -> None: - assert payload["ok"] is True, payload - assert payload["schema_version"] == AUTO_RESEARCH_DEMO_SUPERVISOR_SCHEMA_VERSION, payload - assert payload["mode"] == "dry_run", payload - assert payload["goal_id"] == GOAL_ID, payload - - layering = payload["layer_minimality_contract"] - assert ( - layering["schema_version"] == "multi_agent_three_layer_minimality_contract_v0" - ), layering - assert layering["product_id"] == "auto-research", layering - assert layering["preset_id"] == "auto_research_thin_preset", layering - assert layering["principle"] == ( - "user_and_preset_stay_thin_kernel_owns_reusable_mechanics" - ), layering - assert layering["user_layer"]["fields"] == [ - "topic_or_objective", - "rounds", - "role_overrides", - "data_or_eval_entrypoint", - ], layering - assert "research_roles" in layering["preset_layer"]["owns"], layering - assert "metric_contract_hints" in layering["preset_layer"]["owns"], layering - assert "multi_agent_runner" in layering["preset_layer"]["forbidden"], layering - assert "decentralized_a2a_driver" in layering["preset_layer"]["forbidden"], layering - assert "pane_local_a2a_status_check" in layering["preset_layer"]["forbidden"], layering - assert "default_loopx_skill_bootstrap" in layering["preset_layer"]["forbidden"], layering - assert "fixed_a2a_wake_prompt" in layering["preset_layer"]["forbidden"], layering - assert "decentralized_a2a_driver" in layering["kernel_layer"]["owns"], layering - assert "compact_human_status" in layering["kernel_layer"]["owns"], layering - assert layering["kernel_layer"]["default_skills"] == [ - "loopx-project", - "loopx-doc-registry", - ], layering - assert layering["acceptance"]["preset_has_no_runner_process_logic"] is True, layering - - preset = payload["preset"] - assert preset["schema_version"] == AUTO_RESEARCH_PRESET_SCHEMA_VERSION, preset - assert preset["role_count"] == 4, preset - assert "minimal_a2a_recipe" not in payload, payload - minimal_recipe = preset["minimal_a2a_recipe"] - assert minimal_recipe["schema_version"] == "auto_research_minimal_a2a_recipe_v0", preset - assert minimal_recipe["user_plus_preset_line_count"] == 5, preset - assert preset["owns"] == [ - "research_roles", - "handoff_hints", - "metric_contract_hints", - "domain_defaults", - ], preset - assert "multi_agent_runner" in preset["forbidden"], preset - assert "decentralized_a2a_driver" in preset["forbidden"], preset - assert "pane_local_a2a_status_check" in preset["forbidden"], preset - assert "default_loopx_skill_bootstrap" in preset["forbidden"], preset - assert "fixed_a2a_wake_prompt" in preset["forbidden"], preset - assert preset["worker_skill_scope"] == "role_specific_semantics_and_successor_todos_only", preset - assert preset["successor_routing"] == "role_profile_successor_todos_with_target_agent", preset - - kernel = payload["auto_research"] - assert kernel["schema_version"] == AUTO_RESEARCH_PRESET_SCHEMA_VERSION, payload - assert kernel["uses_generic_runner"] is True, payload - assert kernel["surface_count"] == 4, payload - assert kernel["state_bus"] == "loopx_registry_runtime_todo_quota_frontier", payload - assert kernel["kernel_driver"] == "decentralized_a2a_driver", payload - assert kernel["kernel_driver_schema"] == "multi_agent_decentralized_a2a_driver_contract_v1", payload - assert kernel["worker_turn_owner"] == "generic_multi_agent_kernel", payload - assert "pane_local_a2a_status_check" in kernel["delegated_kernel_mechanics"], payload - assert kernel["presentation_layers_in_kernel"] is False, payload - - coordination = payload["coordination_model"] - assert coordination["leader_agent_required"] is False, payload - assert coordination["pattern"] == "decentralized_state_a2a", payload - assert "quota_should_run" in coordination["source_of_truth"], payload - assert "agent_scoped_frontier" in coordination["source_of_truth"], payload - - shared = payload["shared_goal_surface"] - assert shared["shared_goal_id"] == GOAL_ID, shared - assert shared["shared_state_route"] == "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", shared - assert shared["shared_frontier"] is True, shared - assert shared["all_lane_workspace_isolation"] is False, shared - assert "agent-scoped todo/frontier" in shared["mutation_isolation_policy"], shared - - runner = payload["runner_contract"] - assert runner["runner_surface"] == "tmux_codex_cli_tui", runner - assert runner["coordination_model"]["leader_required"] is False, runner - assert runner["coordination_model"]["state_bus"] == "loopx_registry_runtime_todo_quota_frontier", runner - assert runner["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", runner - assert runner["pane_local_a2a"]["human_default"] == "markdown_status_inside_codex_tui", runner - driver = runner["decentralized_a2a_driver"] - assert driver["owner_layer"] == "generic_multi_agent_kernel", driver - assert driver["broadcaster"]["decides_work"] is False, driver - assert driver["acceptance"]["user_and_preset_do_not_own_tick_driver"] is True, driver - assert runner["role_prompt_and_skill"]["default_kernel_skills"] == [ - "loopx-project", - "loopx-doc-registry", - ], runner - assert ( - runner["role_prompt_and_skill"]["worker_local_skill_scope"] - == "role_specific_semantics_only" - ), runner - - tui = payload["interactive_tui_contract"] - assert tui["codex_surface"] == "interactive_cli_tui", tui - assert tui["visible_json_policy"] == "not_printed_before_tui", tui - assert "pre_codex_character_stream" in tui["forbidden_visible_content"], tui - assert "raw_frontier_json" in tui["forbidden_visible_content"], tui - - lanes = payload["lanes"] - assert [lane["lane_id"] for lane in lanes] == [ - "research-curator", - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - ], payload - assert [lane["role_id"] for lane in lanes] == [ - "research_curator", - "hypothesis_proposer", - "research_executor", - "evaluator_promoter", - ], payload - expected_action_hints = { - "research_curator": "write_research_contract", - "hypothesis_proposer": "propose_hypothesis", - "research_executor": "run_dev_eval", - "evaluator_promoter": "summarize_evidence", - } - for lane in lanes: - profile = lane["role_profile"] - assert profile["schema_version"] == "auto_research_role_profile_v0", profile - assert profile["goal_id"] == GOAL_ID, profile - assert profile["agent_id"] == lane["agent_id"], profile - assert profile["role_id"] == lane["role_id"], profile - assert profile["required_skill"] == "loopx-auto-research", profile - assert profile["skill_distribution"] == "worker_local", profile - assert ( - profile["worker_skill_scope"] - == "role_specific_semantics_and_successor_todos_only" - ), profile - assert profile["default_kernel_skills_owner"] == "generic_multi_agent_kernel", profile - assert profile["fixed_a2a_wake_prompt_owner"] == "generic_multi_agent_kernel", profile - assert profile["output_language"] in {"en", "zh"}, profile - assert lane["output_language"] == profile["output_language"], lane - if profile.get("open_question"): - assert profile["open_question"] == QUESTION, profile - preset_context = profile["preset_context"] - assert preset_context["preset_id"] == "knn-demo", preset_context - assert preset_context["metric_name"] == "speedup", preset_context - assert preset_context["baseline_metric"] == 1.0, preset_context - assert preset_context["question_text_supplies_baseline"] is False, preset_context - assert preset_context["editable_scope"] == ["solution.py"], preset_context - assert preset_context["protected_scope"] == ["task.py", "eval.py", "eval.sh"], preset_context - assert preset_context["dev_eval_command"] == "bash eval.sh dev", preset_context - assert preset_context["holdout_eval_command"] == "bash eval.sh test", preset_context - first_steps = profile["visible_first_steps"] - assert "research_contract.public.json" in " ".join(first_steps), profile - assert "$LOOPX_PANE_A2A_TICK" in " ".join(first_steps), profile - if lane["role_id"] == "research_executor": - assert any("bash eval.sh dev" in item for item in first_steps), profile - assert any("solution.py" in item for item in first_steps), profile - assert any("bash eval.sh test" in item for item in first_steps), profile - if lane["role_id"] == "evaluator_promoter": - assert any("without `bash eval.sh test`" in item for item in first_steps), profile - assert profile["worker_skill_source"].endswith("auto_research/worker_skill/SKILL.md"), profile - assert expected_action_hints[lane["role_id"]] in profile["allowed_actions"], profile - assert profile["write_scope"], profile - assert profile["stop_conditions"], profile - if lane["role_id"] == "research_executor": - successors = profile["successor_todos"] - assert successors[0]["after_action"] == "run_dev_eval", profile - assert successors[0]["target_agent_id"] == "research-executor", profile - assert successors[0]["action_kind"] == "run_holdout_eval", profile - assert "run_holdout_eval" in profile["allowed_actions"], profile - if lane["role_id"] == "evaluator_promoter": - successors = profile["successor_todos"] - assert any( - item["after_action"] == "write_evaluation_summary" - and item["target_role_id"] == "research_curator" - and item["action_kind"] == "review_research_contract" - for item in successors - ), profile - - assert "quota should-run" in lane["quota_guard"], lane - assert f"--agent-id {lane['agent_id']}" in lane["quota_guard"], lane - assert lane["frontier"] == "agent-scoped LoopX todo/quota/frontier projection", lane - assert lane["bootstrap_message"] == "role_prompt_public_artifact_for_first_turn_and_fixed_wake", lane - assert lane["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", lane - assert lane["pane_local_a2a"]["worker_turn_configured"] is False, lane - assert lane["pane_local_a2a"]["auto_start"] is True, lane - assert lane["pane_local_a2a"]["auto_start_owner"] == "codex_tui_first_turn_prompt", lane - assert lane["pane_local_a2a"]["status_check_only"] is True, lane - assert lane["pane_local_a2a"]["counts_as_research_round"] is False, lane - assert lane["lane_timeline"] == [ - "role_profile", - "codex_tui", - "tui_first_turn_quota_frontier_status_check", - "frontier", - ], lane - - command = lane["visible_launch_command"] - assert "LOOPX_PANE_A2A_TICK" in command, lane - assert "LOOPX_PANE_BOOTSTRAP_PROMPT" in command, lane - assert "LOOPX_ROLE_PROFILE_ARTIFACT" in command, lane - assert "export LOOPX_PANE_WORKER_TURN=" not in command, lane - assert "LOOPX_PANE_TICK_ROUNDS=" not in command, lane - assert "LOOPX_PANE_TICK_SLEEP_SECONDS=" not in command, lane - assert "pane-a2a-tick.output.txt" in command, lane - assert "auto-research worker-turn" not in command, lane - assert "export LOOPX_AUTO_RESEARCH_OUTPUT_LANGUAGE=" not in command, lane - assert "LOOPX_PANE_LOOPX_JSON" in command, lane - assert "LOOPX_VISIBLE_LANE_COUNT" in command, lane - assert "LOOPX_VISIBLE_TUI_SILENT_BOOTSTRAP=1" in command, lane - assert "LOOPX_CODEX_TUI_MODE=interactive" in command, lane - assert "LOOPX_CODEX_TUI_PROMPT_ARTIFACT" in command, lane - assert "LOOPX_CODEX_FULL_BOOTSTRAP_ARTIFACT" in command, lane - assert "codex-visible-first-prompt.public.txt" in command, lane - assert "LOOPX_CODEX_BIN=codex" in command, lane - assert "LOOPX_CODEX_REASONING_EFFORT=high" in command, lane - assert "exec python3 -c" in command, lane - assert "codex exec" not in command, lane - assert "codex_stream_filter" not in command, lane - assert "[LoopX auto-research frontier]" not in command, lane - assert "BOOTSTRAP_ARTIFACT" in command, lane - assert "bootstrap-or-stop" not in command, lane - - start_script = "\n".join(payload["commands"]["start_script"]) - assert "tmux new-session" in start_script, start_script - assert "tmux split-window" in start_script, start_script - assert "tmux select-layout" in start_script and "tiled" in start_script, start_script - assert "LOOPX_PROJECT" in start_script, start_script - assert "LOOPX_CODEX_TUI_MODE=interactive" in start_script, start_script - assert "LOOPX_CODEX_BIN=codex" in start_script, start_script - assert "LOOPX_CODEX_REASONING_EFFORT=high" in start_script, start_script - assert "exec python3 -c" in start_script, start_script - assert "[Codex bootstrap]" not in start_script, start_script - assert "[Codex bootstrap prompt]" not in start_script, start_script - assert payload["commands"]["attach"] == "tmux attach -t loopx-auto-research", payload - - one_click = payload["one_click_demo"] - assert one_click["schema_version"] == "auto_research_one_click_demo_v1", payload - assert one_click["mode"] == "visible_codex_tui_lanes", payload - assert one_click["default_safe"] is True, payload - assert "one tmux window with tiled role panes" in " ".join( - one_click["expected_visible_result"] - ), one_click - assert "each pane starts a real interactive Codex CLI TUI" in " ".join( - one_click["expected_visible_result"] - ), one_click - assert "each role reads its projected frontier and authors a visible research artifact" in " ".join( - one_click["expected_visible_result"] - ), one_click - assert "$LOOPX_PANE_A2A_TICK is a guard/status check" in " ".join( - one_click["expected_visible_result"] - ), one_click - - boundary = payload["boundary"] - assert boundary["starts_visible_processes"] is False, payload - assert boundary["runs_agent_processes"] is False, payload - assert boundary["writes_loopx_state"] is False, payload - assert boundary["spends_loopx_quota"] is False, payload - assert boundary["reads_raw_transcripts"] is False, payload - assert boundary["reads_session_files"] is False, payload - assert boundary["reads_credentials"] is False, payload - assert boundary["shared_goal_surface"] is True, payload - assert boundary["public_safe_redaction"] is True, payload - - for removed_key in [ - "acceptance", - "demo_acceptance", - "launch_checks", - "board", - "showcase_projection", - "artifact_packet", - "public_claim_boundary", - ]: - assert removed_key not in payload, payload - assert_no_private_surface(payload) - - -def run_cli_json() -> dict[str, Any]: - result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--format", - "json", - "auto-research", - "demo-supervisor", - "--goal-id", - GOAL_ID, - "--agent", - LANES[0], - "--agent", - LANES[1], - "--agent", - LANES[2], - "--agent", - LANES[3], - ], - cwd=REPO_ROOT, - check=True, - capture_output=True, - text=True, - ) - return json.loads(result.stdout) - - -def main() -> int: - default_payload = build_auto_research_demo_supervisor_plan(goal_id=GOAL_ID) - assert [lane["agent_id"] for lane in default_payload["lanes"]] == [ - "research-curator", - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - ], default_payload - assert [lane["lane_id"] for lane in default_payload["lanes"]] == [ - "research-curator", - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - ], default_payload - assert default_payload["auto_research"]["surface_count"] == 4, default_payload - assert default_payload["shared_goal_surface"]["shared_goal_id"] == GOAL_ID, default_payload - - payload = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - open_question=QUESTION, - preset_context=build_auto_research_preset_context("knn-demo"), - agent_specs=LANES, - output_language="zh", - ) - assert payload["auto_research"]["output_language"] == "zh", payload - assert_supervisor_contract(payload) - - try: - build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - agent_specs=["research-executor:research-executor:not_a_role"], - ) - except ValueError as exc: - assert "unknown auto-research role_id" in str(exc), exc - else: - raise AssertionError("explicit invalid role_id must not silently fallback") - - cli_payload = run_cli_json() - assert_supervisor_contract(cli_payload) - - markdown = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "auto-research", - "demo-supervisor", - "--goal-id", - GOAL_ID, - "--agent", - LANES[0], - ], - cwd=REPO_ROOT, - check=True, - capture_output=True, - text=True, - ).stdout - assert "# LoopX Auto Research Demo Supervisor" in markdown, markdown - assert "leader_agent_required: `False`" in markdown, markdown - assert "## Role Profiles" in markdown, markdown - assert "loopx-auto-research" in markdown, markdown - assert "## Lane Timeline" in markdown, markdown - assert "required_worker_playbook: `loopx-auto-research`" in markdown, markdown - assert "skill_distribution: `worker_local`" in markdown, markdown - assert "worker_skill_source: `demo/auto_research/worker_skill/SKILL.md`" in markdown, markdown - assert "## One-Click Dry Run" in markdown, markdown - assert "visible_codex_tui_lanes" in markdown, markdown - assert "## Demo Acceptance" not in markdown, markdown - assert "accept when:" not in markdown, markdown - assert "tmux attach -t loopx-auto-research" in markdown, markdown - assert "start_script: `machine_json_only`" in markdown, markdown - assert 'loopx auto-research start "" --execute' in markdown, markdown - assert "tmux new-session" not in markdown, markdown - assert "LOOPX_ROLE_PROFILE_JSON" not in markdown, markdown - - print("auto-research-demo-supervisor-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-dev-thin-preset-smoke.py b/examples/auto-research-dev-thin-preset-smoke.py deleted file mode 100644 index c7b1c09daa..0000000000 --- a/examples/auto-research-dev-thin-preset-smoke.py +++ /dev/null @@ -1,93 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-check that auto-research stays a thin developer-facing preset.""" - -from __future__ import annotations - -import sys -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -if str(ROOT) not in sys.path: - sys.path.insert(0, str(ROOT)) - -from demo.auto_research.preset import ( # noqa: E402 - build_auto_research_minimal_a2a_recipe, -) -from demo.multi_agent.recipe import ( # noqa: E402 - DEFAULT_DECENTRALIZED_A2A_PROOF_CONTRACT, - build_minimal_decentralized_a2a_recipe, - parse_multi_agent_role_spec_lines, -) - - -def main() -> int: - preset_source = ROOT / "demo" / "auto_research" / "preset.py" - generic_source = ( - ROOT / "demo" / "multi_agent" / "recipe.py" - ) - preset_text = preset_source.read_text(encoding="utf-8") - generic_text = generic_source.read_text(encoding="utf-8") - - for proof_key in DEFAULT_DECENTRALIZED_A2A_PROOF_CONTRACT: - assert proof_key not in preset_text, proof_key - assert proof_key in generic_text, proof_key - - auto_recipe = build_auto_research_minimal_a2a_recipe( - open_question="Can decentralized A2A improve research quality?", - output_language="zh", - ) - assert auto_recipe["schema_version"] == "auto_research_minimal_a2a_recipe_v0" - assert auto_recipe["product_id"] == "auto-research" - assert auto_recipe["user_line_count"] == 1 - assert auto_recipe["preset_role_spec_line_count"] == 4 - assert auto_recipe["user_plus_preset_line_count"] == 5 - assert auto_recipe["shared_kernel_counted_as_recipe_lines"] is False - assert auto_recipe["a2a_proof_contract"] == DEFAULT_DECENTRALIZED_A2A_PROOF_CONTRACT - assert auto_recipe["owner_layers"]["preset_layer"] == ( - "role_defaults_and_domain_handoff_only" - ) - - generic_recipe = build_minimal_decentralized_a2a_recipe( - product_id="custom-research", - claim="one command plus two roles starts decentralized A2A", - claim_boundary="count only user and preset recipe lines", - user_recipe_lines=["loopx custom-research start '' --execute"], - preset_recipe_lines=["agent-a:curator", "agent-b:runner"], - ) - assert generic_recipe["schema_version"] == "generic_multi_agent_minimal_recipe_v0" - assert generic_recipe["user_plus_preset_line_count"] == 3 - assert generic_recipe["a2a_proof_contract"] == DEFAULT_DECENTRALIZED_A2A_PROOF_CONTRACT - - lanes = parse_multi_agent_role_spec_lines( - agent_specs=["agent-a:plan:planner:Plan a focused slice", "agent-b"], - default_agent_specs=[], - resolve_role_id=lambda raw, index: raw or f"role-{index}", - default_scope_by_lane={"lane-2": "Run the bounded work"}, - ) - assert lanes == [ - { - "agent_id": "agent-a", - "lane_id": "plan", - "role_id": "planner", - "scope": "Plan a focused slice", - }, - { - "agent_id": "agent-b", - "lane_id": "lane-2", - "role_id": "role-2", - "scope": "Run the bounded work", - }, - ] - - preset_line_count = len(preset_text.splitlines()) - generic_line_count = len(generic_text.splitlines()) - assert preset_line_count <= 430, preset_line_count - assert generic_line_count <= 120, generic_line_count - - print("auto-research-dev-thin-preset-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-knn-evidence-normalization-smoke.py b/examples/auto-research-knn-evidence-normalization-smoke.py deleted file mode 100644 index de0b053fc5..0000000000 --- a/examples/auto-research-knn-evidence-normalization-smoke.py +++ /dev/null @@ -1,243 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test KNN benchmark contract normalization into auto-research evidence. - -The smoke is split into three independent phases so that wall-clock timing -is never a test oracle: - -1. **Evaluator correctness** — the real evaluator must produce a valid - payload (ok=True, numeric positive score) but its timing-based score - is NOT used as a pass/fail gate. - -2. **Evidence normalization (deterministic)** — a fixed public eval - fixture with a known score > baseline_score feeds the evidence - pipeline. improved/contradicted status and protected-scope checks - are verified against this fixture so they are reproducible across CI - hosts. - -3. **Protected-scope violation** — mutating a protected file and - re-running evidence against the same fixture must produce - ``contradicted`` / ``protected_scope_clean=False``. This check is - independent of the evaluator output. -""" - -from __future__ import annotations - -import json -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.knn_demo_workspace import ( # noqa: E402 - materialize_knn_demo_workspace, -) - - -HYPOTHESIS_ID = "HYP-KNN-TOPK-HEAP" -TODO_ID = "todo_knn_topk_heap" -AGENT_ID = "research-executor" -GOAL_ID = "loopx-auto-research-knn-smoke" -MECHANISM_FAMILY = "topk_heap_exact" -HYPOTHESIS_TEXT = "Use exact heap top-k selection instead of full sorting." - -FIXTURE_DEV = REPO_ROOT / "examples" / "fixtures" / "knn-eval-dev.public.json" -FIXTURE_HOLDOUT = REPO_ROOT / "examples" / "fixtures" / "knn-eval-holdout.public.json" - - -TOPK_SOLUTION = '''from __future__ import annotations - -from heapq import nsmallest - - -def _squared_distance(left, right) -> float: - return sum((a - b) * (a - b) for a, b in zip(left, right)) - - -def solve(problem): - database = problem["database"] - k = int(problem["k"]) - output = [] - for query in problem["queries"]: - nearest = nsmallest( - k, - ((_squared_distance(row, query), index) for index, row in enumerate(database)), - ) - output.append([index for _distance, index in nearest]) - return output -''' - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "lark" + "office", - "byte" + "dance", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def run_eval(workspace: Path, split: str, output: Path) -> dict[str, Any]: - result = subprocess.run( - [sys.executable, "eval.py", split], - cwd=workspace, - check=True, - capture_output=True, - text=True, - encoding="utf-8", - errors="replace", - ) - payload = json.loads(result.stdout.strip().splitlines()[-1]) - output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") - return payload - - -def run_evidence(contract: Path, dev: Path, holdout: Path) -> dict[str, Any]: - result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--format", - "json", - "auto-research", - "evidence", - "--contract", - str(contract), - "--eval-result", - str(dev), - "--eval-result", - str(holdout), - "--hypothesis-id", - HYPOTHESIS_ID, - "--todo-id", - TODO_ID, - "--agent-id", - AGENT_ID, - "--claimed-by", - AGENT_ID, - "--mechanism-family", - MECHANISM_FAMILY, - "--hypothesis", - HYPOTHESIS_TEXT, - "--grounding-ref", - "contract:research_contract.public.json", - ], - cwd=REPO_ROOT, - check=False, - capture_output=True, - text=True, - encoding="utf-8", - errors="replace", - ) - if result.returncode != 0: - raise AssertionError( - f"auto-research evidence failed rc={result.returncode}\n" - f"stdout={result.stdout}\nstderr={result.stderr}" - ) - return json.loads(result.stdout) - - -def main() -> None: - with tempfile.TemporaryDirectory() as temp_dir: - workspace = Path(temp_dir) / "knn-workspace" - materialize_knn_demo_workspace(workspace, goal_id=GOAL_ID) - (workspace / "solution.py").write_text(TOPK_SOLUTION, encoding="utf-8") - - artifacts = Path(temp_dir) / "artifacts" - artifacts.mkdir() - dev_out = artifacts / "dev.public.json" - holdout_out = artifacts / "holdout.public.json" - - # ------------------------------------------------------------------ - # Phase 1 — evaluator correctness (no timing oracle) - # ------------------------------------------------------------------ - # The real evaluator must produce structurally valid output: ok=True, - # a numeric score, and a positive value. We explicitly do NOT assert - # score > baseline_score because wall-clock timing is non-deterministic - # across CI hosts. - dev_payload = run_eval(workspace, "dev", dev_out) - holdout_payload = run_eval(workspace, "test", holdout_out) - for split_name, split_payload in [ - ("dev", dev_payload), - ("holdout", holdout_payload), - ]: - assert split_payload["ok"] is True, ( - f"{split_name} eval failed: {split_payload}" - ) - assert isinstance(split_payload.get("score"), (int, float)), ( - f"{split_name} score must be numeric: {split_payload}" - ) - assert split_payload["score"] > 0, ( - f"{split_name} score must be positive: {split_payload}" - ) - - # ------------------------------------------------------------------ - # Phase 2 — evidence normalization against deterministic fixtures - # ------------------------------------------------------------------ - # Use fixed eval fixtures with known score > baseline_score so that - # `improved` status and `metric.value > baseline_metric` assertions - # are reproducible regardless of CI host timing. - packet = run_evidence( - workspace / "research_contract.public.json", - FIXTURE_DEV, - FIXTURE_HOLDOUT, - ) - assert packet["ok"] is True, packet - assert packet["schema_version"] == "auto_research_evidence_packet_v0", packet - assert packet["research_contract"]["schema_version"] == "research_contract_v0", packet - assert packet["research_contract"]["goal_id"] == GOAL_ID, packet - assert packet["summary"]["goal_id"] == GOAL_ID, packet - assert packet["hypothesis"]["status"] == "supported", packet - assert packet["summary"]["splits"] == ["dev", "holdout"], packet - assert packet["summary"]["protected_scope_clean"] is True, packet - assert {event["primary_metric_status"] for event in packet["evidence_events"]} == { - "improved" - }, packet - assert all( - event["metric"]["value"] > event["baseline_metric"] - for event in packet["evidence_events"] - ), packet - assert any( - "command:bash-eval.sh-test" in event["artifact_refs"] - for event in packet["evidence_events"] - ), packet - assert_public_safe(packet) - - # ------------------------------------------------------------------ - # Phase 3 — protected-scope violation - # ------------------------------------------------------------------ - # Mutating a protected file (task.py) must flip the evidence verdict - # and mark protected_scope_clean=False. This path is independent of - # eval output — it only checks the contract's protected-scope SHA-256 - # against the workspace. - (workspace / "task.py").write_text("# protected change\n", encoding="utf-8") - dirty_packet = run_evidence( - workspace / "research_contract.public.json", - FIXTURE_DEV, - FIXTURE_HOLDOUT, - ) - assert dirty_packet["hypothesis"]["status"] == "contradicted", dirty_packet - assert dirty_packet["summary"]["protected_scope_clean"] is False, dirty_packet - assert dirty_packet["summary"]["negative_evidence_count"] == 2, dirty_packet - assert_public_safe(dirty_packet) - - print("auto-research-knn-evidence-normalization-smoke ok") - - -if __name__ == "__main__": - main() diff --git a/examples/auto-research-knn-preset-start-smoke.py b/examples/auto-research-knn-preset-start-smoke.py deleted file mode 100644 index 7365de598a..0000000000 --- a/examples/auto-research-knn-preset-start-smoke.py +++ /dev/null @@ -1,186 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the explicit KNN demo preset on auto-research start.""" - -from __future__ import annotations - -import json -import subprocess -import sys -import tempfile -from pathlib import Path - - -REPO_ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.knn_demo_workspace import ( # noqa: E402 - materialize_knn_demo_workspace, -) -from demo.auto_research.demo_e2e import ( # noqa: E402 - _seed_visible_demo_control_plane, -) -from demo.auto_research.demo_supervisor import ( # noqa: E402 - build_auto_research_demo_supervisor_plan, -) -from demo.auto_research.user_contract import ( # noqa: E402 - build_auto_research_preset_context, -) - -QUESTION = "如何提升 KNN 精确近邻检索速度?" - - -def main() -> int: - result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--format", - "json", - "auto-research", - "start", - QUESTION, - "--preset", - "knn-demo", - "--language", - "zh", - ], - cwd=REPO_ROOT, - check=True, - capture_output=True, - text=True, - ) - payload = json.loads(result.stdout) - contract = payload["user_contract"] - preset = contract["preset_context"] - assert preset["preset_id"] == "knn-demo", preset - assert preset["baseline_source"] == "generated_knn_benchmark_workspace", preset - assert preset["question_text_supplies_baseline"] is False, preset - assert preset["metric_name"] == "speedup", preset - assert preset["baseline_metric"] == 1.0, preset - assert preset["benchmark_contract_file"] == "research_contract.public.json", preset - assert preset["editable_scope"] == ["solution.py"], preset - assert preset["protected_scope"] == ["task.py", "eval.py", "eval.sh"], preset - assert preset["dev_eval_command"] == "bash eval.sh dev", preset - assert preset["holdout_eval_command"] == "bash eval.sh test", preset - assert payload["preset_context"] == preset, payload - assert payload["route_contract"]["preset_id"] == "knn-demo", payload - assert payload["route_contract"]["preset_baseline_source"] == ( - "generated_knn_benchmark_workspace" - ), payload - assert "--preset knn-demo" in contract["one_click_start"]["command"], contract - assert "--preset knn-demo" in payload["commands"]["one_question_start"], payload - assert payload["contract_acceptance"]["accepted"] is True, payload - - supervisor = build_auto_research_demo_supervisor_plan( - goal_id="loopx-auto-research-knn-smoke", - open_question=QUESTION, - preset_context=build_auto_research_preset_context("knn-demo"), - output_language="zh", - ) - role_steps = { - role["role_id"]: role["role_profile"]["visible_first_steps"] - for role in supervisor["lanes"] - } - continuation_policies = { - role["role_id"]: role["role_profile"]["continuation_policy"] - for role in supervisor["lanes"] - } - for policy in continuation_policies.values(): - assert policy["successor_source"] == "role_profile.successor_todos", policy - assert policy["required_holdout_improvement_count"] == 2, policy - assert "no-follow-up" in policy["no_followup_rule"], policy - assert "role-specific public-safe artifact" in " ".join( - role_steps["research_curator"] - ), role_steps - assert "two-row hypothesis table" in " ".join( - role_steps["hypothesis_proposer"] - ), role_steps - assert "save the last JSON line as baseline dev evidence" in " ".join( - role_steps["research_executor"] - ), role_steps - assert "feed the real evaluator JSON" in " ".join( - role_steps["research_executor"] - ), role_steps - assert "protected-scope cleanliness" in " ".join( - role_steps["evaluator_promoter"] - ), role_steps - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - _visible_control, registry, _runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, - goal_id="loopx-auto-research-knn-smoke", - objective=QUESTION, - supervisor=supervisor, - preset_context=preset, - ) - registry_payload = json.loads(registry.read_text(encoding="utf-8")) - goal = next( - item - for item in registry_payload["goals"] - if item["id"] == "loopx-auto-research-knn-smoke" - ) - assert "solution.py" in goal["coordination"]["write_scope"], goal - assert "benchmark_runner" in goal["coordination"]["available_capabilities"], goal - - markdown_result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "auto-research", - "start", - QUESTION, - "--preset", - "knn-demo", - "--language", - "zh", - ], - cwd=REPO_ROOT, - check=True, - capture_output=True, - text=True, - ) - markdown = markdown_result.stdout - assert "## Preset Context" in markdown, markdown - assert "- preset_id: `knn-demo`" in markdown, markdown - assert "- baseline_source: `generated_knn_benchmark_workspace`" in markdown, markdown - assert "- question_text_supplies_baseline: `False`" in markdown, markdown - assert "- dev_eval_command: `bash eval.sh dev`" in markdown, markdown - assert "- holdout_eval_command: `bash eval.sh test`" in markdown, markdown - assert "--preset knn-demo" in markdown, markdown - - with tempfile.TemporaryDirectory() as temp_dir: - workspace = Path(temp_dir) / "knn-workspace" - marker = materialize_knn_demo_workspace( - workspace, - goal_id="loopx-auto-research-knn-smoke", - ) - contract_payload = json.loads((workspace / "research_contract.public.json").read_text(encoding="utf-8")) - assert marker["goal_id"] == "loopx-auto-research-knn-smoke", marker - assert contract_payload["goal_id"] == "loopx-auto-research-knn-smoke", contract_payload - assert marker["dev_eval_command"] == "bash eval.sh dev", marker - assert marker["holdout_eval_command"] == "bash eval.sh test", marker - assert (workspace / "solution.py").exists(), marker - assert (workspace / "task.py").exists(), marker - assert (workspace / "research_contract.public.json").exists(), marker - dev_result = subprocess.run( - ["bash", "eval.sh", "dev"], - cwd=workspace, - check=True, - capture_output=True, - text=True, - ) - assert "score:" in dev_result.stdout, dev_result.stdout - dev_payload = json.loads(dev_result.stdout.strip().splitlines()[-1]) - assert dev_payload["ok"] is True, dev_payload - assert dev_payload["metric_name"] == "speedup", dev_payload - assert dev_payload["score"] > 0.0, dev_payload - - print("auto-research-knn-preset-start-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-layered-e2e-acceptance-smoke.py b/examples/auto-research-layered-e2e-acceptance-smoke.py deleted file mode 100644 index 3e0d23652a..0000000000 --- a/examples/auto-research-layered-e2e-acceptance-smoke.py +++ /dev/null @@ -1,170 +0,0 @@ -#!/usr/bin/env python3 -"""Acceptance smoke for the real visible auto-research boundary.""" - -from __future__ import annotations - -import json -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -GOAL_ID = "loopx-auto-research-demo-layered-acceptance" -AGENT_ID = "auto-research-operator" - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - "deterministic_" + "protected_eval_kernel", - "raw transcript", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def assert_three_layer_minimality() -> None: - sys.path.insert(0, str(REPO_ROOT)) - from demo.auto_research.demo_supervisor import ( - build_auto_research_demo_supervisor_plan, - ) - from demo.auto_research.preset import AUTO_RESEARCH_PRESET_SCHEMA_VERSION - - supervisor = build_auto_research_demo_supervisor_plan(goal_id=GOAL_ID) - layering = supervisor["layer_minimality_contract"] - assert layering["principle"] == "user_and_preset_stay_thin_kernel_owns_reusable_mechanics" - assert layering["preset_layer"]["owns"] == [ - "research_roles", - "handoff_hints", - "metric_contract_hints", - "domain_defaults", - ], layering - for mechanic in [ - "multi_agent_runner", - "real_codex_tui_panes", - "workspace_and_trust_safe_launch", - "decentralized_a2a_driver", - "pane_local_a2a_status_check", - "todo_evidence_status_protocol", - "compact_human_status", - ]: - assert mechanic in layering["preset_layer"]["forbidden"], layering - assert mechanic in layering["kernel_layer"]["owns"], layering - - preset = supervisor["preset"] - assert preset["schema_version"] == AUTO_RESEARCH_PRESET_SCHEMA_VERSION, preset - assert preset["minimal_a2a_recipe"]["user_plus_preset_line_count"] == 5, preset - assert preset["owns"] == layering["preset_layer"]["owns"], preset - assert preset["worker_skill_scope"] == "role_specific_semantics_and_successor_todos_only", preset - - driver = supervisor["runner_contract"]["decentralized_a2a_driver"] - assert driver["broadcaster"]["decides_work"] is False, driver - assert driver["broadcaster"]["runs_worker_turn"] is False, driver - assert driver["pane"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", driver - assert driver["layer_budget"]["preset_layer"] == [ - "domain_roles", - "handoff_hints", - "optional_worker_hook", - ], driver - - for lane in supervisor["lanes"]: - assert lane["pane_local_a2a"]["auto_start"] is True, lane - assert lane["pane_local_a2a"]["worker_turn_configured"] is False, lane - assert lane["pane_local_a2a"]["status_check_only"] is True, lane - assert lane["pane_local_a2a"]["counts_as_research_round"] is False, lane - command = lane["visible_launch_command"] - assert "LOOPX_PANE_A2A_TICK" in command, lane - assert "auto-research worker-turn" not in command, lane - assert "--complete-selected-todo" not in command, lane - assert "LOOPX_PANE_TICK_ROUNDS=" not in command, lane - assert "LOOPX_PANE_TICK_SLEEP_SECONDS=" not in command, lane - assert_public_safe(supervisor) - - -def assert_headless_worker_loop_is_not_research_uplift() -> None: - sys.path.insert(0, str(REPO_ROOT)) - from demo.auto_research.demo_e2e import run_auto_research_demo_e2e - from demo.auto_research.human_view import render_auto_research_markdown - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - registry = temp / "registry.json" - runtime_root = temp / "runtime" - registry.write_text( - json.dumps({"common_runtime_root": str(runtime_root), "goals": []}), - encoding="utf-8", - ) - payload = run_auto_research_demo_e2e( - agent_id=AGENT_ID, - goal_id=GOAL_ID, - tracking_goal_id=None, - objective="Verify headless worker loop cannot claim research uplift.", - output_dir="auto_research_lightweight_kernel", - execute=True, - run_worker_loop=True, - worker_loop_rounds=1, - launch_visible=False, - keep_workspace=False, - registry_path=registry, - runtime_root_arg=str(runtime_root), - session_name="loopx-ar-layered-acceptance", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - output_language="en", - live_evidence_path=None, - append_evidence=lambda _packet: {}, - ) - - worker_loop = payload["worker_loop"] - assert worker_loop["turn_count"] == 4, worker_loop - assert worker_loop["executed_turn_count"] == 0, worker_loop - assert worker_loop["completed_turn_count"] == 0, worker_loop - assert all(turn.get("dev_metric") is None for turn in worker_loop["turns"]), worker_loop - assert all(turn.get("holdout_metric") is None for turn in worker_loop["turns"]), worker_loop - - collective = payload["collective_research_rounds"] - assert collective["multi_round_research_verified"] is False, collective - assert collective["visible_role_participation_verified"] is False, collective - assert collective["visible_role_participation_basis"] == "headless_worker_loop_summary_only", collective - assert collective["holdout_improvement_count"] == 0, collective - assert collective["dev_metric_sequence"] == [], collective - assert collective["holdout_metric_sequence"] == [], collective - tonight = payload["tonight_experience"] - assert tonight["positive_result"] is False, tonight - assert tonight["ready"] is False, tonight - assert tonight["positive_result_basis"] == "requires_visible_lane_authored_evidence", tonight - assert tonight["dev_metric"] is None, tonight - assert tonight["holdout_metric"] is None, tonight - - markdown = render_auto_research_markdown(payload) - assert "claim_source: `worker_loop_collective_agent_passes`" in markdown, markdown - assert "visible_role_participation_verified: `False`" in markdown, markdown - assert "dev_metric_sequence: `none`" in markdown, markdown - assert "holdout_metric_sequence: `none`" in markdown, markdown - assert "holdout_improvement_count: `0`" in markdown, markdown - assert "holdout=5.2" not in markdown, markdown - assert_public_safe(payload) - assert_public_safe(markdown) - - -def main() -> int: - assert_three_layer_minimality() - assert_headless_worker_loop_is_not_research_uplift() - print("auto-research-layered-e2e-acceptance-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-live-evidence-capture-smoke.py b/examples/auto-research-live-evidence-capture-smoke.py deleted file mode 100644 index c7a8f02de3..0000000000 --- a/examples/auto-research-live-evidence-capture-smoke.py +++ /dev/null @@ -1,351 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test compact live evidence capture for visible auto-research lanes.""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -GOAL_ID = "loopx-auto-research-demo" -AGENT_ID = "research-executor" - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def run_cli( - args: list[str], - *, - registry: Path, - runtime_root: Path, - cwd: Path, - check: bool = True, -) -> subprocess.CompletedProcess[str]: - env = os.environ.copy() - env["PYTHONDONTWRITEBYTECODE"] = "1" - env["PYTHONPATH"] = str(REPO_ROOT) - return subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "--format", - "json", - *args, - ], - cwd=cwd, - env=env, - check=check, - capture_output=True, - text=True, - ) - - -def write_demo_contract_and_eval(workspace: Path) -> tuple[Path, Path, Path]: - contract_path = workspace / "research-contract.public.json" - dev_path = workspace / "dev-result.public.json" - holdout_path = workspace / "holdout-result.public.json" - contract_path.write_text( - json.dumps( - { - "schema_version": "research_contract_v0", - "goal_id": GOAL_ID, - "research_objective": "Validate compact visible-lane evidence capture.", - "editable_scope": ["candidate_strategy", "todo_handoff"], - "protected_scope": ["metric_definition", "holdout_split"], - "metric": { - "name": "fixture_quality_score", - "direction": "maximize", - "baseline": 1.0, - }, - "dev_eval": "public fixture evaluator on dev split", - "holdout_eval": "public fixture evaluator on holdout split", - "promotion_policy": "dev_and_holdout_improved", - }, - indent=2, - sort_keys=True, - ) - + "\n", - encoding="utf-8", - ) - for path, split, value in ( - (dev_path, "dev", 4.0), - (holdout_path, "holdout", 4.5), - ): - path.write_text( - json.dumps( - { - "schema_version": "auto_research_lightweight_eval_result_v0", - "split": split, - "metric": { - "name": "fixture_quality_score", - "value": value, - "direction": "maximize", - "baseline": 1.0, - }, - "eval_status": "scored", - "primary_metric_status": "improved", - "artifact_refs": [f"public_metric:{split}:state_a2a_round"], - "protected_scope_clean": True, - "no_upload": True, - }, - indent=2, - sort_keys=True, - ) - + "\n", - encoding="utf-8", - ) - return contract_path, dev_path, holdout_path - - -def main() -> int: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - workspace = temp / "workspace" - workspace.mkdir() - registry = temp / "registry.json" - runtime_root = temp / "runtime" - registry.write_text( - json.dumps({"common_runtime_root": str(runtime_root), "goals": []}), - encoding="utf-8", - ) - - contract_path, dev_path, holdout_path = write_demo_contract_and_eval(workspace) - - evidence = run_cli( - [ - "auto-research", - "evidence", - "--contract", - str(contract_path), - "--eval-result", - str(dev_path), - "--eval-result", - str(holdout_path), - "--hypothesis-id", - "hyp_live_lane_state_a2a_round", - "--todo-id", - "todo_live_lane_state_a2a_round", - "--agent-id", - AGENT_ID, - "--claimed-by", - AGENT_ID, - "--mechanism-family", - "state_a2a_iteration", - "--hypothesis", - "Use a small state-mediated handoff loop to improve the shared candidate.", - "--grounding-ref", - "fixture:lane_authored_metric", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - evidence_payload = json.loads(evidence.stdout) - evidence_path = workspace / "evidence.public.json" - evidence_path.write_text(json.dumps(evidence_payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") - - append = run_cli( - [ - "auto-research", - "append-evidence", - "--packet", - str(evidence_path), - "--output", - "append-result.public.json", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - append_payload = json.loads(append.stdout) - append_path = workspace / "append-result.public.json" - assert append_path.is_file(), append_payload - assert json.loads(append_path.read_text(encoding="utf-8")) == append_payload, append_payload - assert append_payload["appended_count"] == 3, append_payload - - live_path = workspace / "live-codex-e2e-evidence.public.json" - rejected = run_cli( - [ - "auto-research", - "capture-live-evidence", - "--packet", - str(evidence_path), - "--append-result", - str(append_path), - "--agent-id", - AGENT_ID, - "--lane-count", - "3", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - check=False, - ) - assert rejected.returncode == 1, rejected.stdout - rejected_payload = json.loads(rejected.stdout) - assert rejected_payload["ok"] is False, rejected_payload - assert "accepted visible lanes" in rejected_payload["error"], rejected_payload - assert_public_safe(rejected_payload) - - capture = run_cli( - [ - "auto-research", - "capture-live-evidence", - "--packet", - str(evidence_path), - "--append-result", - str(append_path), - "--agent-id", - AGENT_ID, - "--lane-count", - "3", - "--visible-lanes-accepted", - "--output", - str(live_path), - "--execute", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - live_payload = json.loads(capture.stdout) - assert live_path.is_file(), live_payload - assert json.loads(live_path.read_text(encoding="utf-8")) == live_payload, live_payload - assert live_payload["schema_version"] == "auto_research_live_codex_lane_e2e_evidence_v0", live_payload - assert live_payload["source"] == "live_codex_lane_output", live_payload - assert live_payload["goal_id"] == GOAL_ID, live_payload - assert live_payload["agent_id"] == AGENT_ID, live_payload - assert live_payload["visible_lanes"]["accepted"] is True, live_payload - assert live_payload["lane_evidence"]["append_status"] == "appended_to_loopx_state", live_payload - assert live_payload["lane_evidence"]["evidence_event_count"] == 2, live_payload - assert live_payload["lane_evidence"]["dev_metric"] == 4.0, live_payload - assert live_payload["lane_evidence"]["holdout_metric"] == 4.5, live_payload - assert_public_safe(live_payload) - - handoff_packet = json.loads(evidence_path.read_text(encoding="utf-8")) - handoff_packet["hypothesis"]["claimed_by"] = "research-curator" - handoff_path = workspace / "handoff-evidence.public.json" - handoff_path.write_text( - json.dumps(handoff_packet, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - - wrong_handoff_agent = run_cli( - [ - "auto-research", - "capture-live-evidence", - "--packet", - str(handoff_path), - "--append-result", - str(append_path), - "--agent-id", - "research-curator", - "--lane-count", - "3", - "--visible-lanes-accepted", - "--execute", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - check=False, - ) - assert wrong_handoff_agent.returncode == 1, wrong_handoff_agent.stdout - wrong_payload = json.loads(wrong_handoff_agent.stdout) - assert "evidence event agent_id" in wrong_payload["error"], wrong_payload - assert_public_safe(wrong_payload) - - handoff_live_path = workspace / "handoff-live-codex-e2e-evidence.public.json" - handoff_capture = run_cli( - [ - "auto-research", - "capture-live-evidence", - "--packet", - str(handoff_path), - "--append-result", - str(append_path), - "--agent-id", - AGENT_ID, - "--lane-count", - "3", - "--visible-lanes-accepted", - "--output", - str(handoff_live_path), - "--execute", - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - handoff_live_payload = json.loads(handoff_capture.stdout) - assert handoff_live_path.is_file(), handoff_live_payload - assert handoff_live_payload["agent_id"] == AGENT_ID, handoff_live_payload - assert handoff_live_payload["lane_evidence"]["dev_metric"] == 4.0, handoff_live_payload - assert handoff_live_payload["lane_evidence"]["holdout_metric"] == 4.5, handoff_live_payload - assert_public_safe(handoff_live_payload) - - claimed = run_cli( - [ - "auto-research", - "demo-e2e", - "--goal-id", - GOAL_ID, - "--agent-id", - AGENT_ID, - "--execute", - "--headless", - "--live-evidence", - str(live_path), - ], - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - ) - claimed_payload = json.loads(claimed.stdout) - proof = claimed_payload["visible_worker_proof"] - live = claimed_payload["live_worker_evidence"] - assert proof["lane_authored_evidence_loaded"] is True, claimed_payload - assert proof["visible_lanes_launched"] is True, claimed_payload - assert proof["visible_lanes_accepted"] is True, claimed_payload - assert proof["evidence_source"] == "live_worker_evidence", claimed_payload - assert live["loaded"] is True, claimed_payload - assert live["source"] == "live_codex_lane_output", claimed_payload - assert live["evidence_event_count"] == 2, claimed_payload - assert live["dev_metric"] == 4.0, claimed_payload - assert live["holdout_metric"] == 4.5, claimed_payload - assert "live_codex_e2e" not in claimed_payload, claimed_payload - assert "claim_summary" not in claimed_payload, claimed_payload - assert_public_safe(claimed_payload) - - print("auto-research-live-evidence-capture-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-minimal-kernel-smoke.py b/examples/auto-research-minimal-kernel-smoke.py deleted file mode 100644 index 30589c5367..0000000000 --- a/examples/auto-research-minimal-kernel-smoke.py +++ /dev/null @@ -1,219 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the minimal auto-research decision kernel. - -The kernel must stay a small evaluator-agnostic decision loop. Public demo and -worker-loop code may wrap it, but the kernel itself must not ship a built-in -protected-eval replay or a user-facing shortcut that looks like a full E2E run. -""" - -from __future__ import annotations - -import json -import subprocess -import sys -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research import ( # noqa: E402 - LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION, - lightweight_hypothesis, - run_lightweight_auto_research, -) - - -KERNEL = REPO_ROOT / "demo/auto_research/kernel.py" -CORE = REPO_ROOT / "demo/auto_research/core.py" -INIT = REPO_ROOT / "demo/auto_research/__init__.py" -EVIDENCE_PACKET = REPO_ROOT / "demo/auto_research/evidence_packet.py" -ROLLOUT_APPEND = REPO_ROOT / "demo/auto_research/rollout_append.py" -LIVE_EVIDENCE = REPO_ROOT / "demo/auto_research/live_evidence.py" -RESEARCH_STATE = REPO_ROOT / "demo/auto_research/research_state.py" -HUMAN_VIEW = REPO_ROOT / "demo/auto_research/human_view.py" -WORKER_RUNTIME = REPO_ROOT / "demo/auto_research/worker_runtime.py" -CLI = REPO_ROOT / "demo/auto_research/cli.py" - -KERNEL_FORBIDDEN_MARKERS = [ - "legacy" + "_core", - "demo_e2e", - "worker_runtime", - "visible_multi_agent_launcher", - "protected_eval", - "quickstart", - "knn", - "showcase", - "tmux", - "codex", -] - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "byte" + "dance", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - "deterministic_" + "protected_eval_kernel", - "protected_eval_result", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def assert_positive_result(payload: dict[str, Any]) -> None: - assert payload["ok"] is True, payload - assert payload["schema_version"] == LIGHTWEIGHT_AUTO_RESEARCH_RESULT_SCHEMA_VERSION, payload - assert payload["candidate_count"] == 2, payload - assert payload["dev_round_count"] == 2, payload - assert payload["evidence_event_count"] == 3, payload - assert payload["selected_hypothesis_id"] == "hyp_partial_selection", payload - assert payload["decision"] == "validated_positive", payload - assert payload["dev_metric"] == 4.0, payload - assert payload["holdout_metric"] == 4.5, payload - assert {event["result_source"] for event in payload["evidence"]} == { - "injected_metric_evaluator" - }, payload - assert payload["public_boundary"]["raw_logs_recorded"] is False, payload - assert payload["public_boundary"]["private_artifacts_recorded"] is False, payload - assert_public_safe(payload) - - -def assert_kernel_boundary() -> None: - kernel_text = KERNEL.read_text(encoding="utf-8") - core_text = CORE.read_text(encoding="utf-8") - init_text = INIT.read_text(encoding="utf-8") - lightweight_surface = kernel_text + "\n" + core_text + "\n" + init_text - assert len(kernel_text.splitlines()) <= 220 - assert len(core_text.splitlines()) <= 40 - assert "run_builtin_lightweight_demo" not in lightweight_surface - leaked_markers = [ - marker - for marker in KERNEL_FORBIDDEN_MARKERS - if marker.lower() in lightweight_surface.lower() - ] - assert not leaked_markers, leaked_markers - - -def assert_evidence_packet_boundary() -> None: - evidence_text = EVIDENCE_PACKET.read_text(encoding="utf-8") - rollout_text = ROLLOUT_APPEND.read_text(encoding="utf-8") - live_text = LIVE_EVIDENCE.read_text(encoding="utf-8") - research_state_text = RESEARCH_STATE.read_text(encoding="utf-8") - human_view_text = HUMAN_VIEW.read_text(encoding="utf-8") - worker_text = WORKER_RUNTIME.read_text(encoding="utf-8") - cli_text = CLI.read_text(encoding="utf-8") - legacy_name = "legacy" + "_core" - quickstart_name = "quickstart" + "_seed" - assert not (REPO_ROOT / f"demo/auto_research/{legacy_name}.py").exists() - assert not (REPO_ROOT / f"demo/auto_research/{quickstart_name}.py").exists() - assert legacy_name not in evidence_text - assert legacy_name not in cli_text - assert f"from .{legacy_name} import" not in rollout_text - assert f"from .{legacy_name} import" not in live_text - assert f"from .{legacy_name} import" not in worker_text - assert f"from .{legacy_name} import" not in cli_text - assert quickstart_name not in worker_text - assert quickstart_name not in cli_text - assert "protected_eval" not in worker_text - assert "auto_research_knn_pack" not in worker_text - assert "RESEARCH_HYPOTHESIS_SCHEMA_VERSION" not in worker_text - assert "load_auto_research_evidence_packet_inputs" not in worker_text - assert "AUTO_RESEARCH_MANUAL_RESEARCH_REQUIRED_MODE" in worker_text - assert "from .evidence_packet import load_auto_research_evidence_packet_inputs" in cli_text - assert "from .research_state import (" in cli_text - assert "auto-research quickstart" not in cli_text - assert "build_live_auto_research_projection" in cli_text - assert "build_auto_research_projection" in research_state_text - assert "load_auto_research_fixture" in research_state_text - assert "render_auto_research_markdown" in human_view_text - for removed_surface in ( - "build_auto_research_board_projection", - "build_auto_research_demo_acceptance_packet", - "build_research_artifact_packet", - "build_research_showcase_projection", - "AUTO_RESEARCH_BOARD_SCHEMA_VERSION", - "AUTO_RESEARCH_DEMO_ACCEPTANCE_PACKET_SCHEMA_VERSION", - "RESEARCH_SHOWCASE_PROJECTION_SCHEMA_VERSION", - "AUTO_RESEARCH_PUBLIC_CLAIM_BOUNDARY_SCHEMA_VERSION", - ): - assert removed_surface not in research_state_text, removed_surface - assert removed_surface not in human_view_text, removed_surface - assert removed_surface not in cli_text, removed_surface - - -def main() -> None: - assert_kernel_boundary() - assert_evidence_packet_boundary() - - candidates = [ - lightweight_hypothesis( - hypothesis_id="hyp_full_sort", - todo_id="todo_auto_research_minimal_001", - claimed_by="research-curator", - text="Keep full sorting as the baseline candidate.", - candidate_key="full_sort", - ), - lightweight_hypothesis( - hypothesis_id="hyp_partial_selection", - todo_id="todo_auto_research_minimal_002", - claimed_by="research-executor", - text="Use exact partial selection before full sorting.", - candidate_key="partial_selection", - ), - ] - - def evaluate(hypothesis: dict[str, Any], split: str) -> dict[str, Any]: - metrics = { - ("full_sort", "dev"): 1.0, - ("partial_selection", "dev"): 4.0, - ("partial_selection", "holdout"): 4.5, - } - return { - "metric": metrics[(hypothesis["candidate_key"], split)], - "exact": True, - "protected_scope_clean": True, - "strategy": hypothesis["candidate_key"], - "artifact_refs": [f"public_metric:{split}:{hypothesis['candidate_key']}"], - "result_source": "injected_metric_evaluator", - } - - payload = run_lightweight_auto_research( - goal_id="loopx-auto-research-minimal-smoke", - hypotheses=candidates, - evaluate=evaluate, - baseline=1.0, - direction="maximize", - max_dev_rounds=2, - ) - assert_positive_result(payload) - - help_result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--format", - "markdown", - "auto-research", - "--help", - ], - cwd=REPO_ROOT, - check=True, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - assert "lite-e2e" not in help_result.stdout - assert " board " not in help_result.stdout - assert " acceptance " not in help_result.stdout - - -if __name__ == "__main__": - main() diff --git a/examples/auto-research-quota-pause-smoke.py b/examples/auto-research-quota-pause-smoke.py deleted file mode 100644 index bf83307f22..0000000000 --- a/examples/auto-research-quota-pause-smoke.py +++ /dev/null @@ -1,148 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke: quota-pause stop reason in auto-research worker-loop. - -When every turn in a round returns ``paused_by_quota`` (quota says -``should_run=False``), the worker-loop must stop with ``quota_paused`` -instead of ``no_executed_turns``. - -This test calls the worker-turn function directly with a frontier whose -quota reports ``should_run=False``, then verifies the turn and loop-level -semantics without needing a real quota budget change. -""" - -from __future__ import annotations - -import json -import sys -import tempfile -from pathlib import Path -from typing import Any - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.worker_runtime import ( - run_auto_research_worker_turn, - load_auto_research_worker_frontier, - AUTO_RESEARCH_WORKER_TURN_SCHEMA_VERSION, -) -from demo.auto_research.worker_loop import ( - run_auto_research_worker_loop, -) -from demo.auto_research.rollout_append import ( - append_auto_research_rollout_events, -) -from demo.auto_research.demo_e2e import _seed_visible_demo_control_plane -from demo.auto_research.demo_supervisor import ( - build_auto_research_demo_supervisor_plan, -) - -GOAL_ID = "loopx-auto-research-demo" -CURATOR_AGENT_ID = "research-curator" -HYPOTHESIS_AGENT_ID = "hypothesis-proposer" -EXECUTOR_AGENT_ID = "research-executor" -EVALUATOR_AGENT_ID = "evaluator-promoter" -AGENT_IDS = [CURATOR_AGENT_ID, HYPOTHESIS_AGENT_ID, EXECUTOR_AGENT_ID, EVALUATOR_AGENT_ID] -LANES = [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", -] - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", "/" + "private/", "/" + "tmp/", - "http" + "://", "https" + "://", - "api" + "_key", "pass" + "word", "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def test_paused_by_quota_turn_mode() -> None: - """When quota.should_run is False and quota.ok is True, the turn returns - ``paused_by_quota`` mode with a public_boundary block.""" - temp = Path(tempfile.mkdtemp(prefix="loopx-smoke-quota-pause-")) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, agent_specs=LANES, - session_name="loopx-smoke-quota-pause", - cli_bin="loopx", codex_bin="codex", tmux_bin="tmux", reasoning_effort="high", - ) - _, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, goal_id=GOAL_ID, - objective="Verify quota pause produces the correct turn mode.", - supervisor=supervisor, - ) - workspace = temp / "shared-research-workspace" - workspace.mkdir() - - # Load the real frontier first so we can graft a synthetic quota onto it. - frontier = load_auto_research_worker_frontier( - registry_path=registry, runtime_root_arg=runtime_root, - goal_id=GOAL_ID, agent_id=CURATOR_AGENT_ID, workspace=workspace, - ) - - # ---- Execute turn: real quota will say should_run=True (fresh goal). ---- - turn = run_auto_research_worker_turn( - registry_path=registry, runtime_root_arg=runtime_root, - goal_id=GOAL_ID, agent_id=CURATOR_AGENT_ID, - workspace=workspace, execute=True, - ) - assert turn["ok"] is True, turn - # Fresh goal with runnable seed todo — should NOT be paused by quota. - assert turn["mode"] != "paused_by_quota", ( - f"fresh goal should not be quota-paused, got mode={turn['mode']}" - ) - assert_public_safe(turn) - - # ---- Simulate quota pause via dry-run path. ---- - # In a dry-run the kernel doesn't complete todos, so quota isn't spent. - # The turn still sees the live quota state; for a fresh goal quota says - # should_run=True. We verify the mode is NOT paused_by_quota. - dry_turn = run_auto_research_worker_turn( - registry_path=registry, runtime_root_arg=runtime_root, - goal_id=GOAL_ID, agent_id=CURATOR_AGENT_ID, - workspace=workspace, execute=False, - ) - assert dry_turn["ok"] is True, dry_turn - assert dry_turn["mode"] != "paused_by_quota", dry_turn - assert_public_safe(dry_turn) - - # ---- Worker-loop with execute: verify quota_paused is distinct from - # no_executed_turns at the stop_reason layer. ---- - # Fresh goal with one runnable seed todo (write_research_contract). - # The seed todo requires manual research, so executed=False. - # The other 3 lanes are blocked. So stop_reason should be - # no_executed_turns, NOT quota_paused (quota is not paused). - loop_payload = run_auto_research_worker_loop( - registry_path=registry, runtime_root_arg=runtime_root, - goal_id=GOAL_ID, agent_ids=AGENT_IDS, workspace=workspace, - objective="Verify quota-pause stop reason is distinct.", - max_rounds=1, execute=True, complete_selected_todo=True, - visible_lanes_accepted=True, lane_count=len(AGENT_IDS), - append_evidence=append_auto_research_rollout_events, - ) - assert loop_payload["ok"] is True, loop_payload - assert loop_payload["stop_reason"] in ("no_executed_turns", "no_runnable_frontier"), ( - f"unexpected stop_reason={loop_payload['stop_reason']}" - ) - # Verify the distinction: quota_paused is NOT returned for a fresh goal. - assert loop_payload["stop_reason"] != "quota_paused", ( - "fresh goal should NOT report quota_paused" - ) - assert_public_safe(loop_payload) - - -def main() -> int: - test_paused_by_quota_turn_mode() - print(" ok paused_by_quota turn mode is distinct from no_executed_turns") - print("auto-research-quota-pause-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-rollout-readpath-smoke.py b/examples/auto-research-rollout-readpath-smoke.py deleted file mode 100644 index b229870efb..0000000000 --- a/examples/auto-research-rollout-readpath-smoke.py +++ /dev/null @@ -1,551 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test reading auto-research rollout events back into live evidence graphs.""" - -from __future__ import annotations - -import json -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.evidence_packet import _compact_public_text # noqa: E402 -from demo.auto_research.research_state import ( # noqa: E402 - _compact_optional_text, - build_auto_research_completion_status, - build_live_auto_research_projection, - build_research_decision_candidates, - build_research_evidence_graph_from_records, - build_research_failure_continuation_resolution, - build_research_evidence_graph_from_rollout_events, -) -from loopx.rollout_event_log import load_rollout_events, rollout_event_log_path # noqa: E402 - -from examples.auto_research_lightweight_fixture import ( # noqa: E402 - AGENT_ID, - GOAL_ID, - GROUNDING_REF, - HYPOTHESIS_ID, - HYPOTHESIS_TEXT, - MECHANISM_FAMILY, - METRIC_NAME, - TODO_ID, - write_contract_and_results, -) - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "lark" + "office", - "byte" + "dance", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def run_json(args: list[str]) -> dict[str, Any]: - result = subprocess.run( - [sys.executable, *args], - cwd=REPO_ROOT, - check=True, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - assert_public_safe(result.stdout) - return json.loads(result.stdout) - - -def evidence_packet(contract: Path, dev: Path, holdout: Path) -> dict[str, Any]: - return run_json( - [ - "-m", - "loopx.cli", - "--format", - "json", - "auto-research", - "evidence", - "--contract", - str(contract), - "--eval-result", - str(dev), - "--eval-result", - str(holdout), - "--hypothesis-id", - HYPOTHESIS_ID, - "--todo-id", - TODO_ID, - "--agent-id", - AGENT_ID, - "--claimed-by", - AGENT_ID, - "--mechanism-family", - MECHANISM_FAMILY, - "--hypothesis", - HYPOTHESIS_TEXT, - "--grounding-ref", - GROUNDING_REF, - "--branch-ref", - "codex/auto-research-rollout-readpath-smoke", - ] - ) - - -def append_packet(registry: Path, packet: Path) -> dict[str, Any]: - return run_json( - [ - "-m", - "loopx.cli", - "--registry", - str(registry), - "--format", - "json", - "auto-research", - "append-evidence", - "--packet", - str(packet), - ] - ) - - -def frontier_projection(registry: Path) -> dict[str, Any]: - return run_json( - [ - "-m", - "loopx.cli", - "--registry", - str(registry), - "--format", - "json", - "auto-research", - "frontier", - "--goal-id", - GOAL_ID, - "--agent-id", - AGENT_ID, - ] - ) - - -def main() -> None: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - runtime_root = temp / "runtime" - project = temp / "project" - project.mkdir() - (project / "ACTIVE_GOAL_STATE.md").write_text( - "\n".join( - [ - "# Active Goal State", - "", - "## Next Action", - "Run one lightweight state-mediated research round.", - "", - "## Agent Todo", - "", - f"- [ ] [P0-auto-research] Run one lightweight state-mediated research round. ", - "", - ] - ), - encoding="utf-8", - ) - registry = temp / "registry.json" - registry.write_text( - json.dumps( - { - "schema_version": "0.1", - "common_runtime_root": str(runtime_root), - "goals": [ - { - "id": GOAL_ID, - "domain": "auto_research_smoke", - "status": "active", - "repo": str(project), - "state_file": "ACTIVE_GOAL_STATE.md", - "adapter": {"kind": "fixture", "status": "connected-read-only"}, - "coordination": { - "agent_model": "peer_v1", - "registered_agents": ["research-curator", AGENT_ID], - }, - "authority_sources": [], - } - ], - }, - indent=2, - sort_keys=True, - ), - encoding="utf-8", - ) - - from loopx.control_plane.testing.canary_harness import import_fixture_authority - import_fixture_authority(registry, GOAL_ID, provider="sqlite") - - packet_path = temp / "packet.json" - contract, dev, holdout = write_contract_and_results(temp) - packet_path.write_text( - json.dumps(evidence_packet(contract, dev, holdout), indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - append_payload = append_packet(registry, packet_path) - assert append_payload["appended_count"] == 3, append_payload - - rollout_events = load_rollout_events(rollout_event_log_path(runtime_root, GOAL_ID)) - graph = build_research_evidence_graph_from_rollout_events( - goal_id=GOAL_ID, - rollout_events=rollout_events, - ) - assert graph["schema_version"] == "research_evidence_graph_v0", graph - assert graph["source_kind"] == "loopx_rollout_event_log", graph - assert graph["hypothesis_count"] == 1, graph - assert graph["evidence_event_count"] == 2, graph - assert graph["metric"]["name"] == METRIC_NAME, graph - assert graph["baseline_metric"] == 1.0, graph - assert graph["best_dev_metric"] and graph["best_dev_metric"] > 1.0, graph - assert graph["best_holdout_metric"] and graph["best_holdout_metric"] > 1.0, graph - assert graph["holdout_improved"] is True, graph - assert graph["nodes"][0]["hypothesis_id"] == HYPOTHESIS_ID, graph - assert graph["nodes"][0]["evidence_event_count"] == 2, graph - assert graph["nodes"][0]["best_dev_metric"] == graph["best_dev_metric"], graph - assert graph["nodes"][0]["best_holdout_metric"] == graph["best_holdout_metric"], graph - assert graph["nodes"][0]["holdout_improved"] is True, graph - graph_decisions = build_research_decision_candidates(graph) - assert graph_decisions["retirement_candidates"] == [], graph_decisions - assert graph_decisions["promotion_candidates"], graph_decisions - assert graph_decisions["promotion_candidates"][0]["hypothesis_id"] == ( - HYPOTHESIS_ID - ), graph_decisions - assert graph_decisions["promotion_candidates"][0]["requires"] == [ - "boundary_scan", - "promotion_decision", - ], graph_decisions - assert_public_safe(graph) - - negative_graph = build_research_evidence_graph_from_records( - goal_id=GOAL_ID, - hypotheses=[ - { - "schema_version": "research_hypothesis_v0", - "hypothesis_id": "hyp_bad_distance_cache", - "parent_hypothesis_id": None, - "todo_id": "todo_auto_research_bad_001", - "claimed_by": AGENT_ID, - "mechanism_family": "distance_cache", - "hypothesis": "Cache distances before normalization.", - "status": "contradicted", - "grounding_refs": [GROUNDING_REF], - "blocked_by": ["evidence_or_boundary_guardrail_failed"], - } - ], - evidence_events=[ - { - "schema_version": "research_evidence_event_v0", - "hypothesis_id": "hyp_bad_distance_cache", - "todo_id": "todo_auto_research_bad_001", - "agent_id": AGENT_ID, - "attempt": 1, - "split": "dev", - "metric": { - "name": METRIC_NAME, - "value": 0.74, - "direction": "maximize", - }, - "baseline_metric": 1.0, - "eval_status": "scored", - "primary_metric_status": "regressed", - "artifact_refs": ["public_eval_summary:bad_distance_cache_dev"], - "protected_scope_clean": True, - } - ], - metric_name=METRIC_NAME, - metric_direction="maximize", - baseline_metric=1.0, - source_kind="public_records", - ) - negative_decisions = build_research_decision_candidates(negative_graph) - assert negative_decisions["promotion_candidates"] == [], negative_decisions - assert negative_decisions["retirement_candidates"], negative_decisions - assert negative_decisions["retirement_candidates"][0]["hypothesis_id"] == ( - "hyp_bad_distance_cache" - ), negative_decisions - assert negative_decisions["retirement_candidates"][0]["negative_evidence_count"] == 1 - assert negative_decisions["retirement_candidates"][0]["failure_kind"] == ( - "mechanism_contradicted" - ), negative_decisions - assert negative_decisions["retirement_candidates"][0]["next_outcome"] == ( - "propose_failure_successor" - ), negative_decisions - negative_completion = build_auto_research_completion_status( - negative_graph, - negative_decisions, - ) - assert negative_completion["status"] == "failure_successor_required", negative_completion - assert negative_completion["next_action"] == "propose_failure_successor", negative_completion - assert negative_completion["failure_continuation"]["monitor_allowed"] is False, negative_completion - assert_public_safe(negative_graph) - assert_public_safe(negative_decisions) - assert_public_safe(negative_completion) - - ordinary_retry_graph = dict(negative_graph) - ordinary_retry_node = dict(negative_graph["nodes"][0]) - ordinary_retry_node.update( - { - "hypothesis_id": "hyp_retry_still_available", - "todo_id": "todo_auto_research_retry_001", - "status": "needs_retry", - "negative_evidence_count": 0, - "failure_kind": None, - } - ) - ordinary_retry_graph["nodes"] = [ordinary_retry_node] - ordinary_retry_decisions = build_research_decision_candidates(ordinary_retry_graph) - assert ordinary_retry_decisions["retirement_candidates"] == [], ordinary_retry_decisions - ordinary_retry_completion = build_auto_research_completion_status( - ordinary_retry_graph, - ordinary_retry_decisions, - ) - assert ordinary_retry_completion["failure_continuation"] is None, ordinary_retry_completion - assert ordinary_retry_completion["status"] == "active", ordinary_retry_completion - assert_public_safe(ordinary_retry_decisions) - assert_public_safe(ordinary_retry_completion) - - ambiguous_failure_graph = dict(negative_graph) - second_failure_node = dict(negative_graph["nodes"][0]) - second_failure_node.update( - { - "hypothesis_id": "hyp_bad_distance_cache_second", - "todo_id": "todo_auto_research_bad_002", - } - ) - ambiguous_failure_graph["nodes"] = [ - dict(negative_graph["nodes"][0]), - second_failure_node, - ] - ambiguous_failure_decisions = build_research_decision_candidates(ambiguous_failure_graph) - ambiguous_failure_completion = build_auto_research_completion_status( - ambiguous_failure_graph, - ambiguous_failure_decisions, - ) - assert ambiguous_failure_completion["status"] == "failure_successor_required", ambiguous_failure_completion - assert ambiguous_failure_completion["failure_continuation"] is None, ambiguous_failure_completion - assert ambiguous_failure_completion["failure_continuation_ambiguous"] is True, ( - ambiguous_failure_completion - ) - assert_public_safe(ambiguous_failure_decisions) - assert_public_safe(ambiguous_failure_completion) - - missing_lineage_resolution = build_research_failure_continuation_resolution( - { - "retirement_candidates": [ - { - "hypothesis_id": "hyp_other_failure", - "todo_id": "todo_other", - "failure_kind": "mechanism_contradicted", - "measurement_scope": None, - "remediation_attempt_count": 0, - "remediation_attempt_limit": 1, - "next_outcome": "propose_failure_successor", - } - ] - }, - selected_todo_id="todo_selected_summary", - selected_lineage_todo_id="todo_expected", - require_selected_failure_match=True, - ) - assert missing_lineage_resolution["failure_continuation"] is None, ( - missing_lineage_resolution - ) - assert missing_lineage_resolution["matched_candidate_count"] == 0, ( - missing_lineage_resolution - ) - assert missing_lineage_resolution["unresolved"] is True, missing_lineage_resolution - assert_public_safe(missing_lineage_resolution) - - selected_todo_fallback_resolution = build_research_failure_continuation_resolution( - { - "retirement_candidates": [ - { - "hypothesis_id": "hyp_selected_summary_failure", - "todo_id": "todo_selected_summary", - "failure_kind": "mechanism_contradicted", - "measurement_scope": None, - "remediation_attempt_count": 0, - "remediation_attempt_limit": 1, - "next_outcome": "propose_failure_successor", - } - ] - }, - selected_todo_id="todo_selected_summary", - require_selected_failure_match=True, - ) - assert selected_todo_fallback_resolution["failure_continuation"] is None, ( - selected_todo_fallback_resolution - ) - assert selected_todo_fallback_resolution["matched_candidate_count"] == 0, ( - selected_todo_fallback_resolution - ) - assert selected_todo_fallback_resolution["unresolved"] is True, ( - selected_todo_fallback_resolution - ) - assert_public_safe(selected_todo_fallback_resolution) - - data_gap_graph = build_research_evidence_graph_from_records( - goal_id=GOAL_ID, - hypotheses=[ - { - "schema_version": "research_hypothesis_v0", - "hypothesis_id": "hyp_data_gap", - "parent_hypothesis_id": None, - "todo_id": "todo_auto_research_data_gap", - "claimed_by": AGENT_ID, - "mechanism_family": "price_normalization", - "hypothesis": "Normalize the declared price field before scoring.", - "status": "contradicted", - "grounding_refs": [GROUNDING_REF], - "blocked_by": ["evidence_or_boundary_guardrail_failed"], - } - ], - evidence_events=[ - { - "schema_version": "research_evidence_event_v0", - "hypothesis_id": "hyp_data_gap", - "todo_id": "todo_auto_research_data_gap", - "agent_id": AGENT_ID, - "attempt": 1, - "split": "dev", - "metric": { - "name": METRIC_NAME, - "value": 0.8, - "direction": "maximize", - }, - "baseline_metric": 1.0, - "eval_status": "scored", - "primary_metric_status": "regressed", - "failure_kind": "data_or_measurement_gap", - "measurement_scope": "adjusted_price_field", - "remediation_attempt": False, - "artifact_refs": ["public_eval_summary:data_gap_initial"], - "protected_scope_clean": True, - }, - { - "schema_version": "research_evidence_event_v0", - "hypothesis_id": "hyp_data_gap", - "todo_id": "todo_auto_research_data_gap", - "agent_id": AGENT_ID, - "attempt": 2, - "split": "holdout", - "metric": { - "name": METRIC_NAME, - "value": 0.7, - "direction": "maximize", - }, - "baseline_metric": 1.0, - "eval_status": "scored", - "primary_metric_status": "regressed", - "failure_kind": "data_or_measurement_gap", - "measurement_scope": "adjusted_price_field", - "remediation_attempt": True, - "artifact_refs": ["public_eval_summary:data_gap_remediation"], - "protected_scope_clean": True, - }, - ], - metric_name=METRIC_NAME, - metric_direction="maximize", - baseline_metric=1.0, - source_kind="public_records", - ) - data_gap_decisions = build_research_decision_candidates(data_gap_graph) - data_gap_candidate = data_gap_decisions["retirement_candidates"][0] - assert data_gap_candidate["remediation_attempt_count"] == 1, data_gap_candidate - assert data_gap_candidate["remediation_allowed"] is False, data_gap_candidate - assert data_gap_candidate["next_outcome"] == "propose_failure_successor", data_gap_candidate - assert_public_safe(data_gap_graph) - assert_public_safe(data_gap_decisions) - - live_payload = build_live_auto_research_projection( - goal_id=GOAL_ID, - agent_id=AGENT_ID, - quota_payload={ - "ok": True, - "agent_lane_next_action": { - "todo_id": TODO_ID, - "title": "Run one lightweight state-mediated research round", - "claimed_by": AGENT_ID, - "status": "open", - "task_class": "advancement_task", - "action_kind": "run_dev_attempt", - }, - "capability_gate": { - "runnable_candidates": [ - { - "todo_id": TODO_ID, - "title": "Run one lightweight state-mediated research round", - "claimed_by": AGENT_ID, - "status": "open", - "task_class": "advancement_task", - "action_kind": "run_dev_attempt", - } - ], - "blocked_candidates": [], - }, - }, - rollout_events=rollout_events, - ) - assert live_payload["evidence_graph"]["source_kind"] == "loopx_rollout_event_log", live_payload - assert live_payload["evidence_graph"]["evidence_event_count"] == 2, live_payload - assert live_payload["frontier"]["promotion_candidates"], live_payload - assert live_payload["frontier"]["promotion_candidates"][0]["hypothesis_id"] == ( - HYPOTHESIS_ID - ), live_payload - assert live_payload["frontier"]["retirement_candidates"] == [], live_payload - assert "showcase_projection" not in live_payload, live_payload - assert "artifact_packet" not in live_payload, live_payload - assert live_payload["public_boundary"]["source"] == "live_quota_status_and_rollout_event_log", live_payload - assert_public_safe(live_payload) - - cli_payload = frontier_projection(registry) - assert cli_payload["ok"], cli_payload - assert cli_payload["evidence_graph"]["source_kind"] == "loopx_rollout_event_log", cli_payload - assert cli_payload["evidence_graph"]["evidence_event_count"] == 2, cli_payload - assert cli_payload["frontier"]["promotion_candidates"], cli_payload - assert cli_payload["frontier"]["promotion_candidates"][0]["requires"] == [ - "boundary_scan", - "promotion_decision", - ], cli_payload - assert "showcase_projection" not in cli_payload, cli_payload - assert "artifact_packet" not in cli_payload, cli_payload - assert_public_safe(cli_payload) - - print("auto-research-rollout-readpath-smoke ok") - - -def _test_compact_optional_text_dot_guard() -> None: - """Regression: compacting long text must not produce '..' (parent-directory marker).""" - long_text = "a" * 219 + "b." - result = _compact_optional_text(long_text, field="live.title", default="fallback", max_len=220) - assert ".." not in result, f"unexpected .. in compacted text: {result!r}" - _compact_public_text(result, field="live.title", max_len=220) - long_text_no_dot = "a" * 219 + "c" - result2 = _compact_optional_text(long_text_no_dot, field="live.title", default="fallback", max_len=220) - assert ".." not in result2, f"unexpected .. in compacted text: {result2!r}" - _compact_public_text(result2, field="live.title", max_len=220) - print("compact-optional-text-dot-guard ok") - - -if __name__ == "__main__": - _test_compact_optional_text_dot_guard() - main() diff --git a/examples/auto-research-skill-contract-smoke.py b/examples/auto-research-skill-contract-smoke.py deleted file mode 100644 index e0f596db90..0000000000 --- a/examples/auto-research-skill-contract-smoke.py +++ /dev/null @@ -1,126 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the worker-local loopx-auto-research playbook contract.""" - -from __future__ import annotations - -import sys -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(ROOT)) - -from loopx.doctor import REQUIRED_INSTALLED_SKILL_PHRASES # noqa: E402 - - -SKILL = ROOT / "demo" / "auto_research" / "worker_skill" / "SKILL.md" -ROLE_PROFILE = ROOT / "docs" / "reference" / "protocols" / "auto-research-role-profile-v0.md" -ROLE_STATE_MACHINE = ROOT / "docs" / "reference" / "protocols" / "auto-research-role-state-machine-v0.md" -LANE_CONTRACT = ROOT / "docs" / "reference" / "protocols" / "auto-research-lane-contract-v1.md" -GETTING_STARTED = ROOT / "docs" / "guides" / "getting-started.md" - - -PRIVATE_MARKERS = [ - "byte" + "dance", - "lark" + "office", - "fei" + "shu.cn", - "/" + "Users" + "/", - "/" + "private" + "/", - "api" + "_key", - "pass" + "word", - "sec" + "ret", -] - - -def _read(path: Path) -> str: - assert path.exists(), f"missing {path}" - return path.read_text(encoding="utf-8") - - -def _assert_public_safe(text: str, label: str) -> None: - lower = text.lower() - leaked = [marker for marker in PRIVATE_MARKERS if marker.lower() in lower] - assert not leaked, f"{label} leaks private marker(s): {leaked}" - - -def main() -> int: - skill = _read(SKILL) - compact_skill = " ".join(skill.split()) - role_profile = _read(ROLE_PROFILE) - role_state_machine = _read(ROLE_STATE_MACHINE) - lane_contract = _read(LANE_CONTRACT) - getting_started = _read(GETTING_STARTED) - - _assert_public_safe( - "\n".join([skill, role_profile, role_state_machine, lane_contract, getting_started]), - "auto-research skill contract", - ) - - assert "loopx-auto-research" not in REQUIRED_INSTALLED_SKILL_PHRASES - - required_skill_terms = [ - "name: loopx-auto-research", - "worker-local role playbook", - "not a global LoopX skill", - "Identity comes from LoopX control-plane metadata", - "Pane Tick Contract", - "generic multi-agent kernel owns the default LoopX project/doc-registry skills", - "fixed A2A wake prompt", - "auto_research_role_profile_v0", - 'loopx --format json auto-research frontier --goal-id "$LOOPX_GOAL_ID" --agent-id "$LOOPX_AGENT_ID"', - "No role owns the full graph", - "Do not infer role from pane title", - "Research Curator", - "Hypothesis Proposer", - "Research Executor", - "Evaluator/Promoter", - "Projection Narrator", - "Control-Plane Guard", - "Shared Stop Conditions", - "research_contract_v0", - "research_hypothesis_v0", - "auto_research_evidence_packet_v0", - "research_evidence_graph_v0", - "quota should-run", - "role_profile.continuation_policy", - "No-follow-up is only valid", - "demo-supervisor", - "--execute", - ] - for term in required_skill_terms: - assert term in compact_skill, f"skill missing {term!r}" - - forbidden_skill_terms = [ - "skill assigns identity", - "pane title is authoritative", - "leader agent required", - "coordinator owns the graph", - "promote without evidence", - "First Commands", - "$loopx-project", - "$loopx-doc-registry", - ] - lowered = compact_skill.lower() - for term in forbidden_skill_terms: - assert term.lower() not in lowered, f"skill drifted into forbidden term {term!r}" - - assert "required_skill\": \"loopx-auto-research\"" in role_profile, role_profile - assert "continuation_policy" in role_profile, role_profile - assert "no-follow-up depend on evidence" in role_profile, role_profile - assert "worker-local" in role_profile, role_profile - assert "auto_research_role_profile_v0" in role_state_machine, role_state_machine - assert "single promoted branch does not close" in role_state_machine, role_state_machine - assert "product_narrator" in lane_contract, lane_contract - assert ( - "`loopx-auto-research` | Running role-scoped auto-research lanes" - not in getting_started - ), getting_started - assert "worker-local" in getting_started, getting_started - assert "~/.codex/skills/loopx-auto-research" not in getting_started, getting_started - - print("auto-research-skill-contract-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-start-markdown-commands-smoke.py b/examples/auto-research-start-markdown-commands-smoke.py deleted file mode 100644 index 74f12a0395..0000000000 --- a/examples/auto-research-start-markdown-commands-smoke.py +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test operator commands in auto-research start Markdown output.""" - -from __future__ import annotations - -import subprocess -import sys -from pathlib import Path - - -REPO_ROOT = Path(__file__).resolve().parents[1] -QUESTION = "How should LoopX make visible multi-agent auto research useful?" - -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.human_view import ( # noqa: E402 - render_auto_research_markdown, -) -from demo.auto_research.user_contract import ( # noqa: E402 - build_auto_research_user_contract, -) - - -def require(needle: str, text: str) -> None: - assert needle in text, f"missing {needle!r}\n{text}" - - -def forbid(needle: str, text: str) -> None: - assert needle not in text, f"unexpected {needle!r}\n{text}" - - -def main() -> int: - result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "auto-research", - "start", - QUESTION, - ], - cwd=REPO_ROOT, - check=True, - capture_output=True, - text=True, - ) - markdown = result.stdout - require("## Operator Commands", markdown) - require("## Minimal A2A Recipe", markdown) - require("- user_plus_preset_lines: `5`", markdown) - require("- shared_kernel_counted: `False`", markdown) - require( - "- preset_role: hypothesis-proposer / 假设生成", - markdown, - ) - require( - f"- visible role start: `loopx auto-research start '{QUESTION}' --execute`", - markdown, - ) - require( - f"- immediate takeover: `loopx auto-research start '{QUESTION}' --execute --attach`", - markdown, - ) - require("--attach means operator takeover first", markdown) - forbid("--no-wake-visible-after-launch", markdown) - forbid("- tmux attach:", markdown) - - payload = { - "ok": True, - "schema_version": "auto_research_demo_e2e_result_v0", - "mode": "execute", - "execution_kind": "visible_worker_launch", - "result_source": "visible_worker_launcher", - "goal_id": "loopx-auto-research-markdown-smoke", - "tracking_goal_id": None, - "route_contract": {"frontier_goal_id": "loopx-auto-research-markdown-smoke"}, - "agent_id": "auto-research-operator", - "reasoning_effort": "high", - "user_contract": build_auto_research_user_contract(QUESTION), - "contract_acceptance": {"accepted": True}, - "commands": {}, - "supervisor": {"lane_count": 4}, - "visible_worker_proof": { - "visible_lanes_launched": True, - "visible_lanes_accepted": True, - }, - "visible_pane_a2a_status": {}, - "visible_readiness": {}, - "visible_launch": { - "launch_result": { - "attach_command": "tmux attach -t loopx-auto-research", - "stop_command": "tmux kill-session -t loopx-auto-research", - } - }, - } - launched = render_auto_research_markdown(payload) - require("- tmux attach: `tmux attach -t loopx-auto-research`", launched) - require("- tmux stop: `tmux kill-session -t loopx-auto-research`", launched) - forbid("--no-wake-visible-after-launch", launched) - - print("auto-research-start-markdown-commands-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-state-aware-wake-smoke.py b/examples/auto-research-state-aware-wake-smoke.py deleted file mode 100644 index ae08f233f9..0000000000 --- a/examples/auto-research-state-aware-wake-smoke.py +++ /dev/null @@ -1,169 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke: state-aware wake filter for visible auto-research worker panes. - -Covers: - - Empty ready_lanes after filtering produces a no-op receipt (does NOT - call the underlying wake function, which would misinterpret [] as - "all lanes"). - - Filtered lanes get clear reason codes with no internal paths. - - The no-op receipt is public-safe and carries the correct schema. -""" - -from __future__ import annotations - -import json -import sys -import tempfile -from pathlib import Path -from typing import Any - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_e2e import _seed_visible_demo_control_plane -from demo.auto_research.demo_supervisor import ( - build_auto_research_demo_supervisor_plan, -) -from demo.auto_research.worker_runtime import ( - load_auto_research_worker_frontier, -) - -GOAL_ID = "loopx-auto-research-demo" -CURATOR_AGENT_ID = "research-curator" -HYPOTHESIS_AGENT_ID = "hypothesis-proposer" -EXECUTOR_AGENT_ID = "research-executor" -EVALUATOR_AGENT_ID = "evaluator-promoter" -AGENT_IDS = [CURATOR_AGENT_ID, HYPOTHESIS_AGENT_ID, EXECUTOR_AGENT_ID, EVALUATOR_AGENT_ID] -LANES = [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", -] - -# Lane ID → agent ID mapping that matches the supervisor lanes above. -LANE_AGENT_MAP = { - "research-curator": "research-curator", - "hypothesis-proposer": "hypothesis-proposer", - "research-executor": "research-executor", - "evaluator-promoter": "evaluator-promoter", -} - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", "/" + "private/", "/" + "tmp/", - "http" + "://", "https" + "://", - "api" + "_key", "pass" + "word", "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def _build_no_op_wake_receipt(session: str) -> dict[str, object]: - """The no-op receipt produced when all lanes are filtered out.""" - return { - "ok": True, - "schema_version": "multi_agent_pane_a2a_wakeup_v0", - "mode": "no_op_all_filtered", - "session_name": session, - "target_lanes": [], - "prompt": "", - "prompt_hash": "", - "coordination_model": "decentralized_state_a2a", - "wakeup_model": "state_aware_filter_no_ready_lanes", - "workflow_driver": False, - "broadcaster_reads_frontier": False, - "broadcaster_reads_todo_readiness": False, - "broadcaster_selects_todo": False, - "prompt_delivery": "skipped_no_ready_lanes", - "prompt_delivered": False, - "auto_wake_backoff_recommended": False, - } - - -def test_empty_ready_lanes_produces_no_op_receipt() -> None: - """When the state-aware filter finds zero ready lanes, it must return a - no-op receipt rather than call the wake function with an empty list - (which legacy code interprets as "all lanes").""" - receipt = _build_no_op_wake_receipt("test-session") - assert receipt["mode"] == "no_op_all_filtered", receipt - assert receipt["target_lanes"] == [], receipt - assert receipt["prompt_delivery"] == "skipped_no_ready_lanes", receipt - assert_public_safe(receipt) - - -def test_frontier_load_failed_produces_safe_error_code() -> None: - """When frontier loading fails, the skipped_lanes entry must only contain - a reason_code — never a raw exception string that could leak paths.""" - skipped_entry: dict[str, object] = { - "lane_id": "test-lane", - "agent_id": "test-agent", - "reason": "frontier_load_failed", - "error_code": "FRONTIER_LOAD_FAILED", - } - assert "error_code" in skipped_entry, skipped_entry - assert "error" not in skipped_entry, ( - "skipped_lanes must not leak raw exception strings" - ) - assert_public_safe(skipped_entry) - - -def test_state_aware_filter_reason_codes_are_public_safe() -> None: - """Every filter reason code must be public-safe — no paths, no URLs.""" - reason_codes = [ - "no_agent_mapping", - "frontier_load_failed", - "quiet_completion_allowed", - "no_selected_todo", - "quota_should_run_false", - ] - for code in reason_codes: - assert_public_safe({"reason": code}) - - -def test_live_frontier_load_is_public_safe() -> None: - """The frontier payload itself must not leak internal paths.""" - temp = Path(tempfile.mkdtemp(prefix="loopx-smoke-state-aware-wake-")) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, agent_specs=LANES, - session_name="loopx-smoke-state-aware-wake", - cli_bin="loopx", codex_bin="codex", tmux_bin="tmux", reasoning_effort="high", - ) - _, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, goal_id=GOAL_ID, - objective="Verify state-aware wake filter produces public-safe payloads.", - supervisor=supervisor, - ) - workspace = temp / "shared-research-workspace" - workspace.mkdir() - - for agent_id in AGENT_IDS: - frontier = load_auto_research_worker_frontier( - registry_path=registry, runtime_root_arg=runtime_root, - goal_id=GOAL_ID, agent_id=agent_id, workspace=workspace, - ) - assert frontier["ok"] is True, frontier - assert "public_boundary" in frontier, ( - f"frontier for {agent_id} must declare a public_boundary" - ) - assert_public_safe(frontier) - - -def main() -> int: - test_empty_ready_lanes_produces_no_op_receipt() - print(" ok empty ready_lanes produces no-op receipt") - test_frontier_load_failed_produces_safe_error_code() - print(" ok frontier_load_failed uses error_code, not raw exception") - test_state_aware_filter_reason_codes_are_public_safe() - print(" ok all filter reason codes are public-safe") - test_live_frontier_load_is_public_safe() - print(" ok live frontier payload is public-safe") - print("auto-research-state-aware-wake-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-stop-marker-smoke.py b/examples/auto-research-stop-marker-smoke.py deleted file mode 100644 index 7be1b85eaf..0000000000 --- a/examples/auto-research-stop-marker-smoke.py +++ /dev/null @@ -1,170 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke: stop marker lifecycle for auto-research worker-loop. - -Covers: - - Placing the stop marker causes ``operator_stop_requested`` before round 1. - - The stop marker is checked before *each* round, not just at start. - - Removing the marker lets the worker-loop resume. -""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_e2e import _seed_visible_demo_control_plane # noqa: E402 -from demo.auto_research.demo_supervisor import ( # noqa: E402 - build_auto_research_demo_supervisor_plan, -) - -GOAL_ID = "loopx-auto-research-demo" -CURATOR_AGENT_ID = "research-curator" -HYPOTHESIS_AGENT_ID = "hypothesis-proposer" -EXECUTOR_AGENT_ID = "research-executor" -EVALUATOR_AGENT_ID = "evaluator-promoter" -AGENT_IDS = [CURATOR_AGENT_ID, HYPOTHESIS_AGENT_ID, EXECUTOR_AGENT_ID, EVALUATOR_AGENT_ID] -LANES = [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", -] - - -def _env() -> dict[str, str]: - env = os.environ.copy() - env["PYTHONDONTWRITEBYTECODE"] = "1" - env["PYTHONPATH"] = f"{REPO_ROOT}{os.pathsep}{env.get('PYTHONPATH', '')}" - return env - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http" + "://", - "https" + "://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def _run_worker_loop( - *, registry: Path, runtime_root: str | None, workspace: Path, max_rounds: int = 2 -) -> dict[str, Any]: - env = _env() - args = [ - sys.executable, "-m", "loopx.cli", "--registry", str(registry), - "--runtime-root", str(runtime_root), "--format", "json", - "auto-research", "worker-loop", "--goal-id", GOAL_ID, - "--lane-count", str(len(AGENT_IDS)), "--max-rounds", str(max_rounds), - "--visible-lanes-accepted", "--complete-selected-todo", "--execute", - ] - for agent_id in AGENT_IDS: - args.extend(["--agent-id", agent_id]) - result = subprocess.run(args, cwd=workspace, env=env, check=False, - capture_output=True, text=True) - if result.returncode != 0: - raise AssertionError( - f"worker-loop failed rc={result.returncode}\nstdout={result.stdout}\nstderr={result.stderr}" - ) - return json.loads(result.stdout) - - -def _stop_marker(workspace: Path) -> Path: - return workspace / ".loopx-auto-research-stop" - - -def test_stop_marker_halts_before_round_one() -> None: - """Placing the stop marker causes the worker-loop to exit before any turns.""" - temp = Path(tempfile.mkdtemp(prefix="loopx-smoke-stop-marker-")) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, agent_specs=LANES, - session_name="loopx-smoke-stop-marker", - cli_bin="loopx", codex_bin="codex", tmux_bin="tmux", reasoning_effort="high", - ) - _, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, goal_id=GOAL_ID, - objective="Verify stop marker halts worker-loop before round 1.", - supervisor=supervisor, - ) - workspace = temp / "shared-research-workspace" - workspace.mkdir() - - # Place stop marker before calling the loop. - _stop_marker(workspace).write_text("stop", encoding="utf-8") - - payload = _run_worker_loop( - registry=registry, runtime_root=runtime_root, workspace=workspace, max_rounds=3, - ) - assert payload["ok"] is True, payload - assert payload["stop_reason"] == "operator_stop_requested", payload["stop_reason"] - assert payload["turn_count"] == 0, payload["turn_count"] - assert_public_safe(payload) - - -def test_stop_marker_checked_before_each_round() -> None: - """Stop marker is checked at the top of every round — not just at start.""" - temp = Path(tempfile.mkdtemp(prefix="loopx-smoke-stop-marker-mid-")) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, agent_specs=LANES, - session_name="loopx-smoke-stop-marker-mid", - cli_bin="loopx", codex_bin="codex", tmux_bin="tmux", reasoning_effort="high", - ) - _, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, goal_id=GOAL_ID, - objective="Verify stop marker is checked before each round.", - supervisor=supervisor, - ) - workspace = temp / "shared-research-workspace" - workspace.mkdir() - - # Run one round without marker. - round1 = _run_worker_loop( - registry=registry, runtime_root=runtime_root, workspace=workspace, max_rounds=1, - ) - assert round1["stop_reason"] in ("no_executed_turns", "no_runnable_frontier"), round1 - - # Place marker — next loop call should stop immediately. - _stop_marker(workspace).write_text("stop", encoding="utf-8") - round2 = _run_worker_loop( - registry=registry, runtime_root=runtime_root, workspace=workspace, max_rounds=1, - ) - assert round2["stop_reason"] == "operator_stop_requested", round2["stop_reason"] - assert round2["turn_count"] == 0, round2 - assert_public_safe(round2) - - # Remove marker — loop can run again. - _stop_marker(workspace).unlink(missing_ok=True) - round3 = _run_worker_loop( - registry=registry, runtime_root=runtime_root, workspace=workspace, max_rounds=1, - ) - assert round3["stop_reason"] != "operator_stop_requested", round3["stop_reason"] - assert_public_safe(round3) - - -def main() -> int: - test_stop_marker_halts_before_round_one() - print(" ok stop marker halts before round one") - test_stop_marker_checked_before_each_round() - print(" ok stop marker checked before each round") - print("auto-research-stop-marker-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-stop-takeover-walkthrough-smoke.py b/examples/auto-research-stop-takeover-walkthrough-smoke.py deleted file mode 100644 index 2ee7caa224..0000000000 --- a/examples/auto-research-stop-takeover-walkthrough-smoke.py +++ /dev/null @@ -1,234 +0,0 @@ -#!/usr/bin/env python3 -"""Walkthrough smoke: documented start identity → stop → resume continuity. - -Drives the shipped CLI with the same explicit arguments the contributor -walkthrough documents (docs/guides/auto-research-stop-takeover-wake- -walkthrough.md): one exported goal/workspace identity, ``start`` targeting -that identity, the stop marker placed in that workspace, and ``worker-loop`` -run from that workspace with the same goal. Marker halting, visible launch -policy, state-aware wake, and quota-pause output shapes stay covered by their -own smokes; this file only proves identity continuity across the cycle. -""" - -from __future__ import annotations - -import importlib.util -import json -import subprocess -import sys -import tempfile -from pathlib import Path -from types import ModuleType -from typing import Any - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_e2e import ( # noqa: E402 - _seed_visible_demo_control_plane, -) -from demo.auto_research.demo_supervisor import ( # noqa: E402 - build_auto_research_demo_supervisor_plan, -) - - -def _load_stop_marker_smoke() -> ModuleType: - path = REPO_ROOT / "examples" / "auto-research-stop-marker-smoke.py" - spec = importlib.util.spec_from_file_location("ar_stop_marker_smoke", path) - if spec is None or spec.loader is None: - raise RuntimeError(f"unable to load stop-marker smoke from {path}") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -_STOP = _load_stop_marker_smoke() -GOAL_ID = _STOP.GOAL_ID -AGENT_IDS = _STOP.AGENT_IDS -LANES = _STOP.LANES -assert_public_safe = _STOP.assert_public_safe -_stop_marker = _STOP._stop_marker -_env = _STOP._env - -QUESTION = "How should we evaluate autonomous research agents?" - - -def _seed_demo() -> tuple[Path, Path, str | None, Path]: - temp = Path(tempfile.mkdtemp(prefix="loopx-smoke-stop-takeover-walkthrough-")) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - agent_specs=LANES, - session_name="loopx-smoke-stop-takeover-walkthrough", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - ) - _, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, - goal_id=GOAL_ID, - objective="Prove stop, takeover command, resume, and state-aware wake.", - supervisor=supervisor, - ) - workspace = temp / "shared-research-workspace" - workspace.mkdir() - return temp, registry, runtime_root, workspace - - -def _cli_command( - registry: Path, runtime_root: str | None, *command: str -) -> list[str]: - args = [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - ] - if runtime_root: - args.extend(["--runtime-root", str(runtime_root)]) - args.extend(["--format", "json", *command]) - return args - - -def _run_documented_start( - *, registry: Path, runtime_root: str | None, workspace: Path -) -> dict[str, Any]: - """Run the documented start path in dry-run mode (no visible lanes).""" - result = subprocess.run( - [ - *_cli_command(registry, runtime_root, "auto-research", "start", QUESTION), - "--goal-id", - GOAL_ID, - "--workspace", - str(workspace), - "--create-workspace", - ], - cwd=workspace, - env=_env(), - check=False, - capture_output=True, - text=True, - ) - assert result.returncode == 0, ( - f"documented start failed rc={result.returncode}\n" - f"stdout={result.stdout}\nstderr={result.stderr}" - ) - return json.loads(result.stdout) - - -def _run_worker_loop( - *, registry: Path, runtime_root: str | None, cwd: Path, max_rounds: int -) -> dict[str, Any]: - """Run the documented worker-loop CLI from an explicit working directory.""" - args = [ - *_cli_command( - registry, - runtime_root, - "auto-research", - "worker-loop", - "--goal-id", - GOAL_ID, - ), - "--lane-count", - str(len(AGENT_IDS)), - "--max-rounds", - str(max_rounds), - "--visible-lanes-accepted", - "--complete-selected-todo", - "--execute", - ] - for agent_id in AGENT_IDS: - args.extend(["--agent-id", agent_id]) - result = subprocess.run( - args, - cwd=cwd, - env=_env(), - check=False, - capture_output=True, - text=True, - ) - assert result.returncode == 0, ( - f"worker-loop failed rc={result.returncode}\n" - f"stdout={result.stdout}\nstderr={result.stderr}" - ) - return json.loads(result.stdout) - - -def test_documented_identity_continuity() -> None: - """The documented start, stop, and resume commands share one identity.""" - _, registry, runtime_root, workspace = _seed_demo() - - # 1) start with the documented explicit flags targets the walkthrough goal. - started = _run_documented_start( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - ) - assert started["ok"] is True, started - assert started["goal_id"] == GOAL_ID, started - route = started["route_contract"] - assert route["goal_surface_mode"] == "explicit_goal", route - assert route["visible_lanes_read_goal_id"] == GOAL_ID, route - assert_public_safe(started) - - # 2) worker-loop runs on the same goal from the same workspace directory. - baseline = _run_worker_loop( - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - max_rounds=1, - ) - assert baseline["ok"] is True, baseline - assert baseline["stop_reason"] != "operator_stop_requested", baseline - assert_public_safe(baseline) - - # 3) The marker in the documented workspace halts that loop before round 1. - _stop_marker(workspace).write_text("stop", encoding="utf-8") - stopped = _run_worker_loop( - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - max_rounds=2, - ) - assert stopped["ok"] is True, stopped - assert stopped["stop_reason"] == "operator_stop_requested", stopped - assert stopped["turn_count"] == 0, stopped - assert_public_safe(stopped) - - # 4) The same marker cannot stop a loop running from another directory; - # this is why the walkthrough requires cd "$WORKSPACE" before resuming. - elsewhere = Path(tempfile.mkdtemp(prefix="loopx-smoke-walkthrough-other-cwd-")) - from_elsewhere = _run_worker_loop( - registry=registry, - runtime_root=runtime_root, - cwd=elsewhere, - max_rounds=1, - ) - assert from_elsewhere["ok"] is True, from_elsewhere - assert from_elsewhere["stop_reason"] != "operator_stop_requested", from_elsewhere - assert_public_safe(from_elsewhere) - - # 5) Resume: removing the marker lets the loop proceed from $WORKSPACE. - _stop_marker(workspace).unlink(missing_ok=True) - resumed = _run_worker_loop( - registry=registry, - runtime_root=runtime_root, - cwd=workspace, - max_rounds=1, - ) - assert resumed["ok"] is True, resumed - assert resumed["stop_reason"] != "operator_stop_requested", resumed - assert_public_safe(resumed) - - -def main() -> int: - test_documented_identity_continuity() - print(" ok documented start identity → stop → resume continuity") - print("auto-research-stop-takeover-walkthrough-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-terminal-results-smoke.py b/examples/auto-research-terminal-results-smoke.py deleted file mode 100644 index b25c5c1af0..0000000000 --- a/examples/auto-research-terminal-results-smoke.py +++ /dev/null @@ -1,500 +0,0 @@ -#!/usr/bin/env python3 -"""Exercise evidence-bound Auto Research terminal results and reviews.""" - -from __future__ import annotations - -import json -import sys -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.evidence_packet import ( # noqa: E402 - RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION, - RESEARCH_HYPOTHESIS_SCHEMA_VERSION, -) -from demo.auto_research.research_state import ( # noqa: E402 - build_research_evidence_graph_from_records, -) -from demo.auto_research.terminal_results import ( # noqa: E402 - build_peer_review_event, - build_terminal_decision_event, - build_terminal_result_explore_events, - build_terminal_result_query, -) -from loopx.capabilities.explore.result_log import ( # noqa: E402 - build_explore_result_projection, -) -GOAL_ID = "terminal-result-fixture" -BASELINE = 0.0 -PRODUCER = "research-executor" -PROMOTER = "evaluator-promoter" -REVIEWER = "independent-reviewer" -REGISTERED_AGENTS = [ - PRODUCER, - PROMOTER, - REVIEWER, - "second-independent-reviewer", -] - - -def hypothesis( - hypothesis_id: str, - *, - text: str, - status: str, - parent: str | None = None, -) -> dict[str, Any]: - return { - "schema_version": RESEARCH_HYPOTHESIS_SCHEMA_VERSION, - "hypothesis_id": hypothesis_id, - "parent_hypothesis_id": parent, - "todo_id": f"todo_{hypothesis_id}", - "claimed_by": PRODUCER, - "mechanism_family": "finance_style_claim", - "hypothesis": text, - "status": status, - "grounding_refs": [], - "novelty_audit_ref": None, - "blocked_by": ( - ["evidence_or_boundary_guardrail_failed"] - if status == "contradicted" - else [] - ), - } - - -def evidence( - hypothesis_id: str, - *, - split: str, - value: float, - metric_status: str, - attempt: int = 1, -) -> dict[str, Any]: - return { - "schema_version": RESEARCH_EVIDENCE_EVENT_SCHEMA_VERSION, - "hypothesis_id": hypothesis_id, - "todo_id": f"todo_{hypothesis_id}", - "agent_id": PRODUCER, - "attempt": attempt, - "split": split, - "metric": { - "name": "research_quality", - "value": value, - "direction": "maximize", - }, - "baseline_metric": BASELINE, - "eval_status": "scored", - "primary_metric_status": metric_status, - "artifact_refs": [f"evidence:{hypothesis_id}:{split}:{attempt}"], - "protected_scope_clean": True, - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - } - - -def fixture_graph() -> tuple[dict[str, Any], list[dict[str, Any]], list[dict[str, Any]]]: - hypotheses = [ - hypothesis( - "value_case", - text="The candidate satisfies the preregistered company-value gate.", - status="supported", - ), - hypothesis( - "persistent_alpha", - text="The candidate has persistent benchmark-adjusted alpha.", - status="contradicted", - ), - hypothesis( - "single_window_alpha", - text="One frozen window contains positive benchmark-adjusted alpha.", - status="supported", - ), - hypothesis( - "walk_forward_alpha", - text="The single-window alpha persists across rolling held-out windows.", - status="contradicted", - parent="single_window_alpha", - ), - hypothesis( - "duplicate_route", - text="A duplicate route should retire without becoming a refuted finding.", - status="active", - ), - ] - evidence_events = [ - evidence("value_case", split="dev", value=0.6, metric_status="improved"), - evidence( - "value_case", - split="holdout", - value=0.4, - metric_status="improved", - ), - evidence( - "persistent_alpha", - split="holdout", - value=-0.5, - metric_status="regressed", - ), - evidence( - "single_window_alpha", - split="dev", - value=0.7, - metric_status="improved", - ), - evidence( - "walk_forward_alpha", - split="holdout", - value=-0.2, - metric_status="regressed", - ), - ] - graph = build_research_evidence_graph_from_records( - goal_id=GOAL_ID, - hypotheses=hypotheses, - evidence_events=evidence_events, - metric_name="research_quality", - metric_direction="maximize", - baseline_metric=BASELINE, - ) - return graph, hypotheses, evidence_events - - -def terminal_events(graph: dict[str, Any]) -> list[dict[str, Any]]: - value = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="value_case", - outcome="promoted", - reason="holdout_validated", - decided_by=PROMOTER, - evidence_refs=["decision:value_case"], - recorded_at="2026-08-08T00:00:00Z", - ) - persistent = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="persistent_alpha", - outcome="retired", - reason="contradicted", - decided_by=PROMOTER, - evidence_refs=["decision:persistent_alpha"], - recorded_at="2026-08-08T00:00:01Z", - ) - rolling = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="walk_forward_alpha", - outcome="retired", - reason="contradicted", - decided_by=PROMOTER, - evidence_refs=["decision:walk_forward_alpha"], - recorded_at="2026-08-08T00:00:02Z", - ) - duplicate = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="duplicate_route", - outcome="retired", - reason="duplicate", - decided_by=PROMOTER, - evidence_refs=["decision:duplicate_route"], - recorded_at="2026-08-08T00:00:03Z", - ) - return [value, persistent, rolling, duplicate] - - -def reviews( - graph: dict[str, Any], - decisions: list[dict[str, Any]], -) -> list[dict[str, Any]]: - self_review = build_peer_review_event( - evidence_graph=graph, - rollout_events=decisions, - hypothesis_id="value_case", - reviewer_agent_id=PROMOTER, - verdict="approve", - evidence_refs=["review:self"], - registered_agent_ids=REGISTERED_AGENTS, - recorded_at="2026-08-08T00:01:00Z", - ) - independent = [ - build_peer_review_event( - evidence_graph=graph, - rollout_events=decisions, - hypothesis_id=hypothesis_id, - reviewer_agent_id=REVIEWER, - verdict="approve", - evidence_refs=[f"review:{hypothesis_id}"], - require_independent=True, - registered_agent_ids=REGISTERED_AGENTS, - recorded_at=f"2026-08-08T00:01:0{index}Z", - ) - for index, hypothesis_id in enumerate( - ["value_case", "persistent_alpha", "walk_forward_alpha"], - start=1, - ) - ] - try: - build_peer_review_event( - evidence_graph=graph, - rollout_events=decisions, - hypothesis_id="value_case", - reviewer_agent_id=PROMOTER, - verdict="approve", - require_independent=True, - registered_agent_ids=REGISTERED_AGENTS, - ) - except ValueError as exc: - assert "distinct" in str(exc), exc - else: - raise AssertionError("same-agent independent review must fail") - try: - build_peer_review_event( - evidence_graph=graph, - rollout_events=decisions, - hypothesis_id="value_case", - reviewer_agent_id="unregistered-reviewer", - verdict="approve", - registered_agent_ids=REGISTERED_AGENTS, - ) - except ValueError as exc: - assert "not registered" in str(exc), exc - else: - raise AssertionError("unregistered peer review must fail") - return [self_review, *independent] - - -def assert_query_semantics( - graph: dict[str, Any], - decisions: list[dict[str, Any]], - review_events: list[dict[str, Any]], -) -> dict[str, Any]: - query = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, *review_events], - registered_agent_ids=REGISTERED_AGENTS, - include_history=True, - ) - by_id = {item["hypothesis_id"]: item for item in query["results"]} - assert by_id["value_case"]["finding_status"] == "confirmed", by_id - assert by_id["persistent_alpha"]["finding_status"] == "refuted", by_id - assert by_id["single_window_alpha"]["finding_status"] == "tentative", by_id - assert by_id["walk_forward_alpha"]["finding_status"] == "refuted", by_id - assert by_id["duplicate_route"]["finding_status"] is None, by_id - assert by_id["value_case"]["peer_review"]["independent"] is True, by_id - assert by_id["value_case"]["terminal_decision"]["outcome"] == "promoted", by_id - - self_review_only = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, review_events[0]], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert self_review_only["finding_status"] == "tentative", self_review_only - assert self_review_only["peer_review"]["state"] == "self_review_only", self_review_only - forged_self_review = { - **review_events[0], - "details": { - **review_events[0]["details"], - "independent": True, - }, - } - forged = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, forged_self_review], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert forged["peer_review"]["state"] == "self_review_only", forged - assert forged["finding_status"] == "tentative", forged - - downgraded_independence_claim = { - **review_events[1], - "details": { - **review_events[1]["details"], - "independent": False, - }, - } - downgraded = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, downgraded_independence_claim], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert downgraded["peer_review"]["state"] == "self_review_only", downgraded - assert downgraded["finding_status"] == "tentative", downgraded - - removed_reviewer = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, *review_events], - registered_agent_ids=[PRODUCER, PROMOTER], - hypothesis_id="value_case", - )["results"][0] - assert removed_reviewer["peer_review"]["state"] == "self_review_only", removed_reviewer - assert removed_reviewer["finding_status"] == "tentative", removed_reviewer - - conflict = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="value_case", - outcome="retired", - reason="duplicate", - decided_by="alternate-promoter", - recorded_at="2026-08-08T00:02:00Z", - ) - conflicted = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, conflict, *review_events], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert conflicted["decision_state"] == "conflict", conflicted - assert conflicted["finding_status"] == "tentative", conflicted - - review_reject = build_peer_review_event( - evidence_graph=graph, - rollout_events=decisions, - hypothesis_id="value_case", - reviewer_agent_id="second-independent-reviewer", - verdict="reject", - require_independent=True, - registered_agent_ids=REGISTERED_AGENTS, - recorded_at="2026-08-08T00:03:00Z", - ) - review_conflict = build_terminal_result_query( - evidence_graph=graph, - rollout_events=[*decisions, *review_events, review_reject], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert review_conflict["peer_review"]["state"] == "conflict", review_conflict - assert review_conflict["finding_status"] == "tentative", review_conflict - - stale_graph = build_research_evidence_graph_from_records( - goal_id=GOAL_ID, - hypotheses=[ - hypothesis( - "value_case", - text="The candidate satisfies the preregistered company-value gate.", - status="supported", - ) - ], - evidence_events=[ - evidence("value_case", split="dev", value=0.6, metric_status="improved"), - evidence( - "value_case", - split="holdout", - value=0.4, - metric_status="improved", - ), - evidence( - "value_case", - split="holdout", - value=0.45, - metric_status="improved", - attempt=2, - ), - ], - metric_name="research_quality", - metric_direction="maximize", - baseline_metric=BASELINE, - ) - stale = build_terminal_result_query( - evidence_graph=stale_graph, - rollout_events=[*decisions, *review_events], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert stale["decision_state"] == "stale", stale - assert stale["finding_status"] == "tentative", stale - - revised_decision = build_terminal_decision_event( - evidence_graph=stale_graph, - hypothesis_id="value_case", - outcome="retired", - reason="duplicate", - decided_by=PROMOTER, - recorded_at="2026-08-08T00:04:00Z", - ) - assert ( - revised_decision["causality"]["decision_id"] - != decisions[0]["causality"]["decision_id"] - ) - assert ( - revised_decision["details"]["hypothesis_evidence_revision"] - != decisions[0]["details"]["hypothesis_evidence_revision"] - ) - revised = build_terminal_result_query( - evidence_graph=stale_graph, - rollout_events=[*decisions, revised_decision, *review_events], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - include_history=True, - )["results"][0] - assert revised["decision_state"] == "current", revised - assert revised["terminal_decision"]["outcome"] == "retired", revised - assert len(revised["decision_history"]) == 2, revised - assert revised["finding_status"] == "tentative", revised - - unrelated_hypothesis = hypothesis( - "unrelated_route", - text="An unrelated hypothesis should not stale another terminal decision.", - status="active", - ) - unrelated_graph = build_research_evidence_graph_from_records( - goal_id=GOAL_ID, - hypotheses=[ - *fixture_graph()[1], - unrelated_hypothesis, - ], - evidence_events=fixture_graph()[2], - metric_name="research_quality", - metric_direction="maximize", - baseline_metric=BASELINE, - ) - still_current = build_terminal_result_query( - evidence_graph=unrelated_graph, - rollout_events=[*decisions, *review_events], - registered_agent_ids=REGISTERED_AGENTS, - hypothesis_id="value_case", - )["results"][0] - assert still_current["decision_state"] == "current", still_current - assert still_current["finding_status"] == "confirmed", still_current - return query - - -def assert_projection(query: dict[str, Any]) -> None: - events = build_terminal_result_explore_events(query=query) - repeated = build_terminal_result_explore_events(query=query) - assert events == repeated - assert [event["event_id"] for event in events] == [ - event["event_id"] for event in repeated - ] - projection = build_explore_result_projection(events, goal_id=GOAL_ID) - findings = { - item["finding"]: item["status"] for item in projection["findings"] - } - assert findings == { - "The candidate satisfies the preregistered company-value gate.": "confirmed", - "The candidate has persistent benchmark-adjusted alpha.": "refuted", - "One frozen window contains positive benchmark-adjusted alpha.": "tentative", - "The single-window alpha persists across rolling held-out windows.": "refuted", - }, findings - edges = [edge for edge in projection["edges"] if edge["edge_type"] == "leads_to"] - assert len(edges) == 1, edges - assert projection["counts"]["finding_count"] == 4, projection - - -def main() -> int: - graph, _, _ = fixture_graph() - decisions = terminal_events(graph) - review_events = reviews(graph, decisions) - query = assert_query_semantics(graph, decisions, review_events) - assert_projection(query) - print("auto-research-terminal-results-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-user-contract-entry-smoke.py b/examples/auto-research-user-contract-entry-smoke.py deleted file mode 100644 index dca64943f3..0000000000 --- a/examples/auto-research-user-contract-entry-smoke.py +++ /dev/null @@ -1,448 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the one-question auto-research user contract entrypoint.""" - -from __future__ import annotations - -import json -import subprocess -import sys -import tempfile -from argparse import Namespace -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.user_contract import ( # noqa: E402 - AUTO_RESEARCH_USER_CONTRACT_SCHEMA_VERSION, - build_auto_research_user_contract, - infer_auto_research_output_language, -) -from demo.auto_research.cli import ( # noqa: E402 - _default_auto_research_start_workspace, -) -from demo.auto_research.bootstrap_contract import ( # noqa: E402 - auto_research_start_command_text, - build_auto_research_contract_acceptance, -) -from demo.multi_agent.visible_launch_policy import ( # noqa: E402 - resolve_codex_trust_workspace, - resolve_visible_launch_policy, -) - - -QUESTION = "How should LoopX make visible multi-agent auto research useful?" - - -def assert_public_boundary(payload: dict[str, Any]) -> None: - text = json.dumps(payload, sort_keys=True) - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "lark" + "office", - "byte" + "dance", - "http" + "://", - "https" + "://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def assert_contract(payload: dict[str, Any]) -> None: - assert payload["ok"] is True, payload - assert payload["schema_version"] == AUTO_RESEARCH_USER_CONTRACT_SCHEMA_VERSION, payload - assert payload["mode"] == "user_contract", payload - assert payload["product_id"] == "auto-research", payload - assert payload["open_question"] == QUESTION, payload - assert payload["output_language"]["resolved"] in {"en", "zh"}, payload - - layering = payload["layering"] - assert layering["user_layer"] == "one open question", layering - assert layering["auto_research_layer"] == "fixed output contract only", layering - assert "multi-agent runner" in layering["kernel_layer"], layering - assert "pane-local tick" in layering["kernel_layer"], layering - - assert "minimal_a2a_recipe" not in payload, payload - assert "thin_surface" not in payload, payload - - command_contract = payload["command_contract"] - assert command_contract["canonical_invocation"] == 'loopx auto-research ""' - assert ( - command_contract["one_click_start_invocation"] - == 'loopx auto-research start "" --execute' - ) - assert command_contract["user_required_inputs"] == ["open_question"], command_contract - assert command_contract["user_optional_inputs"] == ["output_language", "preset"], command_contract - assert command_contract["max_action_plan_todos"] == 5, command_contract - assert command_contract["auto_research_required_outputs"] == [ - "research_brief", - "action_plan", - "evidence_refs", - "next_executable_step", - "gate", - ], command_contract - - one_click_start = payload["one_click_start"] - assert ( - one_click_start["command_template"] - == 'loopx auto-research start "" --execute' - ), one_click_start - assert one_click_start["command"].startswith("loopx auto-research start "), one_click_start - assert one_click_start["command"].endswith(" --execute"), one_click_start - assert ( - one_click_start["operator_takeover_command_template"] - == 'loopx auto-research start "" --execute --attach' - ), one_click_start - assert one_click_start["operator_takeover_command"].endswith(" --execute --attach") - assert "operator takeover first" in one_click_start["attach_semantics"], one_click_start - assert one_click_start["preview_command"].startswith("loopx auto-research start "), one_click_start - assert one_click_start["starts"] == "visible_codex_tui_lanes", one_click_start - assert one_click_start["uses_generic_kernel"] is True, one_click_start - assert one_click_start["coordination_model"] == "decentralized_state_a2a", one_click_start - assert "agent_ids" in one_click_start["user_does_not_choose"], one_click_start - - brief = payload["research_brief"] - assert set(brief) == {"read", "not_read", "claim_boundary"}, brief - assert brief["read"] == [], brief - assert "source code" in brief["not_read"], brief - assert "evidence ref" in brief["claim_boundary"], brief - - action_plan = payload["action_plan"] - assert 1 <= len(action_plan) <= 5, action_plan - assert [item["priority"] for item in action_plan][:2] == ["P0", "P0"], action_plan - assert all(item["owner_layer"] != "user_layer" for item in action_plan), action_plan - - evidence = payload["evidence_refs"] - assert set(evidence) == {"code", "docs", "benchmarks", "issues", "pull_requests"}, evidence - assert all(value == [] for value in evidence.values()), evidence - - step = payload["next_executable_step"] - assert step["can_run_automatically"] is True, step - assert "read-only evidence discovery" in step["step"], step - - gate = payload["gate"] - assert gate["user_judgment_needed"], gate - assert "public/read-only discovery" in gate["default_without_user_gate"], gate - - boundary = payload["public_boundary"] - assert boundary == { - "raw_logs": False, - "private_material": False, - "credentials": False, - "destructive_git": False, - "production_actions": False, - }, boundary - assert_public_boundary(payload) - - -def run_cli(temp_dir: Path, *args: str) -> str: - result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(temp_dir / "registry.json"), - *args, - ], - cwd=REPO_ROOT, - text=True, - capture_output=True, - check=True, - ) - return result.stdout - - -def assert_start_wake_contract() -> None: - cli_source = (REPO_ROOT / "demo/auto_research/cli.py").read_text( - encoding="utf-8" - ) - assert "wake_visible_after_launch=bool(args.wake_visible_after_launch)" not in cli_source - assert "def _start_wake_visible_after_launch" not in cli_source - assert "def _start_attach_visible" not in cli_source - assert "def _start_codex_trust_workspace" not in cli_source - assert "wake_visible_multi_agent_panes" not in cli_source - assert "resolve_visible_launch_policy" in cli_source - assert "make_visible_launcher_callback" in cli_source - assert "--auto-wake" in cli_source - assert "auto_wake=bool(visible_policy.launch_visible and args.auto_wake)" in cli_source - start_source = cli_source.split('elif args.auto_research_command == "start":', 1)[1].split( - 'elif args.auto_research_command == "demo-supervisor":', - 1, - )[0] - assert "if args.codex_trust_workspace is None" not in start_source - demo_e2e_source = cli_source.split('elif args.auto_research_command == "demo-e2e":', 1)[1] - assert "default=True" in cli_source.split('"--auto-wake"', 1)[1].split( - '"--auto-wake-interval-seconds"', - 1, - )[0] - assert "auto_wake=bool(visible_policy.launch_visible and args.auto_wake)" in demo_e2e_source - assert "auto_wake_interval_seconds=args.auto_wake_interval_seconds" in demo_e2e_source - - def start_policy(args: Namespace): - return resolve_visible_launch_policy( - args, - launch_visible=bool(args.execute and not args.headless), - default_wake_allowed=False, - default_attach_allowed=bool(args.execute and not args.headless), - ) - - def trust(args: Namespace) -> bool: - return resolve_codex_trust_workspace( - args, - launch_visible=bool(args.execute and not args.headless), - default=True, - ) - - default_visible = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=None, - attach=False, - no_attach=False, - codex_trust_workspace=None, - auto_wake=True, - ) - default_policy = start_policy(default_visible) - assert default_policy.wake_visible_after_launch is False - assert trust(default_visible) is True - assert default_policy.attach is True - - attach_takeover = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=None, - attach=True, - no_attach=False, - codex_trust_workspace=None, - auto_wake=True, - ) - takeover_policy = start_policy(attach_takeover) - assert takeover_policy.wake_visible_after_launch is False - assert trust(attach_takeover) is True - assert takeover_policy.attach is True - - explicit_wake = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=True, - attach=False, - no_attach=False, - codex_trust_workspace=None, - auto_wake=True, - ) - explicit_wake_policy = start_policy(explicit_wake) - assert explicit_wake_policy.wake_visible_after_launch is True - assert trust(explicit_wake) is True - assert explicit_wake_policy.attach is False - - manual_takeover = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=False, - attach=False, - no_attach=False, - codex_trust_workspace=None, - auto_wake=True, - ) - manual_policy = start_policy(manual_takeover) - assert manual_policy.wake_visible_after_launch is False - assert trust(manual_takeover) is True - assert manual_policy.attach is True - - background_manual = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=False, - attach=False, - no_attach=True, - codex_trust_workspace=None, - auto_wake=False, - ) - background_policy = start_policy(background_manual) - assert background_policy.wake_visible_after_launch is False - assert trust(background_manual) is True - assert background_policy.attach is False - - headless = Namespace( - execute=True, - headless=True, - wake_visible_after_launch=True, - attach=False, - no_attach=False, - codex_trust_workspace=None, - auto_wake=True, - ) - headless_policy = start_policy(headless) - assert headless_policy.wake_visible_after_launch is False - assert trust(headless) is False - assert headless_policy.attach is False - - explicit_no_trust = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=None, - attach=False, - no_attach=False, - codex_trust_workspace=False, - auto_wake=True, - ) - assert trust(explicit_no_trust) is False - - explicit_trust = Namespace( - execute=True, - headless=False, - wake_visible_after_launch=None, - attach=False, - no_attach=False, - codex_trust_workspace=True, - auto_wake=True, - ) - assert trust(explicit_trust) is True - - -def main() -> None: - payload = build_auto_research_user_contract(QUESTION) - assert_contract(payload) - assert infer_auto_research_output_language("如何评估 multi-agent auto research 是否有收益?") == "zh" - zh_payload = build_auto_research_user_contract( - "如何评估 multi-agent auto research 是否有收益?" - ) - assert zh_payload["output_language"]["resolved"] == "zh", zh_payload - assert " --language zh --execute" in zh_payload["one_click_start"]["command"], zh_payload - assert ( - auto_research_start_command_text( - cli_bin="loopx", - objective="如何评估 multi-agent auto research 是否有收益?", - execute=True, - output_language="zh", - ) - == "loopx auto-research start '如何评估 multi-agent auto research 是否有收益?' --language zh --execute" - ) - assert build_auto_research_contract_acceptance(zh_payload)["accepted"] is True - assert_start_wake_contract() - default_workspace = Path( - _default_auto_research_start_workspace("loopx-auto-research-demo-smoke") - ) - assert default_workspace.parts[-3:] == ( - "loopx-auto-research", - "loopx-auto-research-demo-smoke", - "visible-workspace", - ), default_workspace - assert default_workspace.is_absolute(), default_workspace - assert default_workspace == ( - Path.home() / "loopx-auto-research" / "loopx-auto-research-demo-smoke" / "visible-workspace" - ), default_workspace - - with tempfile.TemporaryDirectory() as raw_temp_dir: - temp_dir = Path(raw_temp_dir) - shorthand_json = run_cli(temp_dir, "--format", "json", "auto-research", QUESTION) - assert_contract(json.loads(shorthand_json)) - - group_format_json = run_cli(temp_dir, "auto-research", "--format", "json", QUESTION) - assert_contract(json.loads(group_format_json)) - - explicit_json = run_cli( - temp_dir, - "--format", - "json", - "auto-research", - "contract", - QUESTION, - "--max-todos", - "3", - ) - explicit_payload = json.loads(explicit_json) - assert len(explicit_payload["action_plan"]) == 3, explicit_payload - - start_json = run_cli( - temp_dir, - "--format", - "json", - "auto-research", - "start", - QUESTION, - ) - start_payload = json.loads(start_json) - assert start_payload["ok"] is True, start_payload - assert start_payload["mode"] == "dry_run", start_payload - assert start_payload["execution_kind"] == "worker_loop_preview", start_payload - assert start_payload["contract_acceptance"]["accepted"] is True, start_payload - assert_contract(start_payload["user_contract"]) - assert ( - start_payload["commands"]["one_question_contract"] - == f"loopx auto-research '{QUESTION}'" - ), start_payload - assert ( - start_payload["commands"]["one_question_start"] - == f"loopx auto-research start '{QUESTION}' --execute" - ), start_payload - assert_public_boundary(start_payload) - - start_headless_json = run_cli( - temp_dir, - "--format", - "json", - "auto-research", - "start", - QUESTION, - "--execute", - "--headless", - ) - start_headless_payload = json.loads(start_headless_json) - assert start_headless_payload["ok"] is True, start_headless_payload - assert start_headless_payload["execution_kind"] == "loopx_worker_loop", start_headless_payload - assert start_headless_payload["contract_acceptance"]["accepted"] is True, start_headless_payload - worker_loop = start_headless_payload["worker_loop"] - assert worker_loop["turn_count"] >= 4, start_headless_payload - assert worker_loop["executed_turn_count"] == 0, start_headless_payload - assert worker_loop["completed_turn_count"] == 0, start_headless_payload - assert any( - turn.get("mode") == "manual_research_required" for turn in worker_loop["turns"] - ), start_headless_payload - assert start_headless_payload["tonight_experience"]["positive_result"] is False - assert start_headless_payload["tonight_experience"]["ready"] is False - assert start_headless_payload["tonight_experience"]["positive_result_basis"] == ( - "requires_visible_lane_authored_evidence" - ) - assert start_headless_payload["tonight_experience"]["dev_metric"] is None - assert start_headless_payload["tonight_experience"]["holdout_metric"] is None - assert "auto-research start" in start_headless_payload["commands"]["one_question_start"], start_headless_payload - assert_public_boundary(start_headless_payload) - - markdown = run_cli(temp_dir, "auto-research", QUESTION) - assert "# LoopX Auto Research" in markdown, markdown - for required in [ - "## Research Brief", - "## Action Plan", - "## Evidence Refs", - "## One-Click Start", - "## Next Executable Step", - "## Gate", - ]: - assert required in markdown, markdown - - supervisor_json = run_cli( - temp_dir, - "--format", - "json", - "auto-research", - "demo-supervisor", - "--goal-id", - "loopx-auto-research-contract-entry-smoke", - ) - supervisor_payload = json.loads(supervisor_json) - assert supervisor_payload["schema_version"] == "auto_research_demo_supervisor_plan_v0" - - print("auto-research-user-contract-entry-smoke ok") - - -if __name__ == "__main__": - main() diff --git a/examples/auto-research-visible-launch-policy-smoke.py b/examples/auto-research-visible-launch-policy-smoke.py deleted file mode 100644 index 5046c94d33..0000000000 --- a/examples/auto-research-visible-launch-policy-smoke.py +++ /dev/null @@ -1,239 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parents[1] -if str(ROOT) not in sys.path: - sys.path.insert(0, str(ROOT)) - -from demo.multi_agent.visible_launch_policy import ( - make_visible_wake_callback, - resolve_codex_trust_workspace, - resolve_visible_launch_policy, -) -from demo.auto_research.demo_e2e import _load_visible_wake_into_payload - - -AUTO_RESEARCH_CLI = ROOT / "demo" / "auto_research" / "cli.py" - - -def ns(**values: object) -> argparse.Namespace: - defaults = { - "attach": False, - "no_attach": False, - "wake_visible_after_launch": None, - "codex_trust_workspace": None, - } - defaults.update(values) - return argparse.Namespace(**defaults) - - -def require(condition: bool, message: str) -> None: - if not condition: - raise AssertionError(message) - - -def assert_start_policy() -> None: - policy = resolve_visible_launch_policy( - ns(), - launch_visible=True, - default_wake_allowed=False, - default_attach_allowed=True, - ) - require(policy.launch_visible is True, "start should launch visible panes") - require(policy.wake_visible_after_launch is False, "start first tick belongs to the visible TUI") - require(policy.attach is True, "start should attach for operator-visible research") - require( - resolve_codex_trust_workspace(ns(), launch_visible=True, default=True) is True, - "start default workspace should be trusted to avoid first-screen trust prompt", - ) - - takeover = resolve_visible_launch_policy( - ns(attach=True), - launch_visible=True, - default_wake_allowed=False, - default_attach_allowed=True, - ) - require(takeover.wake_visible_after_launch is False, "operator takeover should not add a background wake") - require(takeover.attach is True, "explicit attach should attach") - - -def assert_demo_policy_preserves_headless_visible_mix() -> None: - policy = resolve_visible_launch_policy( - ns(), - launch_visible=True, - default_wake_allowed=False, - default_attach_allowed=False, - ) - require(policy.launch_visible is True, "explicit visible launch should still launch") - require(policy.wake_visible_after_launch is False, "headless visible mix should not default wake") - require(policy.attach is False, "headless visible mix should not default attach") - - require( - resolve_codex_trust_workspace(ns(), launch_visible=True, default=False) is False, - "user-provided demo workspace should not be trusted by default", - ) - require( - resolve_codex_trust_workspace( - ns(codex_trust_workspace=True), - launch_visible=True, - default=False, - ) - is True, - "explicit trust flag should override the workspace default", - ) - - -def assert_conflicts() -> None: - try: - resolve_visible_launch_policy( - ns(attach=True, no_attach=True), - launch_visible=True, - default_wake_allowed=True, - default_attach_allowed=True, - ) - except ValueError as exc: - require("--attach cannot be combined with --no-attach" in str(exc), str(exc)) - else: - raise AssertionError("expected attach/no-attach conflict") - - try: - resolve_visible_launch_policy( - ns(attach=True, wake_visible_after_launch=True), - launch_visible=True, - default_wake_allowed=True, - default_attach_allowed=True, - ) - except ValueError as exc: - require("--wake-visible-after-launch" in str(exc), str(exc)) - else: - raise AssertionError("expected attach/wake conflict") - - -def assert_auto_research_consumes_generic_policy() -> None: - source = AUTO_RESEARCH_CLI.read_text(encoding="utf-8") - for removed in ( - "def _start_wake_visible_after_launch", - "def _start_attach_visible", - "def _start_codex_trust_workspace", - "wake_visible_multi_agent_panes", - ): - require(removed not in source, f"auto-research CLI still owns generic visible policy: {removed}") - for marker in ( - "resolve_visible_launch_policy", - "resolve_codex_trust_workspace", - "make_visible_wake_callback", - "--auto-wake", - ): - require(marker in source, f"auto-research CLI missing generic policy helper: {marker}") - wake_callback = make_visible_wake_callback(tmux_bin="tmux") - require(callable(wake_callback), "wake helper should return a callback") - - -def assert_visible_wake_loader_requires_prompt_delivery() -> None: - delivered_payload: dict[str, object] = {"visible_worker_proof": {}} - _load_visible_wake_into_payload( - payload=delivered_payload, - wake={ - "schema_version": "multi_agent_pane_a2a_wakeup_v0", - "mode": "execute", - "session_name": "fixture", - "target_lanes": ["research-executor"], - "coordination_model": "decentralized_state_a2a", - "wakeup_model": "fixed_prompt_broadcast", - "workflow_driver": False, - "broadcaster_reads_frontier": False, - "broadcaster_selects_todo": False, - "pane_decision_owner": "codex_tui_agent_via_loopx_state", - "pane_input_ready_verified": True, - "ready_lanes": ["research-executor"], - "not_ready_lanes": [], - "prompt_submit_checks": [{"target": "%2", "retry_count": 0}], - "prompt_delivery": "tmux_paste_buffer_after_codex_tui_first_turn_ready", - "driver_contract": { - "schema_version": "multi_agent_decentralized_a2a_driver_contract_v1", - "owner_layer": "generic_multi_agent_kernel", - }, - }, - ) - delivered_proof = delivered_payload["visible_worker_proof"] - require( - delivered_proof["cadence_wake_verified"] is True, - "delivered wake should verify cadence wake", - ) - require( - delivered_proof["cadence_wake_prompt_delivered"] is True, - "delivered wake should expose prompt delivery", - ) - - pending_payload: dict[str, object] = {"visible_worker_proof": {}} - _load_visible_wake_into_payload( - payload=pending_payload, - wake={ - "schema_version": "multi_agent_pane_a2a_wakeup_v0", - "mode": "execute", - "session_name": "fixture", - "target_lanes": ["research-executor"], - "coordination_model": "decentralized_state_a2a", - "wakeup_model": "fixed_prompt_broadcast", - "workflow_driver": False, - "broadcaster_reads_frontier": False, - "broadcaster_selects_todo": False, - "pane_decision_owner": "codex_tui_agent_via_loopx_state", - "pane_input_ready_verified": False, - "pane_input_ready_checks": [ - { - "lane": "research-executor", - "ready": False, - "not_ready_reason": "codex_tui_busy_or_not_ready", - } - ], - "ready_lanes": [], - "not_ready_lanes": ["research-executor"], - "prompt_submit_checks": [], - "prompt_delivery": "skipped_no_input_ready_panes", - "driver_contract": { - "schema_version": "multi_agent_decentralized_a2a_driver_contract_v1", - "owner_layer": "generic_multi_agent_kernel", - }, - }, - ) - pending_proof = pending_payload["visible_worker_proof"] - pending_wake = pending_payload["visible_wake"] - require( - pending_proof["cadence_wake_verified"] is False, - "busy/queued pane must not count as cadence wake verified", - ) - require( - pending_proof["cadence_wake_prompt_delivered"] is False, - "busy/queued pane must expose no prompt delivery", - ) - require( - pending_proof["cadence_wake_pending_reason"] == "pane_not_ready", - "busy/queued pane should expose a pending reason", - ) - require( - pending_proof["cadence_wake_not_ready_reasons"] == ["codex_tui_busy_or_not_ready"], - "busy/queued pane should expose public-safe not-ready reasons", - ) - require( - pending_wake["prompt_delivered"] is False - and pending_wake["not_ready_reasons"] == ["codex_tui_busy_or_not_ready"], - "visible wake payload should preserve public-safe pending diagnostics", - ) - - -def main() -> None: - assert_start_policy() - assert_demo_policy_preserves_headless_visible_mix() - assert_conflicts() - assert_auto_research_consumes_generic_policy() - assert_visible_wake_loader_requires_prompt_delivery() - print("auto-research-visible-launch-policy-smoke ok") - - -if __name__ == "__main__": - main() diff --git a/examples/auto-research-visible-worker-hook-smoke.py b/examples/auto-research-visible-worker-hook-smoke.py deleted file mode 100644 index a187eb0023..0000000000 --- a/examples/auto-research-visible-worker-hook-smoke.py +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the explicit visible auto-research worker-turn hook path.""" - -from __future__ import annotations - -import json -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_e2e import run_auto_research_demo_e2e # noqa: E402 -from demo.auto_research.demo_supervisor import ( # noqa: E402 - build_auto_research_demo_supervisor_plan, - build_visible_worker_turn_command, -) -from demo.auto_research.user_contract import ( # noqa: E402 - build_auto_research_preset_context, -) -from demo.visible_multi_agent_launcher import validate_worker_command # noqa: E402 - - -GOAL_ID = "loopx-auto-research-visible-worker-hook-smoke" -AGENT_ID = "auto-research-operator" -QUESTION = "如何提升 KNN 精确近邻检索速度？" - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - "raw transcript", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def assert_worker_turn_command_shape() -> None: - command = build_visible_worker_turn_command( - goal_id=GOAL_ID, - agent_id=AGENT_ID, - lane_count=1, - ) - assert validate_worker_command(command, field="worker_turn_command") == command - assert command.startswith("loopx --format json auto-research worker-turn") - assert f"--goal-id {GOAL_ID}" in command - assert f"--agent-id {AGENT_ID}" in command - assert "--lane-count 1" in command - assert "--visible-lanes-accepted" in command - assert "--complete-selected-todo" in command - assert "--execute" in command - assert "$" not in command - assert_public_safe(command) - - -def assert_supervisor_can_configure_worker_turn() -> None: - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - open_question=QUESTION, - preset_context=build_auto_research_preset_context("knn-demo"), - output_language="zh", - configure_visible_worker_turn=True, - ) - assert supervisor["auto_research"]["visible_worker_turn_configured"] is True - assert supervisor["auto_research"]["worker_turn_owner"] == "generic_multi_agent_kernel" - assert supervisor["runner_contract"]["decentralized_a2a_driver"]["broadcaster"][ - "runs_worker_turn" - ] is False - for lane in supervisor["lanes"]: - assert lane["pane_local_a2a"]["worker_turn_configured"] is True, lane - assert lane["pane_local_a2a"]["worker_loop_configured"] is False, lane - assert lane["pane_local_a2a"]["status_check_only"] is True, lane - assert lane["pane_local_a2a"]["counts_as_research_round"] is False, lane - assert "LOOPX_PANE_WORKER_TURN" in lane["visible_launch_command"], lane - assert "export LOOPX_PANE_WORKER_TURN='" in lane["visible_launch_command"], lane - assert "auto-research worker-turn" in lane["visible_launch_command"], lane - assert "--visible-lanes-accepted" in lane["visible_launch_command"], lane - assert "--complete-selected-todo" in lane["visible_launch_command"], lane - profile = lane["role_profile"] - assert profile["visible_worker_turn_configured"] is True, profile - assert_public_safe(supervisor) - - -def assert_demo_e2e_exposes_validation_path() -> None: - captured: dict[str, object] = {} - - def fake_visible_launcher( - supervisor: dict[str, object], - _registry_path: Path, - _runtime_root: str | None, - _demo_root: Path, - ) -> dict[str, object]: - captured["supervisor"] = supervisor - return { - "mode": "execute", - "launch_result": { - "session_name": "loopx-ar-visible-worker-hook-smoke", - "started_lanes": [ - str(lane.get("lane_id") or "") - for lane in supervisor.get("lanes", []) - if isinstance(lane, dict) - ], - "visible_acceptance": {"accepted": True}, - }, - "boundary": { - "starts_visible_processes": False, - "writes_loopx_state": False, - "public_safe_redaction": True, - }, - } - - with tempfile.TemporaryDirectory(prefix="loopx-ar-visible-worker-hook.") as temp_dir: - temp = Path(temp_dir) - registry = temp / "registry.json" - runtime_root = temp / "runtime" - registry.write_text( - json.dumps({"common_runtime_root": str(runtime_root), "goals": []}), - encoding="utf-8", - ) - payload = run_auto_research_demo_e2e( - agent_id=AGENT_ID, - goal_id=GOAL_ID, - tracking_goal_id=None, - preset_id="knn-demo", - objective=QUESTION, - output_dir="auto_research_lightweight_kernel", - execute=True, - run_worker_loop=False, - worker_loop_rounds=1, - configure_visible_worker_turn=True, - launch_visible=True, - keep_workspace=False, - registry_path=registry, - runtime_root_arg=str(runtime_root), - session_name="loopx-ar-visible-worker-hook-smoke", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - output_language="zh", - live_evidence_path=None, - append_evidence=lambda _path: {"ok": True, "appended_count": 0}, - visible_launcher=fake_visible_launcher, - visible_wake=None, - visible_live_evidence_wait_seconds=0.0, - ) - validation = payload["visible_worker_turn_validation"] - assert validation["configured"] is True, validation - assert validation["counts_as_research_evidence_without_role_output"] is False, validation - assert validation["manual_research_required_is_blocker_not_progress"] is True, validation - assert payload["visible_worker_proof"]["visible_lanes_accepted"] is True, payload - assert payload["visible_readiness"]["ready"] is False, payload["visible_readiness"] - assert "lane_authored_evidence_loaded" in payload["visible_readiness"]["missing_requirements"] - captured_supervisor = captured["supervisor"] - for lane in captured_supervisor["lanes"]: - assert lane["pane_local_a2a"]["worker_turn_configured"] is True, lane - assert_public_safe(payload) - - -def main() -> None: - assert_worker_turn_command_shape() - assert_supervisor_can_configure_worker_turn() - assert_demo_e2e_exposes_validation_path() - print("auto-research-visible-worker-hook-smoke ok") - - -if __name__ == "__main__": - main() diff --git a/examples/auto-research-worker-loop-smoke.py b/examples/auto-research-worker-loop-smoke.py deleted file mode 100644 index 64f3c7b955..0000000000 --- a/examples/auto-research-worker-loop-smoke.py +++ /dev/null @@ -1,173 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test that auto-research worker-loop is not a fake metric generator.""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_e2e import _seed_visible_demo_control_plane # noqa: E402 -from demo.auto_research.demo_supervisor import ( # noqa: E402 - build_auto_research_demo_supervisor_plan, -) -from loopx.state_projection import state_projection_gap_warning # noqa: E402 - - -GOAL_ID = "loopx-auto-research-demo" -AGENT_IDS = [ - "research-curator", - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", -] -LANES = [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", -] - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http" + "://", - "https" + "://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def main() -> int: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - agent_specs=LANES, - session_name="loopx-auto-research-worker-loop-smoke", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - ) - visible_control, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, - goal_id=GOAL_ID, - objective="Verify worker-loop cannot manufacture auto-research evidence.", - supervisor=supervisor, - ) - state_file = ( - temp - / "visible-control-plane" - / ".loopx" - / "goals" - / GOAL_ID - / "ACTIVE_GOAL_STATE.md" - ) - state_text = state_file.read_text(encoding="utf-8") - assert "Run `loopx check` against the project registry" not in state_text - assert visible_control["registered_agent_count"] == len(AGENT_IDS) - assert {todo["agent_id"] for todo in visible_control["seeded_todos"]} == set(AGENT_IDS) - assert len({todo["todo_id"] for todo in visible_control["seeded_todos"]}) == len(AGENT_IDS) - assert all(todo["todo_id"] in state_text for todo in visible_control["seeded_todos"]) - route_only_state = ( - "## Agent Todo\n\n" - "- [x] Complete role-frontier auto-research work.\n\n" - "## Next Action\n\n" - "- Goal-level route delegates to role frontier; panes own execution.\n" - ) - assert state_projection_gap_warning(route_only_state) is None - - workspace = temp / "shared-research-workspace" - workspace.mkdir() - env = os.environ.copy() - env["PYTHONDONTWRITEBYTECODE"] = "1" - env["PYTHONPATH"] = f"{REPO_ROOT}{os.pathsep}{env.get('PYTHONPATH', '')}" - args = [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "--format", - "json", - "auto-research", - "worker-loop", - "--goal-id", - GOAL_ID, - "--lane-count", - str(len(AGENT_IDS)), - "--max-rounds", - "1", - "--visible-lanes-accepted", - "--complete-selected-todo", - "--execute", - ] - for agent_id in AGENT_IDS: - args.extend(["--agent-id", agent_id]) - result = subprocess.run( - args, - cwd=workspace, - env=env, - check=False, - capture_output=True, - text=True, - ) - if result.returncode != 0: - raise AssertionError( - f"worker-loop failed rc={result.returncode}\nstdout={result.stdout}\nstderr={result.stderr}" - ) - payload = json.loads(result.stdout) - assert payload["ok"] is True, payload - assert payload["schema_version"] == "auto_research_worker_loop_v0", payload - assert payload["mode"] == "execute", payload - assert payload["max_rounds"] == 1, payload - assert payload["turn_count"] == 4, payload - assert payload["stop_reason"] == "no_executed_turns", payload - assert payload["selected_actions"] == ["write_research_contract"], payload - - manual_turns = [ - turn for turn in payload["turns"] if turn.get("mode") == "manual_research_required" - ] - assert [turn["agent_id"] for turn in manual_turns] == ["research-curator"], payload - assert [turn["selected_action"] for turn in manual_turns] == [ - "write_research_contract" - ], payload - no_action_turns = [turn for turn in payload["turns"] if turn.get("mode") == "no_action"] - assert [turn["agent_id"] for turn in no_action_turns] == [ - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - ], payload - assert no_action_turns[0]["selected_action"] is None, payload - assert all(turn["executed"] is False for turn in manual_turns), payload - assert all(turn.get("completion_status") is None for turn in manual_turns), payload - assert all(turn.get("dev_metric") is None for turn in payload["turns"]), payload - assert all(turn.get("holdout_metric") is None for turn in payload["turns"]), payload - assert all("demo_iteration" not in turn for turn in payload["turns"]), payload - assert payload["executed_turn_count"] == 0, payload - assert payload["completed_turn_count"] == 0, payload - assert_public_safe(payload) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto-research-worker-turn-smoke.py b/examples/auto-research-worker-turn-smoke.py deleted file mode 100644 index 650631c766..0000000000 --- a/examples/auto-research-worker-turn-smoke.py +++ /dev/null @@ -1,1559 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test that auto-research worker-turn does not fake research output.""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.demo_e2e import _seed_visible_demo_control_plane # noqa: E402 -from demo.auto_research.demo_supervisor import ( # noqa: E402 - build_auto_research_demo_supervisor_plan, -) -from demo.auto_research.evidence_packet import ( # noqa: E402 - build_auto_research_evidence_packet, -) -from demo.auto_research.preset import ( # noqa: E402 - auto_research_successor_specs_for_action, -) -from demo.auto_research.rollout_append import ( # noqa: E402 - append_auto_research_rollout_events, -) -from demo.multi_agent.role_successor import ( # noqa: E402 - apply_role_successor_todos, -) -from loopx.todos import add_goal_todo, complete_goal_todo # noqa: E402 -from loopx.control_plane.coordination.local_authority import ( # noqa: E402 - LocalCoordinationAuthorityUnavailable, - read_canonical_todos_if_promoted, -) -from loopx.control_plane.testing.canary_harness import import_fixture_authority # noqa: E402 -from loopx.control_plane.work_items.task_lease import ( # noqa: E402 - TaskLeaseError, - acquire_task_lease, - inspect_task_lease, - release_task_lease, -) -from demo.auto_research.worker_runtime import run_auto_research_worker_turn # noqa: E402 - -from examples.auto_research_lightweight_fixture import ( # noqa: E402 - AGENT_ID as EVIDENCE_AGENT_ID, - GROUNDING_REF, - HYPOTHESIS_ID, - HYPOTHESIS_TEXT, - MECHANISM_FAMILY, - TODO_ID as EVIDENCE_TODO_ID, - eval_result, - research_contract, -) - - -GOAL_ID = "loopx-auto-research-demo" -CURATOR_AGENT_ID = "research-curator" -HYPOTHESIS_AGENT_ID = "hypothesis-proposer" -EXECUTOR_AGENT_ID = "research-executor" -EVALUATOR_AGENT_ID = "evaluator-promoter" -LANES = [ - "research-curator:research-curator:research_curator", - "hypothesis-proposer:hypothesis-proposer:hypothesis_proposer", - "research-executor:research-executor:research_executor", - "evaluator-promoter:evaluator-promoter:evaluator_promoter", -] - - -def assert_public_safe(payload: Any) -> None: - text = json.dumps(payload, sort_keys=True) if not isinstance(payload, str) else payload - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http" + "://", - "https" + "://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def run_worker_turn( - *, - registry: Path, - runtime_root: str | None, - workspace: Path, - agent_id: str, - execute: bool, - complete: bool = False, -) -> dict[str, Any]: - env = os.environ.copy() - env["PYTHONDONTWRITEBYTECODE"] = "1" - env["PYTHONPATH"] = f"{REPO_ROOT}{os.pathsep}{env.get('PYTHONPATH', '')}" - args = [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "--format", - "json", - "auto-research", - "worker-turn", - "--goal-id", - GOAL_ID, - "--agent-id", - agent_id, - "--lane-count", - "4", - "--visible-lanes-accepted", - ] - if execute: - args.append("--execute") - if complete: - args.append("--complete-selected-todo") - result = subprocess.run( - args, - cwd=workspace, - env=env, - check=False, - capture_output=True, - text=True, - ) - if result.returncode != 0: - raise AssertionError( - f"worker-turn failed rc={result.returncode}\nstdout={result.stdout}\nstderr={result.stderr}" - ) - return json.loads(result.stdout) - - -def assert_manual_research_required(payload: dict[str, Any], *, action: str) -> None: - assert payload["schema_version"] == "auto_research_worker_turn_v0", payload - assert payload["mode"] == "manual_research_required", payload - assert payload["selected_action"] == action, payload - assert payload["executed"] is False, payload - assert payload["manual_research_required"] is True, payload - assert payload["completion"]["executed"] is False, payload - assert payload["public_boundary"]["fake_metrics_recorded"] is False, payload - assert "dev_metric" not in payload, payload - assert "holdout_metric" not in payload, payload - assert "live_evidence" not in payload, payload - assert_public_safe(payload) - - -def assert_no_action(payload: dict[str, Any], *, agent_id: str) -> None: - assert payload["schema_version"] == "auto_research_worker_turn_v0", payload - assert payload["mode"] == "no_action", payload - assert payload["agent_id"] == agent_id, payload - assert payload["executed"] is False, payload - assert payload["frontier"]["frontier"]["selected"] is None, payload - assert payload["frontier"]["frontier"]["runnable_count"] == 0, payload - assert_public_safe(payload) - - -def write_summary_state(path: Path) -> None: - path.write_text( - "\n".join( - [ - "# Auto Research Worker Summary Smoke", - "", - "## User Todo / Owner Review Reading Queue", - "", - "## Agent Todo", - "", - ] - ), - encoding="utf-8", - ) - - -def write_registry( - path: Path, *, project: Path, state_file: Path, runtime_root: Path, provider: str, -) -> None: - path.write_text( - json.dumps( - { - "common_runtime_root": str(runtime_root), - "goals": [ - { - "id": GOAL_ID, - "domain": "auto_research_smoke", - "status": "active", - "repo": str(project), - "state_file": str(state_file), - "adapter": { - "kind": "auto_research_demo_local_queue", - "status": "connected", - }, - "coordination": { - "agent_model": "peer_v1", - "registered_agents": [ - CURATOR_AGENT_ID, - HYPOTHESIS_AGENT_ID, - EXECUTOR_AGENT_ID, - EVALUATOR_AGENT_ID, - ] - }, - } - ], - }, - indent=2, - sort_keys=True, - ) - + "\n", - encoding="utf-8", - ) - import_fixture_authority(path, GOAL_ID, provider=provider) - - -def append_real_fixture_evidence(*, registry: Path, runtime_root: Path, temp: Path) -> None: - packet = build_auto_research_evidence_packet( - contract=research_contract(goal_id=GOAL_ID), - eval_results=[eval_result("dev"), eval_result("holdout")], - hypothesis_id=HYPOTHESIS_ID, - todo_id=EVIDENCE_TODO_ID, - agent_id=EVIDENCE_AGENT_ID, - claimed_by=EVIDENCE_AGENT_ID, - mechanism_family=MECHANISM_FAMILY, - hypothesis=HYPOTHESIS_TEXT, - grounding_refs=[GROUNDING_REF], - branch_ref="codex/auto-research-worker-turn-smoke", - ) - packet_path = temp / "real-evidence-packet.public.json" - packet_path.write_text(json.dumps(packet, indent=2, sort_keys=True) + "\n", encoding="utf-8") - appended = append_auto_research_rollout_events( - packet_path=str(packet_path), - registry_path=registry, - runtime_root_arg=str(runtime_root), - dry_run=False, - ) - assert appended["appended_count"] == 3, appended - assert_public_safe(appended) - - -def append_fixture_evidence( - *, - registry: Path, - runtime_root: Path, - temp: Path, - hypothesis_id: str, - todo_id: str, - dev_metric: float, - holdout_metric: float, -) -> None: - packet = build_auto_research_evidence_packet( - contract=research_contract(goal_id=GOAL_ID), - eval_results=[ - eval_result("dev", value=dev_metric), - eval_result("holdout", value=holdout_metric), - ], - hypothesis_id=hypothesis_id, - todo_id=todo_id, - agent_id=EVIDENCE_AGENT_ID, - claimed_by=EVIDENCE_AGENT_ID, - mechanism_family=MECHANISM_FAMILY, - hypothesis=f"{HYPOTHESIS_TEXT} Variant {hypothesis_id}.", - grounding_refs=[GROUNDING_REF], - branch_ref="codex/auto-research-worker-turn-smoke", - ) - packet_path = temp / f"{hypothesis_id}.public.json" - packet_path.write_text(json.dumps(packet, indent=2, sort_keys=True) + "\n", encoding="utf-8") - appended = append_auto_research_rollout_events( - packet_path=str(packet_path), - registry_path=registry, - runtime_root_arg=str(runtime_root), - dry_run=False, - ) - assert appended["appended_count"] == 3, appended - assert_public_safe(appended) - - -def append_negative_fixture_evidence( - *, - registry: Path, - runtime_root: Path, - temp: Path, - hypothesis_id: str, - todo_id: str, - failure_kind: str | None = None, - measurement_scope: str | None = None, - remediation_attempt: bool = False, - eval_status: str = "scored", - primary_metric_status: str = "regressed", -) -> None: - result = eval_result("dev", value=0.75) - result["eval_status"] = eval_status - result["primary_metric_status"] = primary_metric_status - if failure_kind: - result["failure_kind"] = failure_kind - if measurement_scope: - result["measurement_scope"] = measurement_scope - if remediation_attempt: - result["remediation_attempt"] = True - packet = build_auto_research_evidence_packet( - contract=research_contract(goal_id=GOAL_ID), - eval_results=[result], - hypothesis_id=hypothesis_id, - todo_id=todo_id, - agent_id=EVIDENCE_AGENT_ID, - claimed_by=EVIDENCE_AGENT_ID, - mechanism_family=MECHANISM_FAMILY, - hypothesis=f"{HYPOTHESIS_TEXT} Failure {hypothesis_id}.", - grounding_refs=[GROUNDING_REF], - branch_ref="codex/auto-research-worker-turn-smoke", - ) - packet_path = temp / f"{hypothesis_id}.public.json" - packet_path.write_text(json.dumps(packet, indent=2, sort_keys=True) + "\n", encoding="utf-8") - appended = append_auto_research_rollout_events( - packet_path=str(packet_path), - registry_path=registry, - runtime_root_arg=str(runtime_root), - dry_run=False, - ) - assert appended["appended_count"] == 2, appended - assert_public_safe(appended) - - -def assert_seeded_worker_turns() -> None: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - agent_specs=LANES, - session_name="loopx-auto-research-worker-turn-smoke", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - ) - visible_control, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, - goal_id=GOAL_ID, - objective="Verify visible auto-research requires real role-authored evidence.", - supervisor=supervisor, - ) - seeded_todos = visible_control["seeded_todos"] - executor_seed = next(item for item in seeded_todos if item["agent_id"] == EXECUTOR_AGENT_ID) - evaluator_seed = next(item for item in seeded_todos if item["agent_id"] == EVALUATOR_AGENT_ID) - assert evaluator_seed["resume_when"] == f"todo_done:{executor_seed['todo_id']}", visible_control - workspace = temp / "visible-workspace" - workspace.mkdir() - - curator_preview = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=CURATOR_AGENT_ID, - execute=False, - ) - assert curator_preview["mode"] == "dry_run", curator_preview - assert curator_preview["selected_action"] == "write_research_contract", curator_preview - - curator_execute = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=CURATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert_manual_research_required(curator_execute, action="write_research_contract") - - executor_execute = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EXECUTOR_AGENT_ID, - execute=True, - complete=True, - ) - assert_no_action(executor_execute, agent_id=EXECUTOR_AGENT_ID) - - evaluator_execute = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=False, - ) - assert_no_action(evaluator_execute, agent_id=EVALUATOR_AGENT_ID) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - supervisor = build_auto_research_demo_supervisor_plan( - goal_id=GOAL_ID, - agent_specs=LANES, - session_name="loopx-auto-research-resume-lineage-smoke", - cli_bin="loopx", - codex_bin="codex", - tmux_bin="tmux", - reasoning_effort="high", - ) - visible_control, registry, runtime_root = _seed_visible_demo_control_plane( - demo_root=temp, - goal_id=GOAL_ID, - objective="Verify seeded evaluator summaries retain executor lineage.", - supervisor=supervisor, - ) - seeded_todos = visible_control["seeded_todos"] - executor_seed = next(item for item in seeded_todos if item["agent_id"] == EXECUTOR_AGENT_ID) - evaluator_seed = next(item for item in seeded_todos if item["agent_id"] == EVALUATOR_AGENT_ID) - assert evaluator_seed["resume_when"] == f"todo_done:{executor_seed['todo_id']}", visible_control - append_negative_fixture_evidence( - registry=registry, - runtime_root=Path(runtime_root), - temp=temp, - hypothesis_id="hyp_seeded_retry_exhausted", - todo_id=executor_seed["todo_id"], - failure_kind="retry_exhausted", - eval_status="failed_to_run", - primary_metric_status="inconclusive", - ) - # This seeded Goal uses the ordinary canonical SQLite/hard_lease default. - # Admit the exact executor lease before terminal writeback; a claim alone - # is not execution authority under that contract. - for prerequisite, successor in zip(seeded_todos[:2], seeded_todos[1:3]): - lease_key = "seeded-prerequisite-" + str(prerequisite["action_kind"]) - lease = acquire_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=prerequisite["todo_id"], owner=prerequisite["agent_id"], - idempotency_key=lease_key, ttl_seconds=600, - ) - completed = complete_goal_todo( - registry_path=registry, goal_id=GOAL_ID, todo_id=prerequisite["todo_id"], - role="agent", claimed_by=prerequisite["agent_id"], agent_id=prerequisite["agent_id"], - task_lease_idempotency_key=lease_key, task_lease_expected_version=lease["lease"]["version"], - note="synthetic predecessor resolved for exhausted-retry lineage fixture", - evidence="public-safe predecessor fixture; no research metric or result claimed", - successor_todo_ids=[successor["todo_id"]], self_merged=True, dry_run=False, - ) - assert completed["completed"] is True, completed - executor_lease = acquire_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=executor_seed["todo_id"], owner=EXECUTOR_AGENT_ID, - idempotency_key="seeded-executor-completion", ttl_seconds=600, - ) - assert executor_lease["source_authority"] == "sqlite_v0", executor_lease - executor_completion = complete_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - todo_id=executor_seed["todo_id"], - role="agent", - claimed_by=EXECUTOR_AGENT_ID, - agent_id=EXECUTOR_AGENT_ID, - task_lease_idempotency_key="seeded-executor-completion", - task_lease_expected_version=executor_lease["lease"]["version"], - note="lane-authored exhausted retry evidence was appended", - evidence="public-safe evidence packet recorded retry exhaustion", - no_followup=True, - self_merged=True, - dry_run=False, - ) - assert executor_completion["completed"] is True, executor_completion - workspace = temp / "visible-workspace" - workspace.mkdir() - - evaluator_request = dict( - registry_path=registry, runtime_root_arg=runtime_root, goal_id=GOAL_ID, - agent_id=EVALUATOR_AGENT_ID, workspace=workspace, execute=True, - complete_selected_todo=True, - ) - external = acquire_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=evaluator_seed["todo_id"], owner=EVALUATOR_AGENT_ID, - idempotency_key="external-summary-attempt", ttl_seconds=600, - ) - try: - run_auto_research_worker_turn(**evaluator_request) - except TaskLeaseError as exc: - assert exc.code == "todo_lease_conflict", exc - else: - raise AssertionError("a worker must not borrow another execution's lease") - assert not (workspace / ".local").exists(), workspace - retained = inspect_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=evaluator_seed["todo_id"], - ) - assert retained["active"] is True and retained["lease"] == external["lease"], retained - release_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=evaluator_seed["todo_id"], owner=EVALUATOR_AGENT_ID, - idempotency_key="external-summary-attempt", expected_version=external["lease"]["version"], - ) - # A real output failure must retire only this worker's admitted lease. - (workspace / ".local").write_text("blocked output directory", encoding="utf-8") - try: - run_auto_research_worker_turn(**evaluator_request) - except (FileExistsError, NotADirectoryError): - pass - else: - raise AssertionError("the invalid output directory must fail before Todo completion") - (workspace / ".local").unlink() - failed_attempt = inspect_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=evaluator_seed["todo_id"], - ) - assert failed_attempt["active"] is False, failed_attempt - - unfinished_summary = run_auto_research_worker_turn( - **{**evaluator_request, "complete_selected_todo": False} - ) - assert unfinished_summary["selected_todo_id"] == evaluator_seed["todo_id"], unfinished_summary - assert unfinished_summary["executed"] is True, unfinished_summary - assert unfinished_summary["completion"]["requested"] is False, unfinished_summary - unfinished_lease = inspect_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=evaluator_seed["todo_id"], - ) - assert unfinished_lease["active"] is False, unfinished_lease - assert unfinished_lease["lease"]["idempotency_key"].startswith("auto-research-worker-"), unfinished_lease - - seeded_failure_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert seeded_failure_turn["selected_todo_id"] == evaluator_seed["todo_id"], seeded_failure_turn - assert seeded_failure_turn["selected_action"] == "summarize_evidence", seeded_failure_turn - assert ( - seeded_failure_turn["frontier"]["frontier"]["selected"]["resume_when"] - == f"todo_done:{executor_seed['todo_id']}" - ), seeded_failure_turn - assert ( - seeded_failure_turn["evaluation_summary"]["failure_continuation"]["source_todo_id"] - == executor_seed["todo_id"] - ), seeded_failure_turn - successors = seeded_failure_turn["successor_todos"]["successors"] - assert len(successors) == 1, seeded_failure_turn - assert successors[0]["action_kind"] == "propose_failure_successor", seeded_failure_turn - assert successors[0]["unblocks_todo_id"] == executor_seed["todo_id"], seeded_failure_turn - assert seeded_failure_turn["completion"]["successor_todo_ids"] == [ - successors[0]["todo_id"] - ], seeded_failure_turn - completed_attempt = inspect_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=evaluator_seed["todo_id"], - ) - assert completed_attempt["active"] is False, completed_attempt - assert_public_safe(seeded_failure_turn) - - manual_todo = add_goal_todo( - registry_path=registry, goal_id=GOAL_ID, role="agent", - text="Author a real failure successor before further research execution.", - task_class="advancement_task", action_kind="propose_failure_successor", - claimed_by="research-curator", priority="P0", dry_run=False, - ) - manual_successor = run_worker_turn( - registry=registry, runtime_root=runtime_root, workspace=workspace, - agent_id="research-curator", execute=True, complete=True, - ) - assert_manual_research_required(manual_successor, action="propose_failure_successor") - assert manual_successor["selected_todo_id"] == manual_todo["todo_id"], manual_successor - manual_lease = inspect_task_lease( - registry_path=registry, runtime_root=Path(runtime_root), goal_id=GOAL_ID, - todo_id=manual_successor["selected_todo_id"], - ) - assert manual_lease["lease"] is None, manual_lease - - - -def assert_evidence_lineage(provider: str) -> None: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - summary_todo = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text=( - "[P0-auto-research-live] Summarize visible held-out evidence and " - "open the next research round." - ), - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - dry_run=False, - ) - summary_todo_id = summary_todo["todo_id"] - append_real_fixture_evidence(registry=registry, runtime_root=runtime_root, temp=temp) - workspace = project / "visible-workspace" - workspace.mkdir() - - evaluator_summary = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert evaluator_summary["mode"] == "execute", evaluator_summary - assert evaluator_summary["selected_action"] == "summarize_evidence", evaluator_summary - successor_todos = evaluator_summary["successor_todos"] - assert successor_todos["executed"] is True, successor_todos - successor_ids = [ - successor["todo_id"] - for successor in successor_todos["successors"] - if successor.get("todo_id") - ] - assert len(successor_ids) == 1, successor_todos - assert successor_todos["successors"][0]["claimed_by"] == CURATOR_AGENT_ID, successor_todos - assert successor_todos["successors"][0]["action_kind"] == "review_research_contract", successor_todos - completion = evaluator_summary["completion"] - assert completion["executed"] is True, completion - assert completion["successor_todo_ids"] == successor_ids, evaluator_summary - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={summary_todo_id} status=done" in state_text, state_text - assert "no_followup=true" not in state_text, state_text - for successor_id in successor_ids: - assert successor_id in state_text, state_text - assert_public_safe(evaluator_summary) - canonical = read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) - by_id = {row["todo_id"]: row for row in canonical["todos"]} - assert by_id[summary_todo_id]["status"] == "done", canonical - assert by_id[summary_todo_id]["claimed_by"] == EVALUATOR_AGENT_ID - assert by_id[successor_ids[0]]["claimed_by"] == CURATOR_AGENT_ID - assert by_id[successor_ids[0]]["unblocks_todo_id"] == summary_todo_id - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - promotion_todo = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text=( - "[P0-auto-research-live] Review promotion readiness and " - "open the next research round when acceptance is not met." - ), - task_class="advancement_task", - action_kind="review_promotion_readiness", - claimed_by=EVALUATOR_AGENT_ID, - dry_run=False, - ) - promotion_todo_id = promotion_todo["todo_id"] - append_real_fixture_evidence(registry=registry, runtime_root=runtime_root, temp=temp) - workspace = project / "visible-workspace" - workspace.mkdir() - - promotion_review = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert promotion_review["mode"] == "execute", promotion_review - assert promotion_review["selected_action"] == "review_promotion_readiness", promotion_review - assert promotion_review["evaluation_summary"]["holdout_improvement_count"] == 1, promotion_review - successor_todos = promotion_review["successor_todos"] - assert successor_todos["executed"] is True, successor_todos - successor_ids = [ - successor["todo_id"] - for successor in successor_todos["successors"] - if successor.get("todo_id") - ] - assert len(successor_ids) == 1, successor_todos - assert successor_todos["successors"][0]["claimed_by"] == CURATOR_AGENT_ID, successor_todos - assert successor_todos["successors"][0]["action_kind"] == "review_research_contract", successor_todos - completion = promotion_review["completion"] - assert completion["executed"] is True, completion - assert completion["successor_todo_ids"] == successor_ids, promotion_review - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={promotion_todo_id} status=done" in state_text, state_text - assert "no_followup=true" not in state_text, state_text - for successor_id in successor_ids: - assert successor_id in state_text, state_text - assert_public_safe(promotion_review) - - curator_handoff = apply_role_successor_todos( - registry_path=registry, - goal_id=GOAL_ID, - source_todo_id=successor_ids[0], - current_agent_id=CURATOR_AGENT_ID, - role_id="research_curator", - action="review_research_contract", - successor_specs=auto_research_successor_specs_for_action( - role_id="research_curator", - action="review_research_contract", - ), - decision_summary={ - "validated_promotion_candidate_count": 1, - "holdout_improvement_count": 1, - }, - execute=True, - ) - assert curator_handoff["executed"] is True, curator_handoff - curator_successor_ids = [ - successor["todo_id"] - for successor in curator_handoff["successors"] - if successor.get("todo_id") - ] - assert len(curator_successor_ids) == 1, curator_handoff - assert curator_handoff["successors"][0]["claimed_by"] == HYPOTHESIS_AGENT_ID, curator_handoff - assert curator_handoff["successors"][0]["action_kind"] == "propose_hypothesis", curator_handoff - curator_completion = complete_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - todo_id=successor_ids[0], - role="agent", - claimed_by=CURATOR_AGENT_ID, - agent_id=CURATOR_AGENT_ID, - note="auto-research curator review routed the next hypothesis round", - evidence="state-summary agent=research-curator action=review_research_contract linked next hypothesis successor", - successor_todo_ids=curator_successor_ids, - self_merged=True, - dry_run=False, - ) - assert curator_completion["completed"] is True, curator_completion - assert curator_completion["successor_todo_ids"] == curator_successor_ids, curator_completion - state_text = state_file.read_text(encoding="utf-8") - assert "no_followup=true" not in state_text, state_text - for successor_id in curator_successor_ids: - assert successor_id in state_text, state_text - assert_public_safe(curator_handoff) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - first_failure_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate the first failed evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - second_failure_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate the second failed evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - unrelated_summary = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize an independent evidence branch.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_first_failure", - todo_id=first_failure_evidence["todo_id"], - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_second_failure", - todo_id=second_failure_evidence["todo_id"], - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - unrelated_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert unrelated_turn["evaluation_summary"]["failure_continuation"] is None, unrelated_turn - assert unrelated_turn["successor_todos"]["successors"] == [], unrelated_turn - assert unrelated_turn["completion"] == { - "requested": True, - "executed": False, - "reason": "failure_successor_lineage_unresolved", - }, unrelated_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={unrelated_summary['todo_id']} status=done" not in state_text, state_text - assert "propose_failure_successor" not in state_text, state_text - assert_public_safe(unrelated_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - retry_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Retry one pending evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - other_failure_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate a separately retired evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - missing_match_summary = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize the retry evidence branch.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=retry_evidence["todo_id"], - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_retry_pending", - todo_id=retry_evidence["todo_id"], - eval_status="failed_to_run", - primary_metric_status="inconclusive", - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_other_retired", - todo_id=other_failure_evidence["todo_id"], - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - missing_match_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert ( - missing_match_turn["frontier"]["frontier"]["completion"]["status"] - == "failure_successor_required" - ), missing_match_turn - assert missing_match_turn["evaluation_summary"]["failure_continuation"] is None, ( - missing_match_turn - ) - assert missing_match_turn["successor_todos"]["successors"] == [], missing_match_turn - assert missing_match_turn["completion"] == { - "requested": True, - "executed": False, - "reason": "failure_successor_lineage_unresolved", - }, missing_match_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={missing_match_summary['todo_id']} status=done" not in state_text, state_text - assert_public_safe(missing_match_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - self_referential_summary = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize a self-referential retired branch.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_summary_id_failure", - todo_id=self_referential_summary["todo_id"], - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - self_referential_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert ( - self_referential_turn["frontier"]["frontier"]["completion"]["status"] - == "failure_successor_required" - ), self_referential_turn - assert self_referential_turn["evaluation_summary"]["failure_continuation"] is None, ( - self_referential_turn - ) - assert self_referential_turn["successor_todos"]["successors"] == [], self_referential_turn - assert self_referential_turn["completion"] == { - "requested": True, - "executed": False, - "reason": "failure_successor_lineage_unresolved", - }, self_referential_turn - state_text = state_file.read_text(encoding="utf-8") - assert ( - f"todo_id={self_referential_summary['todo_id']} status=done" not in state_text - ), state_text - assert_public_safe(self_referential_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - positive_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate the supported evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - unrelated_failure_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate an unrelated failed evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - resume_gate = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Complete an independent summary gate.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - positive_summary = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize the supported evidence branch.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=positive_evidence["todo_id"], - resume_when=f"todo_done:{resume_gate['todo_id']}", - dry_run=False, - ) - append_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_positive_branch", - todo_id=positive_evidence["todo_id"], - dev_metric=1.4, - holdout_metric=1.5, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_unrelated_failure", - todo_id=unrelated_failure_evidence["todo_id"], - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_resume_gate_failure", - todo_id=resume_gate["todo_id"], - ) - resume_gate_completion = complete_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - todo_id=resume_gate["todo_id"], - role="agent", - claimed_by=EXECUTOR_AGENT_ID, - agent_id=EXECUTOR_AGENT_ID, - note="independent gate completed for summary lineage priority coverage", - evidence="public-safe gate evidence", - no_followup=True, - self_merged=True, - dry_run=False, - ) - assert resume_gate_completion["completed"] is True, resume_gate_completion - workspace = project / "visible-workspace" - workspace.mkdir() - - positive_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert positive_turn["evaluation_summary"]["failure_continuation"] is None, positive_turn - assert ( - positive_turn["frontier"]["frontier"]["completion"]["status"] - == "promotion_review_required" - ), positive_turn - assert ( - positive_turn["frontier"]["frontier"]["selected"]["unblocks_todo_id"] - == positive_evidence["todo_id"] - ), positive_turn - assert ( - positive_turn["frontier"]["frontier"]["selected"]["resume_when"] - == f"todo_done:{resume_gate['todo_id']}" - ), positive_turn - successors = positive_turn["successor_todos"]["successors"] - assert len(successors) == 1, positive_turn - assert successors[0]["claimed_by"] == CURATOR_AGENT_ID, positive_turn - assert successors[0]["action_kind"] == "review_research_contract", positive_turn - assert positive_turn["completion"]["successor_todo_ids"] == [successors[0]["todo_id"]], positive_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={positive_summary['todo_id']} status=done" in state_text, state_text - assert_public_safe(positive_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - failure_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate retired evidence.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - failure_review = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize retired evidence.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=failure_evidence["todo_id"], - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_failure_successor", - todo_id=failure_evidence["todo_id"], - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - failure_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - successors = failure_turn["successor_todos"]["successors"] - assert len(successors) == 1, failure_turn - assert successors[0]["claimed_by"] == HYPOTHESIS_AGENT_ID, failure_turn - assert successors[0]["action_kind"] == "propose_failure_successor", failure_turn - assert failure_turn["evaluation_summary"]["failure_continuation"]["monitor_allowed"] is False, failure_turn - assert ( - failure_turn["evaluation_summary"]["failure_continuation"]["source_todo_id"] - == failure_evidence["todo_id"] - ), failure_turn - assert ( - failure_turn["frontier"]["frontier"]["selected"]["unblocks_todo_id"] - == failure_evidence["todo_id"] - ), failure_turn - assert successors[0]["unblocks_todo_id"] == failure_evidence["todo_id"], failure_turn - assert failure_turn["completion"]["successor_todo_ids"] == [successors[0]["todo_id"]], failure_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={failure_review['todo_id']} status=done" in state_text, state_text - assert "continuous_monitor" not in state_text, state_text - assert_public_safe(failure_turn) - - proposer_successors = auto_research_successor_specs_for_action( - role_id="hypothesis_proposer", - action="propose_failure_successor", - ) - assert proposer_successors == [], proposer_successors - classified_failure_successors = auto_research_successor_specs_for_action( - role_id="evaluator_promoter", - action="classify_evidence", - ) - assert { - successor["action_kind"] for successor in classified_failure_successors - } == { - "remediate_data_measurement", - "propose_failure_successor", - }, classified_failure_successors - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - data_gap_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate the declared data gap.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - data_gap_review = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize a declared data gap.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=data_gap_evidence["todo_id"], - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_data_gap_successor", - todo_id=data_gap_evidence["todo_id"], - failure_kind="data_or_measurement_gap", - measurement_scope="adjusted_price_field", - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - data_gap_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - successors = data_gap_turn["successor_todos"]["successors"] - assert len(successors) == 1, data_gap_turn - assert successors[0]["claimed_by"] == EXECUTOR_AGENT_ID, data_gap_turn - assert successors[0]["action_kind"] == "remediate_data_measurement", data_gap_turn - assert data_gap_turn["evaluation_summary"]["failure_continuation"]["remediation_attempt_limit"] == 1, data_gap_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={data_gap_review['todo_id']} status=done" in state_text, state_text - assert "continuous_monitor" not in state_text, state_text - assert_public_safe(data_gap_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - retry_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Retry the exhausted evidence branch.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - retry_review = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Summarize exhausted retry evidence.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=retry_evidence["todo_id"], - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_retry_exhausted", - todo_id=retry_evidence["todo_id"], - failure_kind="retry_exhausted", - eval_status="failed_to_run", - primary_metric_status="inconclusive", - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - retry_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - successors = retry_turn["successor_todos"]["successors"] - assert len(successors) == 1, retry_turn - assert successors[0]["claimed_by"] == HYPOTHESIS_AGENT_ID, retry_turn - assert successors[0]["action_kind"] == "propose_failure_successor", retry_turn - assert ( - retry_turn["evaluation_summary"]["failure_continuation"]["failure_kind"] - == "retry_exhausted" - ), retry_turn - assert ( - retry_turn["evaluation_summary"]["failure_continuation"]["source_todo_id"] - == retry_evidence["todo_id"] - ), retry_turn - assert successors[0]["unblocks_todo_id"] == retry_evidence["todo_id"], retry_turn - assert retry_turn["completion"]["successor_todo_ids"] == [successors[0]["todo_id"]], retry_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={retry_review['todo_id']} status=done" in state_text, state_text - assert_public_safe(retry_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - generic_todo = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text=( - "[P1] Fix auto-research completion projection after validated " - "promotion and holdout evidence." - ), - task_class="advancement_task", - claimed_by=EVALUATOR_AGENT_ID, - dry_run=False, - ) - generic_todo_id = generic_todo["todo_id"] - append_real_fixture_evidence(registry=registry, runtime_root=runtime_root, temp=temp) - workspace = project / "visible-workspace" - workspace.mkdir() - - generic_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - assert generic_turn["mode"] == "unsupported_action", generic_turn - assert generic_turn["selected_action"] == "advance_todo", generic_turn - assert generic_turn["executed"] is False, generic_turn - state_text = state_file.read_text(encoding="utf-8") - assert f"todo_id={generic_todo_id} status=done" not in state_text, state_text - assert "satisfied_generic_handoff_closed" not in json.dumps(generic_turn), generic_turn - assert_public_safe(generic_turn) - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - append_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_target_reached_first", - todo_id="todo_auto_research_target_first", - dev_metric=1.4, - holdout_metric=1.5, - ) - append_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_target_reached_second", - todo_id="todo_auto_research_target_second", - dev_metric=2.0, - holdout_metric=2.1, - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - target_reached = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=False, - ) - assert target_reached["mode"] == "quiet_completion", target_reached - completion = target_reached["completion"] - assert completion["status"] == "target_reached", target_reached - assert completion["quiet_completion_allowed"] is True, target_reached - assert completion["next_action"] == "quiet_completion", target_reached - assert completion["holdout_improvement_count"] == 2, target_reached - assert completion["required_holdout_improvement_count"] == 2, target_reached - frontier_completion = target_reached["frontier"]["frontier"]["completion"] - assert frontier_completion["status"] == "target_reached", target_reached - assert target_reached["frontier"]["frontier"]["selected"] is None, target_reached - assert_public_safe(target_reached) - - for historical_parent in (False, True): - assert_failure_successor_lineage(provider, historical_parent) - - -def assert_failure_successor_lineage(provider: str, historical_parent: bool) -> None: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - project = temp / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = temp / "registry.json" - runtime_root = temp / "runtime" - write_summary_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root, provider=provider) - failure_evidence = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Evaluate a retired evidence branch for lineage dedup.", - task_class="advancement_task", - action_kind="run_dev_eval", - claimed_by=EXECUTOR_AGENT_ID, - dry_run=False, - ) - first_summary = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] First summary of the retired evidence branch.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=failure_evidence["todo_id"], - dry_run=False, - ) - second_summary = add_goal_todo( - registry_path=registry, - goal_id=GOAL_ID, - role="agent", - text="[P0-auto-research-live] Second summary of the same retired evidence branch.", - task_class="advancement_task", - action_kind="summarize_evidence", - claimed_by=EVALUATOR_AGENT_ID, - unblocks_todo_id=failure_evidence["todo_id"], - dry_run=False, - ) - append_negative_fixture_evidence( - registry=registry, - runtime_root=runtime_root, - temp=temp, - hypothesis_id="hyp_lineage_dedup_failure", - todo_id=failure_evidence["todo_id"], - ) - workspace = project / "visible-workspace" - workspace.mkdir() - - if historical_parent: - # Seed the published pre-cutoff producer intent through native create; - # its parent must survive upgrade even when another summary retries. - historical = apply_role_successor_todos(registry_path=registry, goal_id=GOAL_ID, - source_todo_id=first_summary['todo_id'], text_source_todo_id=failure_evidence['todo_id'], - current_agent_id=EVALUATOR_AGENT_ID, role_id='evaluator_promoter', - action='summarize_evidence', execute=True, - successor_specs=[spec for spec in auto_research_successor_specs_for_action( - role_id='evaluator_promoter', action='summarize_evidence') - if spec['action_kind'] == 'propose_failure_successor'], - decision_summary={'failure_continuation': {'next_outcome': 'propose_failure_successor'}}) - assert historical['successors'][0]['added'] is True - original = read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) - original_source = next(row for row in original['todos'] if row['todo_id'] == failure_evidence['todo_id']) - - first_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - first_successors = first_turn["successor_todos"]["successors"] - assert len(first_successors) == 1, first_turn - assert first_successors[0]["action_kind"] == "propose_failure_successor", first_turn - assert first_successors[0]["claimed_by"] == HYPOTHESIS_AGENT_ID, first_turn - first_successor_text = first_successors[0]["todo_command"] - assert ".." not in first_successor_text, f"successor text contains ..: {first_successor_text!r}" - assert_public_safe(first_turn) - first_snapshot = read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) - successor_id = first_successors[0]['todo_id'] - accepted = next(row for row in first_snapshot['todos'] if row['todo_id'] == successor_id) - assert accepted['unblocks_todo_id'] == (first_summary['todo_id'] if historical_parent else failure_evidence['todo_id']) - - # An explicit attempt to rebind this existing successor remains illegal. - # The producer fix must not weaken the canonical duplicate-intent fence. - try: - add_goal_todo(registry_path=registry, goal_id=GOAL_ID, role='agent', - text=accepted['text'], task_class='advancement_task', - action_kind='propose_failure_successor', claimed_by=HYPOTHESIS_AGENT_ID, - unblocks_todo_id=second_summary['todo_id'], dry_run=False) - except LocalCoordinationAuthorityUnavailable as error: - assert error.code == 'todo_semantic_duplicate_conflict', error - else: - raise AssertionError('same-title successor must not acquire another source binding') - assert read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) == first_snapshot - if historical_parent: - from loopx.control_plane.testing.canary_harness import run_json_cli - run_json_cli('todo', 'archive-completed', '--goal-id', GOAL_ID, '--role', 'agent', - '--max-active-done', '0', '--execute', - registry_path=registry, runtime_root=runtime_root) - - if historical_parent: - from unittest.mock import patch - from demo.auto_research import worker_runtime - observed_after_race = None - - def concurrent_read(**kwargs): - nonlocal observed_after_race - snapshot = read_canonical_todos_if_promoted(**kwargs) - add_goal_todo(registry_path=registry, goal_id=GOAL_ID, role='agent', - text='[P2] Independent concurrent planning work.', task_class='advancement_task', - claimed_by=CURATOR_AGENT_ID, dry_run=False) - observed_after_race = read_canonical_todos_if_promoted(**kwargs) - return snapshot - - # Intervene after a real read; both the update and CAS refusal use - # the real provider. No mock supplies the claimed postcondition. - with patch.object(worker_runtime, 'read_canonical_todos_if_promoted', concurrent_read): - try: - run_auto_research_worker_turn(registry_path=registry, - runtime_root_arg=str(runtime_root), goal_id=GOAL_ID, - agent_id=EVALUATOR_AGENT_ID, workspace=workspace, execute=True, - complete_selected_todo=True, lane_count=4, visible_lanes_accepted=True) - except LocalCoordinationAuthorityUnavailable as error: - assert error.code == 'provider_revision_mismatch', error - else: - raise AssertionError('stale historical-lineage proof must refuse') - assert read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) == observed_after_race - - second_turn = run_worker_turn( - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - agent_id=EVALUATOR_AGENT_ID, - execute=True, - complete=True, - ) - second_successors = second_turn["successor_todos"]["successors"] - assert len(second_successors) == 1, second_turn - second_successor = second_successors[0] - assert second_successor["already_exists"] is True, ( - f"second successor should be deduplicated, got: {second_successor}" - ) - assert second_successor["todo_id"] == first_successors[0]["todo_id"], ( - f"second successor should have the same todo_id, got: {second_successor['todo_id']}" - ) - assert_public_safe(second_turn) - final = read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) - rows = {row['todo_id']: row for row in final['todos']} - assert len(final['todos']) == len(original['todos']) + 1 - assert rows[successor_id] == accepted - assert rows[failure_evidence['todo_id']] == original_source - for summary in (first_summary, second_summary): - stored = rows[summary['todo_id']] - assert stored['status'] == 'done' and stored['claimed_by'] == EVALUATOR_AGENT_ID - assert stored['successor_todo_ids'] == [successor_id] - -def main() -> int: - assert_seeded_worker_turns() - for provider in ("file", "sqlite"): - assert_evidence_lineage(provider) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/auto_research_lightweight_fixture.py b/examples/auto_research_lightweight_fixture.py deleted file mode 100644 index 1c0ad612a7..0000000000 --- a/examples/auto_research_lightweight_fixture.py +++ /dev/null @@ -1,138 +0,0 @@ -from __future__ import annotations - -import json -import subprocess -import sys -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -GOAL_ID = "loopx-auto-research-demo" -METRIC_NAME = "fixture_quality_score" -BASELINE = 1.0 -HYPOTHESIS_ID = "hyp_state_a2a_round" -TODO_ID = "todo_auto_research_demo_001" -AGENT_ID = "research-executor" -MECHANISM_FAMILY = "state_a2a_iteration" -HYPOTHESIS_TEXT = "Use a small state-mediated handoff loop to improve the shared candidate." -GROUNDING_REF = "fixture:lane_authored_metric" - - -def research_contract(*, goal_id: str = GOAL_ID) -> dict[str, Any]: - return { - "schema_version": "research_contract_v0", - "goal_id": goal_id, - "research_objective": "Validate compact multi-agent research evidence.", - "editable_scope": ["candidate_strategy", "hypothesis_text", "todo_handoff"], - "protected_scope": ["metric_definition", "baseline_metric", "holdout_split"], - "metric": { - "name": METRIC_NAME, - "direction": "maximize", - "baseline": BASELINE, - }, - "dev_eval": "public fixture evaluator on dev split", - "holdout_eval": "public fixture evaluator on holdout split", - "promotion_policy": "dev_and_holdout_improved", - } - - -def eval_result(split: str, *, value: float | None | object = None) -> dict[str, Any]: - if value is None and split in {"dev", "holdout"}: - value = 4.0 if split == "dev" else 4.5 - return { - "schema_version": "auto_research_lightweight_eval_result_v0", - "split": split, - "metric": { - "name": METRIC_NAME, - "direction": "maximize", - "value": value, - "baseline": BASELINE, - }, - "eval_status": "scored", - "primary_metric_status": "improved", - "artifact_refs": [f"public_metric:{split}:state_a2a_round"], - "protected_scope_clean": True, - "no_upload": True, - } - - -def write_json(path: Path, payload: dict[str, Any]) -> None: - path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") - - -def write_registry( - temp: Path, - *, - registered_agents: list[str], -) -> Path: - runtime_root = temp / "runtime" - project = temp / "project" - project.mkdir() - registry = temp / "registry.json" - registry.write_text( - json.dumps( - { - "schema_version": "0.1", - "common_runtime_root": str(runtime_root), - "goals": [ - { - "id": GOAL_ID, - "repo": str(project), - "state_file": "ACTIVE_GOAL_STATE.md", - "adapter": { - "kind": "fixture", - "status": "connected-read-only", - }, - "coordination": { - "agent_model": "peer_v1", - "registered_agents": registered_agents, - }, - } - ], - } - ), - encoding="utf-8", - ) - return registry - - -def run_auto_research_cli( - registry: Path, - *args: str, - expect_ok: bool = True, -) -> dict[str, Any]: - result = subprocess.run( - [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--format", - "json", - "auto-research", - *args, - ], - cwd=REPO_ROOT, - check=False, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - payload = json.loads(result.stdout) - if expect_ok: - assert result.returncode == 0 and payload["ok"] is True, (result, payload) - else: - assert result.returncode != 0 and payload["ok"] is False, (result, payload) - return payload - - -def write_contract_and_results(temp: Path) -> tuple[Path, Path, Path]: - contract = temp / "research-contract.public.json" - dev = temp / "dev-result.public.json" - holdout = temp / "holdout-result.public.json" - write_json(contract, research_contract()) - write_json(dev, eval_result("dev")) - write_json(holdout, eval_result("holdout")) - return contract, dev, holdout diff --git a/examples/canary/catalog-planner-smoke.py b/examples/canary/catalog-planner-smoke.py index 53d0601ff6..f27dc612cf 100644 --- a/examples/canary/catalog-planner-smoke.py +++ b/examples/canary/catalog-planner-smoke.py @@ -64,7 +64,6 @@ def assert_profiles_come_from_catalog_matrix() -> None: "first-connect-contract", "runtime-connector-catalog", "frontstage-rollout", - "auto-research-demo", "catalog-canary-contract", "benchmark-toolkit-boundary", } <= domain_profile_ids, payload @@ -718,26 +717,6 @@ def assert_pr_release_and_refactor_profiles_select() -> None: assert runtime_connector_profile["deep_checks_available"] is True, runtime_connector_profile assert runtime_connector_profile["deep_checks_included"] is False, runtime_connector_profile - auto_research_payload = build_catalog_canary_plan( - changed_files=["demo/auto_research/core.py"], - surfaces=["auto-research demo frontier visible launcher"], - ) - auto_research_profiles = { - profile["id"]: profile for profile in auto_research_payload["domain_profiles"] - } - assert "auto-research-demo" in auto_research_profiles, auto_research_payload - auto_research_profile = auto_research_profiles["auto-research-demo"] - auto_research_commands = [check["command"] for check in auto_research_profile["checks"]] - assert ( - "python3 examples/auto-research-minimal-kernel-smoke.py" - in auto_research_commands - ), auto_research_profile - assert ( - "python3 examples/decentralized-auto-research-frontier-smoke.py" in auto_research_commands - ), auto_research_profile - assert all(check["tier"] == "default" for check in auto_research_profile["checks"]), auto_research_profile - assert auto_research_profile["deep_checks_available"] is True, auto_research_profile - explore_payload = build_catalog_canary_plan( changed_files=[ "loopx/capabilities/explore/harness_runtime.py", diff --git a/examples/cli-multi-agent-command-modularization-smoke.py b/examples/cli-multi-agent-command-modularization-smoke.py deleted file mode 100644 index 3dc2daec7a..0000000000 --- a/examples/cli-multi-agent-command-modularization-smoke.py +++ /dev/null @@ -1,78 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import subprocess -import sys -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -CLI = ROOT / "loopx" / "cli.py" -MODULE = ROOT / "demo" / "multi_agent_cli.py" - - -def run_cli(*args: str) -> subprocess.CompletedProcess[str]: - return subprocess.run( - [sys.executable, "-m", "loopx.cli", *args], - cwd=ROOT, - text=True, - capture_output=True, - check=False, - ) - - -def require(condition: bool, message: str) -> None: - if not condition: - raise AssertionError(message) - - -def require_success(result: subprocess.CompletedProcess[str]) -> str: - if result.returncode != 0: - raise AssertionError( - f"expected success, got {result.returncode}\n" - f"stdout:\n{result.stdout}\n\nstderr:\n{result.stderr}" - ) - return result.stdout - - -def main() -> None: - cli_source = CLI.read_text(encoding="utf-8") - module_source = MODULE.read_text(encoding="utf-8") - - require("multi_agent.py" not in cli_source, "cli.py should not embed multi-agent module details") - require( - '("multi-agent", "demo.multi_agent_cli", "register_multi_agent_commands")' - in cli_source, - "cli.py did not register the source-checkout multi-agent demo", - ) - require( - "from demo.multi_agent_cli import handle_multi_agent_command" in cli_source, - "cli.py did not dispatch the source-checkout multi-agent demo", - ) - - for marker in ( - "register_multi_agent_commands", - "handle_multi_agent_command", - "build_visible_multi_agent_payload_from_spec", - "execute_visible_multi_agent_launcher", - "generic_multi_agent_launch_spec_v0", - "--auto-wake", - ): - require(marker in module_source, f"multi-agent module missing {marker}") - - help_text = require_success(run_cli("multi-agent", "launch", "--help")) - for option in ( - "--spec", - "--execute", - "--workspace", - "--attach", - "--auto-wake", - "--codex-trust-workspace", - ): - require(option in help_text, f"multi-agent launch help omitted {option}") - - print("cli-multi-agent-command-modularization-smoke: ok") - - -if __name__ == "__main__": - main() diff --git a/examples/control_plane/bounded-context-namespace-smoke.py b/examples/control_plane/bounded-context-namespace-smoke.py index 85da545a23..67d49c28a6 100644 --- a/examples/control_plane/bounded-context-namespace-smoke.py +++ b/examples/control_plane/bounded-context-namespace-smoke.py @@ -33,7 +33,7 @@ } LEGACY_CAPABILITY_PACKAGES = { "loopx/capabilities/cross_runtime/": "loopx.control_plane.handoff.cross_runtime_impl_review", - "loopx/capabilities/multi_agent/": "demo.multi_agent", + "loopx/capabilities/multi_agent/": None, "loopx/capabilities/review_batch/": "loopx.control_plane.handoff.review_batch", } CANONICAL_MODULES = { @@ -159,10 +159,11 @@ def assert_rehomed_capability_packages_stay_in_bounded_contexts() -> None: "legacy_module": legacy_module, "canonical_module": canonical_module, } - assert importlib.util.find_spec(canonical_module) is not None, { - "reason": "canonical bounded-context import must be available", - "canonical_module": canonical_module, - } + if canonical_module is not None: + assert importlib.util.find_spec(canonical_module) is not None, { + "reason": "canonical bounded-context import must be available", + "canonical_module": canonical_module, + } def assert_moved_root_imports_are_not_shimmed() -> None: diff --git a/examples/decentralized-auto-research-frontier-smoke.py b/examples/decentralized-auto-research-frontier-smoke.py deleted file mode 100644 index 12984a9585..0000000000 --- a/examples/decentralized-auto-research-frontier-smoke.py +++ /dev/null @@ -1,263 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test decentralized auto-research fixture projection.""" - -from __future__ import annotations - -import json -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.auto_research.research_state import ( # noqa: E402 - AUTO_RESEARCH_PROJECTION_SCHEMA_VERSION, - RESEARCH_EVIDENCE_GRAPH_SCHEMA_VERSION, - RESEARCH_FRONTIER_SCHEMA_VERSION, - build_auto_research_projection, - build_live_auto_research_projection, - load_auto_research_fixture, -) - - -FIXTURE = REPO_ROOT / "examples/fixtures/decentralized-auto-research-knn.public.json" - - -def assert_no_private_surface(payload: dict[str, Any]) -> None: - text = json.dumps(payload, sort_keys=True) - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "lark" + "office", - "byte" + "dance", - "http://", - "https://", - "s3://", - "tos://", - "hdfs://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def run_cli(args: list[str], *, check: bool = True) -> subprocess.CompletedProcess[str]: - return subprocess.run( - [sys.executable, "-m", "loopx.cli", *args], - cwd=REPO_ROOT, - check=check, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - - -def main() -> None: - fixture = load_auto_research_fixture(FIXTURE) - payload = build_auto_research_projection(fixture, agent_id="codex-side-bypass") - assert payload["ok"], payload - assert payload["schema_version"] == AUTO_RESEARCH_PROJECTION_SCHEMA_VERSION, payload - assert payload["frontier"]["schema_version"] == RESEARCH_FRONTIER_SCHEMA_VERSION, payload - assert payload["evidence_graph"]["schema_version"] == RESEARCH_EVIDENCE_GRAPH_SCHEMA_VERSION, payload - assert payload["frontier"]["agent_id"] == "codex-side-bypass", payload - assert payload["frontier"]["selected"]["hypothesis_id"] == "hyp_002", payload - assert payload["frontier"]["selected"]["allowed_action"] == "run_dev_attempt", payload - assert payload["frontier"]["blocked"][0]["blocked_by"] == "claimed_by:codex-product-capability", payload - assert payload["evidence_graph"]["holdout_improved"] is True, payload - assert payload["evidence_graph"]["negative_evidence_count"] == 1, payload - assert "showcase_projection" not in payload, payload - assert "artifact_packet" not in payload, payload - assert payload["public_boundary"]["raw_logs_recorded"] is False, payload - assert payload["public_boundary"]["private_artifacts_recorded"] is False, payload - assert_no_private_surface(payload) - - result = run_cli( - [ - "--format", - "json", - "auto-research", - "frontier", - "--fixture", - str(FIXTURE), - "--agent-id", - "codex-side-bypass", - ] - ) - cli_payload = json.loads(result.stdout) - assert cli_payload["ok"], cli_payload - assert cli_payload["frontier"]["selected"]["todo_id"] == "todo_auto_research_002", cli_payload - assert_no_private_surface(cli_payload) - - bad_fixture = json.loads(FIXTURE.read_text(encoding="utf-8")) - bad_fixture["evidence_events"][0]["artifact_refs"] = ["/" + "Users/example/raw.log"] - with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as handle: - json.dump(bad_fixture, handle) - bad_path = handle.name - blocked = run_cli( - [ - "--format", - "json", - "auto-research", - "frontier", - "--fixture", - bad_path, - "--agent-id", - "codex-side-bypass", - ], - check=False, - ) - assert blocked.returncode == 1, blocked.stdout - blocked_payload = json.loads(blocked.stdout) - assert blocked_payload["ok"] is False, blocked_payload - assert blocked_payload["error_code"] == "auto_research_invalid_input", blocked_payload - assert "public alias" in blocked_payload["error"], blocked_payload - - missing_path = f"{bad_path}.missing" - missing_args = [ - "auto-research", - "frontier", - "--fixture", - missing_path, - "--agent-id", - "codex-side-bypass", - ] - missing_json = run_cli( - ["--format", "json", *missing_args], - check=False, - ) - assert missing_json.returncode == 1, missing_json.stdout - missing_payload = json.loads(missing_json.stdout) - assert missing_payload["error_code"] == "auto_research_io_failed", missing_payload - assert missing_payload["error"] == "Auto Research could not access a required resource.", missing_payload - assert missing_path not in missing_json.stdout, missing_json.stdout - - missing_markdown = run_cli(["--format", "markdown", *missing_args], check=False) - assert missing_markdown.returncode == 1, missing_markdown.stdout - assert "error_code: `auto_research_io_failed`" in missing_markdown.stdout - assert "error: `Auto Research could not access a required resource.`" in missing_markdown.stdout - assert missing_path not in missing_markdown.stdout, missing_markdown.stdout - - docs = (REPO_ROOT / "docs/reference/protocols/decentralized-auto-research-state-v0.md").read_text( - encoding="utf-8" - ) - assert "decentralized_research_frontier_v0" in docs, docs - assert "shared control plane" in docs, docs - - live_payload = build_live_auto_research_projection( - goal_id="loopx-meta", - agent_id="codex-side-bypass", - quota_payload={ - "ok": True, - "agent_lane_next_action": { - "todo_id": "todo_auto_live_001", - "title": "Try partial selection for exact k-NN", - "claimed_by": "codex-side-bypass", - "status": "open", - "task_class": "advancement_task", - "action_kind": "run_dev_attempt", - }, - "capability_gate": { - "runnable_candidates": [ - { - "todo_id": "todo_auto_live_001", - "title": "Try partial selection for exact k-NN", - "claimed_by": "codex-side-bypass", - "status": "open", - "task_class": "advancement_task", - "action_kind": "run_dev_attempt", - }, - { - "todo_id": "todo_auto_live_003", - "title": "Other lane candidate must remain context", - "claimed_by": "codex-main-control", - "status": "open", - "task_class": "advancement_task", - "action_kind": "holdout_eval", - } - ], - "blocked_candidates": [ - { - "todo_id": "todo_auto_live_002", - "title": "Audit alternate batch query path", - "claimed_by": "codex-product-capability", - "status": "open", - "task_class": "advancement_task", - "action_kind": "novelty_audit", - } - ], - }, - }, - ) - assert live_payload["ok"], live_payload - assert live_payload["source_schema_version"] == "loopx_live_quota_status_v0", live_payload - assert live_payload["frontier"]["selected"]["todo_id"] == "todo_auto_live_001", live_payload - assert live_payload["frontier"]["selected"]["source_kind"] == "todo_item_v0", live_payload - assert [item["todo_id"] for item in live_payload["frontier"]["runnable"]] == [ - "todo_auto_live_001" - ], live_payload - assert live_payload["frontier"]["blocked"][0]["blocked_by"] == "claimed_by:codex-main-control", live_payload - assert live_payload["frontier"]["blocked"][1]["blocked_by"] == "claimed_by:codex-product-capability", live_payload - assert live_payload["evidence_graph"]["hypothesis_count"] == 3, live_payload - assert "showcase_projection" not in live_payload, live_payload - assert "artifact_packet" not in live_payload, live_payload - assert live_payload["public_boundary"]["source"] == "live_quota_status_projection", live_payload - assert_no_private_surface(live_payload) - - live_from_todo_summary = build_live_auto_research_projection( - goal_id="loopx-auto-research-demo", - agent_id="research-curator", - quota_payload={ - "ok": True, - "agent_todo_summary": { - "claimed_advancement_open_items": [ - { - "todo_id": "todo_auto_summary_001", - "title": "Write the public-safe research contract", - "claimed_by": "research-curator", - "status": "open", - "task_class": "advancement_task", - "action_kind": "write_research_contract", - } - ], - "claimed_by_others_items": [ - { - "todo_id": "todo_auto_summary_002", - "title": "Wait for contract before proposing a hypothesis", - "claimed_by": "hypothesis-proposer", - "status": "open", - "task_class": "advancement_task", - "action_kind": "propose_hypothesis", - "resume_when": "todo_done:todo_auto_summary_001", - "resume_ready": False, - } - ], - }, - }, - ) - assert live_from_todo_summary["frontier"]["selected"]["todo_id"] == ( - "todo_auto_summary_001" - ), live_from_todo_summary - assert live_from_todo_summary["frontier"]["selected"]["allowed_action"] == ( - "write_research_contract" - ), live_from_todo_summary - assert [item["todo_id"] for item in live_from_todo_summary["frontier"]["runnable"]] == [ - "todo_auto_summary_001" - ], live_from_todo_summary - assert live_from_todo_summary["frontier"]["blocked"][0]["blocked_by"] == ( - "claimed_by:hypothesis-proposer" - ), live_from_todo_summary - assert_no_private_surface(live_from_todo_summary) - - print("decentralized-auto-research-frontier-smoke ok") - - -if __name__ == "__main__": - main() diff --git a/examples/decentralized-auto-research-protocol-smoke.py b/examples/decentralized-auto-research-protocol-smoke.py deleted file mode 100644 index 72929da2e5..0000000000 --- a/examples/decentralized-auto-research-protocol-smoke.py +++ /dev/null @@ -1,233 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-check the decentralized auto-research protocol docs. - -This guards the durable product contract: Arbor-inspired auto research must -remain LoopX-native and decentralized, with todo-linked hypotheses, -agent-scoped frontiers, split-aware evidence, and no public/private leakage. -""" - -from __future__ import annotations - -from pathlib import Path -import re - - -ROOT = Path(__file__).resolve().parents[1] -PROTOCOL = ROOT / "docs/reference/protocols/decentralized-auto-research-state-v0.md" -LANE_CONTRACT = ROOT / "docs/reference/protocols/auto-research-lane-contract-v1.md" -ROLE_STATE_MACHINE = ROOT / "docs/reference/protocols/auto-research-role-state-machine-v0.md" -ROLE_PROFILE = ROOT / "docs/reference/protocols/auto-research-role-profile-v0.md" -PROTOCOL_README = ROOT / "docs/reference/protocols/README.md" -BLUEPRINT = ROOT / "docs/product/use-cases/auto-research/decentralized-auto-research-showcase.md" - - -PRIVATE_MARKERS = [ - "byte" + "dance", - "lark" + "office", - "fei" + "shu.cn", - "/" + "Users" + "/", - "/" + "private" + "/", - "api" + "_key", - "pass" + "word", - "sec" + "ret", -] - - -def _read(path: Path) -> str: - assert path.exists(), f"missing {path}" - return path.read_text(encoding="utf-8") - - -def _assert_public_safe(text: str, label: str) -> None: - lower = text.lower() - for marker in PRIVATE_MARKERS: - assert marker.lower() not in lower, f"{label} leaks private marker {marker!r}" - - -def main() -> None: - protocol = _read(PROTOCOL) - lane_contract = _read(LANE_CONTRACT) - role_state_machine = _read(ROLE_STATE_MACHINE) - role_profile = _read(ROLE_PROFILE) - protocol_readme = _read(PROTOCOL_README) - blueprint = _read(BLUEPRINT) - combined = ( - protocol - + "\n" - + lane_contract - + "\n" - + role_state_machine - + "\n" - + role_profile - + "\n" - + protocol_readme - + "\n" - + blueprint - ) - compact_protocol = re.sub(r"\s+", " ", protocol) - compact_lane_contract = re.sub(r"\s+", " ", lane_contract) - compact_role_state_machine = re.sub(r"\s+", " ", role_state_machine) - compact_role_profile = re.sub(r"\s+", " ", role_profile) - - _assert_public_safe(combined, "decentralized auto-research docs") - - required_protocol_terms = [ - "decentralized_auto_research_state_v0", - "research_contract_v0", - "research_hypothesis_v0", - "research_evidence_event_v0", - "decentralized_research_frontier_v0", - "research_evidence_graph_v0", - "agent_lane_next_action_v0", - "claimed_by", - "todo_id", - "agent_id", - "needs_retry", - "held-out", - "grounded_ideation", - "novelty_audit", - ] - for term in required_protocol_terms: - assert term in protocol, f"protocol missing {term!r}" - for removed_term in ( - "auto_research_artifact_packet_v0", - "research_showcase_projection_v0", - "auto_research_public_claim_boundary_v0", - ): - assert removed_term not in protocol, f"protocol kept removed projection {removed_term!r}" - assert "No agent owns the whole research tree" in compact_protocol - assert "auto_research_lane_contract_v1" in protocol - assert "auto_research_role_state_machine_v0" in protocol - assert "auto_research_lane_contract_v1" in protocol_readme - assert "auto_research_role_state_machine_v0" in protocol_readme - assert "auto_research_role_profile_v0" in protocol_readme - - required_lane_terms = [ - "auto_research_lane_contract_v1", - "research_curator", - "hypothesis_proposer", - "research_executor", - "evaluator_promoter", - "product_narrator", - "auto_research_lane_claim_v1", - "research_contract_v0", - "research_hypothesis_v0", - "auto_research_evidence_packet_v0", - "research_evidence_event_v0", - "research_evidence_graph_v0", - "quota should-run --agent-id", - "first-screen review gate", - ] - for term in required_lane_terms: - assert term in lane_contract, f"lane contract missing {term!r}" - assert "No lane is privileged" in compact_lane_contract - assert "not a lock on the full graph" in compact_lane_contract - assert "no public surface needs a leader or coordinator agent" in compact_lane_contract - - required_role_state_machine_terms = [ - "auto_research_role_state_machine_v0", - "auto_research_state_transition_v0", - "Research curator", - "Hypothesis mapper", - "Evidence runner", - "Evidence verifier", - "always-on role set small", - "transition duties", - "Read-only projection builder", - "Future Role Splits", - "Gate steward", - "Synthesis narrator", - "Frontier janitor", - "contract_ready", - "hypothesis_proposed", - "frontier_selected", - "attempt_running", - "evidence_recorded", - "evaluated", - "promotion_gate", - "quota should-run --agent-id", - "operator_gate", - "todo_id", - "claimed_by", - "No role owns the full graph", - ] - for term in required_role_state_machine_terms: - assert term in role_state_machine, f"role state machine missing {term!r}" - assert "not a coordinator" in compact_role_state_machine - assert "not start Codex, write LoopX state, or spend quota" in compact_role_state_machine - assert "Future versions may split gate stewardship" in compact_role_state_machine - assert "outside the v0 always-on role set" in compact_role_state_machine - assert "Promoting any future role requires a smoke update" in compact_role_state_machine - assert "Gate handling is a transition duty" in compact_role_state_machine - for future_role in ("Frontier janitor", "Synthesis narrator", "Gate steward"): - assert f"| {future_role} |" in role_state_machine, f"future split missing {future_role}" - - required_role_profile_terms = [ - "auto_research_role_profile_v0", - "LoopX control plane", - "worker-local role playbooks", - "AGENTS.md", - "Host launcher", - "agent_id", - "role_id", - "phase", - "capability_token", - "allowed_actions", - "write_scope", - "protected_scope", - "required_skill", - "skill_section", - "stop_conditions", - "research_curator", - "hypothesis_proposer", - "research_executor", - "evaluator_promoter", - "loopx-auto-research", - "quota should-run --goal-id", - "The pane title is cosmetic", - ] - for term in required_role_profile_terms: - assert term in role_profile, f"role profile missing {term!r}" - assert "playbooks useful without letting them become a second source of identity" in compact_role_profile - assert "identity comes from the profile and quota/frontier" in compact_role_profile - - required_blueprint_terms = [ - "Decentralized Auto Research: k-NN Speedup", - "not a claim that LoopX has already achieved", - "Research Contract card", - "Decentralized frontier", - "Evidence timeline", - "Promotion decision", - "no leader agent owns the graph", - "auto_research_lane_contract_v1", - "auto_research_role_state_machine_v0", - "curator", - "hypothesis proposer", - "executor", - "evaluator/promoter", - "product narrator", - ] - for term in required_blueprint_terms: - assert term in blueprint, f"blueprint missing {term!r}" - - forbidden_patterns = [ - r"single leader agent owns", - r"leader agent owns the full", - r"leader agent.*full hypothesis graph", - r"central Coordinator owns", - r"Coordinator owns the tree", - r"Coordinator.*promotion decisions live", - r"single agent owns the whole research tree", - r"supervisor owns the full", - ] - for pattern in forbidden_patterns: - assert not re.search(pattern, combined, re.IGNORECASE), ( - f"docs drifted toward centralized wording: {pattern}" - ) - - assert "not one leader Coordinator" in blueprint or "no leader agent" in blueprint - print("decentralized auto-research protocol smoke passed") - - -if __name__ == "__main__": - main() diff --git a/examples/docs-governance-smoke.py b/examples/docs-governance-smoke.py index 0a0ff6585c..c012238aa3 100644 --- a/examples/docs-governance-smoke.py +++ b/examples/docs-governance-smoke.py @@ -981,11 +981,9 @@ def main() -> int: root_readme_zh = read("README.zh-CN.md") governance = read(".github/GOVERNANCE.md") support = read(".github/SUPPORT.md") - auto_research_command_path = read("demo/auto_research/README.md") codex_cli_tui_loop = read("docs/product/runtimes/codex-cli/codex-cli-tui-loop.md") project_agent_contract = read("docs/project-agent-todo-contract.md") status_contract = read("docs/status-data-contract.md") - compact_auto_research_command_path = compact(auto_research_command_path) compact_codex_cli_tui_loop = compact(codex_cli_tui_loop) compact_project_agent_contract = compact(project_agent_contract) compact_status_contract = compact(status_contract) @@ -1137,8 +1135,6 @@ def main() -> int: "docs/development/documentation-layout.md", "docs/development/testing-and-quality.md", "docs/guides/README.md", - "demo/auto_research/README.md", - "docs/guides/multi-agent-product-recipe.md", "docs/integrations/README.md", "docs/reference/README.md", "docs/reference/contracts/README.md", @@ -1269,38 +1265,6 @@ def main() -> int: ]: assert required in compact_status_contract, required - for required in [ - "Start From A Clean Workspace", - "loopx-auto-research-demo", - "auto-research demo-e2e", - "auto-research demo-supervisor", - "auto-research worker-loop", - "research-curator", - "hypothesis-proposer", - "research-executor", - "evaluator-promoter", - "tmux attach -t loopx-auto-research", - "tmux kill-session -t loopx-auto-research", - "not a leader agent", - ]: - assert required in compact_auto_research_command_path, required - - multi_agent_product_recipe = read("docs/guides/multi-agent-product-recipe.md") - compact_multi_agent_product_recipe = compact(multi_agent_product_recipe) - for required in [ - "Multi-Agent Product Recipe", - "Product preset", - "Multi-agent kernel", - "role list", - "agent scope", - "worker-local skill snippet", - "handoff/todo hints", - "One-Command Launch", - "Attach, Stop, Retry", - "Auto-research should stay a reference preset, not the kernel", - ]: - assert required in compact_multi_agent_product_recipe, required - check_rfc_language_mirrors() check_rfc_ledger_entries() check_rfc_status_index() diff --git a/examples/generic-multi-agent-one-command-smoke.py b/examples/generic-multi-agent-one-command-smoke.py deleted file mode 100644 index 8dc732452a..0000000000 --- a/examples/generic-multi-agent-one-command-smoke.py +++ /dev/null @@ -1,478 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the generic visible multi-agent one-command path.""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import tempfile -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -RUNNER_CONTRACT_SCHEMA_VERSION = "tui_multi_agent_runner_contract_v0" - -PRIVATE_MARKERS = [ - "byte" + "dance", - "lark" + "office", - "fei" + "shu.cn", - "/" + "Users" + "/", - "/" + "private" + "/", - "/" + "tmp" + "/", - "api" + "_key", - "pass" + "word", - "sec" + "ret", -] - - -def write_executable(path: Path, content: str) -> None: - path.write_text(content, encoding="utf-8") - path.chmod(0o755) - - -def run_launch_command( - env: dict[str, str], - *, - spec_path: Path, - registry: Path, - runtime_root: Path, - workspace: Path, - execute: bool, -) -> dict[str, object]: - command = [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "--format", - "json", - "multi-agent", - "launch", - "--spec", - str(spec_path), - "--tmux-bin", - "tmux", - "--cli-bin", - "loopx", - "--codex-bin", - "codex", - ] - if execute: - command.extend( - [ - "--execute", - "--workspace", - str(workspace), - "--create-workspace", - "--replace-existing", - ] - ) - completed = subprocess.run( - command, - check=True, - capture_output=True, - text=True, - env=env, - ) - return json.loads(completed.stdout) - - -def run_wake_command( - env: dict[str, str], - *, - registry: Path, - runtime_root: Path, - session_name: str, - execute: bool, - lanes: list[str] | None = None, -) -> dict[str, object]: - command = [ - sys.executable, - "-m", - "loopx.cli", - "--registry", - str(registry), - "--runtime-root", - str(runtime_root), - "--format", - "json", - "multi-agent", - "wake", - "--session-name", - session_name, - "--tmux-bin", - "tmux", - ] - for lane in lanes or []: - command.extend(["--lane", lane]) - if execute: - command.append("--execute") - completed = subprocess.run( - command, - check=True, - capture_output=True, - text=True, - env=env, - ) - return json.loads(completed.stdout) - - -def assert_public_safe(value: object, label: str) -> None: - text = json.dumps(value, sort_keys=True).lower() - leaked = [marker for marker in PRIVATE_MARKERS if marker.lower() in text] - assert not leaked, f"{label} leaks private markers: {leaked}" - - -def fake_tmux_script() -> str: - return "\n".join( - [ - "#!/usr/bin/env python3", - "import json, os, sys", - "record = {", - " 'argv': sys.argv[1:],", - " 'env': {", - " 'LOOPX_PROJECT': os.environ.get('LOOPX_PROJECT'),", - " 'LOOPX_REGISTRY': os.environ.get('LOOPX_REGISTRY'),", - " 'LOOPX_RUNTIME_ROOT': os.environ.get('LOOPX_RUNTIME_ROOT'),", - " },", - "}", - "with open(os.environ['FAKE_TMUX_LOG'], 'a', encoding='utf-8') as f:", - " f.write(json.dumps(record, sort_keys=True) + '\\n')", - "if len(sys.argv) > 1 and sys.argv[1] == 'has-session':", - " raise SystemExit(1)", - "if len(sys.argv) > 1 and sys.argv[1] == 'list-windows':", - " print('planner\\ncritic')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'list-panes':", - " print('%0\\tplanner\\troles')", - " print('%1\\tcritic\\troles')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'split-window':", - " lines = []", - " try:", - " with open(os.environ['FAKE_TMUX_LOG'], encoding='utf-8') as read_log:", - " lines = read_log.readlines()", - " except Exception:", - " pass", - " split_count = sum(1 for line in lines if 'split-window' in line)", - " print(f'%{split_count}')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'capture-pane':", - " print('╭────────────────────────╮')", - " print('│ >_ OpenAI Codex │')", - " print('│ model: gpt-5.5 high │')", - " print('╰────────────────────────╯')", - " print()", - " print('› Implement {feature}')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'display-message':", - " if '#{pane_id}' in sys.argv:", - " print('%0')", - " elif '#{pane_current_command}' in sys.argv:", - " print('codex')", - " raise SystemExit(0)", - "raise SystemExit(0)", - "", - ] - ) - - -def main() -> int: - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - fake_bin = temp / "bin" - fake_bin.mkdir() - registry = temp / "registry.json" - runtime_root = temp / "runtime" - workspace = temp / "workspace" - tmux_log = temp / "tmux.jsonl" - registry.write_text(json.dumps({"common_runtime_root": str(runtime_root)}), encoding="utf-8") - runtime_root.mkdir() - worker_skill = temp / "worker" / "SKILL.md" - worker_skill.parent.mkdir() - worker_skill.write_text( - "# Planner Worker Skill\n\nUse LoopX todo/frontier state for handoff.\n", - encoding="utf-8", - ) - spec_path = temp / "multi-agent-spec.json" - spec_path.write_text( - json.dumps( - { - "schema_version": "generic_multi_agent_launch_spec_v0", - "goal_id": "loopx-meta", - "session_name": "loopx-generic-multi-agent-smoke", - "default_reasoning_effort": "high", - "roles": [ - { - "lane_id": "planner", - "agent_id": "codex-main-control", - "role_id": "planner", - "scope": "Plan the next bounded step from the shared goal surface.", - "skill": { - "name": "loopx-planner-worker", - "source": "worker/SKILL.md", - }, - "handoff_hints": [ - "Create or update a LoopX todo for critic when a plan needs review." - ], - }, - { - "lane_id": "critic", - "agent_id": "codex-side-bypass", - "role_id": "critic", - "scope": "Review the bounded step against the same todo and quota projection.", - "handoff_hints": [ - "Complete the review todo or hand a focused fix todo back to planner." - ], - }, - ], - }, - sort_keys=True, - ), - encoding="utf-8", - ) - - write_executable(fake_bin / "loopx", "#!/usr/bin/env sh\nexit 0\n") - write_executable(fake_bin / "codex", "#!/usr/bin/env sh\nexit 0\n") - write_executable(fake_bin / "tmux", fake_tmux_script()) - - env = dict(os.environ) - env.update( - { - "PATH": f"{fake_bin}{os.pathsep}{env.get('PATH', '')}", - "PYTHONPATH": f"{ROOT}{os.pathsep}{env.get('PYTHONPATH', '')}", - "FAKE_TMUX_LOG": str(tmux_log), - } - ) - - dry_packet = run_launch_command( - env, - spec_path=spec_path, - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - execute=False, - ) - assert dry_packet["mode"] == "dry_run", dry_packet - assert dry_packet["product_spec"]["schema_version"] == "generic_multi_agent_launch_spec_v0", dry_packet - assert dry_packet["product_spec"]["domain_specific"] is False, dry_packet - assert dry_packet["reasoning_contract"]["default_reasoning_effort"] == "high", dry_packet - assert dry_packet["shared_goal_surface"]["shared_state_route"] == "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", dry_packet - assert dry_packet["shared_goal_surface"]["all_lane_workspace_isolation"] is False, dry_packet - assert dry_packet["runner_contract"]["schema_version"] == RUNNER_CONTRACT_SCHEMA_VERSION, dry_packet - assert dry_packet["runner_contract"]["runner_surface"] == "tmux_codex_cli_tui", dry_packet - assert dry_packet["runner_contract"]["coordination_model"]["leader_required"] is False, dry_packet - assert dry_packet["runner_contract"]["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", dry_packet - assert dry_packet["runner_contract"]["pane_local_a2a"]["cadence_wakeup_command"] == ( - "loopx multi-agent wake --session-name " - ), dry_packet - assert dry_packet["runner_contract"]["pane_local_a2a"]["cadence_wakeup_model"] == ( - "todo_readiness_targeted_wake" - ), dry_packet - assert dry_packet["runner_contract"]["pane_local_a2a"]["cadence_broadcaster_decides_work"] is False, dry_packet - driver = dry_packet["runner_contract"]["decentralized_a2a_driver"] - assert driver["schema_version"] == "multi_agent_decentralized_a2a_driver_contract_v1", driver - assert driver["owner_layer"] == "generic_multi_agent_kernel", driver - assert driver["driver_model"] == ( - "todo_readiness_edge_plus_fixed_retry" - ), driver - assert driver["broadcaster"]["decides_work"] is False, driver - assert driver["broadcaster"]["reads_todo_readiness"] is True, driver - assert driver["pane"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", driver - assert driver["prompt"]["tick_summary_ref"] == "$LOOPX_PANE_TICK_SUMMARY", driver - assert "own_prior_status_summary" in driver["pane"]["reads"], driver - assert driver["pane"]["cadence_action"] == ( - "targeted_wakeup_then_own_quota_frontier_check_when_runnable" - ), driver - assert driver["prompt"]["tick_summary_semantics"] == ( - "prior_pane_status_not_research_evidence" - ), driver - assert ( - driver["acceptance"]["tick_summary_does_not_gate_wake_tick"] is True - ), driver - assert driver["acceptance"]["tui_first_turn_owns_tick"] is True, driver - assert driver["acceptance"]["user_and_preset_do_not_own_tick_driver"] is True, driver - assert dry_packet["runner_contract"]["pane_local_a2a"]["tick_summary"] == ( - "$LOOPX_PANE_TICK_SUMMARY" - ), dry_packet - assert dry_packet["runner_contract"]["pane_local_a2a"]["tick_output"] == ( - "$LOOPX_PANE_TICK_OUTPUT_ARTIFACT" - ), dry_packet - assert dry_packet["runner_contract"]["pane_local_a2a"]["machine_json_destination"] == ( - "$LOOPX_PANE_ARTIFACT_DIR/*.public.json" - ), dry_packet - assert dry_packet["runner_contract"]["boundaries"]["domain_specific_research_logic"] is False, dry_packet - compact = dry_packet["compact_human_status"] - assert compact["schema_version"] == "generic_multi_agent_compact_status_v0", compact - assert compact["role_count"] == 2, compact - assert compact["first_action"] == "$LOOPX_PANE_A2A_TICK", compact - assert compact["driver_model"] == ( - "todo_readiness_edge_plus_fixed_retry" - ), compact - assert compact["user_takeover"] == "attach to the session and type into any role pane", compact - assert dry_packet["interactive_tui_contract"]["schema_version"] == "multi_agent_visible_interactive_tui_contract_v0", dry_packet - assert dry_packet["interactive_tui_contract"]["runner_contract"] == RUNNER_CONTRACT_SCHEMA_VERSION, dry_packet - assert dry_packet["interactive_tui_contract"]["machine_json_policy"] == "file_or_explicit_machine_channel_only", dry_packet - assert dry_packet["interactive_tui_contract"]["codex_surface"] == "interactive_cli_tui", dry_packet - assert dry_packet["acceptance"]["runner_contract"] == RUNNER_CONTRACT_SCHEMA_VERSION, dry_packet - assert dry_packet["acceptance"]["machine_json_file_bound"] is True, dry_packet - assert dry_packet["acceptance"]["codex_tui_interactive"] is True, dry_packet - assert dry_packet["boundary"]["starts_visible_processes"] is False, dry_packet - assert dry_packet["boundary"]["spends_loopx_quota"] is False, dry_packet - assert all(lane["reasoning_effort"] == "high" for lane in dry_packet["lanes"]), dry_packet - assert all( - lane["role_profile"].get("default_kernel_skills") - == ["loopx-project", "loopx-doc-registry"] - for lane in dry_packet["lanes"] - ), dry_packet - assert any(lane["role_profile"].get("required_skill") == "loopx-planner-worker" for lane in dry_packet["lanes"]), dry_packet - assert_public_safe(dry_packet, "dry-run packet") - - exec_packet = run_launch_command( - env, - spec_path=spec_path, - registry=registry, - runtime_root=runtime_root, - workspace=workspace, - execute=True, - ) - assert exec_packet["chosen_launcher"] == "tmux", exec_packet - assert exec_packet["workspace_mode"] == "explicit_workspace", exec_packet - launch = exec_packet["launch_result"] - assert exec_packet["mode"] == "execute", exec_packet - assert exec_packet["boundary"]["starts_visible_processes"] is True, exec_packet - assert exec_packet["boundary"]["writes_loopx_state"] is False, exec_packet - assert exec_packet["boundary"]["spends_loopx_quota"] is False, exec_packet - assert launch["schema_version"] == "multi_agent_visible_launch_result_v0", launch - assert launch["started_lanes"] == ["planner", "critic"], launch - assert launch["tmux_layout"] == { - "window_name": "roles", - "single_window": True, - "layout": "tiled", - "recording_friendly": True, - }, launch - assert launch["visible_acceptance"]["accepted"] is True, launch - assert launch["visible_acceptance"]["missing_lanes"] == [], launch - assert all( - item["interactive_codex_tui_script"] for item in launch["visible_acceptance"]["pane_checks"] - ), launch - skill_items = launch["worker_skill_materialization"] - assert skill_items and skill_items[0]["materialized"] is True, launch - assert (workspace / ".codex" / "skills" / "loopx-planner-worker" / "SKILL.md").is_file() - dry_wake = run_wake_command( - env, - registry=registry, - runtime_root=runtime_root, - session_name="loopx-generic-multi-agent-smoke", - execute=False, - lanes=["planner"], - ) - assert dry_wake["schema_version"] == "multi_agent_pane_a2a_wakeup_v0", dry_wake - assert dry_wake["mode"] == "dry_run", dry_wake - assert dry_wake["target_lanes"] == ["planner"], dry_wake - assert dry_wake["coordination_model"] == "decentralized_state_a2a", dry_wake - assert dry_wake["wakeup_model"] == "fixed_prompt_broadcast", dry_wake - assert dry_wake["driver_contract"]["owner_layer"] == "generic_multi_agent_kernel", dry_wake - assert dry_wake["driver_contract"]["broadcaster"]["selects_todo"] is False, dry_wake - assert dry_wake["workflow_driver"] is False, dry_wake - assert dry_wake["broadcaster_reads_frontier"] is False, dry_wake - assert dry_wake["broadcaster_selects_todo"] is False, dry_wake - assert "$LOOPX_PANE_A2A_TICK" in dry_wake["prompt"], dry_wake - assert "LoopX targeted wake" in dry_wake["prompt"], dry_wake - assert "do not inspect skills" in dry_wake["prompt"], dry_wake - assert "not research evidence or completion" in dry_wake["prompt"], dry_wake - assert "First run $LOOPX_PANE_A2A_TICK once" in dry_wake["prompt"], dry_wake - assert "\n" not in dry_wake["prompt"], dry_wake - exec_wake = run_wake_command( - env, - registry=registry, - runtime_root=runtime_root, - session_name="loopx-generic-multi-agent-smoke", - execute=True, - ) - assert exec_wake["mode"] == "execute", exec_wake - assert exec_wake["target_lanes"] == ["planner", "critic"], exec_wake - assert [item["retry_count"] for item in exec_wake["prompt_submit_checks"]] == [ - 0, - 0, - ], exec_wake - assert exec_wake["boundary"]["writes_loopx_state"] is False, exec_wake - assert exec_wake["boundary"]["runs_worker_turn_directly"] is False, exec_wake - assert_public_safe( - { - "product_spec": exec_packet["product_spec"], - "schema_version": launch["schema_version"], - "launcher": launch["launcher"], - "started_lanes": launch["started_lanes"], - "workspace_mode": launch["workspace_mode"], - "attach_requested": launch["attach_requested"], - }, - "launch public summary", - ) - - log_entries = [json.loads(line) for line in tmux_log.read_text(encoding="utf-8").splitlines()] - env_snapshots = [ - entry["env"] - for entry in log_entries - if entry["env"].get("LOOPX_REGISTRY") - ] - assert env_snapshots, log_entries - for snapshot in env_snapshots: - assert snapshot["LOOPX_REGISTRY"] == str(registry), snapshot - assert snapshot["LOOPX_RUNTIME_ROOT"] == str(runtime_root), snapshot - assert Path(snapshot["LOOPX_PROJECT"]).resolve() == workspace.resolve(), snapshot - start_commands = [ - entry["argv"][-1] - for entry in log_entries - if entry["argv"][:1] in (["new-session"], ["split-window"]) - ] - assert len(start_commands) == 2, log_entries - start_payloads = [] - for command in start_commands: - command_path = Path(command) - start_payloads.append( - command_path.read_text(encoding="utf-8") if command_path.is_file() else command - ) - assert all("LOOPX_RESOLVED_CODEX_BIN" in command for command in start_payloads), start_payloads - assert all("LOOPX_CODEX_REASONING_EFFORT=high" in command for command in start_payloads), start_payloads - assert all("exec python3 -c" in command for command in start_payloads), start_payloads - assert all("LOOPX_PANE_ARTIFACT_DIR" in command for command in start_payloads), start_payloads - assert all("LOOPX_PANE_TICK_SUMMARY" in command for command in start_payloads), start_payloads - assert all( - "LOOPX_PANE_TICK_OUTPUT_ARTIFACT" in command - for command in start_payloads - ), start_payloads - assert all("codex exec" not in command for command in start_payloads), start_payloads - wake_entries = [entry["argv"] for entry in log_entries if entry["argv"][:1] in (["set-buffer"], ["paste-buffer"], ["send-keys"])] - assert any(entry[:1] == ["set-buffer"] for entry in wake_entries), wake_entries - assert sum(1 for entry in wake_entries if entry[:1] == ["paste-buffer"]) == 2, wake_entries - assert sum(1 for entry in wake_entries if entry[:1] == ["send-keys"] and entry[-1] == "Enter") == 2, wake_entries - assert any("$LOOPX_PANE_A2A_TICK" in " ".join(entry) for entry in wake_entries), wake_entries - launcher_source = (ROOT / "demo/visible_multi_agent_launcher.py").read_text( - encoding="utf-8" - ) - contract_source = ( - ROOT / "demo/multi_agent/contract.py" - ).read_text( - encoding="utf-8" - ) - assert "scoped_loopx_wrapper" in launcher_source - assert "build_tui_multi_agent_runner_contract" in contract_source - assert "generic_multi_agent_compact_status_v0" in contract_source - assert "demo_local_wrapper" not in launcher_source - - smoke_source = Path(__file__).read_text(encoding="utf-8").lower() - domain_marker = "auto" + "-research" - assert domain_marker not in smoke_source, "generic launcher smoke should not depend on a domain demo" - - print("generic-multi-agent-one-command-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/generic-multi-agent-runner-kernel-smoke.py b/examples/generic-multi-agent-runner-kernel-smoke.py deleted file mode 100644 index a25a90fcd3..0000000000 --- a/examples/generic-multi-agent-runner-kernel-smoke.py +++ /dev/null @@ -1,164 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the generic multi-agent runner kernel module.""" - -from __future__ import annotations - -import json -import sys -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(ROOT)) - -from demo.multi_agent.contract import ( # noqa: E402 - GENERIC_MULTI_AGENT_COMPACT_STATUS_SCHEMA_VERSION, - GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, - THREE_LAYER_MINIMALITY_CONTRACT_SCHEMA_VERSION, - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - build_compact_human_status, - build_generic_role_profile, - build_three_layer_minimality_contract, - build_tui_multi_agent_runner_contract, - generic_role_prompt, - role_skill_profile, -) -from demo.multi_agent.runtime_scripts import ( # noqa: E402 - CODEX_TUI_EXEC_PY, - SCOPED_LOOPX_WRAPPER_PY, -) - - -def main() -> int: - skill = role_skill_profile({"name": "planner-skill", "source": "skills/planner/SKILL.md"}) - profile = build_generic_role_profile( - role_id="planner", - agent_id="codex-main-control", - scope="Plan one bounded handoff.", - responsibility="Turn shared state into the next runnable todo.", - handoff_hints=["Write a todo for builder when ready."], - skill_profile=skill, - ) - assert profile["schema_version"] == GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, profile - assert profile["required_skill"] == "planner-skill", profile - - prompt = generic_role_prompt( - goal_id="loopx-meta", - agent_id="codex-main-control", - role_id="planner", - scope=str(profile["agent_scope"]), - handoff_hints=["Write a todo for builder when ready."], - skill_name=str(profile["required_skill"]), - ) - assert "$LOOPX_PANE_A2A_TICK" in prompt, prompt - assert "$LOOPX_PANE_LOOPX_JSON" in prompt, prompt - assert "normal Codex CLI agent" in prompt, prompt - - runner = build_tui_multi_agent_runner_contract( - session_name="loopx-generic-team", - lane_count=2, - attach_command="tmux attach -t loopx-generic-team", - stop_command="tmux kill-session -t loopx-generic-team", - retry_command="rerun after quota/frontier refresh", - all_lane_workspace_isolation=False, - ) - assert runner["schema_version"] == TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, runner - assert runner["coordination_model"]["leader_required"] is False, runner - assert runner["pane_local_a2a"]["machine_json_policy"] == "file_or_explicit_machine_channel_only", runner - assert runner["boundaries"]["domain_specific_research_logic"] is False, runner - role_prompt = runner["role_prompt_and_skill"] - assert role_prompt["default_kernel_skills"] == ["loopx-project", "loopx-doc-registry"], runner - assert ( - role_prompt["default_kernel_skill_policy"]["owner_layer"] - == "generic_multi_agent_kernel" - ), runner - assert role_prompt["default_kernel_skill_policy"][ - "preset_should_not_repeat_skill_playbooks" - ] is True, runner - assert role_prompt["wake_prompt_owner"] == "generic_multi_agent_kernel", runner - assert ( - runner["decentralized_a2a_driver"]["prompt"]["owner_layer"] - == "generic_multi_agent_kernel" - ), runner - - compact = build_compact_human_status( - { - "goal_id": "loopx-meta", - "mode": "dry_run", - "session_name": "loopx-generic-team", - "commands": { - "attach": "tmux attach -t loopx-generic-team", - "stop": "tmux kill-session -t loopx-generic-team", - }, - "lanes": [ - { - "lane_id": "planner", - "role_id": "planner", - "agent_id": "codex-main-control", - "agent_scope": "Plan one bounded handoff.", - "pane_local_a2a": {"worker_turn_configured": True}, - }, - { - "lane_id": "builder", - "role_id": "builder", - "agent_id": "codex-side-bypass", - "agent_scope": "Execute one claimed todo.", - "pane_local_a2a": {"worker_loop_configured": True}, - }, - ], - } - ) - assert compact["schema_version"] == GENERIC_MULTI_AGENT_COMPACT_STATUS_SCHEMA_VERSION, compact - assert compact["role_count"] == 2, compact - assert compact["roles"][0]["tick"] == "$LOOPX_PANE_A2A_TICK", compact - assert compact["machine_json_policy"] == "artifact_only_in_visible_panes", compact - - layering = build_three_layer_minimality_contract( - product_id="customer-support", - preset_id="support_triage_preset", - user_intent_fields=["inbox", "rounds"], - preset_responsibilities=["triage_roles", "handoff_hints", "resolution_policy"], - extension_points=["role_overrides", "ticket_adapter"], - ) - assert layering["schema_version"] == THREE_LAYER_MINIMALITY_CONTRACT_SCHEMA_VERSION, layering - assert layering["principle"] == ( - "user_and_preset_stay_thin_kernel_owns_reusable_mechanics" - ), layering - assert layering["user_layer"]["fields"] == ["inbox", "rounds"], layering - assert "multi_agent_runner" in layering["kernel_layer"]["owns"], layering - assert layering["kernel_layer"]["default_skills"] == [ - "loopx-project", - "loopx-doc-registry", - ], layering - assert layering["kernel_layer"]["fixed_wake_prompt"] == ( - "PANE_LOCAL_A2A_WAKEUP_PROMPT" - ), layering - assert "pane_local_a2a_status_check" in layering["preset_layer"]["forbidden"], layering - assert ( - layering["preset_layer"]["role_skill_limit"] - == "role_specific_semantics_and_successor_declarations_only" - ), layering - assert layering["acceptance"]["other_multi_agent_products_can_reuse_kernel"] is True, layering - assert "LoopX machine JSON hidden" in SCOPED_LOOPX_WRAPPER_PY - assert "loopx-pane-a2a-tick" in SCOPED_LOOPX_WRAPPER_PY - assert "LOOPX_MACHINE_JSON=1 explicitly" in SCOPED_LOOPX_WRAPPER_PY - assert "LOOPX_CODEX_TRUST_WORKSPACE" in CODEX_TUI_EXEC_PY - assert "trust_level" in CODEX_TUI_EXEC_PY - - source = (ROOT / "demo/multi_agent/contract.py").read_text( - encoding="utf-8" - ) - runtime_source = ( - ROOT / "demo/multi_agent/runtime_scripts.py" - ).read_text( - encoding="utf-8" - ) - assert "auto" + "_research" not in source + runtime_source - assert "quickstart" not in (source + runtime_source).lower() - json.dumps(compact, sort_keys=True) - print("generic-multi-agent-runner-kernel-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/generic-multi-agent-spec-contract-smoke.py b/examples/generic-multi-agent-spec-contract-smoke.py deleted file mode 100644 index e2efeec43d..0000000000 --- a/examples/generic-multi-agent-spec-contract-smoke.py +++ /dev/null @@ -1,228 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the thin generic multi-agent spec contract. - -This intentionally avoids tmux/Codex execution. It checks that the user-facing -role spec is enough to materialize interactive TUI lanes while machine JSON -stays routed to public artifact files. -""" - -from __future__ import annotations - -import json -import sys -from pathlib import Path -from typing import Any - - -ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(ROOT)) - -from demo.visible_multi_agent_launcher import ( # noqa: E402 - INTERACTIVE_TUI_CONTRACT_SCHEMA_VERSION, - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - build_visible_multi_agent_payload_from_spec, -) -from demo.multi_agent.contract import ( # noqa: E402 - DECENTRALIZED_A2A_DRIVER_CONTRACT_SCHEMA_VERSION, - GENERIC_MULTI_AGENT_COMPACT_STATUS_SCHEMA_VERSION, - GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, - build_tui_multi_agent_runner_contract, -) - - -PRIVATE_MARKERS = [ - "byte" + "dance", - "lark" + "office", - "fei" + "shu.cn", - "/" + "Users" + "/", - "/" + "private" + "/", - "/" + "tmp" + "/", - "api" + "_key", - "pass" + "word", - "sec" + "ret", -] - - -def assert_public_safe(payload: dict[str, Any]) -> None: - text = json.dumps(payload, sort_keys=True).lower() - leaked = [marker for marker in PRIVATE_MARKERS if marker.lower() in text] - assert not leaked, leaked - - -def lane_by_id(payload: dict[str, Any], lane_id: str) -> dict[str, Any]: - for lane in payload["lanes"]: - if lane["lane_id"] == lane_id: - return lane - raise AssertionError(f"missing lane {lane_id}") - - -def main() -> int: - payload = build_visible_multi_agent_payload_from_spec( - { - "schema_version": "generic_multi_agent_launch_spec_v0", - "goal_id": "loopx-meta", - "session_name": "loopx-generic-spec-smoke", - "default_reasoning_effort": "medium", - "roles": [ - { - "lane_id": "planner", - "agent_id": "codex-main-control", - "role_id": "research-planner", - "scope": "Plan the next bounded research handoff.", - "skill": { - "name": "loopx-planner-worker", - "source": "skills/planner/SKILL.md", - }, - "handoff_hints": [ - "Write a LoopX todo for builder when the plan is ready." - ], - "worker_turn_command": "loopx auto-research worker-turn --dry-run", - }, - { - "lane_id": "builder", - "agent_id": "codex-side-bypass", - "role_id": "research-builder", - "scope": "Implement the claimed todo and write compact evidence.", - "skill": { - "name": "loopx-builder-worker", - "source": "skills/builder/SKILL.md", - }, - "handoff_hints": [ - "Complete the todo or hand a focused review todo back." - ], - "worker_turn_command": "loopx auto-research worker-turn --execute", - "reasoning_effort": "high", - }, - ], - }, - tmux_bin="tmux", - cli_bin="loopx", - codex_bin="codex", - ) - - assert payload["ok"] is True, payload - assert payload["schema_version"] == "multi_agent_visible_launcher_v0", payload - assert payload["mode"] == "dry_run", payload - assert payload["goal_id"] == "loopx-meta", payload - assert payload["product_spec"] == { - "schema_version": "generic_multi_agent_launch_spec_v0", - "input_shape": ["goal_id", "session_name", "roles"], - "role_fields": [ - "agent_id", - "role_id", - "scope", - "skill", - "handoff_hints", - "output_language", - "reasoning_effort", - "worker_turn_command", - "worker_loop_command", - ], - "role_count": 2, - "role_profile_schema_version": GENERIC_MULTI_AGENT_ROLE_PROFILE_SCHEMA_VERSION, - "uses_generic_runner": True, - "domain_specific": False, - }, payload - - runner = payload["runner_contract"] - assert runner["schema_version"] == TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, runner - assert runner["runner_surface"] == "tmux_codex_cli_tui", runner - assert runner["coordination_model"]["leader_required"] is False, runner - assert runner["coordination_model"]["state_bus"] == "loopx_registry_runtime_todo_quota_frontier", runner - assert runner["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", runner - assert runner["pane_local_a2a"]["machine_json_policy"] == "file_or_explicit_machine_channel_only", runner - assert runner["pane_local_a2a"]["machine_json_destination"] == "$LOOPX_PANE_ARTIFACT_DIR/*.public.json", runner - driver = runner["decentralized_a2a_driver"] - assert driver["schema_version"] == DECENTRALIZED_A2A_DRIVER_CONTRACT_SCHEMA_VERSION, driver - assert driver["owner_layer"] == "generic_multi_agent_kernel", driver - assert driver["coordination_pattern"] == "decentralized_state_a2a", driver - assert driver["schema_version"] == "multi_agent_decentralized_a2a_driver_contract_v1", driver - assert driver["broadcaster"]["model"] == "todo_readiness_targeted_wake", driver - assert driver["broadcaster"]["reads_todo_readiness"] is True, driver - assert driver["broadcaster"]["decides_work"] is False, driver - assert driver["pane"]["decision_owner"] == "codex_tui_agent_via_loopx_state", driver - assert driver["acceptance"]["user_and_preset_do_not_own_tick_driver"] is True, driver - assert runner["role_prompt_and_skill"]["default_kernel_skills"] == [ - "loopx-project", - "loopx-doc-registry", - ], runner - assert ( - runner["role_prompt_and_skill"]["worker_local_skill_scope"] - == "role_specific_semantics_only" - ), runner - assert runner["boundaries"]["domain_specific_research_logic"] is False, runner - direct_runner = build_tui_multi_agent_runner_contract( - session_name="loopx-direct-kernel-smoke", - lane_count=2, - attach_command="tmux attach -t loopx-direct-kernel-smoke", - stop_command="tmux kill-session -t loopx-direct-kernel-smoke", - retry_command="rerun after state refresh", - all_lane_workspace_isolation=False, - ) - assert direct_runner["schema_version"] == TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION - assert direct_runner["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK" - assert direct_runner["decentralized_a2a_driver"]["schema_version"] == ( - DECENTRALIZED_A2A_DRIVER_CONTRACT_SCHEMA_VERSION - ) - assert direct_runner["boundaries"]["domain_specific_research_logic"] is False - - tui = payload["interactive_tui_contract"] - assert tui["schema_version"] == INTERACTIVE_TUI_CONTRACT_SCHEMA_VERSION, tui - assert tui["codex_surface"] == "interactive_cli_tui", tui - assert tui["machine_json_policy"] == "file_or_explicit_machine_channel_only", tui - assert "raw_role_profile_json" in tui["forbidden_visible_content"], tui - assert "pre_codex_character_stream" in tui["forbidden_visible_content"], tui - - shared = payload["shared_goal_surface"] - assert shared["shared_state_route"] == "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", shared - assert shared["lane_identity_source"] == "role_profile_plus_agent_scoped_quota", shared - assert shared["all_lane_workspace_isolation"] is False, shared - - planner = lane_by_id(payload, "planner") - builder = lane_by_id(payload, "builder") - assert planner["role_profile"]["agent_scope"] == "Plan the next bounded research handoff.", planner - assert builder["role_profile"]["agent_scope"] == "Implement the claimed todo and write compact evidence.", builder - assert planner["role_profile"]["required_skill"] == "loopx-planner-worker", planner - assert builder["role_profile"]["required_skill"] == "loopx-builder-worker", builder - assert planner["pane_local_a2a"]["worker_turn_configured"] is True, planner - assert planner["pane_local_a2a"]["status_check_only"] is True, planner - assert planner["pane_local_a2a"]["counts_as_research_round"] is False, planner - assert builder["reasoning_effort"] == "high", builder - assert planner["reasoning_effort"] == "medium", planner - - for lane in (planner, builder): - command = lane["visible_launch_command"] - assert "LOOPX_CODEX_TUI_MODE=interactive" in command, lane - assert "LOOPX_PANE_A2A_TICK" in command, lane - assert "LOOPX_PANE_WORKER_TURN" in command, lane - assert "LOOPX_PANE_LOOPX_JSON" in command, lane - assert "LOOPX_PANE_ARTIFACT_DIR" in command, lane - assert "pane-a2a-tick.output.txt" in command, lane - assert "exec python3 -c" in command, lane - assert "codex exec" not in command, lane - assert "codex_stream_filter" not in command, lane - assert "raw_frontier_json" not in command, lane - assert "$LOOPX_PANE_LOOPX_JSON quota should-run" in lane["quota_guard"], lane - assert "> \"$LOOPX_PANE_ARTIFACT_DIR/quota.public.json\"" in lane["quota_guard"], lane - assert "tui_first_turn_quota_frontier_status_check" in lane["lane_timeline"], lane - - assert payload["cli_contract"]["machine_json_policy"] == "artifact_only_in_visible_panes", payload - compact = payload["compact_human_status"] - assert compact["schema_version"] == GENERIC_MULTI_AGENT_COMPACT_STATUS_SCHEMA_VERSION, compact - assert compact["role_count"] == 2, compact - assert compact["first_action"] == "$LOOPX_PANE_A2A_TICK", compact - assert compact["driver_model"] == "todo_readiness_edge_plus_fixed_retry", compact - assert compact["coordination_pattern"] == "decentralized_state_a2a", compact - assert compact["machine_json_policy"] == "artifact_only_in_visible_panes", compact - assert [role["lane_id"] for role in compact["roles"]] == ["planner", "builder"], compact - assert payload["boundary"]["starts_visible_processes"] is False, payload - assert payload["boundary"]["writes_loopx_state"] is False, payload - assert payload["boundary"]["spends_loopx_quota"] is False, payload - assert_public_safe(payload) - - print("generic-multi-agent-spec-contract-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/multi-agent-collective-round-ledger-smoke.py b/examples/multi-agent-collective-round-ledger-smoke.py deleted file mode 100644 index 53a4de40f1..0000000000 --- a/examples/multi-agent-collective-round-ledger-smoke.py +++ /dev/null @@ -1,216 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the generic multi-agent collective-round ledger.""" - -from __future__ import annotations - -import json -import sys -from pathlib import Path - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.multi_agent.collective_round_ledger import ( # noqa: E402 - MULTI_AGENT_COLLECTIVE_ROUND_LEDGER_SCHEMA_VERSION, - build_multi_agent_collective_round_ledger, -) - - -def assert_public_safe(payload: dict[str, object]) -> None: - text = json.dumps(payload, sort_keys=True) - forbidden = [ - "/" + "Users/", - "/" + "private/", - "/" + "tmp/", - "http://", - "https://", - "api" + "_key", - "pass" + "word", - "sec" + "ret", - ] - leaked = [needle for needle in forbidden if needle.lower() in text.lower()] - assert not leaked, leaked - - -def main() -> int: - ledger = build_multi_agent_collective_round_ledger( - source="smoke", - expected_lanes=[ - {"agent_id": "agent-a", "lane_id": "curator", "role_id": "research_curator"}, - {"agent_id": "agent-b", "lane_id": "runner", "role_id": "research_executor"}, - ], - lane_outcomes=[ - { - "round": 1, - "agent_id": "agent-a", - "selected_todo_id": "todo_contract", - "selected_action": "write_research_contract", - "executed": True, - "completion_status": "done", - }, - { - "round": 1, - "agent_id": "agent-b", - "selected_todo_id": "todo_dev", - "selected_action": "run_dev_eval", - "executed": True, - "completion_status": "done", - "dev_metric": 4.0, - "appended_count": 2, - }, - { - "round": 2, - "agent_id": "agent-a", - "executed": True, - "completion_status": "done", - }, - { - "round": 2, - "agent_id": "agent-b", - "selected_todo_id": "todo_holdout", - "selected_action": "run_holdout_eval", - "executed": True, - "completion_status": "done", - "holdout_metric": 4.5, - "appended_count": 1, - }, - ], - integrated_evidence={ - "evidence_event_count": 3, - "dev_metric": 4.0, - "holdout_metric": 4.5, - "dev_metric_sequence": [4.0], - "holdout_metric_sequence": [4.5], - "holdout_improvement_count": 1, - "protected_scope_clean": True, - }, - role_declared_successor_todos=[ - { - "todo_id": "todo_holdout", - "target_agent_id": "agent-b", - "target_role_id": "research_executor", - "source_todo_id": "todo_dev", - "action_kind": "run_holdout_eval", - } - ], - baseline_metric=1.0, - required_full_participation_round_count=1, - required_holdout_improvement_count=1, - ) - assert ledger["schema_version"] == MULTI_AGENT_COLLECTIVE_ROUND_LEDGER_SCHEMA_VERSION - assert ledger["owner_layer"] == "generic_multi_agent_kernel", ledger - assert ledger["coordination_model"] == "decentralized_state_a2a", ledger - assert ledger["round_unit"] == "collective_agent_pass", ledger - assert ledger["expected_lane_count"] == 2, ledger - assert ledger["expected_agent_ids"] == ["agent-a", "agent-b"], ledger - assert ledger["lane_outcome_count"] == 4, ledger - assert ledger["completed_lane_turn_count"] == 3, ledger - assert ledger["completed_turn_count_by_agent"] == {"agent-a": 1, "agent-b": 2}, ledger - assert ledger["collective_round_indexes"] == [1, 2], ledger - assert ledger["collective_round_count"] == 2, ledger - assert ledger["full_participation_round_indexes"] == [1], ledger - assert ledger["synchronous_full_participation_round_count"] == 1, ledger - assert ledger["asynchronous_full_participation_round_count"] == 1, ledger - assert ledger["full_participation_round_count"] == 1, ledger - assert ledger["full_participation_count_basis"] == "synchronous_and_asynchronous", ledger - assert ledger["full_participation_requirement_gap"] == { - "schema_version": "multi_agent_full_participation_gap_v0", - "required_count": 1, - "count_basis": "synchronous_and_asynchronous", - "completed_turn_count_by_agent": {"agent-a": 1, "agent-b": 2}, - "shortfall_by_agent": {}, - "missing_agent_count": 0, - "met": True, - }, ledger - assert ledger["full_participation_verified"] is False, ledger - assert ledger["multi_round_interaction_verified"] is True, ledger - assert ledger["integrated_evidence"]["evidence_event_count"] == 3, ledger - assert ledger["integrated_evidence"]["dev_metric"] == 4.0, ledger - assert ledger["integrated_evidence"]["holdout_metric"] == 4.5, ledger - assert ledger["integrated_evidence"]["dev_metric_sequence"] == [4.0], ledger - assert ledger["integrated_evidence"]["holdout_metric_sequence"] == [4.5], ledger - assert ledger["integrated_evidence"]["holdout_improvement_count"] == 1, ledger - verification = ledger["collective_research_verification"] - assert verification["schema_version"] == "multi_agent_collective_research_verification_v0" - assert verification["baseline_metric"] == 1.0, ledger - assert verification["completed_turn_count_by_agent"] == {"agent-a": 1, "agent-b": 2} - assert verification["full_participation_requirement_gap"]["met"] is True - assert verification["full_participation_requirement_met"] is True, ledger - assert verification["holdout_improvement_requirement_met"] is True, ledger - assert verification["verified"] is True, ledger - assert ledger["successor_todo_count"] == 1, ledger - assert ledger["role_declared_successor_todos"][0]["target_agent_id"] == "agent-b" - assert ledger["public_boundary"] == { - "raw_logs_recorded": False, - "private_artifacts_recorded": False, - "absolute_paths_recorded": False, - "credentials_recorded": False, - }, ledger - assert_public_safe(ledger) - - gap_ledger = build_multi_agent_collective_round_ledger( - source="smoke-gap", - expected_lanes=[ - {"agent_id": "agent-a", "lane_id": "curator", "role_id": "research_curator"}, - {"agent_id": "agent-b", "lane_id": "runner", "role_id": "research_executor"}, - ], - lane_outcomes=[ - { - "round": 1, - "agent_id": "agent-a", - "selected_todo_id": "todo_contract", - "selected_action": "write_research_contract", - "executed": True, - "completion_status": "done", - }, - { - "round": 1, - "agent_id": "agent-b", - "selected_todo_id": "todo_dev", - "selected_action": "run_dev_eval", - "executed": True, - "completion_status": "done", - }, - { - "round": 2, - "agent_id": "agent-a", - "selected_todo_id": "todo_review", - "selected_action": "review_research_contract", - "executed": True, - "completion_status": "done", - }, - { - "round": 2, - "agent_id": "agent-b", - "executed": True, - "completion_status": "done", - }, - ], - baseline_metric=1.0, - required_full_participation_round_count=2, - ) - assert gap_ledger["completed_turn_count_by_agent"] == { - "agent-a": 2, - "agent-b": 1, - }, gap_ledger - assert gap_ledger["full_participation_round_count"] == 1, gap_ledger - gap = gap_ledger["full_participation_requirement_gap"] - assert gap["count_basis"] == "synchronous_and_asynchronous", gap_ledger - assert gap["shortfall_by_agent"] == {"agent-b": 1}, gap_ledger - assert gap["missing_agent_count"] == 1, gap_ledger - assert gap["met"] is False, gap_ledger - assert ( - gap_ledger["collective_research_verification"][ - "full_participation_requirement_met" - ] - is False - ), gap_ledger - assert_public_safe(gap_ledger) - print("multi-agent-collective-round-ledger-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/multi-agent-product-recipe-smoke.py b/examples/multi-agent-product-recipe-smoke.py deleted file mode 100644 index 14f23ace00..0000000000 --- a/examples/multi-agent-product-recipe-smoke.py +++ /dev/null @@ -1,132 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-check the reusable multi-agent product recipe documentation.""" - -from __future__ import annotations - -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -RECIPE = ROOT / "docs" / "guides" / "multi-agent-product-recipe.md" -DOCS_INDEX = ROOT / "docs" / "README.md" -GUIDES_INDEX = ROOT / "docs" / "guides" / "README.md" -AUTO_RESEARCH_GUIDE = ROOT / "demo" / "auto_research" / "README.md" - - -PRIVATE_MARKERS = [ - "byte" + "dance", - "lark" + "office", - "fei" + "shu.cn", - "/" + "Users" + "/", - "/" + "private" + "/", - "/" + "tmp" + "/", - "api" + "_key", - "pass" + "word", - "sec" + "ret", -] - - -def read(path: Path) -> str: - assert path.is_file(), f"missing {path}" - return path.read_text(encoding="utf-8") - - -def require(text: str, snippets: list[str], *, source: Path) -> None: - compact = " ".join(text.split()) - missing = [ - snippet for snippet in snippets if snippet not in text and " ".join(snippet.split()) not in compact - ] - assert not missing, f"{source}: missing {missing}" - - -def assert_public_safe(text: str, label: str) -> None: - lower = text.lower() - leaked = [marker for marker in PRIVATE_MARKERS if marker.lower() in lower] - assert not leaked, f"{label} leaks private markers: {leaked}" - - -def main() -> int: - recipe = read(RECIPE) - docs_index = read(DOCS_INDEX) - guides_index = read(GUIDES_INDEX) - auto_research_guide = read(AUTO_RESEARCH_GUIDE) - - assert_public_safe(recipe, "multi-agent product recipe") - require( - recipe, - [ - "# Multi-Agent Product Recipe", - "User layer", - "Product preset", - "Multi-agent kernel", - "It is not enough to make the user command\nshort if the auto-research preset becomes a second runner", - "Both the user layer\nand the product preset must stay thin", - "role list", - "agent scope", - "worker-local skill snippet", - "$loopx-project", - "$loopx-doc-registry", - "should not copy\nthose generic project/doc-registry instructions", - "handoff/todo hints", - "One-Command Launch", - "Attach, Stop, Retry", - "loopx multi-agent launch", - 'loopx auto-research start "" --execute', - "real interactive Codex CLI TUI panes", - "first action is the pane-local A2A tick", - "machine JSON is written to public artifacts", - "todos and evidence are the only handoff authority", - "validated visible proof promotes the existing preset as\nthe reference recipe", - "fixed-prompt wake, pane-local A2A tick,\nrunner lifecycle, compact status, and public artifact routing stay in the\ngeneric multi-agent kernel", - "must not grow a private\nlauncher or hidden workflow driver", - "a promoted auto-research proof demonstrates the same one-command visible\n path", - "Auto-research should stay a reference preset, not the kernel", - "implement it in the\nmulti-agent kernel first", - ], - source=RECIPE, - ) - require( - docs_index, - ["guides/"], - source=DOCS_INDEX, - ) - require( - guides_index, - ["Multi-agent product recipe", "multi-agent-product-recipe.md"], - source=GUIDES_INDEX, - ) - require( - auto_research_guide, - [ - "Multi-agent product recipe", - "multi-agent-product-recipe.md", - "copy the pattern without copying auto-research code", - "The validated visible proof is promoted into this existing command path", - "the operator runs one command", - "the generic multi-agent kernel supplies runner, wake, pane-local tick, status", - "fixed-prompt wake causes each pane to run\nits local A2A tick", - "Concrete KNN visible command", - "`--preset knn-demo` is the public signal that LoopX should materialize a small\ngenerated KNN benchmark workspace", - "The visible panes get `solution.py` as the\nonly editable file", - "dev/holdout uplift is absent until visible roles write real public-safe\n evidence", - "it is not a claim that visible Codex panes authored the research result", - ], - source=AUTO_RESEARCH_GUIDE, - ) - - forbidden = [ - "auto-research owns the runner", - "preset owns tmux", - "hidden workflow engine", - "auto-research private launcher", - "raw JSON should be visible first", - ] - for phrase in forbidden: - assert phrase not in recipe, f"forbidden phrase present: {phrase}" - - print("multi-agent-product-recipe-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/multi-agent-role-successor-todos-smoke.py b/examples/multi-agent-role-successor-todos-smoke.py deleted file mode 100644 index 8b2682e3cf..0000000000 --- a/examples/multi-agent-role-successor-todos-smoke.py +++ /dev/null @@ -1,212 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test generic role-declared successor todo execution.""" - -from __future__ import annotations - -import json -import sys -import tempfile -from pathlib import Path - - -REPO_ROOT = Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from demo.multi_agent.role_successor import ( # noqa: E402 - MULTI_AGENT_ROLE_SUCCESSOR_TODOS_SCHEMA_VERSION, - apply_role_successor_todos, - first_successor_followup, -) -from loopx.control_plane.coordination.local_authority import read_canonical_todos_if_promoted # noqa: E402 -from loopx.control_plane.testing.canary_harness import import_fixture_authority # noqa: E402 - - -GOAL_ID = "multi-agent-successor-smoke" -SOURCE_TODO_ID = "todo_source123456" - - -def write_minimal_state(path: Path) -> None: - path.write_text( - "\n".join( - [ - "# Multi-Agent Successor Smoke", - "", - "## User Todo / Owner Review Reading Queue", - "", - "## Agent Todo", - "", - "- [ ] [P0] Run source action.", - " ", - "", - ] - ), - encoding="utf-8", - ) - - -def write_registry(path: Path, *, project: Path, state_file: Path, runtime_root: Path) -> None: - path.write_text( - json.dumps( - { - "common_runtime_root": str(runtime_root), - "goals": [ - { - "id": GOAL_ID, - "repo": str(project), - "state_file": str(state_file), - "coordination": { - "registered_agents": ["agent-a", "agent-b"], - }, - } - ], - }, - indent=2, - sort_keys=True, - ), - encoding="utf-8", - ) - - -def successor_spec(*, target_agent_id: str = "agent-b", condition: object = "always") -> dict[str, object]: - return { - "condition": condition, - "target_agent_id": target_agent_id, - "target_role_id": "evidence_runner", - "task_class": "advancement_task", - "action_kind": "run_holdout_eval", - "text": "[P0] Run holdout evidence.", - "todo_command_template": ( - "loopx todo add --goal-id {goal_id_shell} --role agent " - "--text {text_shell} --task-class {task_class_shell} " - "--action-kind {action_kind_shell} --claimed-by {target_agent_id_shell} " - "--unblocks-todo-id {source_todo_id_shell}" - ), - } - - -def main() -> int: - with tempfile.TemporaryDirectory() as temp_dir: - root = Path(temp_dir) - project = root / "project" - project.mkdir() - state_file = project / "ACTIVE_GOAL_STATE.md" - registry = root / "registry.json" - runtime_root = root / "runtime" - write_minimal_state(state_file) - write_registry(registry, project=project, state_file=state_file, runtime_root=runtime_root) - import_fixture_authority(registry, GOAL_ID) - before = read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) - display_before = state_file.read_bytes() - - preview = apply_role_successor_todos( - registry_path=registry, - goal_id=GOAL_ID, - source_todo_id=SOURCE_TODO_ID, - current_agent_id="agent-a", - role_id="evidence_runner", - action="run_dev_eval", - successor_specs=[successor_spec()], - decision_summary={"dev_promotion_candidate_count": 1}, - execute=False, - ) - assert preview["schema_version"] == MULTI_AGENT_ROLE_SUCCESSOR_TODOS_SCHEMA_VERSION, preview - assert preview["source"] == "role_profile_todo_command_template", preview - assert preview["needed"] is True, preview - assert preview["executed"] is False, preview - successor = preview["successors"][0] - assert successor["target_agent_id"] == "agent-b", successor - assert "--claimed-by agent-b" in str(successor["todo_command"]), successor - assert first_successor_followup(preview)["action_kind"] == "run_holdout_eval", preview - - skipped = apply_role_successor_todos( - registry_path=registry, - goal_id=GOAL_ID, - source_todo_id=SOURCE_TODO_ID, - current_agent_id="agent-a", - role_id="evidence_runner", - action="run_dev_eval", - successor_specs=[ - successor_spec( - condition={ - "all": [ - { - "path": "decision_summary.dev_promotion_candidate_count", - "op": "gt", - "value": 1, - "fail_reason": "no_second_candidate", - } - ] - } - ) - ], - decision_summary={"dev_promotion_candidate_count": 1}, - execute=False, - ) - assert skipped["needed"] is False, skipped - assert skipped["reason"] == "no_second_candidate", skipped - assert read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) == before - assert state_file.read_bytes() == display_before - - executed = apply_role_successor_todos( - registry_path=registry, - goal_id=GOAL_ID, - source_todo_id=SOURCE_TODO_ID, - current_agent_id="agent-a", - role_id="evidence_runner", - action="run_dev_eval", - successor_specs=[successor_spec()], - decision_summary={"dev_promotion_candidate_count": 1}, - execute=True, - ) - assert executed["executed"] is True, executed - added = executed["successors"][0] - assert added["added"] is True, added - assert added["claimed_by"] == "agent-b", added - state_text = state_file.read_text(encoding="utf-8") - assert "claimed_by=agent-b" in state_text, state_text - assert f"unblocks_todo_id={SOURCE_TODO_ID}" in state_text, state_text - after = read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) - assert after["source_authority"] == "file_v0", after - records = {todo["todo_id"]: todo for todo in after["todos"]} - assert len(records) == 2, records - assert records[added["todo_id"]]["claimed_by"] == "agent-b", records - assert records[added["todo_id"]]["unblocks_todo_id"] == SOURCE_TODO_ID, records - original_source = next(todo for todo in before["todos"] if todo["todo_id"] == SOURCE_TODO_ID) - assert records[SOURCE_TODO_ID] == original_source, records - replay = apply_role_successor_todos( - registry_path=registry, goal_id=GOAL_ID, source_todo_id=SOURCE_TODO_ID, - current_agent_id="agent-a", role_id="evidence_runner", action="run_dev_eval", - successor_specs=[successor_spec()], decision_summary={"dev_promotion_candidate_count": 1}, - execute=True, - ) - assert replay["successors"][0]["already_exists"] is True, replay - assert replay["successors"][0]["todo_id"] == added["todo_id"], replay - assert read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) == after - display_after = state_file.read_bytes() - - try: - apply_role_successor_todos( - registry_path=registry, - goal_id=GOAL_ID, - source_todo_id=SOURCE_TODO_ID, - current_agent_id="agent-a", - role_id="evidence_runner", - action="run_dev_eval", - successor_specs=[successor_spec(target_agent_id="missing-agent")], - decision_summary={}, - execute=False, - ) - except ValueError as exc: - assert "successor target_agent_id 'missing-agent' is not registered" in str(exc), exc - else: - raise AssertionError("missing successor target agent should fail closed") - assert read_canonical_todos_if_promoted(runtime_root=runtime_root, goal_id=GOAL_ID) == after - assert state_file.read_bytes() == display_after - - print("multi-agent-role-successor-todos-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/multi-agent-visible-launcher-protocol-smoke.py b/examples/multi-agent-visible-launcher-protocol-smoke.py deleted file mode 100644 index 2da1ccd502..0000000000 --- a/examples/multi-agent-visible-launcher-protocol-smoke.py +++ /dev/null @@ -1,217 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-check the generic multi-agent visible launcher protocol contract.""" - -from __future__ import annotations - -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -PROTOCOL = ROOT / "docs/reference/protocols/multi-agent-visible-launcher-v0.md" -THREE_LAYER = ROOT / "docs/reference/protocols/multi-agent-three-layer-minimality-v0.md" -LOCAL_PLAN = ROOT / "docs/reference/protocols/local-agent-launch-plan-v1.md" -AUTO_RESEARCH_PROFILE = ROOT / "docs/reference/protocols/auto-research-role-profile-v0.md" -AUTO_RESEARCH_GUIDE = ROOT / "demo/auto_research/README.md" -PROTOCOL_INDEX = ROOT / "docs/reference/protocols/README.md" -DOCS_INDEX = ROOT / "docs/README.md" - - -PRIVATE_MARKERS = [ - "byte" + "dance", - "lark" + "office", - "fei" + "shu.cn", - "/" + "Users" + "/", - "/" + "private" + "/", - "/" + "tmp" + "/", - "api" + "_key", - "pass" + "word", - "sec" + "ret", -] - - -def read(path: Path) -> str: - assert path.exists(), f"missing {path}" - return path.read_text(encoding="utf-8") - - -def require(text: str, snippets: list[str], *, source: Path) -> None: - compact = " ".join(text.split()) - missing = [ - snippet for snippet in snippets if snippet not in text and " ".join(snippet.split()) not in compact - ] - assert not missing, f"{source}: missing {missing}" - - -def assert_public_safe(text: str, label: str) -> None: - lower = text.lower() - leaked = [marker for marker in PRIVATE_MARKERS if marker.lower() in lower] - assert not leaked, f"{label} leaks private markers: {leaked}" - - -def main() -> int: - protocol = read(PROTOCOL) - three_layer = read(THREE_LAYER) - local_plan = read(LOCAL_PLAN) - auto_research_profile = read(AUTO_RESEARCH_PROFILE) - auto_research_guide = read(AUTO_RESEARCH_GUIDE) - protocol_index = read(PROTOCOL_INDEX) - docs_index = read(DOCS_INDEX) - changed_public_docs = "\n".join( - [ - protocol, - three_layer, - protocol_index, - ] - ) - - assert_public_safe(changed_public_docs, "multi-agent visible launcher docs") - - require( - protocol, - [ - "multi_agent_visible_launcher_v0", - "local_agent_launch_plan_v1", - "Domain capabilities", - "not become a leader agent", - "multi_agent_three_layer_minimality_contract_v0", - "both the user-facing recipe and the domain preset stay thin", - "Kernel Module", - "demo/multi_agent/", - "generic_multi_agent_role_profile_v0", - "multi_agent_three_layer_minimality_contract_v0", - "compact human status", - "runtime scripts for scoped LoopX wrappers", - "pane-local tick scripts", - "machine JSON wrapper policy", - "visible_multi_agent_launcher.py", - "Ownership Split", - "LoopX control plane", - "Multi-agent visible launcher", - "Host shell or app", - "schema_version", - "reasoning_contract", - "default_reasoning_effort", - "model_reasoning_effort", - "shared_goal_surface", - "LOOPX_REGISTRY_and_LOOPX_RUNTIME_ROOT", - "agent-scoped `quota should-run`", - "todo projection, frontier projection, and run history", - "public-safe evidence", - "all_lane_workspace_isolation=false", - "only mutating attempts require a claimed worktree", - "role_profile", - "quota_guard", - "frontier", - "bootstrap_message", - "visible_launch_command", - "reasoning_effort", - "lane_timeline", - "The pane title is cosmetic", - "Start Order", - "Prepare the role profile, scoped LoopX wrappers, and bootstrap prompt as\n local artifacts", - "Start one fresh interactive Codex CLI TUI per role window", - "Let the Codex role read quota/frontier/todo state through the pane-local\n LoopX wrapper inside the TUI", - "Host Controls", - "attach", - "stop", - "retry", - "visible acceptance proof", - "Interactive TUI Contract", - "no extra frontier/status JSON window by default", - "not `codex exec`", - "no pre-Codex character stream", - "Boundary", - "hidden_prompt_injection", - "public_safe_redaction", - "Domain Adapter Responsibilities", - "Acceptance Checks", - ], - source=PROTOCOL, - ) - require( - protocol, - [ - "dry-run mode starts no process, runs no agent, writes no LoopX state, and\n spends no quota", - "execute mode still writes state and spends quota only through normal LoopX\n writeback after validation", - "workspace isolation scoped to mutating attempts rather than\n splitting the shared goal surface", - "Visible Codex TUI panes must not default into a generated demo-local git\nworktree or control-plane repository", - "generated workspace trust prompt", - ], - source=PROTOCOL, - ) - forbidden = [ - "launcher owns promotion decisions", - "all lanes must use separate goal state", - "may hide guard output", - "is a hidden scheduler", - ] - for phrase in forbidden: - assert phrase not in protocol, f"forbidden phrase present: {phrase}" - - require( - three_layer, - [ - "multi_agent_three_layer_minimality_contract_v0", - "User layer", - "Preset layer", - "Kernel layer", - "user_and_preset_stay_thin_kernel_owns_reusable_mechanics", - "The goal is not only to minimize the user's snippet", - "Auto-research is one preset on top of the generic kernel", - "another multi-agent product can reuse the same kernel", - ], - source=THREE_LAYER, - ) - require( - local_plan, - [ - "local_agent_launch_plan_v1", - "mode=dry_run", - "It must not start a process", - ], - source=LOCAL_PLAN, - ) - require( - auto_research_profile, - [ - "Host launcher", - "Visible panes", - "attach/stop controls", - "The pane title is cosmetic", - "start one fresh interactive Codex CLI TUI per role", - ], - source=AUTO_RESEARCH_PROFILE, - ) - require( - auto_research_guide, - [ - "The panes share the same LoopX\ngoal surface", - "isolate only mutating research-executor attempts", - "Each Codex TUI role must route through its own quota/frontier/worker-turn path", - "first show the Codex CLI TUI", - "should not default into the demo-local\ncontrol-plane repository or generated lane worktrees", - ], - source=AUTO_RESEARCH_GUIDE, - ) - require( - protocol_index, - [ - "multi_agent_three_layer_minimality_contract_v0", - "multi-agent-three-layer-minimality-v0.md", - "multi_agent_visible_launcher_v0", - "multi-agent-visible-launcher-v0.md", - ], - source=PROTOCOL_INDEX, - ) - require( - docs_index, - ["reference/README.md"], - source=DOCS_INDEX, - ) - - print("multi-agent-visible-launcher-protocol-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/project/goal-vision-replan-contract-smoke.py b/examples/project/goal-vision-replan-contract-smoke.py index 3061606b08..8ea8116482 100644 --- a/examples/project/goal-vision-replan-contract-smoke.py +++ b/examples/project/goal-vision-replan-contract-smoke.py @@ -12,7 +12,6 @@ DOCS_INDEX = ROOT / "docs" / "README.md" STATE_MACHINE = ROOT / "docs" / "product" / "core-control-plane" / "state-machine.md" RULE_SEAM = ROOT / "docs" / "product" / "core-control-plane" / "rule-seam-map.md" -THREE_LAYER = ROOT / "docs" / "reference" / "protocols" / "multi-agent-three-layer-minimality-v0.md" SELF_REPAIR_SKILL = ROOT / "skills" / "loopx-self-repair" / "SKILL.md" REPAIR_PATTERNS = ROOT / "skills" / "loopx-self-repair" / "references" / "repair-patterns.md" @@ -56,7 +55,6 @@ def main() -> int: docs_index = read(DOCS_INDEX) state_machine = read(STATE_MACHINE) rule_seam = read(RULE_SEAM) - three_layer = read(THREE_LAYER) self_repair_skill = read(SELF_REPAIR_SKILL) repair_patterns = read(REPAIR_PATTERNS) @@ -68,7 +66,6 @@ def main() -> int: docs_index, state_machine, rule_seam, - three_layer, self_repair_skill, repair_patterns, ] @@ -177,16 +174,6 @@ def main() -> int: ], source=RULE_SEAM, ) - require( - three_layer, - [ - "per-agent vision/replan state", - "per-agent vision budgets", - "vision/replan state transitions", - "product-specific fork of per-agent vision/replan mechanics", - ], - source=THREE_LAYER, - ) require( self_repair_skill, [ diff --git a/examples/visible-codex-executable-resolution-smoke.py b/examples/visible-codex-executable-resolution-smoke.py deleted file mode 100644 index 373167c304..0000000000 --- a/examples/visible-codex-executable-resolution-smoke.py +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test durable Codex executable selection for visible agents.""" - -from __future__ import annotations - -import json -import os -import sys -import tempfile -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(ROOT)) - -from demo.multi_agent.codex_executable import ( # noqa: E402 - resolve_codex_executable, - write_codex_compatibility_shim, -) - - -def write_codex(path: Path, version: str) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text( - "#!/bin/sh\n" - f"if [ \"${{1:-}}\" = --version ]; then echo 'codex-cli {version}'; exit 0; fi\n" - "exit 0\n", - encoding="utf-8", - ) - path.chmod(0o755) - - -def version_order(version: str) -> tuple[int, int, int, int, str]: - release, separator, prerelease = version.partition("-") - major, minor, patch = (int(part) for part in release.split(".")) - return major, minor, patch, 1 if not separator else 0, prerelease - - -def main() -> int: - with tempfile.TemporaryDirectory(prefix="loopx-codex-resolution-smoke.") as tmp: - temp = Path(tmp) - path_bin = temp / "path-bin" - home = temp / "home" - old_codex = path_bin / "codex" - new_codex = home / ".npm-global" / "bin" / "codex" - explicit = temp / "explicit-codex" - write_codex(old_codex, "0.142.0-alpha.7") - write_codex(new_codex, "0.144.1") - write_codex(explicit, "0.141.0") - env = { - "PATH": str(path_bin), - "HOME": str(home), - } - - selected, packet = resolve_codex_executable("codex", env=env) - assert packet["selection_policy"] == ( - "newest_version_across_host_candidates" - ), packet - candidates = [ - item for item in packet["candidate_versions"] if item["version"] - ] - expected = max(candidates, key=lambda item: version_order(item["version"])) - assert packet["selected_source"] == expected["source"], packet - assert packet["selected_version"] == expected["version"], packet - expected_paths = { - "path": old_codex, - "user_npm_global": new_codex, - "chatgpt_app_bundle": Path( - "/Applications/ChatGPT.app/Contents/Resources/codex" - ), - } - assert Path(selected) == expected_paths[expected["source"]], (selected, packet) - candidate_versions = { - item["source"]: item["version"] for item in candidates - } - assert candidate_versions["path"] == "0.142.0-alpha.7", packet - assert candidate_versions["user_npm_global"] == "0.144.1", packet - assert packet["path_default_bypassed"] is True, packet - assert packet["path_frozen"] is True, packet - assert packet["candidate_count"] >= 2, packet - - selected_explicit, explicit_packet = resolve_codex_executable( - str(explicit), env=env - ) - assert Path(selected_explicit) == explicit, explicit_packet - assert explicit_packet["selection_policy"] == "explicit_authoritative" - assert explicit_packet["selected_version"] == "0.141.0" - - shim = write_codex_compatibility_shim( - directory=temp / "shim-bin", - executable=str(explicit), - ) - assert shim.name == "codex" and os.access(shim, os.X_OK), shim - assert str(explicit) in shim.read_text(encoding="utf-8"), shim - - public_packet = json.dumps(packet, sort_keys=True) - assert str(temp) not in public_packet, public_packet - assert not public_packet.startswith("/"), public_packet - - print("visible-codex-executable-resolution-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/visible-multi-agent-launcher-smoke.py b/examples/visible-multi-agent-launcher-smoke.py deleted file mode 100644 index c931d0fd0a..0000000000 --- a/examples/visible-multi-agent-launcher-smoke.py +++ /dev/null @@ -1,946 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test the reusable visible multi-agent launcher seam.""" - -from __future__ import annotations - -import json -import os -import pty -import select -import subprocess -import sys -import tempfile -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(ROOT)) - -from demo.multi_agent.runtime_scripts import ( # noqa: E402 - CODEX_TUI_EXEC_PY as _CODEX_TUI_EXEC_PY, - SCOPED_LOOPX_WRAPPER_PY as _SCOPED_LOOPX_WRAPPER_PY, -) -from demo.visible_multi_agent_launcher import ( # noqa: E402 - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION, - build_visible_multi_agent_payload, - build_visible_multi_agent_payload_from_spec, - execute_visible_multi_agent_launcher, - wake_visible_multi_agent_panes, -) - - -def write_executable(path: Path, content: str) -> None: - path.write_text(content, encoding="utf-8") - path.chmod(0o755) - - -def run_with_pty(command: list[str], *, env: dict[str, str]) -> tuple[int, str]: - master, slave = pty.openpty() - try: - proc = subprocess.Popen( - command, - env=env, - stdin=subprocess.DEVNULL, - stdout=slave, - stderr=slave, - close_fds=True, - ) - os.close(slave) - chunks: list[bytes] = [] - while True: - ready, _, _ = select.select([master], [], [], 0.2) - if master in ready: - try: - data = os.read(master, 4096) - except OSError: - data = b"" - if data: - chunks.append(data) - if proc.poll() is not None: - while True: - try: - data = os.read(master, 4096) - except OSError: - break - if not data: - break - chunks.append(data) - return proc.returncode or 0, b"".join(chunks).decode(errors="replace") - finally: - try: - os.close(master) - except OSError: - pass - - -def main() -> int: - auto_research_cli = (ROOT / "demo/auto_research/cli.py").read_text( - encoding="utf-8" - ) - launcher_source = (ROOT / "demo/visible_multi_agent_launcher.py").read_text( - encoding="utf-8" - ) - tmux_source = (ROOT / "demo/visible_multi_agent_tmux.py").read_text( - encoding="utf-8" - ) - contract_source = (ROOT / "demo/multi_agent/contract.py").read_text( - encoding="utf-8" - ) - runtime_source = ( - ROOT / "demo/multi_agent/runtime_scripts.py" - ).read_text( - encoding="utf-8" - ) - assert "from ..visible_multi_agent_launcher import" in auto_research_cli - assert "execute_visible_multi_agent_launcher" in auto_research_cli - forbidden_defs = [ - "def _launch_auto_research_with_tmux", - "def _tmux_visible_launch_acceptance", - "def _resolve_demo_workspace", - "def _resolve_auto_research_launcher", - ] - leaked_defs = [name for name in forbidden_defs if name in auto_research_cli] - assert not leaked_defs, leaked_defs - assert "demo_local_wrapper" not in launcher_source - assert "_SCOPED_LOOPX_WRAPPER_PY = r" not in launcher_source - assert "_CODEX_TUI_EXEC_PY = r" not in launcher_source - assert "SCOPED_LOOPX_WRAPPER_PY = r" in runtime_source - assert "CODEX_TUI_EXEC_PY = r" in runtime_source - assert "scoped_loopx_wrapper" in launcher_source - assert "LOOPX_PANE_LOOPX_JSON" in launcher_source - assert "LOOPX_PANE_ARTIFACT_DIR" in launcher_source - assert "LOOPX_PANE_A2A_TICK" in launcher_source - assert "loopx-pane-a2a-tick" in launcher_source - assert "LOOPX_PANE_WORKER_TURN" in launcher_source - assert "LOOPX_PANE_TICK_ROUNDS" not in launcher_source - assert "LOOPX_PANE_TICK_SLEEP_SECONDS" not in launcher_source - assert "LOOPX_PANE_TICK_SUMMARY" in launcher_source - assert "LOOPX_PANE_TICK_OUTPUT_ARTIFACT" in launcher_source - assert "LOOPX_PANE_BOOTSTRAP_PROMPT" in launcher_source - assert "auto-wake.public.jsonl" in launcher_source - assert "AUTO_WAKE_LOOP_SCHEMA_VERSION" in launcher_source - assert "loopx-build-codex-bootstrap-prompt" in runtime_source - assert "LoopX Agent Bootstrap Context" in runtime_source - assert "### Role First Move" in runtime_source - assert "visible_first_steps" in runtime_source - assert "### Continuation Target" in runtime_source - assert "before no-follow-up" in runtime_source - assert "codex-visible-first-prompt.public.txt" in launcher_source - assert "LOOPX_CODEX_FULL_BOOTSTRAP_ARTIFACT" in launcher_source - assert "不要把完整 bootstrap 合约当作用户首屏复述" in runtime_source - assert "等待 resume gate" in runtime_source - assert "waiting on a resume gate" in runtime_source - assert "只有发现未 gate 的本角色 open todo" in runtime_source - assert "ungated open todo for this role" in runtime_source - assert "selected_todo_id" in runtime_source - assert "claim_allowed_rule" in runtime_source - assert "Human output language:" in contract_source - assert "receives an initial or targeted wake prompt" in contract_source - assert "tick reads your quota/frontier; it runs a role worker only when" in contract_source - assert "manual research is required" in contract_source - assert "Treat `$LOOPX_PANE_TICK_SUMMARY` as previous pane-local evidence" in contract_source - assert "not a gate that cancels later targeted retries" in contract_source - assert "Use $loopx-project" in contract_source - assert "Use $loopx-doc-registry" in contract_source - assert "LOOPX_VISIBLE_FORCE_MARKDOWN" in launcher_source - assert "machine_json_command=$LOOPX_PANE_LOOPX_JSON artifact_dir=$LOOPX_PANE_ARTIFACT_DIR" in runtime_source - assert "$LOOPX_PANE_LOOPX_JSON is a command path, not an output file." in runtime_source - assert "It injects --format json unless you pass an explicit --format." in runtime_source - assert "Never write output to `$LOOPX_PANE_LOOPX_JSON`" in contract_source - assert "LoopX machine JSON hidden" in runtime_source - assert "LOOPX_ALLOW_TTY_JSON" in runtime_source - assert "stat.S_ISREG" in runtime_source - assert "LOOPX_MACHINE_JSON=1 explicitly" in runtime_source - assert "multi_agent_visible_interactive_tui_contract_v0" in contract_source - assert TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION in contract_source - assert "build_tui_multi_agent_runner_contract" in launcher_source - assert "LOOPX_CODEX_TUI_MODE=interactive" in launcher_source - assert "LOOPX_CODEX_TRUST_WORKSPACE" in launcher_source - assert "LOOPX_CODEX_BIN" in launcher_source - assert "LOOPX_CODEX_REASONING_EFFORT" in launcher_source - assert "export BOOTSTRAP_PROMPT" in launcher_source - assert "VISIBLE_PROMPT_ARTIFACT" in launcher_source - assert "pane_input_ready_verified" in tmux_source - assert "tmux_paste_buffer_after_codex_tui_first_turn_ready" in tmux_source - assert "prompt_submit_checks" in tmux_source - assert "TMUX_LANE_ID_OPTION" in launcher_source - assert "TMUX_LANE_ID_OPTION" in tmux_source - assert '"list-panes", "-s", "-t", session' in launcher_source - assert '"list-panes"' in tmux_source - assert '"-s"' in tmux_source and '"-t"' in tmux_source - assert "#{{pane_id}}" in tmux_source and "TMUX_LANE_ID_OPTION" in tmux_source - assert '"list-panes", "-a", "-t", session' not in launcher_source - assert '"list-panes", "-a", "-t", session' not in tmux_source - assert '"set-option", "-p", "-t", pane_id, TMUX_LANE_ID_OPTION, lane_id' in launcher_source - assert "_persist_codex_workspace_trust" in launcher_source - assert "rev-parse\", \"--show-toplevel" in runtime_source - assert "trust_level=" in runtime_source and "trusted" in runtime_source - assert "trust_prompt_blocked" in launcher_source - assert "Do you trust the contents of this directory?" in launcher_source - assert "pre_codex_character_stream" in launcher_source - assert "build_visible_frontier_command" not in launcher_source - assert 'FRONTIER_ARTIFACT_NAME="frontier.public.json"' not in launcher_source - assert "_VISIBLE_CODEX_STREAM_FILTER_PY" not in launcher_source - assert "_HUMAN_VIEW_PACKET_PY" not in launcher_source - assert "worker-local skill block hidden" not in launcher_source - assert "codex_stream_filter" not in launcher_source - assert " codex exec " not in launcher_source - - dry_packet = build_visible_multi_agent_payload( - goal_id="loopx-meta", - session_name="loopx-visible-launcher-contract-smoke", - lanes=[ - { - "lane_id": "planner", - "frontier": "true", - "visible_launch_command": "true", - } - ], - ) - assert dry_packet["commands"]["attach"] == "tmux attach -t loopx-visible-launcher-contract-smoke" - assert dry_packet["commands"]["stop"] == "tmux kill-session -t loopx-visible-launcher-contract-smoke" - assert "retry" in dry_packet["commands"], dry_packet - runner_contract = dry_packet["runner_contract"] - assert runner_contract["schema_version"] == TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION - assert runner_contract["coordination_model"]["leader_required"] is False, runner_contract - assert runner_contract["coordination_model"]["state_bus"] == ( - "loopx_registry_runtime_todo_quota_frontier" - ), runner_contract - assert runner_contract["tmux_lifecycle"]["one_window_per_role"] is False, runner_contract - assert runner_contract["tmux_lifecycle"]["single_window_tiled_role_panes"] is True, runner_contract - assert runner_contract["tmux_lifecycle"]["attach_command"] == dry_packet["commands"]["attach"] - assert runner_contract["tmux_lifecycle"]["stop_command"] == dry_packet["commands"]["stop"] - assert runner_contract["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK" - assert runner_contract["pane_local_a2a"]["cadence_wakeup_command"] == ( - "loopx multi-agent wake --session-name " - ) - assert runner_contract["pane_local_a2a"]["cadence_wakeup_model"] == "todo_readiness_targeted_wake" - assert runner_contract["pane_local_a2a"]["cadence_broadcaster_decides_work"] is False - driver = runner_contract["decentralized_a2a_driver"] - assert driver["schema_version"] == "multi_agent_decentralized_a2a_driver_contract_v1", driver - assert driver["owner_layer"] == "generic_multi_agent_kernel", driver - assert driver["driver_model"] == "todo_readiness_edge_plus_fixed_retry", driver - assert driver["broadcaster"]["reads_frontier"] is False, driver - assert driver["broadcaster"]["reads_todo_readiness"] is True, driver - assert driver["broadcaster"]["selects_todo"] is False, driver - assert driver["broadcaster"]["runs_worker_turn"] is False, driver - assert driver["pane"]["decision_owner"] == "codex_tui_agent_via_loopx_state", driver - assert driver["prompt"]["tick_summary_ref"] == "$LOOPX_PANE_TICK_SUMMARY", driver - assert "own_prior_status_summary" in driver["pane"]["reads"], driver - assert driver["pane"]["cadence_action"] == ( - "targeted_wakeup_then_own_quota_frontier_check_when_runnable" - ), driver - assert driver["prompt"]["wake_round"] == "fresh_agent_scoped_quota_frontier_check", driver - assert ( - driver["prompt"]["tick_summary_semantics"] - == "prior_pane_status_not_research_evidence" - ), driver - assert driver["acceptance"]["each_pane_decides_from_state"] is True, driver - assert driver["acceptance"]["tick_summary_does_not_gate_wake_tick"] is True, driver - assert driver["acceptance"]["tui_first_turn_owns_tick"] is True, driver - assert runner_contract["pane_local_a2a"]["first_action"] == ( - "Codex TUI first turn runs $LOOPX_PANE_A2A_TICK as a status check; " - "later live wakes check own quota/frontier when runnable" - ) - assert runner_contract["pane_local_a2a"]["counts_as_research_round"] is False - assert "LOOPX_PANE_WORKER_TURN" in runner_contract["lane_runtime_env"]["pane_tools"] - assert "LOOPX_PANE_ARTIFACT_DIR" in runner_contract["lane_runtime_env"]["pane_tools"] - assert "LOOPX_PANE_TICK_SUMMARY" in runner_contract["lane_runtime_env"]["pane_tools"] - assert ( - "LOOPX_PANE_TICK_OUTPUT_ARTIFACT" - in runner_contract["lane_runtime_env"]["pane_tools"] - ) - assert runner_contract["pane_local_a2a"]["machine_json_destination"] == ( - "$LOOPX_PANE_ARTIFACT_DIR/*.public.json" - ) - assert runner_contract["pane_local_a2a"]["status_artifact"] == ( - "$LOOPX_PANE_ARTIFACT_DIR/pane-a2a-status.public.json" - ) - assert runner_contract["pane_local_a2a"]["tick_summary"] == ( - "$LOOPX_PANE_TICK_SUMMARY" - ) - assert runner_contract["pane_local_a2a"]["tick_output"] == ( - "$LOOPX_PANE_TICK_OUTPUT_ARTIFACT" - ) - assert runner_contract["role_prompt_and_skill"]["default_kernel_skills"] == [ - "loopx-project", - "loopx-doc-registry", - ] - assert ( - runner_contract["role_prompt_and_skill"]["worker_local_skill_scope"] - == "role_specific_semantics_only" - ) - assert runner_contract["debug_artifacts"]["machine_json"] == ( - "redirected_public_artifacts_only" - ) - assert runner_contract["boundaries"]["domain_specific_research_logic"] is False - assert ( - dry_packet["interactive_tui_contract"]["schema_version"] - == "multi_agent_visible_interactive_tui_contract_v0" - ), dry_packet - assert dry_packet["interactive_tui_contract"]["runner_contract"] == ( - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION - ) - assert dry_packet["interactive_tui_contract"]["machine_json_policy"] == ( - "file_or_explicit_machine_channel_only" - ), dry_packet - assert dry_packet["interactive_tui_contract"]["codex_surface"] == "interactive_cli_tui", dry_packet - assert dry_packet["acceptance"]["runner_contract"] == ( - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION - ) - assert dry_packet["acceptance"]["machine_json_file_bound"] is True, dry_packet - assert dry_packet["acceptance"]["codex_tui_interactive"] is True, dry_packet - assert dry_packet["boundary"]["hidden_prompt_injection"] is False, dry_packet - assert dry_packet["boundary"]["spends_loopx_quota"] is False, dry_packet - assert dry_packet["boundary"]["all_lane_workspace_isolation"] is False, dry_packet - - generic_packet = build_visible_multi_agent_payload_from_spec( - { - "goal_id": "loopx-meta", - "roles": [ - { - "agent_id": "codex-side-bypass", - "role_id": "planner", - "scope": "plan one state-backed handoff", - "worker_turn_command": "printf turn-streamed", - } - ], - } - ) - assert generic_packet["runner_contract"]["schema_version"] == ( - TUI_MULTI_AGENT_RUNNER_CONTRACT_SCHEMA_VERSION - ) - assert generic_packet["runner_contract"]["tmux_lifecycle"]["lane_count"] == 1 - generic_lane = generic_packet["lanes"][0] - assert generic_lane["pane_local_a2a"]["tick_command"] == "$LOOPX_PANE_A2A_TICK", generic_lane - assert generic_lane["pane_local_a2a"]["worker_turn_configured"] is True, generic_lane - assert generic_lane["pane_local_a2a"]["auto_start"] is True, generic_lane - assert generic_lane["pane_local_a2a"]["auto_start_owner"] == ( - "codex_tui_first_turn_prompt" - ), generic_lane - assert generic_lane["pane_local_a2a"]["status_check_only"] is True, generic_lane - assert generic_lane["pane_local_a2a"]["counts_as_research_round"] is False, generic_lane - assert "tui_first_turn_quota_frontier_status_check" in generic_lane["lane_timeline"], generic_lane - assert "LOOPX_PANE_A2A_TICK" in generic_lane["visible_launch_command"], generic_lane - assert "LOOPX_PANE_WORKER_TURN" in generic_lane["visible_launch_command"], generic_lane - assert "LOOPX_PANE_ARTIFACT_DIR" in generic_lane["visible_launch_command"], generic_lane - assert "LOOPX_PANE_TICK_SUMMARY" in generic_lane["visible_launch_command"], generic_lane - assert "LOOPX_PANE_TICK_OUTPUT_ARTIFACT" in generic_lane["visible_launch_command"], generic_lane - assert "$LOOPX_PANE_ARTIFACT_DIR/quota.public.json" in generic_lane["quota_guard"], generic_lane - assert "LOOPX_PANE_TICK_ROUNDS=" not in generic_lane["visible_launch_command"], generic_lane - assert "LOOPX_PANE_TICK_SLEEP_SECONDS=" not in generic_lane["visible_launch_command"], generic_lane - assert "pane-a2a-tick.output.txt" in generic_lane["visible_launch_command"], generic_lane - - with tempfile.TemporaryDirectory() as temp_dir: - temp = Path(temp_dir) - fake_bin = temp / "bin" - fake_bin.mkdir() - registry = temp / "registry.json" - runtime_root = temp / "runtime" - workspace = temp / "workspace" - tmux_log = temp / "tmux.jsonl" - codex_home = temp / "codex-home" - codex_argv_log = temp / "codex-argv.json" - worker_skill = temp / "worker" / "SKILL.md" - wrapper_arg_log = temp / "wrapper-args.jsonl" - worker_skill.parent.mkdir(parents=True) - worker_skill.write_text("# Worker-local playbook\n", encoding="utf-8") - registry.write_text(json.dumps({"common_runtime_root": str(runtime_root)}), encoding="utf-8") - write_executable( - fake_bin / "loopx", - "\n".join( - [ - "#!/usr/bin/env python3", - "import json, os, sys", - "path = os.environ.get('WRAPPER_ARG_LOG')", - "if path:", - " with open(path, 'a', encoding='utf-8') as f:", - " f.write(json.dumps(sys.argv[1:]) + '\\n')", - "print('fake-loopx ' + ' '.join(sys.argv[1:]))", - "", - ] - ), - ) - write_executable( - fake_bin / "codex", - "\n".join( - [ - "#!/usr/bin/env python3", - "import json, os, sys", - "if '--version' in sys.argv:", - " print('codex-cli 9.9.9')", - " raise SystemExit(0)", - "path = os.environ.get('FAKE_CODEX_ARGV_LOG')", - "if path:", - " with open(path, 'w', encoding='utf-8') as f:", - " json.dump(sys.argv, f)", - "print('fake codex tui')", - "raise SystemExit(0)", - "", - ] - ), - ) - write_executable( - fake_bin / "tmux", - "\n".join( - [ - "#!/usr/bin/env python3", - "import json, os, sys", - "with open(os.environ['FAKE_TMUX_LOG'], 'a', encoding='utf-8') as f:", - " f.write(json.dumps(sys.argv[1:]) + '\\n')", - "if len(sys.argv) > 1 and sys.argv[1] == 'has-session':", - " raise SystemExit(1)", - "if len(sys.argv) > 1 and sys.argv[1] == 'list-windows':", - " print('planner\\nreviewer')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'list-panes':", - " print('%0\\tplanner\\tfour-up')", - " print('%1\\treviewer\\tfour-up')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'split-window':", - " lines = []", - " try:", - " with open(os.environ['FAKE_TMUX_LOG'], encoding='utf-8') as read_log:", - " lines = read_log.readlines()", - " except Exception:", - " pass", - " split_count = sum(1 for line in lines if 'split-window' in line)", - " print(f'%{split_count}')", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'capture-pane':", - " text = os.environ.get('FAKE_TMUX_CAPTURE_TEXT', '')", - " if text:", - " print(text)", - " raise SystemExit(0)", - "if len(sys.argv) > 1 and sys.argv[1] == 'display-message':", - " if '#{pane_id}' in sys.argv:", - " print('%0')", - " elif '#{pane_current_command}' in sys.argv:", - " print('codex')", - " raise SystemExit(0)", - "raise SystemExit(0)", - "", - ] - ), - ) - - original_path = os.environ.get("PATH", "") - original_codex_home = os.environ.get("CODEX_HOME") - os.environ["PATH"] = f"{fake_bin}{os.pathsep}{original_path}" - os.environ["CODEX_HOME"] = str(codex_home) - os.environ["FAKE_TMUX_LOG"] = str(tmux_log) - os.environ["FAKE_CODEX_ARGV_LOG"] = str(codex_argv_log) - os.environ["WRAPPER_ARG_LOG"] = str(wrapper_arg_log) - try: - scoped_env = dict(os.environ) - scoped_env.update( - { - "LOOPX_PROJECT": str(workspace), - "LOOPX_REAL_CLI": str(fake_bin / "loopx"), - "LOOPX_REGISTRY": str(registry), - "LOOPX_RUNTIME_ROOT": str(runtime_root), - "LOOPX_PANE_ARTIFACT_DIR": str(workspace / ".local" / "pane-artifacts" / "reviewer"), - } - ) - workspace.mkdir(parents=True, exist_ok=True) - subprocess.run( - [sys.executable, "-c", _SCOPED_LOOPX_WRAPPER_PY], - env=scoped_env, - check=True, - capture_output=True, - text=True, - ) - human = subprocess.run( - [str(workspace / ".local/bin/loopx"), "--format", "json", "status"], - env=scoped_env, - check=True, - capture_output=True, - text=True, - ) - visible_pipe = subprocess.run( - [str(workspace / ".local/bin/loopx-json"), "--format", "json", "status"], - env=scoped_env, - capture_output=True, - text=True, - ) - machine_artifact = workspace / ".local" / "pane-artifacts" / "reviewer" / "status.public.json" - machine_artifact.parent.mkdir(parents=True, exist_ok=True) - with machine_artifact.open("w", encoding="utf-8") as handle: - subprocess.run( - [str(workspace / ".local/bin/loopx-json"), "--format", "json", "status"], - env=scoped_env, - stdout=handle, - stderr=subprocess.STDOUT, - check=True, - text=True, - ) - implicit_machine_artifact = ( - workspace / ".local" / "pane-artifacts" / "reviewer" / "implicit-status.public.json" - ) - with implicit_machine_artifact.open("w", encoding="utf-8") as handle: - subprocess.run( - [str(workspace / ".local/bin/loopx-json"), "status"], - env=scoped_env, - stdout=handle, - stderr=subprocess.STDOUT, - check=True, - text=True, - ) - explicit_machine = subprocess.run( - [str(workspace / ".local/bin/loopx-json"), "status"], - env={**scoped_env, "LOOPX_MACHINE_JSON": "1"}, - check=True, - capture_output=True, - text=True, - ) - tty_status, tty_output = run_with_pty( - [str(workspace / ".local/bin/loopx-json"), "--format", "json", "status"], - env=scoped_env, - ) - tick = subprocess.run( - [str(workspace / ".local/bin/loopx-pane-a2a-tick")], - env={ - **scoped_env, - "LOOPX_PANE_LOOPX": str(workspace / ".local/bin/loopx"), - "LOOPX_GOAL_ID": "loopx-meta", - "LOOPX_AGENT_ID": "codex-side-bypass", - "LOOPX_ROLE_ID": "planner", - "LOOPX_PANE_WORKER_TURN": "printf worker-turn-streamed", - }, - check=True, - capture_output=True, - text=True, - ) - assert "machine_json_command=$LOOPX_PANE_LOOPX_JSON artifact_dir=$LOOPX_PANE_ARTIFACT_DIR" in human.stdout, human.stdout - assert "--format markdown status" in human.stdout, human.stdout - assert visible_pipe.returncode == 2, visible_pipe.stdout - assert "LoopX machine JSON hidden" in visible_pipe.stdout, visible_pipe.stdout - assert "LOOPX_PANE_ARTIFACT_DIR" in visible_pipe.stdout, visible_pipe.stdout - assert "command path, not an output file" in visible_pipe.stdout, visible_pipe.stdout - assert "injects --format json" in visible_pipe.stdout, visible_pipe.stdout - assert "fake-loopx" not in visible_pipe.stdout, visible_pipe.stdout - assert "--format json status" in machine_artifact.read_text(encoding="utf-8") - assert "--format json status" in implicit_machine_artifact.read_text(encoding="utf-8") - assert (workspace / ".local/bin/loopx-json").read_text(encoding="utf-8").startswith( - "#!/usr/bin/env python3" - ) - assert "--format json status" in explicit_machine.stdout, explicit_machine.stdout - assert tty_status == 2, tty_output - assert "LoopX machine JSON hidden" in tty_output, tty_output - assert "fake-loopx" not in tty_output, tty_output - assert "role=planner agent=codex-side-bypass" in tick.stdout, tick.stdout - assert "round 1/2" not in tick.stdout and "round 2/2" not in tick.stdout, tick.stdout - assert "quota should-run" in tick.stdout, tick.stdout - assert "--format markdown" in tick.stdout, tick.stdout - assert "worker-turn-streamed" in tick.stdout, tick.stdout - status_artifact = ( - workspace - / ".local" - / "pane-artifacts" - / "reviewer" - / "pane-a2a-status.public.json" - ) - status_payload = json.loads(status_artifact.read_text(encoding="utf-8")) - assert status_payload["schema_version"] == "pane_local_a2a_status_check_v0", status_payload - assert status_payload["coordination_model"] == "decentralized_state_a2a", status_payload - assert status_payload["workflow_driver"] is False, status_payload - assert status_payload["status_check_count"] == 1, status_payload - assert status_payload["counts_as_research_round"] is False, status_payload - assert status_payload["worker_configured"] is True, status_payload - assert [item["check_index"] for item in status_payload["checks"]] == [1], status_payload - assert status_payload["public_boundary"]["raw_logs_recorded"] is False, status_payload - assert "/" + "tmp/" not in json.dumps(status_payload, sort_keys=True), status_payload - - try: - execute_visible_multi_agent_launcher( - payload={ - "session_name": "loopx-visible-launcher-missing-skill-smoke", - "lanes": [ - { - "lane_id": "reviewer", - "visible_launch_command": "true", - "role_profile": { - "required_skill": "loopx-auto-research", - "worker_skill_source": "worker/MISSING.md", - }, - }, - ], - }, - registry=registry, - runtime_root=runtime_root, - requested_launcher="tmux", - tmux_bin="tmux", - cli_bin="loopx", - codex_bin="codex", - attach=False, - replace_existing=False, - workspace=str(temp / "missing-skill-workspace"), - create_workspace=True, - cwd=temp, - ) - raise AssertionError("missing worker-local skill source should fail closed") - except ValueError as exc: - assert "worker-local skill materialization failed" in str(exc), exc - assert "worker/MISSING.md" in str(exc), exc - assert not tmux_log.exists(), tmux_log - - launch, chosen, workspace_mode = execute_visible_multi_agent_launcher( - payload={ - "session_name": "loopx-visible-launcher-smoke", - "lanes": [ - { - "lane_id": "planner", - "frontier": "true", - "visible_launch_command": "exec codex -c model_reasoning_effort=high -C \"$LOOPX_PROJECT\" planner", - }, - { - "lane_id": "reviewer", - "visible_launch_command": "exec codex -c model_reasoning_effort=high -C \"$LOOPX_PROJECT\" reviewer", - "role_profile": { - "required_skill": "loopx-auto-research", - "worker_skill_source": "worker/SKILL.md", - }, - }, - ], - }, - registry=registry, - runtime_root=runtime_root, - requested_launcher="tmux", - tmux_bin="tmux", - cli_bin="loopx", - codex_bin="codex", - attach=False, - replace_existing=False, - workspace=str(workspace), - create_workspace=True, - cwd=temp, - codex_trust_workspace=True, - auto_wake=True, - auto_wake_interval_seconds=5.0, - ) - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "╭────────────────────────╮\n" - "│ >_ OpenAI Codex │\n" - "│ model: gpt-5.5 high │\n" - "╰────────────────────────╯\n\n" - "› Write tests for @filename\n" - ) - wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - ) - assert wake["schema_version"] == "multi_agent_pane_a2a_wakeup_v0", wake - assert wake["mode"] == "execute", wake - assert wake["wakeup_model"] == "fixed_prompt_broadcast", wake - assert "LoopX targeted wake" in wake["prompt"], wake - assert "do not inspect skills" in wake["prompt"], wake - assert "not research evidence or completion" in wake["prompt"], wake - assert "First run $LOOPX_PANE_A2A_TICK once" in wake["prompt"], wake - assert wake["pane_input_ready_verified"] is True, wake - assert wake["prompt_delivery"] == ( - "tmux_paste_buffer_after_codex_tui_first_turn_ready" - ), wake - assert [item["lane"] for item in wake["pane_input_ready_checks"]] == [ - "planner", - "reviewer", - ], wake - assert [item["target"] for item in wake["pane_input_ready_checks"]] == [ - "%0", - "%1", - ], wake - assert all( - item["ready_marker"] == "codex_tui_first_turn_ready" - for item in wake["pane_input_ready_checks"] - ), wake - assert [item["retry_count"] for item in wake["prompt_submit_checks"]] == [ - 0, - 0, - ], wake - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "\n" - "› Implement {feature}\n\n" - " gpt-5.5 high · /tmp/loopx-visible-workspace\n" - ) - footer_ready_wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - ) - assert footer_ready_wake["pane_input_ready_verified"] is True, footer_ready_wake - assert footer_ready_wake["prompt_delivery"] == ( - "tmux_paste_buffer_after_codex_tui_first_turn_ready" - ), footer_ready_wake - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "\n" - "› quiet with a brief no-action note. This wake is not research evidence or " - "completion. LoopX state, not the scheduler, decides the work.\n\n" - " gpt-5.5 high · /tmp/loopx-visible-workspace\n" - ) - tail_retry_wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - ) - assert [item["retry_count"] for item in tail_retry_wake["prompt_submit_checks"]] == [ - 1, - 1, - ], tail_retry_wake - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "\n" - "› [Pasted Content 1010 chars]\n\n" - " gpt-5.5 high · /tmp/loopx-visible-workspace\n" - ) - pasted_content_retry_wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - ) - assert [ - item["retry_count"] - for item in pasted_content_retry_wake["prompt_submit_checks"] - ] == [1, 1], pasted_content_retry_wake - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "╭────────────────────────╮\n" - "│ >_ OpenAI Codex │\n" - "│ model: gpt-5.5 high │\n" - "╰────────────────────────╯\n\n" - "• Working (12s • esc to interrupt)\n" - ) - busy_wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - input_ready_timeout_seconds=0.25, - ) - assert busy_wake["ok"] is True, busy_wake - assert busy_wake["pane_input_ready_verified"] is False, busy_wake - assert busy_wake["prompt_delivery"] == "skipped_no_input_ready_panes", busy_wake - assert busy_wake["prompt_submit_checks"] == [], busy_wake - assert busy_wake["ready_lanes"] == [], busy_wake - assert busy_wake["not_ready_lanes"] == ["planner", "reviewer"], busy_wake - assert all( - item["not_ready_reason"] == "codex_tui_busy_or_not_ready" - for item in busy_wake["pane_input_ready_checks"] - ), busy_wake - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "╭────────────────────────╮\n" - "│ >_ OpenAI Codex │\n" - "│ model: gpt-5.6-sol high│\n" - "╰────────────────────────╯\n\n" - "• Working (45s • esc to interrupt)\n\n" - "› Queue another request\n\n" - " gpt-5.6-sol high · /tmp/loopx-visible-workspace\n" - ) - queueable_busy_wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - input_ready_timeout_seconds=0.25, - ) - assert queueable_busy_wake["prompt_delivery"] == ( - "skipped_no_input_ready_panes" - ), queueable_busy_wake - assert queueable_busy_wake["ready_lanes"] == [], queueable_busy_wake - assert queueable_busy_wake["prompt_submit_checks"] == [], queueable_busy_wake - before_usage_limit_log_count = len( - tmux_log.read_text(encoding="utf-8").splitlines() - ) - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = ( - "\n" - "■ You've hit your usage limit. Visit\n" - "https://chatgpt.com/codex/settings/usage\n" - "to purchase more credits or try again at Jul 12th, 2026 6:15 PM.\n\n" - "› LoopX pane-local A2A wakeup: read $LOOPX_CODEX_TUI_PROMPT_ARTIFACT\n\n" - " gpt-5.5 high · /tmp/loopx-visible-workspace\n" - ) - usage_limited_wake = wake_visible_multi_agent_panes( - session_name="loopx-visible-launcher-smoke", - tmux_bin="tmux", - lanes=["planner", "reviewer"], - execute=True, - input_ready_timeout_seconds=0.25, - ) - after_usage_limit_log_entries = [ - json.loads(line) - for line in tmux_log.read_text(encoding="utf-8").splitlines()[ - before_usage_limit_log_count: - ] - ] - assert usage_limited_wake["prompt_delivery"] == ( - "skipped_terminal_pane_backoff" - ), usage_limited_wake - assert usage_limited_wake["auto_wake_backoff_recommended"] is True, ( - usage_limited_wake - ) - assert usage_limited_wake["prompt_submit_checks"] == [], usage_limited_wake - assert all( - item["not_ready_reason"] == "codex_tui_usage_or_rate_limited" - and item["backoff_recommended"] is True - for item in usage_limited_wake["pane_input_ready_checks"] - ), usage_limited_wake - assert not any( - entry[:1] in (["paste-buffer"], ["send-keys"]) - for entry in after_usage_limit_log_entries - ), after_usage_limit_log_entries - - git_root_workspace = temp / "git-root" / "lanes" / "planner" - git_root_workspace.mkdir(parents=True, exist_ok=True) - subprocess.run( - ["git", "init", str(temp / "git-root")], - check=True, - capture_output=True, - text=True, - ) - prompt_artifact = temp / "bootstrap.prompt.txt" - prompt_artifact.write_text("planner prompt", encoding="utf-8") - subprocess.run( - [sys.executable, "-c", _CODEX_TUI_EXEC_PY], - env={ - **os.environ, - "LOOPX_CODEX_BIN": str(fake_bin / "codex"), - "LOOPX_PROJECT": str(git_root_workspace), - "LOOPX_CODEX_REASONING_EFFORT": "high", - "LOOPX_CODEX_TUI_PROMPT_ARTIFACT": str(prompt_artifact), - "LOOPX_CODEX_TRUST_WORKSPACE": "1", - }, - check=True, - capture_output=True, - text=True, - ) - codex_args = json.loads(codex_argv_log.read_text(encoding="utf-8")) - assert "-C" in codex_args and str(git_root_workspace) in codex_args, codex_args - assert codex_args[-1] == "planner prompt", codex_args - trust_args = [ - item - for index, item in enumerate(codex_args) - if index > 0 and codex_args[index - 1] == "-c" and ".trust_level=" in item - ] - assert any(str(git_root_workspace) in item for item in trust_args), codex_args - assert any(str(temp / "git-root") in item for item in trust_args), codex_args - - os.environ["FAKE_TMUX_CAPTURE_TEXT"] = "Do you trust the contents of this directory?" - blocked_launch, _, _ = execute_visible_multi_agent_launcher( - payload={ - "session_name": "loopx-visible-launcher-trust-block-smoke", - "lanes": [ - { - "lane_id": "planner", - "visible_launch_command": "exec codex -c model_reasoning_effort=high -C \"$LOOPX_PROJECT\" planner", - } - ], - }, - registry=registry, - runtime_root=runtime_root, - requested_launcher="tmux", - tmux_bin="tmux", - cli_bin="loopx", - codex_bin="codex", - attach=False, - replace_existing=False, - workspace=str(workspace), - create_workspace=True, - cwd=temp, - codex_trust_workspace=True, - ) - finally: - os.environ["PATH"] = original_path - if original_codex_home is None: - os.environ.pop("CODEX_HOME", None) - else: - os.environ["CODEX_HOME"] = original_codex_home - os.environ.pop("FAKE_TMUX_LOG", None) - os.environ.pop("FAKE_CODEX_ARGV_LOG", None) - os.environ.pop("WRAPPER_ARG_LOG", None) - os.environ.pop("FAKE_TMUX_CAPTURE_TEXT", None) - - assert chosen == "tmux", launch - assert workspace_mode == "explicit_workspace", launch - assert workspace.is_dir(), workspace - assert launch["schema_version"] == "multi_agent_visible_launch_result_v0", launch - assert launch["codex_trust_workspace"] is True, launch - assert launch["codex_trust_scope"] == "persisted_selected_workspace_and_git_root", launch - assert launch["codex_trust_config"]["trusted_path_count"] >= 1, launch - config_text = (codex_home / "config.toml").read_text(encoding="utf-8") - assert str(workspace) in config_text, config_text - skills = launch["worker_skill_materialization"] - assert skills == [ - { - "skill": "loopx-auto-research", - "source": "worker/SKILL.md", - "destination": ".codex/skills/loopx-auto-research/SKILL.md", - "materialized": True, - "workspace_count": 1, - "source_resolution": "source_root", - } - ], skills - assert (workspace / ".codex/skills/loopx-auto-research/SKILL.md").read_text( - encoding="utf-8" - ) == "# Worker-local playbook\n" - assert launch["started_lanes"] == ["planner", "reviewer"], launch - assert launch["tmux_layout"] == { - "window_name": "roles", - "single_window": True, - "layout": "tiled", - "recording_friendly": True, - }, launch - assert set(launch["started_lane_targets"]) == {"planner", "reviewer"}, launch - assert launch["surviving_lanes"] == ["planner", "reviewer"], launch - assert launch["script_mode"] == "runtime_local_files", launch - assert launch["launcher_script_count"] == 2, launch - assert launch["auto_wake"]["schema_version"] == "multi_agent_auto_wake_loop_v0", launch - assert launch["auto_wake"]["enabled"] is True, launch - assert launch["auto_wake"]["interval_seconds"] == 5.0, launch - assert launch["auto_wake"]["target_lanes"] == ["planner", "reviewer"], launch - assert launch["auto_wake"]["workflow_driver"] is False, launch - assert launch["auto_wake"]["broadcaster_reads_frontier"] is False, launch - assert launch["auto_wake"]["broadcaster_reads_todo_readiness"] is True, launch - assert launch["auto_wake"]["broadcaster_selects_todo"] is False, launch - assert launch["auto_wake"]["artifact"] == "auto-wake.public.jsonl", launch - codex_resolution = launch["codex_executable_resolution"] - assert codex_resolution["selected_version"] == "9.9.9", codex_resolution - assert codex_resolution["path_frozen"] is True, codex_resolution - assert "selected_path" not in codex_resolution, codex_resolution - acceptance = launch["visible_acceptance"] - assert acceptance["schema_version"] == "multi_agent_visible_launch_acceptance_v0", acceptance - assert acceptance["accepted"] is True, acceptance - assert acceptance["missing_lanes"] == [], acceptance - assert all(item["accepted"] for item in acceptance["pane_checks"]), acceptance - assert all(item["interactive_codex_tui_script"] for item in acceptance["pane_checks"]), acceptance - assert not any(item["trust_prompt_blocked"] for item in acceptance["pane_checks"]), acceptance - - blocked_acceptance = blocked_launch["visible_acceptance"] - assert blocked_acceptance["accepted"] is False, blocked_acceptance - assert blocked_acceptance["pane_checks"][0]["trust_prompt_blocked"] is True, blocked_acceptance - - log_entries = [json.loads(line) for line in tmux_log.read_text(encoding="utf-8").splitlines()] - assert any(entry[:1] == ["new-session"] for entry in log_entries), log_entries - assert sum(1 for entry in log_entries if entry[:1] == ["split-window"]) == 1, log_entries - assert any(entry[:3] == ["list-panes", "-s", "-t"] for entry in log_entries), log_entries - assert not any(entry[:2] == ["list-panes", "-a"] for entry in log_entries), log_entries - assert any( - entry[:4] == ["set-option", "-p", "-t", "%0"] - and entry[-2:] == ["@loopx_lane_id", "planner"] - for entry in log_entries - ), log_entries - assert any( - entry[:4] == ["set-option", "-p", "-t", "%1"] - and entry[-2:] == ["@loopx_lane_id", "reviewer"] - for entry in log_entries - ), log_entries - assert any(entry[:1] == ["select-layout"] and entry[-1] == "tiled" for entry in log_entries), log_entries - tmux_starts = [entry for entry in log_entries if entry[:1] in (["new-session"], ["split-window"])] - assert all("-lc" not in entry for entry in tmux_starts), tmux_starts - - print("visible-multi-agent-launcher-smoke ok") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/visible-multi-agent-state-aware-wake-smoke.py b/examples/visible-multi-agent-state-aware-wake-smoke.py deleted file mode 100644 index 4adb49f8d6..0000000000 --- a/examples/visible-multi-agent-state-aware-wake-smoke.py +++ /dev/null @@ -1,240 +0,0 @@ -#!/usr/bin/env python3 -"""Smoke-test readiness-targeted visible multi-agent wake scheduling.""" - -from __future__ import annotations - -import json -import os -import sys -import tempfile -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(ROOT)) - -from demo.multi_agent.runtime_scripts import ( # noqa: E402 - CODEX_TUI_EXEC_PY, -) -from demo.multi_agent.visible_wake_scheduler import ( # noqa: E402 - lane_agent_map, - run_state_aware_wake_loop, - todo_readiness_projection, -) - - -def write_executable(path: Path, content: str) -> None: - path.write_text(content, encoding="utf-8") - path.chmod(0o755) - - -def main() -> int: - lanes = [ - {"lane_id": "curator", "agent_id": "research-curator"}, - {"lane_id": "proposer", "agent_id": "hypothesis-proposer"}, - {"lane_id": "fallback-agent"}, - ] - agents = lane_agent_map(lanes) - assert agents == { - "curator": "research-curator", - "proposer": "hypothesis-proposer", - "fallback-agent": "fallback-agent", - }, agents - - initial = todo_readiness_projection( - { - "todos": [ - { - "todo_id": "todo_curate", - "status": "open", - "done": False, - "claimed_by": "research-curator", - }, - { - "todo_id": "todo_propose", - "status": "open", - "done": False, - "claimed_by": "hypothesis-proposer", - "resume_ready": False, - }, - ] - }, - lane_agents=agents, - ) - assert initial["recognized"] is True, initial - assert initial["goal_todo_count"] == 2, initial - assert initial["mapped_todo_count"] == 2, initial - assert initial["runnable_lane_tokens"] == {"curator": ("todo_curate",)}, initial - - successor = todo_readiness_projection( - { - "todos": [ - { - "todo_id": "todo_curate", - "status": "done", - "done": True, - "claimed_by": "research-curator", - }, - { - "todo_id": "todo_propose_v2", - "status": "open", - "done": False, - "claimed_by": "hypothesis-proposer", - "resume_ready": True, - }, - ] - }, - lane_agents=agents, - ) - assert successor["runnable_lane_tokens"] == { - "proposer": ("todo_propose_v2",) - }, successor - - unmapped = todo_readiness_projection( - {"todos": [{"todo_id": "todo_other", "claimed_by": "other-agent"}]}, - lane_agents=agents, - ) - assert unmapped["recognized"] is True, unmapped - assert unmapped["runnable_lane_tokens"] == {}, unmapped - - unclaimed = todo_readiness_projection( - { - "todos": [ - { - "todo_id": "todo_unclaimed", - "status": "open", - "done": False, - "task_class": "advancement_task", - "excluded_agents": ["hypothesis-proposer"], - } - ] - }, - lane_agents=agents, - ) - assert unclaimed["runnable_lane_tokens"] == { - "curator": ("todo_unclaimed",), - "fallback-agent": ("todo_unclaimed",), - }, unclaimed - - with tempfile.TemporaryDirectory(prefix="loopx-state-aware-wake-smoke.") as tmp: - temp = Path(tmp) - fake_bin = temp / "bin" - fake_bin.mkdir() - state = temp / "todo-state.json" - wake_log = temp / "wake-argv.json" - stop = temp / "stop" - artifact = temp / "auto-wake.public.jsonl" - state.write_text( - json.dumps( - { - "todos": [ - { - "todo_id": "todo_propose_v2", - "status": "open", - "done": False, - "claimed_by": "hypothesis-proposer", - "resume_ready": True, - } - ] - } - ), - encoding="utf-8", - ) - write_executable( - fake_bin / "tmux", - """#!/usr/bin/env python3 -import os -import sys -raise SystemExit(1 if os.path.exists(os.environ['FAKE_STOP']) else 0) -""", - ) - write_executable( - fake_bin / "loopx", - """#!/usr/bin/env python3 -import json -import os -import sys -args = sys.argv[1:] -if 'todo' in args and 'list' in args: - print(open(os.environ['FAKE_TODO_STATE'], encoding='utf-8').read()) - raise SystemExit(0) -if 'multi-agent' in args and 'wake' in args: - with open(os.environ['FAKE_WAKE_LOG'], 'w', encoding='utf-8') as handle: - json.dump(args, handle) - open(os.environ['FAKE_STOP'], 'w', encoding='utf-8').write('stop') - print(json.dumps({'ok': True, 'auto_wake_backoff_recommended': False})) - raise SystemExit(0) -raise SystemExit(2) -""", - ) - old_env = os.environ.copy() - os.environ.update( - { - "FAKE_STOP": str(stop), - "FAKE_TODO_STATE": str(state), - "FAKE_WAKE_LOG": str(wake_log), - } - ) - try: - run_state_aware_wake_loop( - cli_bin=str(fake_bin / "loopx"), - tmux_bin=str(fake_bin / "tmux"), - registry=temp / "registry.json", - runtime_root=temp / "runtime", - goal_id="demo-goal", - session="demo-session", - lane_agents={ - "curator": "research-curator", - "proposer": "hypothesis-proposer", - }, - initial_projection=initial, - artifact=artifact, - retry_interval_seconds=45.0, - poll_interval_seconds=0.5, - ) - finally: - os.environ.clear() - os.environ.update(old_env) - wake_args = json.loads(wake_log.read_text(encoding="utf-8")) - assert wake_args.count("--lane") == 1, wake_args - assert wake_args[wake_args.index("--lane") + 1] == "proposer", wake_args - events = [ - json.loads(line) - for line in artifact.read_text(encoding="utf-8").splitlines() - ] - assert events[0]["mode"] == "todo_readiness_edge_plus_fixed_retry", events - assert events[1]["trigger"] == "readiness_edge", events - assert events[1]["target_lanes"] == ["proposer"], events - - codex_args = temp / "codex-args.json" - prompt = temp / "prompt.txt" - prompt.write_text("must-not-run-yet", encoding="utf-8") - write_executable( - fake_bin / "codex", - """#!/usr/bin/env python3 -import json -import os -import sys -open(os.environ['FAKE_CODEX_ARGS'], 'w', encoding='utf-8').write(json.dumps(sys.argv[1:])) -""", - ) - env = { - **os.environ, - "FAKE_CODEX_ARGS": str(codex_args), - "LOOPX_CODEX_BIN": str(fake_bin / "codex"), - "LOOPX_PROJECT": str(temp), - "LOOPX_CODEX_REASONING_EFFORT": "high", - "LOOPX_CODEX_TUI_PROMPT_ARTIFACT": str(prompt), - "LOOPX_CODEX_INITIAL_PROMPT_ENABLED": "0", - } - import subprocess - - subprocess.run([sys.executable, "-c", CODEX_TUI_EXEC_PY], env=env, check=True) - assert "must-not-run-yet" not in json.loads(codex_args.read_text(encoding="utf-8")) - - print("visible multi-agent state-aware wake smoke passed") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/loopx/canary/planner.py b/loopx/canary/planner.py index d2ac7ed152..29af2a36eb 100644 --- a/loopx/canary/planner.py +++ b/loopx/canary/planner.py @@ -1050,51 +1050,6 @@ }, ], }, - { - "id": "auto-research-demo", - "title": "Auto-research demo and frontier route", - "purpose": ( - "Check the minimal auto-research kernel and shared frontier projection; " - "keep legacy visible/demo wrappers out of the default canary path." - ), - "catalog_families": ["Work Routing", "Evidence Lifecycle", "State And Boundary", "Human Decision"], - "trigger_hints": ( - "auto-research", - "auto research", - "demo-supervisor", - "demo e2e", - "frontier", - "visible launcher", - "demo/auto_research", - ), - "checks": [ - { - "command": "python3 examples/auto-research-minimal-kernel-smoke.py", - "tier": "default", - "reason": "checks the minimal evaluator-agnostic kernel and rejects public shortcut replay paths", - }, - { - "command": "python3 examples/decentralized-auto-research-frontier-smoke.py", - "tier": "default", - "reason": "checks shared frontier, evidence graph, and public boundary fixtures", - }, - { - "command": "python3 examples/auto-research-demo-supervisor-smoke.py", - "tier": "deep", - "reason": "samples the full dry-run supervisor packet and lane bootstrap contract", - }, - { - "command": "python3 examples/auto-research-rollout-readpath-smoke.py", - "tier": "deep", - "reason": "checks rollout event read-path projection into live evidence graphs", - }, - { - "command": "python3 examples/auto-research-live-evidence-capture-smoke.py", - "tier": "deep", - "reason": "checks compact live evidence capture fixtures for visible lanes", - }, - ], - }, { "id": "explore-harness", "title": "Explore Harness configuration and runtime", diff --git a/loopx/cli.py b/loopx/cli.py index b71955fec0..265fb7705c 100644 --- a/loopx/cli.py +++ b/loopx/cli.py @@ -13,7 +13,6 @@ register_automation_cadence_command, handle_automation_cadence_command, ) -import argparse import sys from .cli_commands.agent_capabilities import register_agent_capabilities @@ -218,46 +217,6 @@ ) -def _demo_not_available_message(command: str) -> str: - return ( - f"`{command}` is a LoopX **demo** showcase that ships only with a " - "source checkout (demo/), not in installed builds. Clone the loopx " - "repository to run it." - ) - - -def _register_demo_commands( - subparsers: argparse._SubParsersAction, - add_subcommand_format, -) -> None: - """Register demo-only commands backed by the non-shipped demo/ package. - - In a source checkout the full demo command surface is wired; in an - installed build the demo package is absent, so only a stub is registered and - dispatch reports the demo is not available. - """ - - for demo_command, module, register_name in ( - ("auto-research", "demo.auto_research.cli", "register_auto_research_commands"), - ("multi-agent", "demo.multi_agent_cli", "register_multi_agent_commands"), - ): - try: - register = getattr( - __import__(module, fromlist=[register_name]), - register_name, - ) - register(subparsers, add_subcommand_format) - except Exception: - stub = subparsers.add_parser( - demo_command, - help=( - f"(demo) {demo_command} showcase — requires a loopx " - "source checkout" - ), - ) - add_subcommand_format(stub) - - def build_parser() -> LoopXArgumentParser: parser, sub = build_cli_parser() @@ -333,8 +292,6 @@ def build_parser() -> LoopXArgumentParser: register_ml_experiment_commands(sub, add_subcommand_format) - _register_demo_commands(sub, add_subcommand_format) - register_turn_commands(sub, add_subcommand_format) register_host_mode_plan_command(sub, add_subcommand_format) register_preset_commands(sub, add_subcommand_format) @@ -396,12 +353,6 @@ def _main(argv: list[str] | None = None) -> int: if top_level_help_requested(raw_argv): print(render_concise_help(sys.argv[0] if argv is None else "loopx"), end="") return 0 - try: - from demo.auto_research.cli import rewrite_auto_research_question_argv - - raw_argv = rewrite_auto_research_question_argv(raw_argv) - except Exception: - pass # demo package absent in installed builds; no question rewrite parser = build_parser() args = parser.parse_args(raw_argv) from .usage_ping import select_operation @@ -880,34 +831,4 @@ def _main(argv: list[str] | None = None) -> int: if goal_amendment_proposal_result is not None: return goal_amendment_proposal_result - if args.command == "auto-research": - try: - from demo.auto_research.cli import handle_auto_research_command - - return handle_auto_research_command( - args, - registry_path=registry_path, - runtime_root_arg=args.runtime_root, - output_format=output_format, - print_payload=print_payload, - ) - except Exception: - print(_demo_not_available_message("auto-research")) - return 1 - - if args.command == "multi-agent": - try: - from demo.multi_agent_cli import handle_multi_agent_command - - return handle_multi_agent_command( - args, - registry_path=registry_path, - runtime_root_arg=args.runtime_root, - output_format=output_format, - print_payload=print_payload, - ) - except Exception: - print(_demo_not_available_message("multi-agent")) - return 1 - return 2 diff --git a/loopx/help_surface.py b/loopx/help_surface.py index f8c36b6d1a..3fa0c916d6 100644 --- a/loopx/help_surface.py +++ b/loopx/help_surface.py @@ -325,7 +325,6 @@ def register_command_reference(subparsers: argparse._SubParsersAction) -> None: "command": "loopx periodic-report", "purpose": "Resolve a portable weekly profile, evaluate report triggers, and compose provider-neutral run receipts.", }, - {"command": "loopx auto-research", "purpose": "Project public-safe research frontiers."}, { "command": "loopx deepresearch", "purpose": "Run a bounded, citation-auditable research ledger and report workflow.", @@ -334,7 +333,6 @@ def register_command_reference(subparsers: argparse._SubParsersAction) -> None: "command": "loopx agent-turn-recall", "purpose": "Prepare goal-scoped memory guidance for one admitted autonomous agent turn.", }, - {"command": "loopx multi-agent", "purpose": "Launch visible role-scoped Codex TUI agents."}, {"command": "loopx canary", "purpose": "Plan or run catalog-informed smoke profiles."}, { "command": "loopx performance-diagnosis --help", diff --git a/mkdocs.yaml b/mkdocs.yaml index ee2b8c3c65..d6dbf4fec2 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -99,7 +99,6 @@ nav: - Long-Running Coding Agents: guides/long-running-coding-agents.md - Minimal Custom Runtime Example: guides/minimal-custom-runtime-example.md - Custom Runner Integration: guides/custom-agent-runner-integration.md - - Auto Research Stop/Takeover/Wake: guides/auto-research-stop-takeover-wake-walkthrough.md - Concepts: - concepts/README.md - Interaction Patterns: concepts/interaction-pattern-catalog.md diff --git a/tests/test_auto_research_artifact_receipt.py b/tests/test_auto_research_artifact_receipt.py deleted file mode 100644 index 612852842b..0000000000 --- a/tests/test_auto_research_artifact_receipt.py +++ /dev/null @@ -1,558 +0,0 @@ -from __future__ import annotations - -from copy import deepcopy -from typing import Any - -from demo.auto_research.artifact_receipt import ( - AUTO_RESEARCH_ARTIFACT_RECEIPT_SCHEMA_VERSION, - build_auto_research_artifact_receipt, -) -from demo.auto_research.delivery_contract import ( - AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION, - normalize_auto_research_delivery_contract, -) -from demo.auto_research.evidence_packet import ( - build_auto_research_evidence_packet, - build_auto_research_rollout_events, -) -from demo.auto_research.research_state import ( - build_research_evidence_graph_from_rollout_events, -) -from demo.auto_research.terminal_result_contract import ( - build_peer_review_event, - build_terminal_decision_event, -) - - -GOAL_ID = "wish-artifact-fixture" -PRODUCER = "research-executor" -PROMOTER = "evaluator-promoter" -REVIEWER = "independent-reviewer" -REGISTERED_AGENTS = [PRODUCER, PROMOTER, REVIEWER] - - -def _research_contract() -> dict[str, Any]: - return { - "schema_version": "research_contract_v0", - "goal_id": GOAL_ID, - "research_objective": ( - "Test whether a reusable skill improves held-out tool use." - ), - "editable_scope": ["candidate_skill", "experiment_driver"], - "protected_scope": ["heldout_tasks", "metric_definition"], - "metric": { - "name": "heldout_success_rate", - "direction": "maximize", - "baseline": 0.5, - }, - "dev_eval": "run public dev evaluator", - "holdout_eval": "run public holdout evaluator", - "promotion_policy": "dev_and_holdout_improved", - } - - -def _delivery_contract( - *, - include_report: bool = True, - reentry_conditions: list[str] | None = None, -) -> dict[str, Any]: - artifacts = [ - { - "artifact_ref": "artifact:experiment", - "kind": "experiment", - "description": "Runnable comparison experiment.", - "required": True, - }, - ] - required_refs = ["artifact:experiment"] - if include_report: - artifacts.append( - { - "artifact_ref": "artifact:report", - "kind": "report", - "description": "Evidence-backed result and limitation report.", - "required": True, - } - ) - required_refs.append("artifact:report") - return { - "schema_version": AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION, - "contract_ref": "contract:skill-library-generalization", - "wish": { - "original_text": ( - "Verify whether a durable skill library improves general tool " - "use instead of memorizing one repository." - ), - "intent_summary": ( - "Compare a fixed base agent with and without the skill library " - "on held-out repositories." - ), - "assumptions": ["A public held-out task split is available."], - "non_goals": ["Do not train or deploy a production model."], - }, - "research_contract": _research_contract(), - "required_artifacts": artifacts, - "acceptance_criteria": [ - { - "criterion_id": "heldout_generalization", - "description": ( - "Held-out performance is terminally decided from clean " - "evidence and an independent review." - ), - "hypothesis_id": "hyp_generalization", - "required_artifact_refs": required_refs, - "requires_independent_review": True, - "required": True, - } - ], - "failure_policy": { - "fallback_artifact_refs": ["artifact:report"] - if include_report - else [], - "reentry_conditions": reentry_conditions - or ["Provide a new held-out repository split."], - }, - } - - -def _eval_result( - *, - split: str, - value: float, - metric_status: str, - artifact_refs: list[str], - failure_kind: str | None = None, -) -> dict[str, Any]: - return { - "schema_version": "auto_research_lightweight_eval_result_v0", - "split": split, - "metric": { - "name": "heldout_success_rate", - "direction": "maximize", - "value": value, - "baseline": 0.5, - }, - "eval_status": "scored", - "primary_metric_status": metric_status, - "failure_kind": failure_kind, - "artifact_refs": artifact_refs, - "protected_scope_clean": True, - "no_upload": True, - } - - -def _packet( - delivery_contract: dict[str, Any], - *, - supported: bool, - include_report: bool = True, -) -> dict[str, Any]: - refs = ["artifact:experiment"] - if include_report: - refs.append("artifact:report") - return build_auto_research_evidence_packet( - contract=delivery_contract, - eval_results=[ - _eval_result( - split="dev", - value=0.7 if supported else 0.4, - metric_status="improved" if supported else "regressed", - artifact_refs=refs, - failure_kind=None if supported else "mechanism_contradicted", - ), - _eval_result( - split="holdout", - value=0.65 if supported else 0.35, - metric_status="improved" if supported else "regressed", - artifact_refs=refs, - failure_kind=None if supported else "mechanism_contradicted", - ), - ], - hypothesis_id="hyp_generalization", - todo_id="todo_generalization", - agent_id=PRODUCER, - claimed_by=PRODUCER, - mechanism_family="skill_library_generalization", - hypothesis="The skill library improves held-out tool use.", - ) - - -def _terminal_events( - packet: dict[str, Any], - *, - supported: bool, -) -> list[dict[str, Any]]: - events = build_auto_research_rollout_events( - packet, - recorded_at="2026-08-25T00:00:00Z", - ) - graph = build_research_evidence_graph_from_rollout_events( - goal_id=GOAL_ID, - rollout_events=events, - ) - decision = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="hyp_generalization", - outcome="promoted" if supported else "retired", - reason="holdout_validated" if supported else "contradicted", - decided_by=PROMOTER, - evidence_refs=["decision:generalization"], - recorded_at="2026-08-25T00:01:00Z", - ) - review = build_peer_review_event( - evidence_graph=graph, - rollout_events=[*events, decision], - hypothesis_id="hyp_generalization", - reviewer_agent_id=REVIEWER, - verdict="approve", - evidence_refs=["review:independent"], - require_independent=True, - registered_agent_ids=REGISTERED_AGENTS, - recorded_at="2026-08-25T00:02:00Z", - ) - return [*events, decision, review] - - -def _tampered_evidence_lineage( - events: list[dict[str, Any]], - *, - field: str, - value: str | None, -) -> list[dict[str, Any]]: - tampered = deepcopy(events) - for event in tampered: - if event["event_kind"] != "research_evidence": - continue - if value is None: - event["details"].pop(field, None) - else: - event["details"][field] = value - return tampered - - -def _assert_lineage_tampering_fails_closed( - events: list[dict[str, Any]], -) -> None: - graph = build_research_evidence_graph_from_rollout_events( - goal_id=GOAL_ID, - rollout_events=events, - ) - assert graph["evidence_event_count"] == 0 - assert all(not node["artifact_refs"] for node in graph["nodes"]) - - receipt = build_auto_research_artifact_receipt( - delivery_contract=_delivery_contract(), - rollout_events=events, - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] != "verified" - assert receipt["acceptance_results"][0]["status"] != "verified" - assert "artifact:experiment" in receipt["artifacts"][ - "missing_required_refs" - ] - - -def test_delivery_contract_normalization_is_stable() -> None: - normalized = normalize_auto_research_delivery_contract(_delivery_contract()) - assert normalize_auto_research_delivery_contract(normalized) == normalized - assert normalized["wish"]["wish_id"].startswith("wish_") - assert normalized["contract_revision"].startswith("sha256:") - - -def test_legacy_research_contract_keeps_existing_event_shape() -> None: - packet = build_auto_research_evidence_packet( - contract=_research_contract(), - eval_results=[ - _eval_result( - split="dev", - value=0.7, - metric_status="improved", - artifact_refs=["artifact:experiment"], - ) - ], - hypothesis_id="hyp_generalization", - todo_id="todo_generalization", - agent_id=PRODUCER, - claimed_by=PRODUCER, - mechanism_family="skill_library_generalization", - hypothesis="The skill library improves held-out tool use.", - ) - events = build_auto_research_rollout_events(packet) - assert "delivery_contract" not in packet - assert [event["event_kind"] for event in events] == [ - "research_hypothesis", - "research_evidence", - ] - assert all( - "contract_revision" not in event.get("details", {}) - for event in events - ) - - -def test_verified_artifact_receipt_closes_the_delivery_contract() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=True) - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=_terminal_events(packet, supported=True), - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["schema_version"] == AUTO_RESEARCH_ARTIFACT_RECEIPT_SCHEMA_VERSION - assert receipt["status"] == "verified" - assert receipt["failure_feedback"] is None - assert receipt["contract"]["state"] == "current" - assert receipt["artifacts"]["missing_required_refs"] == [] - assert receipt["acceptance_results"][0]["status"] == "verified" - assert receipt["learning_disposition"] == "candidate" - assert receipt["boundary"]["user_acceptance_inferred"] is False - - -def test_terminally_unfulfilled_wish_returns_actionable_feedback() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=False) - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=_terminal_events(packet, supported=False), - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "not_fulfilled" - feedback = receipt["failure_feedback"] - assert feedback["failure_kinds"] == ["mechanism_contradicted"] - assert feedback["unmet_criteria"] == ["heldout_generalization"] - assert feedback["verified_boundary"] == [] - assert feedback["fallback_artifact_refs"] == ["artifact:report"] - assert feedback["reentry_conditions"] == [ - "Provide a new held-out repository split.", - "resolve:heldout_generalization:mechanism_contradicted", - ] - assert receipt["learning_disposition"] == "candidate" - - -def test_retired_result_without_required_review_remains_inconclusive() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=False) - events = build_auto_research_rollout_events(packet) - graph = build_research_evidence_graph_from_rollout_events( - goal_id=GOAL_ID, - rollout_events=events, - ) - decision = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="hyp_generalization", - outcome="retired", - reason="contradicted", - decided_by=PROMOTER, - ) - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=[*events, decision], - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "inconclusive" - assert receipt["failure_feedback"]["failure_kinds"] == [ - "independent_review_missing" - ] - - -def test_missing_artifact_is_partial_not_verified() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=True, include_report=False) - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=_terminal_events(packet, supported=True), - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "partial" - assert receipt["artifacts"]["missing_required_refs"] == ["artifact:report"] - assert receipt["failure_feedback"]["failure_kinds"] == [ - "required_artifact_missing" - ] - assert receipt["failure_feedback"]["missing_required_artifact_refs"] == [ - "artifact:report" - ] - - -def test_contract_change_invalidates_prior_terminal_result() -> None: - old_contract = _delivery_contract() - old_packet = _packet(old_contract, supported=True) - events = _terminal_events(old_packet, supported=True) - changed_contract = _delivery_contract( - reentry_conditions=[ - "Provide two new held-out repository families.", - ] - ) - receipt = build_auto_research_artifact_receipt( - delivery_contract=changed_contract, - rollout_events=events, - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "stale" - assert receipt["contract"]["state"] == "stale" - assert receipt["failure_feedback"]["failure_kinds"] == [ - "contract_revision_changed", - "required_artifact_missing", - ] - assert ( - receipt["failure_feedback"]["reentry_conditions"][0] - == "Provide two new held-out repository families." - ) - - -def test_latest_contract_revision_replaces_prior_hypothesis_evidence() -> None: - old_contract = _delivery_contract() - changed_contract = _delivery_contract( - reentry_conditions=["Provide two new held-out repository families."] - ) - old_events = build_auto_research_rollout_events( - _packet(old_contract, supported=True) - ) - new_events = build_auto_research_rollout_events( - _packet(changed_contract, supported=False) - ) - graph = build_research_evidence_graph_from_rollout_events( - goal_id=GOAL_ID, - rollout_events=[*old_events, *new_events], - ) - node = graph["nodes"][0] - assert graph["evidence_event_count"] == 2 - assert node["status"] == "contradicted" - assert node["contract_revision"] != old_events[0]["details"][ - "contract_revision" - ] - - -def test_tampered_evidence_wish_id_cannot_verify_same_revision() -> None: - contract = _delivery_contract() - events = _terminal_events(_packet(contract, supported=True), supported=True) - _assert_lineage_tampering_fails_closed( - _tampered_evidence_lineage( - events, - field="wish_id", - value="wish_tampered", - ) - ) - - -def test_tampered_evidence_contract_ref_cannot_verify_same_revision() -> None: - contract = _delivery_contract() - events = _terminal_events(_packet(contract, supported=True), supported=True) - _assert_lineage_tampering_fails_closed( - _tampered_evidence_lineage( - events, - field="contract_ref", - value="contract:tampered", - ) - ) - - -def test_partial_evidence_lineage_cannot_verify_same_revision() -> None: - contract = _delivery_contract() - events = _terminal_events(_packet(contract, supported=True), supported=True) - _assert_lineage_tampering_fails_closed( - _tampered_evidence_lineage( - events, - field="contract_ref", - value=None, - ) - ) - - -def test_partial_hypothesis_lineage_cannot_verify_or_raise_key_error() -> None: - contract = _delivery_contract() - events = deepcopy( - _terminal_events(_packet(contract, supported=True), supported=True) - ) - for event in events: - if event["event_kind"] != "research_hypothesis": - continue - event["details"].pop("contract_ref") - event["details"].pop("contract_revision") - _assert_lineage_tampering_fails_closed(events) - - -def test_missing_terminal_decision_is_inconclusive() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=True) - events = build_auto_research_rollout_events(packet) - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=events, - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "inconclusive" - assert receipt["failure_feedback"]["failure_kinds"] == [ - "terminal_decision_missing" - ] - - -def test_missing_contract_record_returns_reentry_feedback() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=True) - events = [ - event - for event in _terminal_events(packet, supported=True) - if event["classification"] != AUTO_RESEARCH_DELIVERY_CONTRACT_SCHEMA_VERSION - ] - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=events, - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "inconclusive" - assert "contract_record_missing" in receipt["failure_feedback"][ - "failure_kinds" - ] - assert receipt["failure_feedback"]["reentry_conditions"][0] == ( - "Provide a new held-out repository split." - ) - assert "record_current_delivery_contract" in receipt["failure_feedback"][ - "reentry_conditions" - ] - - -def test_independent_review_rejection_is_inconclusive_feedback() -> None: - contract = _delivery_contract() - packet = _packet(contract, supported=True) - events = build_auto_research_rollout_events(packet) - graph = build_research_evidence_graph_from_rollout_events( - goal_id=GOAL_ID, - rollout_events=events, - ) - decision = build_terminal_decision_event( - evidence_graph=graph, - hypothesis_id="hyp_generalization", - outcome="promoted", - reason="holdout_validated", - decided_by=PROMOTER, - ) - review = build_peer_review_event( - evidence_graph=graph, - rollout_events=[*events, decision], - hypothesis_id="hyp_generalization", - reviewer_agent_id=REVIEWER, - verdict="reject", - require_independent=True, - registered_agent_ids=REGISTERED_AGENTS, - ) - receipt = build_auto_research_artifact_receipt( - delivery_contract=contract, - rollout_events=[*events, decision, review], - registered_agent_ids=REGISTERED_AGENTS, - ) - assert receipt["status"] == "inconclusive" - assert receipt["failure_feedback"]["failure_kinds"] == [ - "independent_review_rejected" - ] - - -def test_delivery_contract_rejects_undeclared_artifact_reference() -> None: - contract = _delivery_contract() - contract["acceptance_criteria"][0]["required_artifact_refs"].append( - "artifact:undeclared" - ) - try: - normalize_auto_research_delivery_contract(contract) - except ValueError as exc: - assert "undeclared artifacts" in str(exc) - else: - raise AssertionError("undeclared artifact references must fail closed") diff --git a/tests/test_cli_entrypoint.py b/tests/test_cli_entrypoint.py index 01bb65aeca..44f6c15192 100644 --- a/tests/test_cli_entrypoint.py +++ b/tests/test_cli_entrypoint.py @@ -15,6 +15,32 @@ REPO_ROOT = Path(__file__).resolve().parents[1] +@pytest.mark.parametrize("module", ["loopx.entrypoint", "loopx.cli"]) +@pytest.mark.parametrize("command", ["auto-research", "multi-agent"]) +def test_retired_demo_commands_reject_execution_without_writing_state( + tmp_path: Path, module: str, command: str, +) -> None: + registry, runtime = write_command_fixture(tmp_path) + before = {p.relative_to(tmp_path): p.read_bytes() for p in tmp_path.rglob("*") if p.is_file()} + result = run_cli_main(module, [ + "--registry", str(registry), "--runtime-root", str(runtime), + command, "start" if command == "auto-research" else "launch", "--execute", + ], environment={"LOOPX_USAGE_PING": "0"}) + assert result.returncode == 2, result.stderr + assert "invalid choice" in result.stderr and command in result.stderr + assert not result.stdout + assert {p.relative_to(tmp_path): p.read_bytes() for p in tmp_path.rglob("*") if p.is_file()} == before + + +def test_command_reference_omits_retired_demo_launchers() -> None: + result = run_cli_main("loopx.entrypoint", ["commands", "--format", "json"]) + assert result.returncode == 0, result.stderr + reference = json.loads(result.stdout) + assert reference["groups"] + assert "loopx auto-research" not in result.stdout + assert "loopx multi-agent" not in result.stdout + + def test_machine_json_keeps_values_capture_and_human_renderer( capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/tests/test_worker_command_validation.py b/tests/test_worker_command_validation.py deleted file mode 100644 index 2b2240c50e..0000000000 --- a/tests/test_worker_command_validation.py +++ /dev/null @@ -1,110 +0,0 @@ -from __future__ import annotations - -import pytest - -from demo.visible_multi_agent_launcher import ( - build_visible_multi_agent_payload_from_spec, - validate_worker_command, -) - - -@pytest.mark.parametrize( - "command", - [ - None, - "", - "loopx turn run-once", - "python3 worker.py --flag 1", - "printf turn-streamed", - "/usr/bin/env python3", - ], -) -def test_validate_worker_command_accepts_safe_commands(command: str | None) -> None: - assert validate_worker_command(command, field="worker_turn_command") == ( - command if command is not None else "" - ) - - -@pytest.mark.parametrize( - "command", - [ - "evil; id", - "evil | cat", - "evil > /tmp/pwned", - "evil $(id)", - "`id`", - "printf 'x'", - "a\nb", - "a&b", - "a${x}", - "a b'c", - ], -) -def test_validate_worker_command_rejects_unsafe_commands(command: str) -> None: - with pytest.raises(ValueError, match="unsafe shell metacharacters"): - validate_worker_command(command, field="worker_turn_command") - - -@pytest.mark.parametrize( - "command", - [ - "ENV=x python3 worker.py", - "LD_PRELOAD=/tmp/x python3", - "evil *", - "evil ?", - "a\tb", - ], -) -def test_validate_worker_command_rejects_env_assignment_and_glob_variants( - command: str, -) -> None: - with pytest.raises(ValueError, match="unsafe shell metacharacters"): - validate_worker_command(command, field="worker_turn_command") - - -def test_validate_worker_command_rejects_input_redirect() -> None: - # Output redirect (>) is already covered; input redirect must fail closed too. - with pytest.raises(ValueError, match="unsafe shell metacharacters"): - validate_worker_command("evil < /tmp/x", field="worker_turn_command") - - -def test_spec_builder_rejects_unsafe_worker_command() -> None: - spec = { - "goal_id": "loopx-meta", - "roles": [ - { - "agent_id": "codex-side-bypass", - "worker_turn_command": "evil; id", - } - ], - } - - with pytest.raises(ValueError, match="unsafe shell metacharacters"): - build_visible_multi_agent_payload_from_spec(spec) - - -def test_auto_research_supervisor_visible_worker_turn_hook_is_safe() -> None: - from demo.auto_research.demo_supervisor import ( - build_auto_research_demo_supervisor_plan, - build_visible_worker_turn_command, - ) - from demo.auto_research.user_contract import ( - build_auto_research_preset_context, - ) - - command = build_visible_worker_turn_command( - goal_id="loopx-auto-research-safe-hook", - agent_id="auto-research-observer", - lane_count=1, - ) - assert validate_worker_command(command, field="worker_turn_command") == command - - payload = build_auto_research_demo_supervisor_plan( - goal_id="loopx-auto-research-safe-hook", - open_question="如何提升 KNN 精确近邻检索速度？", - preset_context=build_auto_research_preset_context("knn-demo"), - configure_visible_worker_turn=True, - ) - assert payload["auto_research"]["visible_worker_turn_configured"] is True - for lane in payload["lanes"]: - assert lane["pane_local_a2a"]["worker_turn_configured"] is True