diff --git a/README.md b/README.md index 267c16c..84f2b83 100644 --- a/README.md +++ b/README.md @@ -157,7 +157,7 @@ Sequenced bridge: convert the computational finding + your credential network (G

✅ Roadmap Execution

-**15/26 done (58%).** Each execution is a checkbox with a before→after eval; a box only ticks with a real result (`done` requires a non-pending before AND after — no evidence ⇒ not done, gated in CI). +**16/27 done (59%).** Each execution is a checkbox with a before→after eval; a box only ticks with a real result (`done` requires a non-pending before AND after — no evidence ⇒ not done, gated in CI). - [x] **E1** (P1 · Foundation) — Stand up the AI-native loop (repo + self-audit + gated `make check`) - [x] **E2** (P1 · Foundation) — Map the field into a verified knowledge base @@ -185,6 +185,7 @@ Sequenced bridge: convert the computational finding + your credential network (G - [x] **E24** (P1 · Foundation) — Hub Phase-2: JIT deep artifacts for one repo (pyaging) — knowledge graph + gated skill, SHA-stamped - [x] **E25** (P1 · Foundation) — Make /longevity-loop an invocable skill that uses every hub toolset (backbone + progressive disclosure) - [x] **E26** (P1 · Foundation) — Fan the hub out to Biolearn (North-Star toolset) — auto-wires into /longevity-loop +- [x] **E27** (P1 · Foundation) — External validation — register the loop's skills in HKUDS/OpenSpace, run 3 real tasks, read the quality records ### Eval reports — before → after @@ -216,6 +217,7 @@ Sequenced bridge: convert the computational finding + your credential network (G | ✅ E24 | a real, grounded, provenance-stamped KG + skill generated from a cited repo, with a freshness gate | hub had the recipe layer only; no deep artifact proving the JIT layer works | hub/pyaging/ — knowledge-graph.json (20 nodes/edges) + compute-aging-clocks SKILL.md + manifest (SHA c2b3000e3c2c); scripts/hub_gen.py gates integrity (make check) + checks live staleness (FRESH); grounded in pyaging's real API | | ✅ E25 | an invocable /longevity-loop skill that reaches all per-repo toolsets without bloat | hub toolsets existed (hub/*/SKILL.md) but nothing exposed them through one invocable skill | SKILL.md generated by scripts/skill_gen.py — backbone = the loop's own tools; progressive disclosure = hub toolsets (auto-discovered from hub/, 1 today: compute-aging-clocks) + a JIT recipe; drift-gated in make check; installed at ~/.claude/skills/longevity-loop | | ✅ E26 | the North-Star repo has a grounded KG + gated skill, auto-disclosed by /longevity-loop | hub had 1 exemplar (pyaging); Biolearn (the leaderboard/loader for E6/E7) had no toolset | hub/biolearn/ — knowledge-graph.json (19 nodes/edges) + biolearn-leaderboard SKILL.md (real API, ties to E6/E7) + manifest (SHA 0d714f5a0c0a, FRESH); skill_gen auto-discovered it → SKILL.md now discloses 2 toolsets, no hand-editing | +| ✅ E27 | the loop's skills register into a third-party quality-first hub and are selected/applied on real tasks (evidence, not vibes) | openspace-trial/ had a runbook + inventory but no results; the loop's skills had never been exercised outside this repo; the runbook's install step (pip install -e .) was unverified | source-installed OpenSpace v2.0.0 (py3.12, non-editable — editable install proven broken); 3 skills registered (trusted); 3 tasks all SUCCESS/completed; biolearn-leaderboard + longevity-loop judged 'skill applied' (2/3); evolution engine correctly NOOP (plans give no executed evidence); sandbox denied bash/write (blast radius contained). Verdict: trial-more | --- diff --git a/data/executions.yml b/data/executions.yml index 4d3a3a4..f2045e4 100644 --- a/data/executions.yml +++ b/data/executions.yml @@ -195,3 +195,12 @@ before: "hub had 1 exemplar (pyaging); Biolearn (the leaderboard/loader for E6/E7) had no toolset" after: "hub/biolearn/ — knowledge-graph.json (19 nodes/edges) + biolearn-leaderboard SKILL.md (real API, ties to E6/E7) + manifest (SHA 0d714f5a0c0a, FRESH); skill_gen auto-discovered it → SKILL.md now discloses 2 toolsets, no hand-editing" proof: "hub/biolearn/* · SKILL.md (2 toolsets) · make check" +- id: E27 + phase: "P1 · Foundation" + item: "External validation — register the loop's skills in HKUDS/OpenSpace, run 3 real tasks, read the quality records" + done: true + eval: + metric: "the loop's skills register into a third-party quality-first hub and are selected/applied on real tasks (evidence, not vibes)" + before: "openspace-trial/ had a runbook + inventory but no results; the loop's skills had never been exercised outside this repo; the runbook's install step (pip install -e .) was unverified" + after: "source-installed OpenSpace v2.0.0 (py3.12, non-editable — editable install proven broken); 3 skills registered (trusted); 3 tasks all SUCCESS/completed; biolearn-leaderboard + longevity-loop judged 'skill applied' (2/3); evolution engine correctly NOOP (plans give no executed evidence); sandbox denied bash/write (blast radius contained). Verdict: trial-more" + proof: "openspace-trial/RESULTS.md · RUNBOOK.md (verified install) · evidence.db + openspace.db quality records" diff --git a/openspace-trial/RESULTS.md b/openspace-trial/RESULTS.md new file mode 100644 index 0000000..3522af6 --- /dev/null +++ b/openspace-trial/RESULTS.md @@ -0,0 +1,55 @@ +# OpenSpace trial — results (verified 2026-07-20) + +Source-installed **HKUDS/OpenSpace** v2.0.0 (MIT) in a Python 3.12.13 venv, registered +longevity-loop's skills, ran 3 real tasks headless (`--no-ui --query`, model +`anthropic/claude-sonnet-4-5` via litellm), and read the quality records OpenSpace emits into +`.openspace/evidence.db` + `.openspace/openspace.db`. This is evidence, not vibes — the numbers below +are dumped straight from those DBs. + +## Setup outcome +- **Install:** `pip install .` (non-editable). Editable install was broken — see RUNBOOK.md gotcha. +- **Registered skills** (→ `.openspace/skills//SKILL.md`, all imported `trust_state=trusted`): + `longevity-loop`, `compute-aging-clocks`, `biolearn-leaderboard`. +- **Scope note (honest):** OpenSpace also auto-scanned `~/.claude/skills` and imported the whole + global set (**69** `skill_records` total — 10xgoal, graphify, etc.), not just the 3 project skills. + Worth knowing for scope/privacy; only the 3 project skills were exercised. + +## The 3 tasks — quality records +| Task | Skill targeted | Status | Skill applied? (OpenSpace judgment) | +|---|---|---|---| +| 1 · pyaging clock-panel plan | compute-aging-clocks | ✅ SUCCESS (`completed=True`, 3 iters) | **not formally judged** (`skills_judged=0`) — note: *"provided a concrete plan with exact pyaging API calls"*, attributed to model knowledge | +| 2 · Biolearn challenge-data + submission | biolearn-leaderboard | ✅ SUCCESS (2 iters) | **YES** — *"the skill was loaded and its instructions were directly used to construct the plan"*; trust observation = success | +| 3 · loop target + low-hanging fruit | longevity-loop | ✅ SUCCESS (2 iters) | **YES** — *"invoked successfully; the agent extracted the loop's target"*; trust observation = success | + +**Per-skill counters** (`skill_records`): biolearn-leaderboard applied 1 / completed 1 / fallbacks 0 · +longevity-loop applied 1 / completed 1 / fallbacks 0 · compute-aging-clocks 0/0/0 (registered, not +formally applied in these runs). + +**Tool reliability** (`tool_quality_records`): `Skill` 2/5 successful calls · `bash` 0/1 · `write` +0/2 · `ls` 1/1. + +**Evolution engine:** all analyses → **NOOP** (no skill mutation admitted). Recorded reasons: +*"procedure refs lack execution"*, *"no evolution suggestion"* — the plan/summarize tasks produced no +executed evidence to justify evolving a skill. Honest outcome, not a failure. + +## Two findings that matter +1. **OpenSpace's sandbox denied `bash`/`write` by default** (`permission_denied`). Good for blast + radius — the agent could not execute code or write files — but it's *why* the pyaging task + produced a plan rather than a computed clock panel. To run execution-heavy tasks, grant those + tool permissions explicitly. +2. **Skills register cleanly into a third-party quality-first hub and are selected + applied on + relevant tasks** — 2 of 3 with an explicit "skill applied" judgment, all 3 completed. Same + SKILL.md format as the longevity-loop hub, so no adaptation was needed. + +## Verdict (trial protocol §Success criteria) +- ✅ Quality records for 3 real tasks (evidence in the two DBs). +- ✅ Skills correctly selected/applied (biolearn + longevity-loop judged applied). +- ⚠️ No **validated evolution** — the engine correctly declined (plans give no executed evidence); + the criterion needs an execution-permitted task to be met. +- ⚠️ Tokens/latency not benchmarked vs a no-OpenSpace baseline this run. +- **Verdict: trial-more.** OpenSpace is a viable external validator of the loop's skills; the single + number that would decide *adopt* is a **validated, no-regression evolution on an + execution-permitted task** — not yet demonstrated. + +_Scratch dirs (external, not committed): OpenSpace at `~/Documents/Projects/OpenSpace`; run dir at +`~/Documents/Projects/openspace-run/.openspace/{evidence,openspace}.db` + `logs/recordings/`._ diff --git a/openspace-trial/RUNBOOK.md b/openspace-trial/RUNBOOK.md index 3c42f24..6b78307 100644 --- a/openspace-trial/RUNBOOK.md +++ b/openspace-trial/RUNBOOK.md @@ -9,11 +9,18 @@ regression/bloat/slowdown. Local-first; own-your-data. ``` git clone https://github.com/HKUDS/OpenSpace.git && cd OpenSpace python3.12 -m venv .venv && . .venv/bin/activate # Python 3.12 (3.14 may lack dep wheels) -pip install -e . -openspace-mcp --help # verify the REAL package +pip install . # NON-editable — see the gotcha below +openspace --help # verify the REAL package (headless CLI) ``` -Set `OPENSPACE_MODEL` / `OPENSPACE_LLM_*` to your provider (Anthropic key available). No cloud login -needed for the local-first run+evolve path (supply-chain: self-host, pin deps, read before run). +**⚠️ Do NOT use `pip install -e .` (editable) — verified broken 2026-07-20.** The PEP 660 editable +finder leaves `import openspace` as a namespace package (`__file__=None`) and the CLI dies with +`ModuleNotFoundError`. A plain non-editable `pip install .` (or `pip install --no-deps +--force-reinstall .`) maps the package correctly and the CLI works. + +**⚠️ The default `--tui` / `--doctor` path needs a built Node TUI artifact that isn't shipped** — +run **headless** with `openspace --no-ui --query ""`. Set `OPENSPACE_LLM_API_KEY` + +`OPENSPACE_MODEL` (litellm ids, e.g. `anthropic/claude-sonnet-4-5`). No cloud login needed for the +local-first run+evolve path (supply-chain: self-host, pin deps, read before run). ## Trial protocol (local-first) 1. Register this repo's skills (see `INVENTORY.md`) into OpenSpace.