diff --git a/AGENTS.md b/AGENTS.md index a9e71a3..522fa51 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,15 +1,35 @@ # Repository Instructions -This repository owns the canonical PathMX skills. - -- `pathmx` owns PathMX authoring and tooling. -- `path` owns the personal learning workflow and uses `pathmx` for syntax. -- `work-log/` holds design briefs for reference; it is not synced skill content. -- Keep all checked-in content self-contained. Do not depend on another checkout. -- Keep prose short and plain. -- Add only syntax supported by the pinned PathMX version and a local fixture. -- Add question syntax only when it passes the pinned fixture. -- Do not add general actions or spaceholders until they are ready for this skill. -- Test sync changes with temporary repositories, including check mode, conflicts, - containment, and rollback. -- Run `bun run check` before handoff. +This repository owns the canonical PathMX skills and the hosted bootstrap +instructions for a personal learning space. + +- `bootstrap.md` gets a nontechnical learner from an agent prompt to a working + learning repository. +- `pathmx` owns PathMX authoring, tooling, Player use, annotations, and review. +- `path` owns the agent-led personal learning workflow and uses `pathmx` for + authoring. +- `work-log/` keeps design history. It is not installed as skill content. +- The public `pathmx-learning-starter` is a consumer, not a source of truth. + +Keep checked-in content self-contained. Keep prose short and plain. Put core +procedure in `SKILL.md`; put detailed syntax and examples in references. + +Add only syntax supported by the pinned PathMX version and a local fixture. +Treat Player interactions, annotations, questions, components, routes, and CLI +claims as version-sensitive. Verify them against fixtures or the installed CLI. +The exact dependency in `package.json` is the fixture baseline. Keep bootstrap +instructions pointed at latest, but update this baseline only after the complete +suite passes. + +The learning workflow is buffered, not Block-at-a-time: + +- map the whole path as visible milestones; +- fully author the current 2–4-session module; +- let a learner finish a session without waiting for an agent; +- adapt at useful session or module boundaries; +- keep annotations and durable evidence in Sources. + +Keep learner fixtures fictional. Do not add real personal or sensitive data. + +Test sync changes with temporary repositories, including check mode, conflicts, +containment, and rollback. Run `bun run check` before handoff. diff --git a/README.md b/README.md index 2106054..3eb45d6 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,70 @@ # PathMX Skills -This is the canonical, self-contained collection of PathMX agent skills. +Canonical agent instructions for authoring PathMX and running a personal +learning space. + +Give an agent [the bootstrap instructions](./bootstrap.md) to create a new +learning repository. After setup, repository instructions and these two skills +carry the workflow: | Skill | Use | | --- | --- | -| `/pathmx` | PathMX syntax, tooling, authoring, review, Play, and verification. | -| `/path` | One opinionated, adaptive personal learning path. Uses `/pathmx` for authoring. | +| `/pathmx` | Author, play, review, and verify PathMX. Invoked automatically for PathMX work. | +| `/path` | Start or resume a buffered adaptive learning path for one learner. | + +Skills install under `.agents/skills/`. Codex discovers that directory +directly. Claude Code uses the matching `.claude/skills` discovery link and a +small `CLAUDE.md` that imports `AGENTS.md`. ## Develop -Requires Bun. Dependencies include a pinned PathMX CLI. +Requires Bun. The repository pins a published PathMX version so every syntax +claim and fixture is reproducible. `package.json` names that exact compatibility +baseline. It is the safe fallback for latest-after-verification updates, not +the version of this skills repository. ```sh bun install --frozen-lockfile bun run check ``` -Examples and syntax claims must pass the local checks. - -## Evals - -Use [the agent rubric](./evals/pathmx-skills.rubric.md) to score task output and -verification evidence. - -## Work log - -Design briefs that informed `/path` live under [work-log/](./work-log/). They -are reference notes, not synced skill packages. - ## Sync -Check a target without writing: +Check a target repository without writing: ```sh bun run sync-skills -- --check ``` -Apply the canonical copies: +Apply the canonical packages: ```sh bun run sync-skills -- --write ``` -Write mode manages only the skill packages declared in -`skills/manifest.json` and their Claude discovery links. It leaves unrelated -target content and skills alone. Edit canonical skill content here; a later -sync replaces edits made inside managed target copies. +Write mode owns only the packages declared in `skills/manifest.json` and their +Claude discovery links. It leaves unrelated target content and skills alone. + +## Evals + +The [eval harness](./evals/README.md) drives the real Codex CLI through a +multi-turn bootstrap and learning flow, grades the resulting repository, and +can add an independent structured model judge. + +```sh +bun run eval:check +bun run eval -- list +bun run eval -- run sql-beginner +bun run eval -- run sql-beginner --profile instruction-floor +``` + +## Design history and contributors + +Design briefs live under [work-log](./work-log/). They are reference notes, not +installed skill content. + +The buffered adaptive learning loop grew from early hands-on testing by Tram Le +and Mark Johnson, which exposed the limits of Block-by-Block curriculum +generation. Tram also contributed the original math, media, code, tooling, +styling, and adaptive-path reference work that this repository continues to +build on. diff --git a/ROADMAP.md b/ROADMAP.md index 0d91f81..fd011c0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,7 +1,39 @@ # Roadmap -- Refine durable question authoring through starter tests and evals. -- Add general actions after their authoring contract is ready. -- Add spaceholders after their authoring and generation flow is stable. -- Test scheduled learning check-ins after the `/path` loop has real use. -- Add more learning-path examples only when they cover a distinct scenario. +## Release gates + +- After the skill pack and learning starter pass candidate and published-flow + evals, audit their commit history and remove unwanted Claude-generated commit + attribution or trailers before the public release. Confirm the exact history + rewrite and affected remotes before changing published history. +- Sync the canonical skills and `AGENTS.md` instructions into + `pathmx-build-week-2026` and `pathmx-learning-starter`. Replace stale managed + copies in both repositories and verify them with the sync checker and each + repository's own checks. +- Rewrite or replace + `pathmx-build-week-2026/paths/guides/self-learning-manual-test.guide.md` so a + teammate can manually reproduce the automated scenario from `bootstrap.md` + in a fresh Codex Desktop task. Keep its learner messages and phase checks + aligned with the scored eval, then add Desktop-only observations for + permission requests, integrated Browser routing, Player appearance and Play + mode, annotations, waiting time, and the learner's sense of progress. Remove + the old direct-init flow and stale minimum-version language. +- Verify bootstrap and installed skills against the latest stable PathMX + release before publishing. The current candidate is `0.1.21`; keep the + bootstrap on `latest`, advance fixture baselines only after the full suite + passes, and do not leave prose or examples pinned to an older release. +- Review `evals/profiles.json` against the current Codex Desktop model guide + before release so the primary experience lane still matches the documented + default Power setting. + +## Later work + +- Forward-test bootstrap and both implicit skill triggers in clean Codex and + Claude Code sessions. +- Expand stable annotation review and reply workflows after field use. +- Add more learning-path examples only when they cover a distinct domain or + learner need. +- Explore an optional, more playful home and milestone map without making it a + dependency of the minimal starter. +- Add general actions or spaceholders only after their public authoring + contracts are stable and fixture-backed. diff --git a/bootstrap.md b/bootstrap.md new file mode 100644 index 0000000..fa672c6 --- /dev/null +++ b/bootstrap.md @@ -0,0 +1,138 @@ +# Start a PathMX Learning Space + +Follow these instructions as an agent. Carry out the setup instead of only +explaining it. Ask for permission when the environment requires it. + +The result is one private, local Git repository where one learner can keep +multiple learning paths, use the PathMX Player, and return to an agent over +time. + +## 1. Choose the location + +Ask where the learner wants the repository and what to call it. Suggest a short +name such as `my-learning-space`. Do not replace or merge into an existing +directory. + +## 2. Ensure Bun and PathMX are ready + +Check `bun --version`. If Bun is missing, ask to install it with the official +installer for the learner's operating system: + +```sh +# macOS or Linux +curl -fsSL https://bun.com/install | bash +``` + +```powershell +# Windows PowerShell +powershell -c "irm bun.sh/install.ps1|iex" +``` + +Open a fresh shell or update the current shell path when the installer asks. +Verify `bun --version` before continuing. + +If `pathmx` already exists, update its native command: + +```sh +pathmx self-update +``` + +Otherwise install the current native command through the latest published +package: + +```sh +bunx @fellowhumans/pathmx@latest self-update +``` + +Verify `pathmx --version` and retain that version for the project comparison +below. + +## 3. Create the learning repository + +Create the chosen directory from the official starter. This command also +installs the current official `/path` and `/pathmx` skills: + +```sh +pathmx init --template pathmx-learning-starter +cd +bun install --frozen-lockfile +``` + +Read `AGENTS.md` and, when present, the instructions for the current agent +harness. Confirm that `.agents/skills/path/SKILL.md` and +`.agents/skills/pathmx/SKILL.md` exist. + +Shell tool calls may not preserve a prior `cd`. Run every following project +command from the new repository explicitly, either by setting its working +directory or by using `cd && ...`. + +Initialize a fresh local history and make the scaffold commit. If Git identity +is not configured, explain that clearly and keep the staged scaffold as the +rollback point without inventing an identity. + +```sh +git init +git add . +git commit -m "Create personal PathMX learning space" +``` + +Do not create or push a remote unless the learner asks. Treat the repository as +private personal data. + +Compare the exact project dependency with the updated native `pathmx --version`. +When they already match, do not reinstall the same package; run `bun run check` +once and continue. When they differ, attempt to bring the project dependency +to the latest release: + +```sh +bun add --exact @fellowhumans/pathmx@latest +bun run check:candidate +``` + +Keep and commit that update only if the full build succeeds and the Player +tutorial, questions, annotations, and bundled component behavior pass a smoke +test. Then set `pathmxCompatibility.baseline` in `package.json` to the exact +installed dependency and run `bun run check` before committing. If verification +fails, run `git restore package.json bun.lock` to recover those files from the +scaffold commit or staged baseline, run `bun install --frozen-lockfile`, and +continue on that verified baseline. Report the incompatibility; do not rewrite +learner content to make an update pass. + +## 4. Start the Player + +Run the repository's Player command in a long-lived terminal. Reuse a healthy +Player that already belongs to this repository; do not stop an unknown server. + +```sh +bun run play +``` + +Use `bunx pathmx route` and the server's printed base URL to resolve the exact +Source route with the project dependency. Prefer the most useful Source, Block, +or Beat position instead of linking only to the home page. + +When an integrated browser is available, open and review the Player there. In +Codex, prefer `@Browser`; in Claude Code, use its Chrome integration when it is +already configured. Otherwise open the system browser. If no browser can be +opened, give the learner a clickable URL. + +Open the bundled Player tutorial for a first-time learner. They may skip it. + +## 5. Begin learning + +Use `/path` implicitly. Ask a few questions at a time about: + +- what the learner wants to be able to do and why; +- what they already know, with one small piece of evidence; +- their available time and preferred pace; +- the visual mood, color direction, light or dark preference, and any + readability or motion needs. + +Confirm the learner's profile, milestone map, and first module before teaching. +When showing the proposed map, first write and link a proposed Path Source with +3–7 visibly statused milestones and evidence targets. Give its exact Player +URL, but do not create session, review, or checkpoint Sources yet. Wait for +explicit confirmation, then fully prepare the current module so the learner +can complete each session without waiting for another agent turn. Before +handoff, verify the embedded help and give an exact Player URL with brief +Play-mode and return instructions. diff --git a/bun.lock b/bun.lock index adb5bf7..f3cf83b 100644 --- a/bun.lock +++ b/bun.lock @@ -5,13 +5,13 @@ "": { "name": "pathmx-skills", "devDependencies": { - "@fellowhumans/pathmx": "0.1.16", + "@fellowhumans/pathmx": "0.1.21", "yaml": "2.9.0", }, }, }, "packages": { - "@fellowhumans/pathmx": ["@fellowhumans/pathmx@0.1.16", "", { "bin": { "pathmx": "dist/pathmx.js", "pmx": "dist/pathmx.js" } }, "sha512-EU4HBsnyfiVgUgLxNIrmTQ924o6pFwUlSTFA3kaFBLwVZA7M5OmQc3WK+1Z8CoRYGT1ASzBsjiGhZYSxT+7OQQ=="], + "@fellowhumans/pathmx": ["@fellowhumans/pathmx@0.1.21", "", { "bin": { "pathmx": "dist/pathmx.js", "pmx": "dist/pathmx.js" } }, "sha512-xd8R1+bK5L1vZr+1wku29p8qS7iwpZD7J3+r6wuwmmv7f1EJEdHG8hu4R60yTikT0M7kzMC1T3qDs1DJubE6VA=="], "yaml": ["yaml@2.9.0", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA=="], } diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..6c852b3 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,156 @@ +# PathMX learning-space evals + +These evals exercise the same entry point given to a learner: `bootstrap.md`. +The subject is a real resumable Codex CLI session. It sees only the bootstrap, +the learner's messages, and the repository it creates. Deterministic checks and +an optional independent judge score the result afterward. + +The latest checked development summary is +[candidate-smoke-2026-07-21](./results/candidate-smoke-2026-07-21.md). + +Run workspaces live outside this repository by default so the subject cannot +read hidden scenarios or inherit this repository's `AGENTS.md`. + +## Quick start + +Validate the suite without calling a model: + +```sh +bun run eval:check +bun run eval -- list +bun run eval -- plan sql-beginner +``` + +Run one subject session using the Codex Desktop-like default profile and +deterministic grading: + +```sh +bun run eval -- run sql-beginner +``` + +Run the deliberately weaker instruction floor: + +```sh +bun run eval -- run sql-beginner \ + --profile instruction-floor +``` + +Add the independent model judge: + +```sh +bun run eval -- run sql-beginner --judge +``` + +Use `--runs 3` to measure variance. Use `--artifacts ` to retain +runs in a chosen location. The default is the operating system's temporary +directory under `pathmx-evals/`. + +For release-gating runs, point `--codex-home` at a dedicated, already +authenticated Codex home containing no personal skills or configuration. The +runner also ignores config and rules and disables plugins. It does not copy or +manage authentication files. Without this option, the CLI uses the caller's +normal authentication and may still discover global instruction surfaces that +live outside `config.toml`. + +## What a run contains + +Each run retains: + +- the exact scenario and CLI configuration; +- one JSONL event stream and final message per learner turn; +- file inventories, Git state, and a structural grade captured after each turn; +- deterministic checks and command output; +- a structured judge result when requested; +- `report.json` and a concise `report.md`. + +Reports show learner wait time per turn and for the full run. They also measure +time to the first learner-visible agent update, update count, and the longest +silent gap. A turn over five minutes or silent gap over one minute is an +experience warning, not a quality-score penalty: model latency, visible +progress, and curriculum correctness are separate release signals. + +The rubric is never copied into the subject workspace. Fictional learner +profiles must not contain real personal information. + +## Model profiles + +The machine-readable profiles in `profiles.json` keep model roles separate: + +- `desktop-power` is the primary experience lane. It currently uses the Codex + Desktop app's documented default Power setting: `gpt-5.6-sol` with medium + reasoning. +- `desktop-fast` approximates moving that Desktop setting toward Faster by + retaining `gpt-5.6-sol` with low reasoning. +- `instruction-floor` uses `gpt-5.4-mini` with low reasoning to expose weak or + overly implicit instructions. +- `judge-quality` uses `gpt-5.6-sol` with high reasoning only for independent + qualitative scoring. + +Use `--model` or `--reasoning` only for an intentional one-off override; the +selected profile and effective overrides are recorded in every run. Review the +profile catalog whenever Codex changes its documented default. The current +default is documented in the [Codex model +guide](https://learn.chatgpt.com/docs/models). + +## Lanes + +The default `local` bootstrap lane copies this checkout's `bootstrap.md` into +an otherwise empty subject repository. It tests the instructions under review +while still using the real package and template installation flow. + +Before a starter release, stage a local candidate while retaining the rest of +the real bootstrap flow: + +```sh +bun run eval -- run sql-beginner \ + --candidate-starter \ + --profile instruction-floor +``` + +The runner copies the candidate into the isolated subject root without its +Git history, dependencies, or build output. The subject is told to use that +copy in place of the published `pathmx init` template. This candidate lane is +not the release gate: rerun without `--candidate-starter` after publication to +test the exact learner installation path. + +A scenario may use a hosted bootstrap URL instead. Hosted runs test URL access +and the published installation path, so they are slower and more sensitive to +network and registry failures. + +Codex runs with `workspace-write`, approval policy `never`, and network access +enabled because bootstrap must install packages and the starter. Never point an +eval at a directory containing personal work. Use an externally isolated +runner before considering broader sandbox permissions. + +A truly fresh-machine lane may need to update global tooling outside the +workspace. Run that only inside an externally isolated VM or container and opt +in with `--sandbox danger-full-access`. A permission failure in the default +lane remains visible in the transcript instead of silently broadening access. + +## Scoring + +Deterministic checks are hard evidence: repository creation, local Git safety, +installed skills, version policy, PathMX verification, durable learner state, +and buffered-module structure. A failed critical check caps the result. + +The optional judge scores qualities such as clarity, coherent progress, +nontechnical language, immediate learning support, and proportional adaptation. +It must cite transcript turns or repository files for every score. The subject +model never sees these criteria. + +Run each release-gating scenario at least three times. Compare critical-pass +rate, median score, worst score, median total model time, the slowest turn, and +silent-gap warnings rather than relying on one average. + +## Codex App coverage + +The CLI covers instruction discovery, skills, filesystem behavior, commands, +and resumable conversations. It does not reproduce the Codex App's integrated +Browser or permission UI exactly. Keep a small manual App smoke pass for Player +appearance, exact-route opening, and browser fallback until those surfaces have +a stable automation interface. + +The runner uses the documented [Codex non-interactive +mode](https://learn.chatgpt.com/docs/non-interactive-mode) and [`codex +exec`](https://learn.chatgpt.com/docs/developer-commands?surface=cli#cli-codex-exec) +session-resume and JSONL interfaces. diff --git a/evals/pathmx-skills.rubric.md b/evals/pathmx-skills.rubric.md deleted file mode 100644 index 453305c..0000000 --- a/evals/pathmx-skills.rubric.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -type: pathmx/rubric -status: active ---- - -# PathMX Skills Agent Rubric - -Score observable work and verification evidence. Do not infer hidden work. - -## Scoring - -Score each applicable row: - -- `0` — missing, wrong, or unsupported -- `1` — partly complete -- `2` — complete and supported by evidence - -Mark a row `N/A` only when it does not apply. Calculate the percentage from -applicable points. - -| Result | Score | -| --- | ---: | -| Meets | 90–100% | -| Mostly meets | 80–89% | -| Partial | 60–79% | -| Fails | Below 60% or any critical failure | - -## Task profiles - -| Task | Sections to score | -| --- | --- | -| Author or revise PathMX content | Core and PathMX | -| Review or diagnose PathMX content | Core and PathMX review | -| Work on components, styling, or Play | Core, PathMX, and experience | -| Author durable assessment questions | Core, PathMX, and questions | -| Start or resume a personal learning path | Core and personal path | -| Sync these skills into a target repository | Core and distribution | - -## Critical failures - -Any of these makes the result `Fails`: - -- Invents PathMX syntax or uses syntax unsupported by the installed version. -- Authors general actions or spaceholders as supported features. -- Writes diagnostic build output into a live `.pathmx` directory. -- Claims a build or test passed without evidence. -- Makes the repository depend on another checkout or named consumer. -- Advances a personal path without assessment evidence and synthesis. -- Records unconfirmed or unnecessary sensitive learner data. - -## Core - -| Criterion | Score | -| --- | ---: | -| Reads the nearest instructions and stays within the task scope. | 0–2 | -| Uses local files, installed versions, and command output as evidence. | 0–2 | -| Keeps changes focused, plain, and readable. | 0–2 | -| Preserves unrelated work and existing repository conventions. | 0–2 | -| Reports changed files, verification, and any remaining gap. | 0–2 | - -## PathMX - -| Criterion | Score | -| --- | ---: | -| Inspects local config, scripts, PathMX version, entry Source, and nearby examples. | 0–2 | -| Identifies the Source role and audience before changing structure. | 0–2 | -| Starts with ordinary Markdown and uses Blocks or Beats only when they help. | 0–2 | -| Uses relative links and reuses local directives, components, and styles. | 0–2 | -| Keeps the Source useful as plain Markdown and uses source-facing `type`. | 0–2 | -| Builds into scratch output and checks warnings or errors. | 0–2 | - -## PathMX review - -| Criterion | Score | -| --- | ---: | -| Checks the installed version and relevant local examples before judging syntax. | 0–2 | -| Gives specific findings tied to files, output, or rendered behavior. | 0–2 | -| Separates defects from optional improvements and states impact. | 0–2 | -| Runs relevant read-only checks or explains why a check could not run. | 0–2 | - -## Questions - -| Criterion | Score | -| --- | ---: | -| Uses one stable `type: question` Block and one heading per response. | 0–2 | -| Uses the correct built-in mapping and question type for the control. | 0–2 | -| Keeps prompts and options readable as plain Markdown. | 0–2 | -| Preserves learner-owned responses and does not prefill or auto-grade them. | 0–2 | -| Verifies rendered controls and, when practical, submit and reload behavior. | 0–2 | - -## Experience - -| Criterion | Score | -| --- | ---: | -| Components and styles follow local contracts and preserve semantic HTML. | 0–2 | -| Play pacing uses Blocks and Beats intentionally. | 0–2 | -| The result is checked for keyboard use, narrow screens, and readable contrast. | 0–2 | -| Rendered or interactive behavior is reviewed when the task changes it. | 0–2 | - -## Personal path - -| Criterion | Score | -| --- | ---: | -| Uses one learner, one goal, and learner-confirmed context. | 0–2 | -| Collects Point A evidence, then establishes Point A, Point B, rubric, and a confirmed plan before teaching. | 0–2 | -| Reads the latest activity, assessment, and synthesis before resuming a path. | 0–2 | -| Builds only the first or next lesson at the learner's proximal edge, one Block per turn. | 0–2 | -| Records an observable `start`, `destination`, review, practice, retrieval, and feedback. | 0–2 | -| Uses a plain Markdown assessment with core/peripheral tags and durable question responses, linked evidence, or both. | 0–2 | -| Gates progress on no-core-miss evidence; retries variants, remediates after repeated fails, and renegotiates after failed remediation. | 0–2 | -| Records synthesis and logs explicit goal changes without rewriting history. | 0–2 | - -## Distribution - -| Criterion | Score | -| --- | ---: | -| Runs check mode before write mode. | 0–2 | -| Validates the target, containment, and conflicts before writing. | 0–2 | -| Updates only declared skill packages and managed discovery links. | 0–2 | -| Preserves unrelated target content and verifies rollback behavior. | 0–2 | - -## Evaluator report - -```text -Task profile: -Score: -Result: -Critical failures: -Evidence: -Gaps: -``` diff --git a/evals/profiles.json b/evals/profiles.json new file mode 100644 index 0000000..2c6ca55 --- /dev/null +++ b/evals/profiles.json @@ -0,0 +1,40 @@ +{ + "schema": "pathmx-eval/profiles", + "version": 1, + "defaultSubject": "desktop-power", + "defaultJudge": "judge-quality", + "profiles": [ + { + "id": "desktop-power", + "role": "subject", + "model": "gpt-5.6-sol", + "reasoning": "medium", + "description": "Codex Desktop's documented default Power setting.", + "evidence": "https://learn.chatgpt.com/docs/models", + "verifiedOn": "2026-07-20" + }, + { + "id": "desktop-fast", + "role": "subject", + "model": "gpt-5.6-sol", + "reasoning": "low", + "description": "CLI approximation of moving Desktop Power toward Faster: the same Sol model with lighter reasoning.", + "evidence": "https://learn.chatgpt.com/docs/models", + "verifiedOn": "2026-07-20" + }, + { + "id": "instruction-floor", + "role": "subject", + "model": "gpt-5.4-mini", + "reasoning": "low", + "description": "A deliberately weaker, lower-reasoning lane for instruction robustness." + }, + { + "id": "judge-quality", + "role": "judge", + "model": "gpt-5.6-sol", + "reasoning": "high", + "description": "A stronger independent judge for evidence-backed qualitative scoring." + } + ] +} diff --git a/evals/results/candidate-smoke-2026-07-21.md b/evals/results/candidate-smoke-2026-07-21.md new file mode 100644 index 0000000..09e2acd --- /dev/null +++ b/evals/results/candidate-smoke-2026-07-21.md @@ -0,0 +1,76 @@ +# Candidate learning-space smoke evals — 2026-07-21 + +These results use the real resumable Codex CLI with the candidate learning +starter. They are development evidence, not a published-template release gate. +Raw workspaces were written under the operating system's temporary +`pathmx-evals` directory. + +## Outcome + +The four representative `desktop-power` smoke scenarios passed their critical +and deterministic checks. A weaker-model loop then exposed three instruction +defects: premature module authoring, a chat-only map, and accidental +registration of nested learner Paths as extra Player roots. Each finding +produced a checked instruction, fixture, starter guardrail, or phase contract. + +After those changes, the ambiguous-goal scenario passed at 100% deterministic +and 100% independent-judge quality on `desktop-fast`. The complete flow took +6m56s; its slowest turn was the 3m22s module build. The learner saw the first +visible update in 4–13 seconds on every turn. One 1m54s silent gap remains an +experience warning. + +The corrected return-and-adapt scenario also passed at 100% deterministic and +100% independent-judge quality on `desktop-fast`. Its two complete modules, +map-first boundary, checkpoint record, open learner feedback, and exact +next-session route all passed. The five-turn flow took 10m07s with no +individual turn over five minutes. + +## Representative runs + +| Profile | Scenario | Result | Total time | Main evidence | +| --- | --- | --- | --- | --- | +| `desktop-power` | SQL beginner | 100% deterministic, critical pass | 8m55s | Buffered module and exact route passed. | +| `desktop-power` | Ambiguous AI goal | 100% deterministic and judge, critical pass | 18m55s | Quality passed, but the final turn took 12m43s. | +| `desktop-power` | Offline guitar | 100% deterministic, critical pass | 10m51s | Non-screen practice constraints survived the flow. | +| `desktop-power` | Return with confusion | 100% deterministic and judge, critical pass | 13m46s | Completed work and reported annotation were preserved honestly. | +| `instruction-floor` | Ambiguous AI goal, initial | 90.9% combined, critical pass | 5m10s | Authored the module before map confirmation and under-supplied help. | +| `instruction-floor` | Ambiguous AI goal, map-only revision | 86.5% deterministic, critical pass | 5m29s | Avoided the module, but left the proposed map only in chat. | +| `instruction-floor` | Ambiguous AI goal, persisted-map revision | 59% capped, critical fail | 13m56s | Map phase passed; module turn added an invalid extra Player root and lacked support. | +| `desktop-fast` | Ambiguous AI goal, current candidate | 100% deterministic and judge, critical pass | 6m56s | Map and module phase contracts passed. | +| `desktop-fast` | Return with confusion, current candidate | 100% deterministic and judge, critical pass | 10m07s | Three phase contracts passed; completed work and open feedback were preserved. | + +The repeated floor failures are retained because they show the +self-improvement loop working; they are not results from the final candidate. + +## Changes driven by eval evidence + +- Map-first requests now persist and link a proposed Path with statused + milestones before confirmation, while session and checkpoint Sources remain + absent. +- The skill pack ships Path and two-session module scaffolds so agents start + from useful artifacts instead of blank files. +- The learning starter keeps one configured Player root. Nested learner Paths + are linked from home and the starter check rejects extra config roots. +- Multiple-root Source handle syntax is documented as `@name` and verified by + a live Player-start fixture. +- The starter check rejects modules without 2–4 sessions, review, checkpoint, + examples, hints, smaller attempts, self-check or rationale, and stretch work. +- Exact-route guidance now forbids ambiguous basename queries and broad + recursive searches of generated `.pathmx` caches. +- Eval phase contracts catch work authored too early. Reports separately track + total duration, first visible update, update count, longest silence, and + five-minute turns. +- Bootstrap compares the updated native version with the exact project + dependency before reinstalling and makes the project working directory + explicit for every shell call. + +## Remaining release work + +- Repeat final-candidate scenarios to measure variance; the current post-fix + fast-model result is one run. +- Run the published-template lane after the starter and skills are released. +- Keep the manual Codex Desktop Player review because the CLI does not reproduce + the integrated Browser or permission UI exactly. +- Treat silent gaps over one minute as an experience warning even when quality + passes. The buffered module prevents that delay from interrupting ordinary + learner progress after handoff, but initial authoring latency still matters. diff --git a/evals/rubrics/learning-space.json b/evals/rubrics/learning-space.json new file mode 100644 index 0000000..4e99a35 --- /dev/null +++ b/evals/rubrics/learning-space.json @@ -0,0 +1,54 @@ +{ + "schema": "pathmx-eval/rubric", + "version": 1, + "criteria": [ + { + "id": "onboarding.clarity", + "phase": "onboarding", + "weight": 2, + "description": "Questions are brief, plain, and collect Point B, Point A evidence, rhythm, learning preferences, and relevant access needs without unnecessary personal data." + }, + { + "id": "map.achievable-progress", + "phase": "map", + "weight": 3, + "description": "The learner gets a comprehensible 3–7 milestone capability map with evidence targets, statuses, and one foreground path." + }, + { + "id": "module.coherence", + "phase": "module", + "weight": 3, + "description": "The current module is a coherent runway of 2–4 complete sessions; later work remains provisional." + }, + { + "id": "session.immediate-support", + "phase": "session", + "weight": 3, + "description": "Sessions have a finishable arc and embed hints, examples, rationale or rubrics, smaller options, and optional stretch work so ordinary progress does not wait for the agent." + }, + { + "id": "presentation.personal-readable", + "phase": "module", + "weight": 1, + "description": "Confirmed style choices shape a restrained, attractive, readable presentation without destabilizing navigation." + }, + { + "id": "checkpoint.evidence", + "phase": "checkpoint", + "weight": 2, + "description": "Before learner submission, the checkpoint requests observable evidence, gives success criteria, and makes no unsupported progress claim. When the scenario includes submitted checkpoint evidence, the resulting decision uses that evidence and makes the next state understandable. Do not deduct merely because a pre-checkpoint scenario ends before the learner completes it." + }, + { + "id": "adaptation.proportional", + "phase": "adaptation", + "weight": 2, + "description": "The agent adapts at a useful boundary, preserves history and annotations, and avoids unnecessary remediation loops." + }, + { + "id": "handoff.next-action", + "phase": "return", + "weight": 2, + "description": "The learner is given the narrowest useful Player route and knows exactly what to do next and how to use Play mode." + } + ] +} diff --git a/evals/scenarios/ambiguous-ai-goal.yaml b/evals/scenarios/ambiguous-ai-goal.yaml new file mode 100644 index 0000000..7f3aa27 --- /dev/null +++ b/evals/scenarios/ambiguous-ai-goal.yaml @@ -0,0 +1,41 @@ +schema: pathmx-eval/scenario +version: 1 +id: ambiguous-ai-goal +title: Ambiguous AI goal that needs narrowing +objective: Turn an unclear aspiration into an evidence-based, learner-confirmed path. +tags: [bootstrap, ambiguous-goal, nontechnical] +bootstrap: + source: local +learner: + learningSpace: learning-space + initialMessage: >- + Follow the bootstrap instructions and create ./learning-space. I keep hearing + about AI and feel behind. I want to learn AI, but I do not know what that + should mean for me. + hiddenNotes: >- + The agent should narrow the outcome before building substantial material. + The learner mainly writes community-newsletter copy and wants practical, + critical use rather than programming or model training. +turns: + - id: context + phase: onboarding + message: >- + I write a weekly community newsletter. I would like to use AI to brainstorm + and revise without losing my voice, and to recognize when an answer is + unreliable. I do not want to learn programming right now. + - id: rhythm-style + phase: map + message: >- + I have two 20-minute sessions a week. I prefer conversational examples and + side-by-side comparisons. Use a warm paper-like light theme with high + contrast and no animation. Show me a small milestone map first. + expect: + passed: [path.milestones] + notPassed: [module.buffered] + - id: confirm + phase: module + message: >- + Yes, that outcome and map fit. Prepare the complete first module and point + me to the exact place to start. + expect: + passed: [module.buffered, session.immediate-support, player.exact-route] diff --git a/evals/scenarios/offline-guitar.yaml b/evals/scenarios/offline-guitar.yaml new file mode 100644 index 0000000..646e088 --- /dev/null +++ b/evals/scenarios/offline-guitar.yaml @@ -0,0 +1,41 @@ +schema: pathmx-eval/scenario +version: 1 +id: offline-guitar +title: Offline physical practice +objective: Build a PathMX runway that supports guitar practice away from the screen. +tags: [bootstrap, physical-practice, media] +bootstrap: + source: local +learner: + learningSpace: learning-space + initialMessage: >- + Follow the bootstrap instructions and create ./learning-space. I want to play + a simple song smoothly on acoustic guitar instead of only knowing disconnected + chord shapes. + hiddenNotes: >- + The material should coordinate short off-screen attempts with useful Player + guidance. Avoid pretending the repository can hear or assess performance. +turns: + - id: evidence + phase: onboarding + message: >- + I can form G, C, D, and Em slowly, but I pause for several seconds between + changes and lose the beat. I can practice for 20 minutes most days. I want + direct instructions, diagrams when useful, and no autoplaying media. + - id: style + phase: map + message: >- + A dark charcoal and amber style would feel good, but keep text large and + contrast strong. My goal song uses G, C, D, and Em. Show me achievable + milestones and how I will know each one is working. + expect: + passed: [path.milestones] + notPassed: [module.buffered] + - id: confirm + phase: module + message: >- + I confirm the map. Prepare the first full practice module with immediate + help for when a chord change breaks down, then give me the exact starting + link. + expect: + passed: [module.buffered, session.immediate-support, player.exact-route] diff --git a/evals/scenarios/return-with-confusion.yaml b/evals/scenarios/return-with-confusion.yaml new file mode 100644 index 0000000..9c87227 --- /dev/null +++ b/evals/scenarios/return-with-confusion.yaml @@ -0,0 +1,50 @@ +schema: pathmx-eval/scenario +version: 1 +id: return-with-confusion +title: Return with confusion at a module boundary +objective: Verify evidence-based adaptation without tightening the loop around every activity. +tags: [return, annotations, checkpoint, adaptation] +bootstrap: + source: local +learner: + learningSpace: learning-space + initialMessage: >- + Follow the bootstrap instructions and create ./learning-space. I want a + beginner SQL path for exploring customer data. I can study for 25 minutes + twice a week. Use a quiet light theme and prepare the first module. + hiddenNotes: >- + The learner later reports a specific misconception. The agent should preserve + completed work, record the evidence, and adapt at the checkpoint rather than + rewriting history or blocking ordinary session flow. +turns: + - id: baseline + phase: map + message: >- + I understand rows and columns but have never written SQL. I like worked + examples followed by a similar attempt. Please show me the milestone map + before teaching. + expect: + passed: [path.milestones] + notPassed: [module.buffered] + - id: confirm-map + phase: module + message: >- + I confirm the map. Prepare the first complete module with worked examples, + similar attempts, and immediate help, then give me the exact starting link. + expect: + passed: [module.buffered, session.immediate-support, player.exact-route] + - id: simulated-completion + phase: checkpoint + message: >- + Assume I completed the first module. My exercises show I can select columns + and filter simple values, but I keep putting aggregate conditions in WHERE + and cannot explain when HAVING is needed. Record this as checkpoint evidence. + I also left an annotation saying the first explanation was too abstract. + - id: adapt + phase: adaptation + message: >- + Update my current position and prepare the next useful module. Preserve the + completed module and annotation. Do not make me repeat work I demonstrated. + Give me the exact next Player location when it is ready. + expect: + passed: [module.buffered, session.immediate-support, player.exact-route] diff --git a/evals/scenarios/sql-beginner.yaml b/evals/scenarios/sql-beginner.yaml new file mode 100644 index 0000000..7c8ed1c --- /dev/null +++ b/evals/scenarios/sql-beginner.yaml @@ -0,0 +1,41 @@ +schema: pathmx-eval/scenario +version: 1 +id: sql-beginner +title: SQL beginner with a concrete work goal +objective: Bootstrap a personal learning space and prepare a coherent first SQL module. +tags: [bootstrap, technical, beginner, buffered-module] +bootstrap: + source: local +learner: + learningSpace: learning-space + initialMessage: >- + Please follow the bootstrap instructions and carry them out. Create the new + repository at ./learning-space. I want to learn enough SQL to answer useful + questions from customer data at work. I am not very technical. + hiddenNotes: >- + The learner can read a spreadsheet and recognize rows and columns but has + never written a query. Favor visible, low-jargon progress and concrete + evidence over database theory. +turns: + - id: preferences + phase: onboarding + message: >- + I can usually do 25 minutes three evenings a week. I learn best from one + worked example and then trying something similar. Keep it calm, practical, + light, and blue-green. Larger text is helpful and I dislike motion. + - id: point-a + phase: map + message: >- + For current evidence: I understand that a customer table has one row per + customer and columns like name or signup date, but I do not know how to ask + the database a question. Please show me the proposed map before teaching. + expect: + passed: [path.milestones] + notPassed: [module.buffered] + - id: confirm-map + phase: module + message: >- + The map makes sense. Confirm it and fully prepare the first module so I can + work through all of its sessions without waiting for another agent turn. + expect: + passed: [module.buffered, session.immediate-support, player.exact-route] diff --git a/evals/schemas/judge-result.schema.json b/evals/schemas/judge-result.schema.json new file mode 100644 index 0000000..8a5e933 --- /dev/null +++ b/evals/schemas/judge-result.schema.json @@ -0,0 +1,26 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "pathmx-eval/judge-result", + "type": "object", + "required": ["summary", "criteria", "strengths", "risks"], + "properties": { + "summary": { "type": "string" }, + "criteria": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "score", "evidence", "reason"], + "properties": { + "id": { "type": "string" }, + "score": { "type": "integer", "minimum": 0, "maximum": 2 }, + "evidence": { "type": "array", "items": { "type": "string" }, "minItems": 1 }, + "reason": { "type": "string" } + }, + "additionalProperties": false + } + }, + "strengths": { "type": "array", "items": { "type": "string" } }, + "risks": { "type": "array", "items": { "type": "string" } } + }, + "additionalProperties": false +} diff --git a/evals/schemas/scenario.schema.json b/evals/schemas/scenario.schema.json new file mode 100644 index 0000000..ca6427c --- /dev/null +++ b/evals/schemas/scenario.schema.json @@ -0,0 +1,63 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "pathmx-eval/scenario", + "type": "object", + "required": ["schema", "version", "id", "title", "objective", "bootstrap", "learner", "turns"], + "properties": { + "schema": { "const": "pathmx-eval/scenario" }, + "version": { "const": 1 }, + "id": { "type": "string", "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$" }, + "title": { "type": "string", "minLength": 1 }, + "objective": { "type": "string", "minLength": 1 }, + "tags": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }, + "bootstrap": { + "type": "object", + "required": ["source"], + "properties": { + "source": { "enum": ["local", "hosted"] }, + "url": { "type": "string", "format": "uri" } + }, + "additionalProperties": false + }, + "learner": { + "type": "object", + "required": ["learningSpace", "initialMessage", "hiddenNotes"], + "properties": { + "learningSpace": { "type": "string", "pattern": "^[a-zA-Z0-9][a-zA-Z0-9._-]*$" }, + "initialMessage": { "type": "string", "minLength": 1 }, + "hiddenNotes": { "type": "string", "minLength": 1 } + }, + "additionalProperties": false + }, + "turns": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "phase", "message"], + "properties": { + "id": { "type": "string", "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$" }, + "phase": { "enum": ["onboarding", "map", "module", "session", "checkpoint", "adaptation", "return"] }, + "message": { "type": "string", "minLength": 1 }, + "expect": { + "type": "object", + "properties": { + "passed": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "uniqueItems": true + }, + "notPassed": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "uniqueItems": true + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false +} diff --git a/package.json b/package.json index 5ab78fa..f7307fc 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,13 @@ { "name": "pathmx-skills", "private": true, + "pathmxCompatibility": { + "baseline": "0.1.21", + "updatePolicy": "latest-after-verification" + }, "scripts": { + "eval": "bun run scripts/evals.ts", + "eval:check": "bun run scripts/evals.ts check", "sync-skills": "bun run scripts/sync-skills.ts", "test": "bun test", "check:links": "bun run scripts/check-markdown-links.ts", @@ -11,7 +17,7 @@ "check": "bun run check:links && bun run check:skills && bun run check:self-contained && bun run check:pathmx && bun test && git diff --check HEAD" }, "devDependencies": { - "@fellowhumans/pathmx": "0.1.16", + "@fellowhumans/pathmx": "0.1.21", "yaml": "2.9.0" } } diff --git a/scripts/check-markdown-links.test.ts b/scripts/check-markdown-links.test.ts index 0397bf7..0adb17b 100644 --- a/scripts/check-markdown-links.test.ts +++ b/scripts/check-markdown-links.test.ts @@ -61,4 +61,12 @@ describe("Markdown links", () => { { line: 3, target: "./missing.md" }, ]) }) + + it("does not treat footnote or annotation definitions as links", async () => { + expect( + await check( + "Sentence.[^c1]\n\n[^c1]: **@learner** (2026-07-20 10:30 -04:00): A note.\n", + ), + ).toEqual([]) + }) }) diff --git a/scripts/check-markdown-links.ts b/scripts/check-markdown-links.ts index a38d7e1..dd3f943 100644 --- a/scripts/check-markdown-links.ts +++ b/scripts/check-markdown-links.ts @@ -65,7 +65,7 @@ export function extractMarkdownLinks(markdown: string): MarkdownLink[] { if (target) links.push({ line: index + 1, target }) } - const definition = line.match(/^\s*\[[^\]]+\]:\s*(<[^>]+>|\S+)/) + const definition = line.match(/^\s*\[(?!\^)[^\]]+\]:\s*(<[^>]+>|\S+)/) if (definition?.[1]) { links.push({ line: index + 1, diff --git a/scripts/check-pathmx-docs.ts b/scripts/check-pathmx-docs.ts index e979ee1..572bdc5 100644 --- a/scripts/check-pathmx-docs.ts +++ b/scripts/check-pathmx-docs.ts @@ -19,6 +19,19 @@ type SourceManifest = { sources: Record }> } +type GraphIndex = { + sources: Array<{ + id: string + blocks: Array<{ + id: string + props?: { + question?: { type?: string } + actions?: { submit?: string } + } + }> + }> +} + type BuildResult = { outputDir: string paths: PathManifest @@ -30,11 +43,11 @@ const repoRoot = path.resolve(import.meta.dir, "..") const pathmxBin = path.join(repoRoot, "node_modules", ".bin", "pathmx") const referenceEntries = [ - "evals/pathmx-skills.rubric.md", "skills/pathmx/references/pathmx-markdown.md", "skills/pathmx/references/pathmx-player.md", "skills/pathmx/references/pathmx-directives.md", "skills/pathmx/references/pathmx-questions.md", + "skills/pathmx/references/pathmx-annotations.md", "skills/pathmx/references/pathmx-literate-components.md", "skills/pathmx/references/pathmx-code.md", "skills/pathmx/references/pathmx-math.md", @@ -98,6 +111,39 @@ async function outputForEntry(result: BuildResult, entry: string) { return path.join(result.outputDir, builtPath.outputPath) } +async function requirePlayerStarts(cwd: string, outputDir: string) { + const child = Bun.spawn( + [pathmxBin, "play", "--print-url", "--port", "0", "-o", outputDir, "--clean"], + { cwd, stdout: "pipe", stderr: "pipe" }, + ) + const reader = child.stdout.getReader() + const decoder = new TextDecoder() + const stderrPromise = new Response(child.stderr).text() + let stdout = "" + let timedOut = false + const timer = setTimeout(() => { + timedOut = true + child.kill() + }, 10_000) + try { + while (!/https?:\/\/\S+/.test(stdout)) { + const { done, value } = await reader.read() + if (done) break + stdout += decoder.decode(value, { stream: true }) + } + } finally { + clearTimeout(timer) + child.kill("SIGINT") + } + const exitCode = await child.exited + const stderr = await stderrPromise + if (timedOut || !/https?:\/\/\S+/.test(stdout)) { + throw new Error( + `PathMX Player did not start for the multi-root handle fixture (${exitCode}):\n${stdout}\n${stderr}`, + ) + } +} + export async function checkPathmxDocs() { const tempRoot = await mkdtemp(path.join(os.tmpdir(), "pathmx-skills-check-")) const relativeTemp = path.relative(repoRoot, tempRoot) @@ -121,7 +167,7 @@ export async function checkPathmxDocs() { ), "utf8", ) - if (!codeMap.includes('"type": "code-step"')) { + if (!/"type"\s*:\s*"code-step"/.test(codeMap)) { throw new Error("Code reference did not produce code-step Beats") } @@ -158,6 +204,7 @@ export async function checkPathmxDocs() { const configRoot = path.join(repoRoot, "tests", "fixtures", "pathmx", "config") const config = await build(configRoot, path.join(tempRoot, "config"), []) requireEntries(config, ["index.path.md", "workshop.path.md"]) + await requirePlayerStarts(configRoot, path.join(tempRoot, "config-player")) const questionsRoot = path.join(repoRoot, "tests", "fixtures", "pathmx", "questions") const questions = await build(questionsRoot, path.join(tempRoot, "questions"), [ @@ -169,18 +216,56 @@ export async function checkPathmxDocs() { path.join(questionsOutput, "index.quiz.html"), "utf8", ) + const questionsGraph = await readJson( + path.join(questionsOutput, "graph-index.json"), + ) + const questionSource = questionsGraph.sources.find( + (source) => source.id === "index.quiz", + ) + if (!questionSource) throw new Error("Question fixture missing graph Source") + const expectedQuestions = [ + ["index.quiz#source-of-truth", "questions.submitSingleChoice", undefined], + ["index.quiz#define-beat", "questions.submitText", "short"], + ["index.quiz#explain-navigation", "questions.submitText", "long"], + ["index.quiz#study-plan", "questions.submitFields", "fields"], + ] as const + for (const [id, action, type] of expectedQuestions) { + const block = questionSource.blocks.find((candidate) => candidate.id === id) + if (block?.props?.actions?.submit !== action || (type && block.props.question?.type !== type)) { + throw new Error(`Question fixture missing graph contract: ${id}`) + } + } + for (const expected of ["@response.field id=minutes", "@response.field id=topic"]) { + if (!questionsHtml.includes(expected)) { + throw new Error(`Question fixture missing authored field: ${expected}`) + } + } + + const annotationsRoot = path.join( + repoRoot, + "tests", + "fixtures", + "pathmx", + "annotations", + ) + const annotations = await build( + annotationsRoot, + path.join(tempRoot, "annotations"), + ["index.path.md"], + ) + requireEntries(annotations, ["index.path.md"]) + const annotationsOutput = await outputForEntry(annotations, "index.path.md") + const annotationsHtml = await readFile( + path.join(annotationsOutput, "index.path.html"), + "utf8", + ) for (const expected of [ - 'data-pathmx-action="questions.submitSingleChoice"', - 'data-pathmx-action="questions.submitText"', - 'data-pathmx-action="questions.submitFields"', - 'name="question.source-of-truth"', - 'name="question.define-beat"', - 'name="question.explain-navigation"', - 'name="question.study-plan.minutes"', - 'name="question.study-plan.topic"', + 'data-pathmx-annotation-thread="c1"', + 'data-pathmx-annotation-status="resolved"', + 'data-pathmx-annotation-text="c2"', ]) { - if (!questionsHtml.includes(expected)) { - throw new Error(`Question fixture missing rendered control: ${expected}`) + if (!annotationsHtml.includes(expected)) { + throw new Error(`Annotation fixture missing rendered evidence: ${expected}`) } } @@ -210,8 +295,12 @@ export async function checkPathmxDocs() { const expectedPathSources = [ "sql-foundations/index.path.md", "sql-foundations/path.outcome.md", - "sql-foundations/lessons/joins/index.lesson.md", - "sql-foundations/lessons/joins/lesson.assessment.md", + "sql-foundations/modules/01-combine-tables/index.path.md", + "sql-foundations/modules/01-combine-tables/01-rows-that-survive.lesson.md", + "sql-foundations/modules/01-combine-tables/02-choose-the-join.lesson.md", + "sql-foundations/modules/01-combine-tables/review.practice.md", + "sql-foundations/modules/01-combine-tables/milestone.assessment.md", + "learner.profile.md", "learning.activity.md", ] for (const source of expectedPathSources) { @@ -220,55 +309,39 @@ export async function checkPathmxDocs() { } } - const adaptivePath = await build(pathRoot, path.join(tempRoot, "path-adaptive"), [ - "paths/chess-opening-principles/index.path.md", - ]) - requireEntries(adaptivePath, ["chess-opening-principles/index.path.md"]) - const expectedAdaptiveSources = [ - "chess-opening-principles/index.path.md", - "chess-opening-principles/path.outcome.md", - "chess-opening-principles/onboarding/index.lesson.md", - "chess-opening-principles/onboarding/point-a-evidence.md", - "chess-opening-principles/onboarding/confirm-plan.md", - "chess-opening-principles/lessons/control-center-development/index.lesson.md", - "chess-opening-principles/lessons/control-center-development/lesson.review.md", - "chess-opening-principles/lessons/control-center-development/lesson.practice.md", - "chess-opening-principles/lessons/control-center-development/lesson.assessment.md", - "chess-opening-principles/lessons/develop-before-queen/index.lesson.md", - "chess-opening-principles/lessons/reteach-opening-principles/index.lesson.md", - "chess-opening-principles/references/index.references.md", - "learner.profile.md", - "learning.activity.md", - "assets/learning.components.md", - ] - for (const source of expectedAdaptiveSources) { - if (!adaptivePath.sourcePaths.has(source)) { - throw new Error(`Adaptive path fixture missing ${source}`) - } - } - const adaptiveOutput = await outputForEntry( - adaptivePath, - "chess-opening-principles/index.path.md", + const personalOutput = await outputForEntry( + personalPath, + "sql-foundations/index.path.md", ) - const adaptiveAssessmentHtml = await readFile( + const personalAssessmentHtml = await readFile( path.join( - adaptiveOutput, - "chess-opening-principles", - "lessons", - "control-center-development", - "lesson.assessment.html", + personalOutput, + "sql-foundations", + "modules", + "01-combine-tables", + "milestone.assessment.html", ), "utf8", ) - for (const expected of [ - 'data-pathmx-action="questions.submitSingleChoice"', - 'data-pathmx-action="questions.submitText"', - 'name="question.mc-center"', - 'name="question.short-justify"', - ]) { - if (!adaptiveAssessmentHtml.includes(expected)) { - throw new Error(`Adaptive assessment missing rendered control: ${expected}`) - } + if (!personalAssessmentHtml.includes("Explain why the other join would drop rows")) { + throw new Error("Personal path checkpoint missing authored prompt") + } + const personalGraph = await readJson( + path.join(personalOutput, "graph-index.json"), + ) + const assessmentSource = personalGraph.sources.find( + (source) => + source.id === + "sql-foundations/modules/01-combine-tables/milestone.assessment", + ) + const explanation = assessmentSource?.blocks.find( + (block) => block.id.endsWith("#join-explanation"), + ) + if ( + explanation?.props?.actions?.submit !== "questions.submitText" || + explanation.props.question?.type !== "long" + ) { + throw new Error("Personal path checkpoint missing question graph contract") } return { @@ -277,17 +350,17 @@ export async function checkPathmxDocs() { Object.keys(core.paths.paths).length + Object.keys(config.paths.paths).length + Object.keys(questions.paths.paths).length + + Object.keys(annotations.paths.paths).length + Object.keys(example.paths.paths).length + - Object.keys(personalPath.paths.paths).length + - Object.keys(adaptivePath.paths.paths).length, + Object.keys(personalPath.paths.paths).length, sources: new Set([ ...references.sourcePaths, ...core.sourcePaths, ...config.sourcePaths, ...questions.sourcePaths, + ...annotations.sourcePaths, ...example.sourcePaths, ...personalPath.sourcePaths, - ...adaptivePath.sourcePaths, ]).size, } } finally { diff --git a/scripts/check-self-contained.ts b/scripts/check-self-contained.ts index 04a6471..9779438 100644 --- a/scripts/check-self-contained.ts +++ b/scripts/check-self-contained.ts @@ -61,7 +61,6 @@ async function listRepositoryFiles(repoRoot: string) { .split(/\r?\n/) .filter(Boolean) .filter((file) => existsSync(path.join(repoRoot, file))) - .filter((file) => !file.startsWith("plans/")) .filter((file) => file !== "scripts/check-self-contained.test.ts") .sort() } diff --git a/scripts/evals.test.ts b/scripts/evals.test.ts new file mode 100644 index 0000000..b05359e --- /dev/null +++ b/scripts/evals.test.ts @@ -0,0 +1,250 @@ +import { afterEach, describe, expect, it } from "bun:test" +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises" +import os from "node:os" +import path from "node:path" + +import { judgeCodexArgs, subjectCodexArgs, summarizeVisibleUpdates } from "./evals/codex" +import { formatDuration, summarizeTimings } from "./evals/report" +import { + gradeLearningSpace, + evaluatePhaseContract, + findEvalProfile, + loadEvalProfiles, + loadScenarios, + parseCodexEvents, + subjectPrompt, + summarizeDeterministicChecks, + validateScenario, +} from "./evals/core" + +const temporaryRoots: string[] = [] + +afterEach(async () => { + await Promise.all(temporaryRoots.splice(0).map((root) => rm(root, { recursive: true, force: true }))) +}) + +describe("eval scenarios", () => { + it("loads and validates every checked-in scenario", async () => { + const scenarios = await loadScenarios() + expect(scenarios.length).toBeGreaterThanOrEqual(4) + for (const scenario of scenarios) expect(validateScenario(scenario)).toEqual([]) + }) + + it("keeps hidden learner notes out of the subject prompt", async () => { + const [scenario] = await loadScenarios() + const prompt = subjectPrompt(scenario, "./bootstrap.md") + expect(prompt).toContain(scenario.learner.initialMessage) + expect(prompt).not.toContain(scenario.learner.hiddenNotes) + }) + + it("makes the candidate-starter substitution explicit", async () => { + const [scenario] = await loadScenarios() + const prompt = subjectPrompt(scenario, "./bootstrap.md", true) + expect(prompt).toContain("./candidate-starter") + expect(prompt).toContain("instead of fetching the published template") + }) + + it("defines separate Desktop-default, floor, and judge profiles", async () => { + const catalog = await loadEvalProfiles() + const desktop = findEvalProfile(catalog, catalog.defaultSubject, "subject") + const fast = findEvalProfile(catalog, "desktop-fast", "subject") + const floor = findEvalProfile(catalog, "instruction-floor", "subject") + const judge = findEvalProfile(catalog, catalog.defaultJudge, "judge") + expect(desktop).toMatchObject({ model: "gpt-5.6-sol", reasoning: "medium" }) + expect(fast).toMatchObject({ model: "gpt-5.6-sol", reasoning: "low" }) + expect(floor).toMatchObject({ model: "gpt-5.4-mini", reasoning: "low" }) + expect(judge).toMatchObject({ model: "gpt-5.6-sol", reasoning: "high" }) + }) +}) + +describe("Codex event and command contract", () => { + it("extracts the thread and usage from JSONL", () => { + const parsed = parseCodexEvents( + [ + JSON.stringify({ type: "thread.started", thread_id: "thread-1" }), + JSON.stringify({ type: "turn.completed", usage: { input_tokens: 10, output_tokens: 2 } }), + ].join("\n"), + ) + expect(parsed.threadId).toBe("thread-1") + expect(parsed.usage).toEqual({ input_tokens: 10, output_tokens: 2 }) + expect(parsed.invalid).toEqual([]) + }) + + it("uses explicit low-permission automation flags and resumable sessions", () => { + const args = subjectCodexArgs( + { model: "floor-model", reasoning: "low", timeoutMs: 1 }, + "/tmp/subject", + "/tmp/final.md", + "learner reply", + "thread-1", + ) + expect(args).toContain("workspace-write") + expect(args).toContain("never") + expect(args).toContain("--ignore-user-config") + expect(args).toContain("--ignore-rules") + expect(args).toContain("plugins") + expect(args).toContain("resume") + expect(args).toContain("thread-1") + expect(args).not.toContain("--ephemeral") + }) + + it("keeps the judge read-only and permits an isolated artifact directory", () => { + const args = judgeCodexArgs( + { model: "judge-model", reasoning: "high", timeoutMs: 1 }, + "/tmp/run", + "/tmp/result.json", + "/tmp/schema.json", + ) + expect(args).toContain("read-only") + expect(args).toContain("--skip-git-repo-check") + expect(args).toContain("--ephemeral") + expect(args).toContain("--output-schema") + expect(args).not.toContain("workspace-write") + }) +}) + +describe("eval timing signal", () => { + it("keeps learner wait time separate and identifies long turns", () => { + const timing = summarizeTimings([ + { turn: "03-module", durationMs: 763_385, timedOut: false }, + { turn: "00-bootstrap", durationMs: 322_688, timedOut: false }, + { turn: "01-context", durationMs: 11_113, timedOut: false }, + ]) + expect(timing.totalDurationMs).toBe(1_097_186) + expect(timing.slowestTurn?.turn).toBe("03-module") + expect(timing.turnsOverFiveMinutes).toBe(2) + expect(timing.turns.map((turn) => turn.turn)).toEqual([ + "00-bootstrap", + "01-context", + "03-module", + ]) + expect(formatDuration(763_385)).toBe("12m 43s") + }) + + it("measures learner-visible progress updates and silent gaps", () => { + const timing = summarizeVisibleUpdates( + [ + { elapsedMs: 15_000, line: JSON.stringify({ type: "turn.started" }) }, + { + elapsedMs: 25_000, + line: JSON.stringify({ + type: "item.completed", + item: { type: "agent_message", text: "I have the scaffold ready." }, + }), + }, + { + elapsedMs: 80_000, + line: JSON.stringify({ + type: "item.completed", + item: { type: "agent_message", text: "Verification is starting." }, + }), + }, + ], + 100_000, + ) + expect(timing).toEqual({ + updateCount: 2, + firstUpdateMs: 25_000, + longestSilentGapMs: 55_000, + }) + }) +}) + +describe("phase contracts", () => { + it("catches a module authored before a map-first confirmation", () => { + const base = summarizeDeterministicChecks([ + { + id: "path.milestones", + label: "milestones", + passed: true, + critical: false, + weight: 1, + evidence: "present", + }, + { + id: "module.buffered", + label: "module", + passed: true, + critical: true, + weight: 1, + evidence: "present", + }, + ]) + const contract = evaluatePhaseContract( + { + id: "show-map", + phase: "map", + message: "Show the map first.", + expect: { passed: ["path.milestones"], notPassed: ["module.buffered"] }, + }, + base, + ) + expect(contract?.passed).toBe(false) + expect(contract?.checks.find((check) => check.id === "module.buffered")).toMatchObject({ + expected: "not-passed", + actual: "passed", + passed: false, + }) + }) +}) + +describe("deterministic grader", () => { + it("recognizes a private buffered learning repository", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "pathmx-eval-test-")) + temporaryRoots.push(root) + for (const directory of [ + ".git", + ".agents/skills/path", + ".agents/skills/pathmx", + "paths/sql/modules/01-basics", + ]) { + await mkdir(path.join(root, directory), { recursive: true }) + } + await writeFile(path.join(root, ".agents/skills/path/SKILL.md"), "path") + await writeFile(path.join(root, ".agents/skills/pathmx/SKILL.md"), "pathmx") + await writeFile( + path.join(root, "package.json"), + JSON.stringify({ + pathmxCompatibility: { baseline: "1.2.3" }, + dependencies: { "@fellowhumans/pathmx": "1.2.3" }, + }), + ) + await writeFile(path.join(root, "paths/index.path.md"), "# Home") + await writeFile( + path.join(root, "paths/learner.profile.md"), + "**Foreground path:** SQL\n**Point B:** Ask questions\n**Point A evidence:** Reads tables", + ) + await writeFile( + path.join(root, "paths/learning.activity.md"), + "**Foreground path:** SQL\n", + ) + await writeFile( + path.join(root, "paths/sql/index.path.md"), + "---\nstatus: ready\n---\n\n## Milestone map\n\n- Select rows: ready\n- Filter rows: planned\n- Combine rows: planned", + ) + const moduleRoot = path.join(root, "paths/sql/modules/01-basics") + await writeFile(path.join(moduleRoot, "index.path.md"), "# Basics") + const support = "Hint. Worked example. Self-check rationale. Try a smaller version. Stretch task." + await writeFile(path.join(moduleRoot, "01-select.lesson.md"), support) + await writeFile(path.join(moduleRoot, "02-filter.lesson.md"), support) + await writeFile(path.join(moduleRoot, "review.practice.md"), "Review") + await writeFile(path.join(moduleRoot, "milestone.assessment.md"), "Checkpoint") + + const result = await gradeLearningSpace(root, { + runVerification: false, + transcript: "Open http://localhost:3000/paths/sql/index.path.html", + }) + for (const id of [ + "repo.exists", + "skills.installed", + "version.verified-baseline", + "path.foreground", + "path.milestones", + "module.buffered", + "session.immediate-support", + "player.exact-route", + ]) { + expect(result.checks.find((check) => check.id === id)?.passed).toBe(true) + } + }) +}) diff --git a/scripts/evals.ts b/scripts/evals.ts new file mode 100644 index 0000000..90cc238 --- /dev/null +++ b/scripts/evals.ts @@ -0,0 +1,364 @@ +#!/usr/bin/env bun + +import { existsSync } from "node:fs" +import { mkdir, readFile, writeFile } from "node:fs/promises" +import path from "node:path" + +import { + defaultArtifactsRoot, + evalsRoot, + findEvalProfile, + gradeLearningSpace, + loadEvalProfiles, + loadRubric, + loadScenario, + loadScenarios, + repoRoot, + runCommand, + selectedWorkspaceContent, + summarizeDeterministicChecks, + subjectPrompt, + validateScenario, +} from "./evals/core" +import { runJudge, runSubject } from "./evals/codex" +import { formatDuration, writeReport } from "./evals/report" + +type Options = Record + +function parseArgs(args: string[]) { + const positional: string[] = [] + const options: Options = {} + for (let index = 0; index < args.length; index += 1) { + const argument = args[index] + if (!argument.startsWith("--")) { + positional.push(argument) + continue + } + const key = argument.slice(2) + if (key.startsWith("no-")) { + options[key.slice(3)] = false + continue + } + const next = args[index + 1] + if (next && !next.startsWith("--")) { + options[key] = next + index += 1 + } else options[key] = true + } + return { positional, options } +} + +function usage() { + return `PathMX eval harness + +Usage: + bun run eval -- check + bun run eval -- list + bun run eval -- plan + bun run eval -- run [options] + bun run eval -- grade [--judge] + +Run options: + --profile Subject profile (default: desktop-power) + --model Override the subject profile model + --reasoning Override the subject profile reasoning effort + --runs Repetitions (default: 1) + --judge Run independent model judge + --judge-profile Judge profile (default: judge-quality) + --judge-model Override the judge profile model + --judge-reasoning Override the judge profile effort + --bootstrap-url Override local/hosted bootstrap source + --candidate-starter Stage a local starter candidate instead of fetching + --artifacts Artifact root outside the subject workspace + --codex-home Optional clean, already-authenticated CODEX_HOME + --sandbox workspace-write (default) or danger-full-access + --timeout-minutes Per-turn timeout (default: 30) +` +} + +async function checkSuite() { + const findings: string[] = [] + const scenarios = await loadScenarios().catch((error) => { + findings.push(error instanceof Error ? error.message : String(error)) + return [] + }) + for (const scenario of scenarios) { + findings.push(...validateScenario(scenario, scenario.id)) + } + await loadRubric().catch((error) => findings.push(error instanceof Error ? error.message : String(error))) + const profiles = await loadEvalProfiles().catch((error) => { + findings.push(error instanceof Error ? error.message : String(error)) + return undefined + }) + for (const file of ["scenario.schema.json", "judge-result.schema.json"]) { + try { + JSON.parse(await readFile(path.join(evalsRoot, "schemas", file), "utf8")) + } catch (error) { + findings.push(`${file}: ${error instanceof Error ? error.message : String(error)}`) + } + } + const codex = await runCommand(["codex", "--version"], { cwd: repoRoot, timeoutMs: 10_000 }) + if (codex.exitCode !== 0) findings.push("codex CLI is not available") + if (codex.exitCode === 0 && profiles) { + const models = await runCommand(["codex", "debug", "models"], { + cwd: repoRoot, + timeoutMs: 10_000, + }) + if (models.exitCode !== 0) findings.push("could not read the Codex model catalog") + else { + const available = new Set( + (JSON.parse(models.stdout).models as Array<{ slug: string }>).map((model) => model.slug), + ) + for (const profile of profiles.profiles) { + if (!available.has(profile.model)) { + findings.push(`eval profile ${profile.id} model is unavailable: ${profile.model}`) + } + } + } + } + if (findings.length > 0) throw new Error(findings.join("\n")) + console.log(`Checked ${scenarios.length} scenarios and the shared rubric.`) + console.log(codex.stdout.trim()) +} + +async function gradeRun(runDir: string, withJudge: boolean, options: Options) { + const metadata = JSON.parse(await readFile(path.join(runDir, "run.json"), "utf8")) + const scenarioFile = path.join(runDir, "scenario.json") + const scenario = existsSync(scenarioFile) + ? (JSON.parse(await readFile(scenarioFile, "utf8")) as Awaited>) + : await loadScenario(metadata.scenarioId) + const rubric = await loadRubric() + const profiles = await loadEvalProfiles() + const judgeProfileId = String( + options["judge-profile"] ?? metadata.judgeProfile ?? profiles.defaultJudge, + ) + const judgeProfile = findEvalProfile(profiles, judgeProfileId, "judge") + const workspace = path.join(runDir, "subject", scenario.learner.learningSpace) + const transcript = await readFile(path.join(runDir, "transcript.md"), "utf8").catch(() => "") + let deterministic = await gradeLearningSpace(workspace, { transcript }) + const phaseContracts = JSON.parse( + await readFile(path.join(runDir, "phase-contracts.json"), "utf8").catch(() => "[]"), + ) as Array<{ + turnId: string + passed: boolean + checks: Array<{ id: string; expected: string; actual: string; passed: boolean }> + }> + if (phaseContracts.length > 0) { + const failures = phaseContracts.flatMap((contract) => + contract.checks + .filter((check) => !check.passed) + .map( + (check) => + `${contract.turnId}: ${check.id} expected ${check.expected}, got ${check.actual}`, + ), + ) + deterministic = summarizeDeterministicChecks([ + ...deterministic.checks, + { + id: "sequence.phase-contract", + label: "Learning work appears at the intended phase", + passed: failures.length === 0, + critical: false, + weight: 3, + evidence: failures.length ? failures.join("; ") : `${phaseContracts.length} phase contracts passed`, + }, + ]) + } + await writeFile( + path.join(runDir, "deterministic.json"), + `${JSON.stringify(deterministic, null, 2)}\n`, + ) + await writeFile( + path.join(runDir, "workspace-content.md"), + await selectedWorkspaceContent(workspace), + ) + const judge = withJudge + ? await runJudge(runDir, rubric, scenario, { + model: String(options["judge-model"] ?? metadata.judgeModel ?? judgeProfile.model), + reasoning: String( + options["judge-reasoning"] ?? metadata.judgeReasoning ?? judgeProfile.reasoning, + ), + timeoutMs: Number(options["timeout-minutes"] ?? 30) * 60_000, + codexHome: + typeof options["codex-home"] === "string" + ? path.resolve(options["codex-home"]) + : metadata.codexHome, + }) + : undefined + const report = await writeReport(runDir, scenario, deterministic, rubric, judge) + console.log(`${scenario.id}: ${report.combined.percentage}% (${report.combined.criticalPassed ? "critical pass" : "critical fail"})`) + console.log(path.join(runDir, "report.md")) + return report +} + +async function main() { + const { positional, options } = parseArgs(Bun.argv.slice(2)) + const command = positional[0] ?? "help" + if (command === "help" || command === "--help") { + console.log(usage()) + return + } + if (command === "check") { + await checkSuite() + return + } + if (command === "list") { + for (const scenario of await loadScenarios()) { + console.log(`${scenario.id.padEnd(24)} ${scenario.title}`) + } + return + } + if (command === "plan") { + const scenario = await loadScenario(positional[1] ?? "") + const bootstrap = scenario.bootstrap.source === "local" ? "./bootstrap.md" : scenario.bootstrap.url ?? "" + console.log(subjectPrompt(scenario, bootstrap)) + for (const turn of scenario.turns) console.log(`\n[${turn.phase}:${turn.id}]\n${turn.message}`) + return + } + if (command === "grade") { + const runDir = path.resolve(positional[1] ?? "") + if (!existsSync(path.join(runDir, "run.json"))) throw new Error("Run directory has no run.json") + await gradeRun(runDir, options.judge === true, options) + return + } + if (command !== "run") throw new Error(`Unknown command: ${command}\n\n${usage()}`) + + const scenario = await loadScenario(positional[1] ?? "") + const artifactsRoot = path.resolve(String(options.artifacts ?? defaultArtifactsRoot())) + const repetitions = Number(options.runs ?? 1) + if (!Number.isInteger(repetitions) || repetitions < 1 || repetitions > 20) { + throw new Error("--runs must be an integer from 1 to 20") + } + const timeoutMs = Number(options["timeout-minutes"] ?? 30) * 60_000 + const profiles = await loadEvalProfiles() + const subjectProfileId = String(options.profile ?? profiles.defaultSubject) + const subjectProfile = findEvalProfile(profiles, subjectProfileId, "subject") + const model = String(options.model ?? subjectProfile.model) + const reasoning = String(options.reasoning ?? subjectProfile.reasoning) + const sandbox = String(options.sandbox ?? "workspace-write") + if (sandbox !== "workspace-write" && sandbox !== "danger-full-access") { + throw new Error("--sandbox must be workspace-write or danger-full-access") + } + const judgeProfileId = String(options["judge-profile"] ?? profiles.defaultJudge) + const judgeProfile = findEvalProfile(profiles, judgeProfileId, "judge") + const judgeModel = String(options["judge-model"] ?? judgeProfile.model) + const judgeReasoning = String(options["judge-reasoning"] ?? judgeProfile.reasoning) + const codexVersion = await runCommand(["codex", "--version"], { + cwd: repoRoot, + timeoutMs: 10_000, + }) + const packageJson = JSON.parse(await readFile(path.join(repoRoot, "package.json"), "utf8")) + await mkdir(artifactsRoot, { recursive: true }) + const stamp = new Date().toISOString().replaceAll(":", "-").replace(/\.\d{3}Z$/, "Z") + const batchDir = path.join(artifactsRoot, `${scenario.id}-${stamp}`) + await mkdir(batchDir, { recursive: false }) + const reports: Awaited>[] = [] + + for (let repetition = 1; repetition <= repetitions; repetition += 1) { + const runDir = path.join(batchDir, `run-${String(repetition).padStart(2, "0")}`) + await mkdir(runDir, { recursive: false }) + const metadata = { + schema: "pathmx-eval/run", + version: 1, + scenarioId: scenario.id, + scenarioFile: `evals/scenarios/${scenario.id}.yaml`, + startedAt: new Date().toISOString(), + repetition, + subjectProfile: subjectProfileId, + subjectModel: model, + subjectReasoning: reasoning, + subjectSandbox: sandbox, + codexVersion: codexVersion.stdout.trim(), + pathmxCompatibility: packageJson.pathmxCompatibility, + judgeRequested: options.judge === true, + judgeProfile: judgeProfileId, + judgeModel, + judgeReasoning, + bootstrapUrl: options["bootstrap-url"], + candidateStarter: + typeof options["candidate-starter"] === "string" + ? path.resolve(options["candidate-starter"]) + : undefined, + codexHome: + typeof options["codex-home"] === "string" + ? path.resolve(options["codex-home"]) + : undefined, + } + await writeFile(path.join(runDir, "run.json"), `${JSON.stringify(metadata, null, 2)}\n`) + await writeFile(path.join(runDir, "scenario.json"), `${JSON.stringify(scenario, null, 2)}\n`) + console.log(`Starting ${scenario.id} run ${repetition}/${repetitions}`) + console.log(runDir) + await runSubject(scenario, runDir, { + model, + reasoning, + timeoutMs, + sandbox, + codexHome: + typeof options["codex-home"] === "string" + ? path.resolve(options["codex-home"]) + : undefined, + bootstrapUrl: typeof options["bootstrap-url"] === "string" ? options["bootstrap-url"] : undefined, + candidateStarter: + typeof options["candidate-starter"] === "string" + ? path.resolve(options["candidate-starter"]) + : undefined, + }) + reports.push(await gradeRun(runDir, options.judge === true, options)) + } + + const percentages = reports + .map((report) => report.combined.percentage) + .sort((a, b) => a - b) + const middle = Math.floor(percentages.length / 2) + const median = percentages.length % 2 + ? percentages[middle] + : (percentages[middle - 1] + percentages[middle]) / 2 + const batch = { + scenarioId: scenario.id, + subjectProfile: subjectProfileId, + subjectModel: model, + subjectReasoning: reasoning, + runs: reports.length, + criticalPasses: reports.filter((report) => report.combined.criticalPassed).length, + criticalPassRate: + Math.round( + (reports.filter((report) => report.combined.criticalPassed).length / reports.length) * + 1000, + ) / 10, + median, + worst: percentages[0], + best: percentages.at(-1), + medianTotalDurationMs: 0, + slowestTurnDurationMs: Math.max( + ...reports.map((report) => report.timing.slowestTurn?.durationMs ?? 0), + ), + runsWithTurnsOverFiveMinutes: reports.filter( + (report) => report.timing.turnsOverFiveMinutes > 0, + ).length, + longestSilentGapMs: Math.max( + ...reports.map((report) => report.timing.longestSilentTurn?.longestSilentGapMs ?? 0), + ), + runsWithSilentMinute: reports.filter( + (report) => report.timing.turnsWithSilentMinute > 0, + ).length, + } + const totalDurations = reports + .map((report) => report.timing.totalDurationMs) + .sort((left, right) => left - right) + const durationMiddle = Math.floor(totalDurations.length / 2) + batch.medianTotalDurationMs = totalDurations.length % 2 + ? totalDurations[durationMiddle] + : (totalDurations[durationMiddle - 1] + totalDurations[durationMiddle]) / 2 + await writeFile(path.join(batchDir, "batch-report.json"), `${JSON.stringify(batch, null, 2)}\n`) + await writeFile( + path.join(batchDir, "batch-report.md"), + `# Eval batch: ${scenario.title}\n\n- Subject profile: ${batch.subjectProfile}\n- Model: ${batch.subjectModel}\n- Reasoning: ${batch.subjectReasoning}\n- Runs: ${batch.runs}\n- Critical pass rate: ${batch.criticalPassRate}%\n- Median: ${batch.median}%\n- Worst: ${batch.worst}%\n- Best: ${batch.best}%\n- Median total model time: ${formatDuration(batch.medianTotalDurationMs)}\n- Slowest turn: ${formatDuration(batch.slowestTurnDurationMs)}\n- Runs with a turn over five minutes: ${batch.runsWithTurnsOverFiveMinutes}/${batch.runs}\n- Longest silent gap: ${formatDuration(batch.longestSilentGapMs)}\n- Runs with a silent gap over one minute: ${batch.runsWithSilentMinute}/${batch.runs}\n`, + ) + console.log(`Batch: ${batch.criticalPassRate}% critical pass; median ${batch.median}%; worst ${batch.worst}%`) + console.log(path.join(batchDir, "batch-report.md")) +} + +await main().catch((error) => { + console.error(error instanceof Error ? error.stack ?? error.message : String(error)) + process.exit(1) +}) diff --git a/scripts/evals/codex.ts b/scripts/evals/codex.ts new file mode 100644 index 0000000..57c2adc --- /dev/null +++ b/scripts/evals/codex.ts @@ -0,0 +1,375 @@ +import { cp, mkdir, readFile, writeFile } from "node:fs/promises" +import path from "node:path" + +import { + captureWorkspace, + evaluatePhaseContract, + evalsRoot, + gradeLearningSpace, + parseCodexEvents, + repoRoot, + runCommand, + subjectPrompt, + type EvalScenario, + type JudgeResult, + type PhaseContractResult, + type Rubric, +} from "./core" + +export type CodexOptions = { + model: string + reasoning: string + timeoutMs: number + codexHome?: string + sandbox?: "workspace-write" | "danger-full-access" +} + +export function subjectCodexArgs( + options: CodexOptions, + cwd: string, + finalFile: string, + prompt: string, + threadId?: string, +) { + const global = [ + "-a", + "never", + "-C", + cwd, + "-s", + options.sandbox ?? "workspace-write", + "-m", + options.model, + "-c", + `model_reasoning_effort=${JSON.stringify(options.reasoning)}`, + "-c", + "sandbox_workspace_write.network_access=true", + "--disable", + "plugins", + ] + const exec = ["exec", "--ignore-user-config", "--ignore-rules", "--json", "-o", finalFile] + if (threadId) exec.push("resume", threadId, prompt) + else exec.push(prompt) + return ["codex", ...global, ...exec] +} + +export function judgeCodexArgs( + options: CodexOptions, + runDir: string, + outputFile: string, + schemaFile: string, +) { + return [ + "codex", + "-a", + "never", + "-C", + runDir, + "-s", + "read-only", + "-m", + options.model, + "-c", + `model_reasoning_effort=${JSON.stringify(options.reasoning)}`, + "--disable", + "plugins", + "exec", + "--skip-git-repo-check", + "--ignore-user-config", + "--ignore-rules", + "--ephemeral", + "--json", + "--output-schema", + schemaFile, + "-o", + outputFile, + "Read judge/judge-input.md and return the required structured evaluation.", + ] +} + +export function summarizeVisibleUpdates( + lines: Array<{ elapsedMs: number; line: string }>, + durationMs: number, +) { + const updates = lines.flatMap(({ elapsedMs, line }) => { + try { + const event = JSON.parse(line) as { + type?: string + item?: { type?: string; text?: string } + } + return event.type === "item.completed" && + event.item?.type === "agent_message" && + event.item.text?.trim() + ? [elapsedMs] + : [] + } catch { + return [] + } + }) + const boundaries = [0, ...updates, durationMs] + let longestSilentGapMs = 0 + for (let index = 1; index < boundaries.length; index += 1) { + longestSilentGapMs = Math.max(longestSilentGapMs, boundaries[index] - boundaries[index - 1]) + } + return { + updateCount: updates.length, + firstUpdateMs: updates[0], + longestSilentGapMs, + } +} + +async function runCodexTurn( + args: string[], + cwd: string, + artifactDir: string, + timeoutMs: number, + codexHome?: string, +) { + await mkdir(artifactDir, { recursive: true }) + const taskTemp = path.join(cwd, ".eval-tmp") + await mkdir(taskTemp, { recursive: true }) + const result = await runCommand(args, { + cwd, + timeoutMs, + env: { + TMPDIR: taskTemp, + BUN_TMPDIR: taskTemp, + ...(codexHome ? { CODEX_HOME: codexHome } : {}), + }, + }) + const visibleUpdates = summarizeVisibleUpdates(result.stdoutLineTimings, result.durationMs) + await writeFile(path.join(artifactDir, "events.jsonl"), result.stdout) + await writeFile(path.join(artifactDir, "stderr.log"), result.stderr) + await writeFile( + path.join(artifactDir, "process.json"), + `${JSON.stringify( + { + command: result.command.map((part, index) => (index === result.command.length - 1 ? "" : part)), + exitCode: result.exitCode, + timedOut: result.timedOut, + durationMs: result.durationMs, + ...visibleUpdates, + }, + null, + 2, + )}\n`, + ) + if (result.exitCode !== 0 || result.timedOut) { + throw new Error( + `Codex turn failed (${result.timedOut ? "timeout" : `exit ${result.exitCode}`}):\n${result.stderr.slice(-3000)}`, + ) + } + const parsed = parseCodexEvents(result.stdout) + if (parsed.invalid.length > 0) throw new Error(`Invalid Codex JSONL: ${parsed.invalid.join(", ")}`) + await writeFile( + path.join(artifactDir, "turn.json"), + `${JSON.stringify( + { threadId: parsed.threadId, usage: parsed.usage, eventCount: parsed.events.length }, + null, + 2, + )}\n`, + ) + return parsed +} + +async function capturePhaseGrade( + workspace: string, + turnDir: string, + transcript: Array<{ id: string; phase: string; learner: string; agent: string }>, +) { + const result = await gradeLearningSpace(workspace, { + runVerification: false, + transcript: transcript.map((turn) => turn.agent).join("\n"), + }) + await writeFile( + path.join(turnDir, "phase-grade.json"), + `${JSON.stringify(result, null, 2)}\n`, + ) + return result +} + +export async function runSubject( + scenario: EvalScenario, + runDir: string, + options: CodexOptions & { bootstrapUrl?: string; candidateStarter?: string }, +) { + const subjectRoot = path.join(runDir, "subject") + const turnsRoot = path.join(runDir, "turns") + await mkdir(subjectRoot, { recursive: true }) + const git = await runCommand(["git", "init", "-q"], { cwd: subjectRoot, timeoutMs: 10_000 }) + if (git.exitCode !== 0) throw new Error(`Could not initialize subject root: ${git.stderr}`) + + let bootstrapReference: string + if (options.bootstrapUrl || scenario.bootstrap.source === "hosted") { + bootstrapReference = options.bootstrapUrl ?? scenario.bootstrap.url ?? "" + } else { + const destination = path.join(subjectRoot, "bootstrap.md") + await Bun.write(destination, Bun.file(path.join(repoRoot, "bootstrap.md"))) + bootstrapReference = "./bootstrap.md" + } + + if (options.candidateStarter) { + const source = path.resolve(options.candidateStarter) + const destination = path.join(subjectRoot, "candidate-starter") + await cp(source, destination, { + recursive: true, + dereference: false, + filter(candidate) { + const relative = path.relative(source, candidate) + return !relative + .split(path.sep) + .some((part) => [".git", "node_modules", ".pathmx", ".pathmx-check"].includes(part)) + }, + }) + } + + const initialPrompt = subjectPrompt( + scenario, + bootstrapReference, + Boolean(options.candidateStarter), + ) + await writeFile(path.join(runDir, "subject-prompt.md"), `${initialPrompt}\n`) + const transcript: Array<{ id: string; phase: string; learner: string; agent: string }> = [] + const phaseContracts: PhaseContractResult[] = [] + let threadId: string | undefined + const initialDir = path.join(turnsRoot, "00-bootstrap") + const initialFinal = path.join(initialDir, "final.md") + const initial = await runCodexTurn( + subjectCodexArgs(options, subjectRoot, initialFinal, initialPrompt), + subjectRoot, + initialDir, + options.timeoutMs, + options.codexHome, + ) + threadId = initial.threadId + if (!threadId) throw new Error("Codex did not emit a thread.started event") + const initialAgent = await readFile(initialFinal, "utf8").catch(() => "") + transcript.push({ id: "bootstrap", phase: "bootstrap", learner: initialPrompt, agent: initialAgent }) + await captureWorkspace( + path.join(subjectRoot, scenario.learner.learningSpace), + path.join(initialDir, "snapshot"), + ) + await capturePhaseGrade( + path.join(subjectRoot, scenario.learner.learningSpace), + initialDir, + transcript, + ) + + for (const [index, turn] of scenario.turns.entries()) { + const turnDir = path.join( + turnsRoot, + `${String(index + 1).padStart(2, "0")}-${turn.id}`, + ) + const finalFile = path.join(turnDir, "final.md") + await runCodexTurn( + subjectCodexArgs(options, subjectRoot, finalFile, turn.message, threadId), + subjectRoot, + turnDir, + options.timeoutMs, + options.codexHome, + ) + const agent = await readFile(finalFile, "utf8").catch(() => "") + transcript.push({ id: turn.id, phase: turn.phase, learner: turn.message, agent }) + await captureWorkspace( + path.join(subjectRoot, scenario.learner.learningSpace), + path.join(turnDir, "snapshot"), + ) + const phaseGrade = await capturePhaseGrade( + path.join(subjectRoot, scenario.learner.learningSpace), + turnDir, + transcript, + ) + const phaseContract = evaluatePhaseContract(turn, phaseGrade) + if (phaseContract) { + phaseContracts.push(phaseContract) + await writeFile( + path.join(turnDir, "phase-contract.json"), + `${JSON.stringify(phaseContract, null, 2)}\n`, + ) + } + } + + await writeFile(path.join(runDir, "transcript.json"), `${JSON.stringify(transcript, null, 2)}\n`) + await writeFile( + path.join(runDir, "transcript.md"), + `${transcript + .map( + (turn) => + `# Turn: ${turn.id} (${turn.phase})\n\n## Learner\n\n${turn.learner}\n\n## Agent\n\n${turn.agent}`, + ) + .join("\n\n")}\n`, + ) + const phaseEvidence: string[] = [] + for (const [index, turn] of transcript.entries()) { + const directoryName = + index === 0 + ? "00-bootstrap" + : `${String(index).padStart(2, "0")}-${scenario.turns[index - 1].id}` + const phaseGrade = JSON.parse( + await readFile(path.join(turnsRoot, directoryName, "phase-grade.json"), "utf8"), + ) + phaseEvidence.push( + `## ${turn.id} (${turn.phase})\n\nStructural score: ${phaseGrade.percentage}%\n\nPassed: ${phaseGrade.checks + .filter((check: { passed: boolean }) => check.passed) + .map((check: { id: string }) => check.id) + .join(", ") || "none"}\n\nNot yet passed: ${phaseGrade.checks + .filter((check: { passed: boolean }) => !check.passed) + .map((check: { id: string }) => check.id) + .join(", ") || "none"}`, + ) + } + await writeFile(path.join(runDir, "phase-evidence.md"), `${phaseEvidence.join("\n\n")}\n`) + await writeFile( + path.join(runDir, "phase-contracts.json"), + `${JSON.stringify(phaseContracts, null, 2)}\n`, + ) + return { + threadId, + subjectRoot, + workspace: path.join(subjectRoot, scenario.learner.learningSpace), + transcript, + } +} + +export async function runJudge( + runDir: string, + rubric: Rubric, + scenario: EvalScenario, + options: CodexOptions, +): Promise { + const judgeDir = path.join(runDir, "judge") + await mkdir(judgeDir, { recursive: true }) + const promptFile = path.join(judgeDir, "judge-input.md") + const outputFile = path.join(judgeDir, "result.json") + const eventsFile = path.join(judgeDir, "events.jsonl") + const deterministic = await readFile(path.join(runDir, "deterministic.json"), "utf8") + const transcript = await readFile(path.join(runDir, "transcript.md"), "utf8") + const workspace = await readFile(path.join(runDir, "workspace-content.md"), "utf8") + const phases = await readFile(path.join(runDir, "phase-evidence.md"), "utf8") + const prompt = `# PathMX learning-space judge\n\nYou are an independent evaluator. Score only the supplied evidence. Do not repair the subject's work. The subject never saw this rubric.\n\nUse exactly the criterion IDs below. Score each 0, 1, or 2: 0 = absent or harmful, 1 = partial or inconsistent, 2 = clearly successful. Cite a transcript turn ID or repository file for every score. Deterministic failures are facts and must not be contradicted.\n\n## Scenario objective\n\n${scenario.objective}\n\n## Hidden learner notes\n\n${scenario.learner.hiddenNotes}\n\n## Rubric\n\n${JSON.stringify(rubric.criteria, null, 2)}\n\n## Final deterministic result\n\n\`\`\`json\n${deterministic}\n\`\`\`\n\n## Structural evidence by turn\n\n${phases}\n\n## Transcript\n\n${transcript}\n\n## Repository content\n\n${workspace}\n` + await writeFile(promptFile, prompt) + + const args = judgeCodexArgs( + options, + runDir, + outputFile, + path.join(evalsRoot, "schemas", "judge-result.schema.json"), + ) + const result = await runCommand(args, { + cwd: runDir, + timeoutMs: options.timeoutMs, + env: options.codexHome ? { CODEX_HOME: options.codexHome } : undefined, + }) + await writeFile(eventsFile, result.stdout) + await writeFile(path.join(judgeDir, "stderr.log"), result.stderr) + if (result.exitCode !== 0 || result.timedOut) { + throw new Error(`Judge failed: ${result.stderr.slice(-3000)}`) + } + const judged = JSON.parse(await readFile(outputFile, "utf8")) as JudgeResult + const expectedIds = rubric.criteria.map((criterion) => criterion.id).sort() + const actualIds = judged.criteria.map((criterion) => criterion.id).sort() + if (JSON.stringify(expectedIds) !== JSON.stringify(actualIds)) { + throw new Error(`Judge criterion IDs did not match rubric`) + } + return judged +} diff --git a/scripts/evals/core.ts b/scripts/evals/core.ts new file mode 100644 index 0000000..77eec52 --- /dev/null +++ b/scripts/evals/core.ts @@ -0,0 +1,764 @@ +import { existsSync } from "node:fs" +import { mkdir, readFile, readdir, writeFile } from "node:fs/promises" +import os from "node:os" +import path from "node:path" +import { parse } from "yaml" + +export const repoRoot = path.resolve(import.meta.dir, "../..") +export const evalsRoot = path.join(repoRoot, "evals") + +export const phases = [ + "onboarding", + "map", + "module", + "session", + "checkpoint", + "adaptation", + "return", +] as const + +export type Phase = (typeof phases)[number] + +export type EvalTurn = { + id: string + phase: Phase + message: string + expect?: { + passed?: string[] + notPassed?: string[] + } +} + +export type EvalScenario = { + schema: "pathmx-eval/scenario" + version: 1 + id: string + title: string + objective: string + tags: string[] + bootstrap: { source: "local" | "hosted"; url?: string } + learner: { + learningSpace: string + initialMessage: string + hiddenNotes: string + } + turns: EvalTurn[] +} + +export type RubricCriterion = { + id: string + phase: Phase + weight: number + description: string +} + +export type Rubric = { + schema: "pathmx-eval/rubric" + version: 1 + criteria: RubricCriterion[] +} + +export type EvalProfile = { + id: string + role: "subject" | "judge" + model: string + reasoning: string + description: string + evidence?: string + verifiedOn?: string +} + +export type EvalProfiles = { + schema: "pathmx-eval/profiles" + version: 1 + defaultSubject: string + defaultJudge: string + profiles: EvalProfile[] +} + +export type CodexEvent = Record & { + type?: string + thread_id?: string + usage?: Record + item?: { type?: string; text?: string } +} + +export type DeterministicCheck = { + id: string + label: string + passed: boolean + critical: boolean + weight: number + evidence: string +} + +export type DeterministicResult = { + score: number + maximum: number + percentage: number + criticalPassed: boolean + checks: DeterministicCheck[] +} + +export type PhaseContractResult = { + turnId: string + phase: Phase + passed: boolean + checks: Array<{ + id: string + expected: "passed" | "not-passed" + actual: "passed" | "not-passed" | "missing" + passed: boolean + }> +} + +export type JudgeCriterion = { + id: string + score: number + evidence: string[] + reason: string +} + +export type JudgeResult = { + summary: string + criteria: JudgeCriterion[] + strengths: string[] + risks: string[] +} + +function isRecord(value: unknown): value is Record { + return Boolean(value) && typeof value === "object" && !Array.isArray(value) +} + +function requireString( + value: unknown, + location: string, + findings: string[], +): value is string { + if (typeof value !== "string" || !value.trim()) { + findings.push(`${location} must be a non-empty string`) + return false + } + return true +} + +export function validateScenario(value: unknown, file = "scenario") { + const findings: string[] = [] + if (!isRecord(value)) return [`${file} must contain a mapping`] + if (value.schema !== "pathmx-eval/scenario") findings.push(`${file}: invalid schema`) + if (value.version !== 1) findings.push(`${file}: unsupported version`) + requireString(value.id, `${file}.id`, findings) + if (typeof value.id === "string" && !/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(value.id)) { + findings.push(`${file}.id must be kebab-case`) + } + requireString(value.title, `${file}.title`, findings) + requireString(value.objective, `${file}.objective`, findings) + if (!Array.isArray(value.tags) || value.tags.some((tag) => typeof tag !== "string")) { + findings.push(`${file}.tags must be an array of strings`) + } + + if (!isRecord(value.bootstrap)) { + findings.push(`${file}.bootstrap must be a mapping`) + } else { + if (value.bootstrap.source !== "local" && value.bootstrap.source !== "hosted") { + findings.push(`${file}.bootstrap.source must be local or hosted`) + } + if (value.bootstrap.source === "hosted") { + requireString(value.bootstrap.url, `${file}.bootstrap.url`, findings) + } + } + + if (!isRecord(value.learner)) { + findings.push(`${file}.learner must be a mapping`) + } else { + requireString(value.learner.learningSpace, `${file}.learner.learningSpace`, findings) + if ( + typeof value.learner.learningSpace === "string" && + !/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/.test(value.learner.learningSpace) + ) { + findings.push(`${file}.learner.learningSpace must be a simple directory name`) + } + requireString(value.learner.initialMessage, `${file}.learner.initialMessage`, findings) + requireString(value.learner.hiddenNotes, `${file}.learner.hiddenNotes`, findings) + } + + if (!Array.isArray(value.turns) || value.turns.length === 0) { + findings.push(`${file}.turns must contain at least one learner turn`) + } else { + const ids = new Set() + value.turns.forEach((turn, index) => { + const location = `${file}.turns[${index}]` + if (!isRecord(turn)) { + findings.push(`${location} must be a mapping`) + return + } + requireString(turn.id, `${location}.id`, findings) + if (typeof turn.id === "string") { + if (ids.has(turn.id)) findings.push(`${location}.id must be unique`) + ids.add(turn.id) + } + if (!phases.includes(turn.phase as Phase)) findings.push(`${location}.phase is invalid`) + requireString(turn.message, `${location}.message`, findings) + if (turn.expect !== undefined) { + if (!isRecord(turn.expect)) findings.push(`${location}.expect must be a mapping`) + else { + for (const key of ["passed", "notPassed"] as const) { + const expected = turn.expect[key] + if ( + expected !== undefined && + (!Array.isArray(expected) || expected.some((id) => typeof id !== "string" || !id)) + ) { + findings.push(`${location}.expect.${key} must be an array of check IDs`) + } + } + } + } + }) + } + return findings +} + +export function evaluatePhaseContract( + turn: EvalTurn, + result: DeterministicResult, +): PhaseContractResult | undefined { + if (!turn.expect) return undefined + const actual = new Map(result.checks.map((check) => [check.id, check.passed])) + const checks: PhaseContractResult["checks"] = [] + for (const id of turn.expect.passed ?? []) { + const state = actual.get(id) + checks.push({ + id, + expected: "passed", + actual: state === undefined ? "missing" : state ? "passed" : "not-passed", + passed: state === true, + }) + } + for (const id of turn.expect.notPassed ?? []) { + const state = actual.get(id) + checks.push({ + id, + expected: "not-passed", + actual: state === undefined ? "missing" : state ? "passed" : "not-passed", + passed: state === false, + }) + } + return { + turnId: turn.id, + phase: turn.phase, + passed: checks.every((check) => check.passed), + checks, + } +} + +export function summarizeDeterministicChecks( + checks: DeterministicCheck[], +): DeterministicResult { + const maximum = checks.reduce((sum, check) => sum + check.weight, 0) + const score = checks + .filter((check) => check.passed) + .reduce((sum, check) => sum + check.weight, 0) + return { + score, + maximum, + percentage: maximum ? Math.round((score / maximum) * 1000) / 10 : 0, + criticalPassed: checks.filter((check) => check.critical).every((check) => check.passed), + checks, + } +} + +export async function loadScenario(idOrFile: string): Promise { + const file = idOrFile.endsWith(".yaml") + ? path.resolve(idOrFile) + : path.join(evalsRoot, "scenarios", `${idOrFile}.yaml`) + const value = parse(await readFile(file, "utf8")) + const findings = validateScenario(value, path.relative(repoRoot, file)) + if (findings.length > 0) throw new Error(findings.join("\n")) + return value as EvalScenario +} + +export async function loadScenarios() { + const root = path.join(evalsRoot, "scenarios") + const files = (await readdir(root)) + .filter((file) => file.endsWith(".yaml")) + .sort() + return Promise.all(files.map((file) => loadScenario(path.join(root, file)))) +} + +export async function loadRubric(): Promise { + const file = path.join(evalsRoot, "rubrics", "learning-space.json") + const value = JSON.parse(await readFile(file, "utf8")) as Rubric + if (value.schema !== "pathmx-eval/rubric" || value.version !== 1) { + throw new Error("Unsupported eval rubric") + } + const ids = new Set() + for (const criterion of value.criteria) { + if (!criterion.id || ids.has(criterion.id)) throw new Error("Rubric IDs must be unique") + if (!phases.includes(criterion.phase)) throw new Error(`Invalid rubric phase: ${criterion.phase}`) + if (!Number.isFinite(criterion.weight) || criterion.weight <= 0) { + throw new Error(`Invalid rubric weight: ${criterion.id}`) + } + ids.add(criterion.id) + } + return value +} + +export async function loadEvalProfiles(): Promise { + const file = path.join(evalsRoot, "profiles.json") + const value = JSON.parse(await readFile(file, "utf8")) as EvalProfiles + if (value.schema !== "pathmx-eval/profiles" || value.version !== 1) { + throw new Error("Unsupported eval profile catalog") + } + const ids = new Set() + for (const profile of value.profiles) { + if (!profile.id || ids.has(profile.id)) throw new Error("Eval profile IDs must be unique") + if (profile.role !== "subject" && profile.role !== "judge") { + throw new Error(`Invalid eval profile role: ${profile.id}`) + } + for (const key of ["model", "reasoning", "description"] as const) { + if (!profile[key]?.trim()) throw new Error(`Eval profile ${profile.id} needs ${key}`) + } + ids.add(profile.id) + } + const subject = value.profiles.find((profile) => profile.id === value.defaultSubject) + const judge = value.profiles.find((profile) => profile.id === value.defaultJudge) + if (subject?.role !== "subject") throw new Error("defaultSubject must name a subject profile") + if (judge?.role !== "judge") throw new Error("defaultJudge must name a judge profile") + return value +} + +export function findEvalProfile( + catalog: EvalProfiles, + id: string, + role: "subject" | "judge", +) { + const profile = catalog.profiles.find((entry) => entry.id === id) + if (!profile) throw new Error(`Unknown eval profile: ${id}`) + if (profile.role !== role) throw new Error(`Eval profile ${id} cannot be used as ${role}`) + return profile +} + +export function parseCodexEvents(jsonl: string) { + const events: CodexEvent[] = [] + const invalid: string[] = [] + for (const [index, line] of jsonl.split(/\r?\n/).entries()) { + if (!line.trim()) continue + try { + events.push(JSON.parse(line) as CodexEvent) + } catch { + invalid.push(`line ${index + 1}`) + } + } + const threadId = events.find((event) => event.type === "thread.started")?.thread_id + const usage = events + .filter((event) => event.type === "turn.completed" && isRecord(event.usage)) + .map((event) => event.usage ?? {}) + .reduce>((total, current) => { + for (const [key, value] of Object.entries(current)) { + if (typeof value === "number") total[key] = (total[key] ?? 0) + value + } + return total + }, {}) + return { events, invalid, threadId, usage } +} + +export function subjectPrompt( + scenario: EvalScenario, + bootstrapReference: string, + candidateStarter = false, +) { + const parts = [ + `Follow the bootstrap instructions at ${bootstrapReference} and carry them out rather than only explaining them.`, + scenario.learner.initialMessage, + "Treat this as a normal learner interaction. Ask only for information or permission the instructions require.", + ] + if (candidateStarter) { + parts.push( + "Candidate-starter lane: for bootstrap step 3, create the requested learning repository by copying ./candidate-starter to the requested location instead of fetching the published template with pathmx init. Do not edit ./candidate-starter itself. Then continue every remaining bootstrap step normally, including installation, Git initialization, candidate-version verification, Player review, and onboarding.", + ) + } + return parts.join("\n\n") +} + +export function defaultArtifactsRoot() { + return path.join(os.tmpdir(), "pathmx-evals") +} + +export async function listFiles(root: string, prefix = ""): Promise { + if (!existsSync(root)) return [] + const files: string[] = [] + for (const entry of await readdir(root, { withFileTypes: true })) { + if ([".git", "node_modules", ".pathmx", ".pathmx-check"].includes(entry.name)) continue + const relative = path.join(prefix, entry.name) + if (entry.isDirectory()) files.push(...(await listFiles(path.join(root, entry.name), relative))) + else if (entry.isFile()) files.push(relative) + } + return files.sort() +} + +export async function runCommand( + command: string[], + options: { cwd: string; timeoutMs?: number; env?: Record }, +) { + const started = Date.now() + const child = Bun.spawn(command, { + cwd: options.cwd, + env: { ...process.env, ...options.env }, + stdout: "pipe", + stderr: "pipe", + }) + let timedOut = false + const timer = setTimeout(() => { + timedOut = true + child.kill() + }, options.timeoutMs ?? 20 * 60 * 1000) + const stdoutReader = child.stdout.getReader() + const stdoutDecoder = new TextDecoder() + const stdoutParts: string[] = [] + const stdoutLineTimings: Array<{ elapsedMs: number; line: string }> = [] + let pendingLine = "" + const readStdout = async () => { + while (true) { + const { done, value } = await stdoutReader.read() + if (done) break + const chunk = stdoutDecoder.decode(value, { stream: true }) + stdoutParts.push(chunk) + pendingLine += chunk + let newline = pendingLine.indexOf("\n") + while (newline >= 0) { + stdoutLineTimings.push({ + elapsedMs: Date.now() - started, + line: pendingLine.slice(0, newline), + }) + pendingLine = pendingLine.slice(newline + 1) + newline = pendingLine.indexOf("\n") + } + } + const tail = stdoutDecoder.decode() + if (tail) { + stdoutParts.push(tail) + pendingLine += tail + } + if (pendingLine) { + stdoutLineTimings.push({ elapsedMs: Date.now() - started, line: pendingLine }) + } + return stdoutParts.join("") + } + const [stdout, stderr, exitCode] = await Promise.all([ + readStdout(), + new Response(child.stderr).text(), + child.exited, + ]) + clearTimeout(timer) + return { + command, + stdout, + stderr, + stdoutLineTimings, + exitCode, + timedOut, + durationMs: Date.now() - started, + } +} + +export async function captureWorkspace(workspace: string, destination: string) { + await mkdir(destination, { recursive: true }) + const files = await listFiles(workspace) + const gitStatus = existsSync(path.join(workspace, ".git")) + ? await runCommand(["git", "status", "--short"], { cwd: workspace, timeoutMs: 10_000 }) + : undefined + const gitLog = existsSync(path.join(workspace, ".git")) + ? await runCommand(["git", "log", "--oneline", "--decorate", "-10"], { + cwd: workspace, + timeoutMs: 10_000, + }) + : undefined + await writeFile( + path.join(destination, "workspace.json"), + `${JSON.stringify({ exists: existsSync(workspace), files, gitStatus, gitLog }, null, 2)}\n`, + ) +} + +async function readIfPresent(file: string) { + return existsSync(file) ? readFile(file, "utf8") : "" +} + +function addCheck( + checks: DeterministicCheck[], + id: string, + label: string, + passed: boolean, + evidence: string, + options: { critical?: boolean; weight?: number } = {}, +) { + checks.push({ + id, + label, + passed, + evidence, + critical: options.critical ?? false, + weight: options.weight ?? 1, + }) +} + +async function pathDirectories(pathsRoot: string) { + if (!existsSync(pathsRoot)) return [] + const ignored = new Set(["assets", "getting-started", ".fixtures"]) + const directories: string[] = [] + for (const entry of await readdir(pathsRoot, { withFileTypes: true })) { + if ( + entry.isDirectory() && + !ignored.has(entry.name) && + existsSync(path.join(pathsRoot, entry.name, "index.path.md")) + ) { + directories.push(path.join(pathsRoot, entry.name)) + } + } + return directories.sort() +} + +async function moduleDirectories(pathRoots: string[]) { + const modules: string[] = [] + for (const pathRoot of pathRoots) { + const root = path.join(pathRoot, "modules") + if (!existsSync(root)) continue + for (const entry of await readdir(root, { withFileTypes: true })) { + if (entry.isDirectory()) modules.push(path.join(root, entry.name)) + } + } + return modules.sort() +} + +export async function gradeLearningSpace( + workspace: string, + options: { runVerification?: boolean; transcript?: string } = {}, +): Promise { + const checks: DeterministicCheck[] = [] + const pathsRoot = path.join(workspace, "paths") + const exists = existsSync(workspace) + addCheck(checks, "repo.exists", "Learning repository exists", exists, workspace, { + critical: true, + weight: 3, + }) + const gitExists = existsSync(path.join(workspace, ".git")) + addCheck(checks, "git.initialized", "Local Git repository initialized", gitExists, ".git", { + critical: true, + weight: 2, + }) + + let remotes = "" + let remoteCommandPassed = false + if (gitExists) { + const result = await runCommand(["git", "remote"], { cwd: workspace, timeoutMs: 10_000 }) + remotes = result.stdout.trim() + remoteCommandPassed = result.exitCode === 0 + } + addCheck( + checks, + "git.no-remotes", + "No remote was created", + gitExists && remoteCommandPassed && !remotes, + remoteCommandPassed ? remotes || "none" : "git remote failed", + { critical: true, weight: 2 }, + ) + + const skills = ["path", "pathmx"].filter((name) => + existsSync(path.join(workspace, ".agents", "skills", name, "SKILL.md")), + ) + addCheck( + checks, + "skills.installed", + "Path and PathMX skills installed", + skills.length === 2, + skills.join(", ") || "none", + { critical: true, weight: 3 }, + ) + + let packageJson: Record = {} + try { + packageJson = JSON.parse(await readFile(path.join(workspace, "package.json"), "utf8")) + } catch {} + const baseline = packageJson.pathmxCompatibility?.baseline + const dependency = packageJson.dependencies?.["@fellowhumans/pathmx"] + const exactVersions = + /^\d+\.\d+\.\d+$/.test(baseline ?? "") && + /^\d+\.\d+\.\d+$/.test(dependency ?? "") && + baseline === dependency + addCheck( + checks, + "version.verified-baseline", + "Project dependency matches an exact verified baseline", + exactVersions, + `baseline=${baseline ?? "missing"}; dependency=${dependency ?? "missing"}`, + { critical: true, weight: 3 }, + ) + + const coreFiles = ["index.path.md", "learner.profile.md", "learning.activity.md"] + const presentCore = coreFiles.filter((file) => existsSync(path.join(pathsRoot, file))) + addCheck( + checks, + "learning.durable-state", + "Home, learner profile, and activity history exist", + presentCore.length === coreFiles.length, + presentCore.join(", ") || "none", + { critical: true, weight: 3 }, + ) + + const profile = await readIfPresent(path.join(pathsRoot, "learner.profile.md")) + const activity = await readIfPresent(path.join(pathsRoot, "learning.activity.md")) + const confirmedProfile = + profile.length > 0 && + !/Point B:\*\* Not confirmed/i.test(profile) && + !/Point A evidence:\*\* Not collected/i.test(profile) + addCheck( + checks, + "learner.confirmed-profile", + "Point A and Point B are confirmed", + confirmedProfile, + confirmedProfile ? "learner.profile.md contains confirmed state" : "starter placeholders remain", + { weight: 2 }, + ) + + const pathRoots = await pathDirectories(pathsRoot) + const foregroundNamed = + pathRoots.length > 0 && + !/Foreground path:\*\* (?:None|Not chosen yet)/i.test(`${profile}\n${activity}`) + addCheck( + checks, + "path.foreground", + "A foreground learning path exists", + foregroundNamed, + pathRoots.map((root) => path.basename(root)).join(", ") || "none", + { critical: true, weight: 3 }, + ) + + const pathIndexes = await Promise.all( + pathRoots.map((root) => readIfPresent(path.join(root, "index.path.md"))), + ) + const milestoneLines = pathIndexes + .flatMap((content) => { + const afterHeading = content.split(/^## Milestone map\s*$/im)[1] + return afterHeading?.split(/^##\s/m)[0]?.split(/\r?\n/) ?? [] + }) + .filter((line) => /\b(?:planned|ready|in progress|demonstrated|paused)\b/i.test(line)) + const milestoneCount = milestoneLines.length + addCheck( + checks, + "path.milestones", + "Milestone map has 3–7 visible status entries", + milestoneCount >= 3 && milestoneCount <= 7, + `${milestoneCount} candidate milestone lines`, + { weight: 2 }, + ) + + const modules = await moduleDirectories(pathRoots) + const sessionCounts = await Promise.all( + modules.map(async (moduleRoot) => { + const files = await readdir(moduleRoot).catch(() => []) + return { + root: moduleRoot, + count: files.filter((file) => file.endsWith(".lesson.md")).length, + checkpoint: files.some((file) => /(?:milestone|checkpoint).*\.assessment\.md$/i.test(file)), + review: files.some((file) => /(?:review|practice).*\.practice\.md$/i.test(file)), + } + }), + ) + const readyModule = sessionCounts.find( + (module) => module.count >= 2 && module.count <= 4 && module.checkpoint, + ) + addCheck( + checks, + "module.buffered", + "A module has 2–4 sessions and a checkpoint", + Boolean(readyModule), + sessionCounts + .map((module) => `${path.basename(module.root)}:${module.count}`) + .join(", ") || "none", + { critical: true, weight: 3 }, + ) + addCheck( + checks, + "module.review-ready", + "Buffered review or practice is ready", + sessionCounts.some((module) => module.review), + sessionCounts.filter((module) => module.review).map((module) => path.basename(module.root)).join(", ") || "none", + { weight: 1 }, + ) + + const lessonFiles = (await listFiles(pathsRoot)).filter((file) => file.endsWith(".lesson.md")) + const lessonText = ( + await Promise.all(lessonFiles.map((file) => readIfPresent(path.join(pathsRoot, file)))) + ).join("\n") + const supportKinds = [ + /\bhint\b/i, + /worked example|\bexample\b/i, + /\b(?:rationale|rubric|self-check)\b/i, + /\bsmaller\b/i, + /\bstretch\b/i, + ].filter((pattern) => pattern.test(lessonText)).length + addCheck( + checks, + "session.immediate-support", + "Sessions contain several kinds of immediate support", + supportKinds >= 3, + `${supportKinds}/5 support kinds found`, + { weight: 2 }, + ) + + const transcript = options.transcript ?? "" + const routeEvidence = /https?:\/\/(?:localhost|127\.0\.0\.1):\d+\/\S+/i.test(transcript) + addCheck( + checks, + "player.exact-route", + "Agent supplied a specific local Player URL", + routeEvidence, + routeEvidence ? "local Player URL found in transcript" : "no local Player URL found", + { weight: 1 }, + ) + + if (options.runVerification !== false && exists && packageJson.scripts?.check) { + const verification = await runCommand(["bun", "run", "check"], { + cwd: workspace, + timeoutMs: 3 * 60 * 1000, + }) + const diagnostics = `${verification.stdout}\n${verification.stderr}`.trim() + addCheck( + checks, + "pathmx.verification", + "Repository check passes", + verification.exitCode === 0 && !verification.timedOut, + diagnostics.slice(-2000) || `exit ${verification.exitCode}`, + { critical: true, weight: 4 }, + ) + } else if (options.runVerification !== false) { + addCheck( + checks, + "pathmx.verification", + "Repository check passes", + false, + "check script missing", + { critical: true, weight: 4 }, + ) + } + + return summarizeDeterministicChecks(checks) +} + +export async function selectedWorkspaceContent(workspace: string) { + const files = (await listFiles(workspace)).filter( + (file) => + file === "AGENTS.md" || + file === "package.json" || + (file.startsWith("paths/") && /\.(?:md|css)$/.test(file)), + ) + const chunks: string[] = [] + let budget = 80_000 + for (const file of files) { + const content = await readIfPresent(path.join(workspace, file)) + const chunk = `\n## FILE: ${file}\n\n${content}\n` + if (chunk.length > budget) break + chunks.push(chunk) + budget -= chunk.length + } + return chunks.join("") +} diff --git a/scripts/evals/report.ts b/scripts/evals/report.ts new file mode 100644 index 0000000..70ac440 --- /dev/null +++ b/scripts/evals/report.ts @@ -0,0 +1,158 @@ +import { readFile, readdir, writeFile } from "node:fs/promises" +import path from "node:path" + +import type { + DeterministicResult, + EvalScenario, + JudgeResult, + Rubric, +} from "./core" + +export type TurnTiming = { + turn: string + durationMs: number + timedOut: boolean + updateCount?: number + firstUpdateMs?: number + longestSilentGapMs?: number +} + +export type TimingSummary = { + turns: TurnTiming[] + totalDurationMs: number + slowestTurn?: TurnTiming + turnsOverFiveMinutes: number + turnsWithSilentMinute: number + longestSilentTurn?: TurnTiming +} + +export function summarizeTimings(turns: TurnTiming[]): TimingSummary { + const sorted = [...turns].sort((left, right) => left.turn.localeCompare(right.turn)) + return { + turns: sorted, + totalDurationMs: sorted.reduce((total, turn) => total + turn.durationMs, 0), + slowestTurn: sorted.reduce( + (slowest, turn) => (!slowest || turn.durationMs > slowest.durationMs ? turn : slowest), + undefined, + ), + turnsOverFiveMinutes: sorted.filter((turn) => turn.durationMs > 5 * 60_000).length, + turnsWithSilentMinute: sorted.filter((turn) => (turn.longestSilentGapMs ?? 0) > 60_000).length, + longestSilentTurn: sorted.reduce( + (longest, turn) => + !longest || (turn.longestSilentGapMs ?? 0) > (longest.longestSilentGapMs ?? 0) + ? turn + : longest, + undefined, + ), + } +} + +async function readTimingSummary(runDir: string) { + const turnsRoot = path.join(runDir, "turns") + const directories = await readdir(turnsRoot, { withFileTypes: true }).catch(() => []) + const turns: TurnTiming[] = [] + for (const entry of directories) { + if (!entry.isDirectory()) continue + const processFile = path.join(turnsRoot, entry.name, "process.json") + const process = JSON.parse(await readFile(processFile, "utf8").catch(() => "{}")) as { + durationMs?: unknown + timedOut?: unknown + updateCount?: unknown + firstUpdateMs?: unknown + longestSilentGapMs?: unknown + } + if (typeof process.durationMs !== "number") continue + turns.push({ + turn: entry.name, + durationMs: process.durationMs, + timedOut: process.timedOut === true, + updateCount: typeof process.updateCount === "number" ? process.updateCount : undefined, + firstUpdateMs: typeof process.firstUpdateMs === "number" ? process.firstUpdateMs : undefined, + longestSilentGapMs: + typeof process.longestSilentGapMs === "number" ? process.longestSilentGapMs : undefined, + }) + } + return summarizeTimings(turns) +} + +export function formatDuration(durationMs: number) { + const totalSeconds = Math.round(durationMs / 1000) + const minutes = Math.floor(totalSeconds / 60) + const seconds = totalSeconds % 60 + return minutes ? `${minutes}m ${String(seconds).padStart(2, "0")}s` : `${seconds}s` +} + +export function scoreJudge(judge: JudgeResult, rubric: Rubric) { + const weights = new Map(rubric.criteria.map((criterion) => [criterion.id, criterion.weight])) + let score = 0 + let maximum = 0 + for (const criterion of judge.criteria) { + const weight = weights.get(criterion.id) ?? 0 + score += criterion.score * weight + maximum += 2 * weight + } + return { + score, + maximum, + percentage: maximum ? Math.round((score / maximum) * 1000) / 10 : 0, + } +} + +export async function writeReport( + runDir: string, + scenario: EvalScenario, + deterministic: DeterministicResult, + rubric: Rubric, + judge?: JudgeResult, +) { + const judgeScore = judge ? scoreJudge(judge, rubric) : undefined + const timing = await readTimingSummary(runDir) + let combined = deterministic.percentage + if (judgeScore) combined = deterministic.percentage * 0.6 + judgeScore.percentage * 0.4 + if (!deterministic.criticalPassed) combined = Math.min(combined, 59) + combined = Math.round(combined * 10) / 10 + + const report = { + scenario: { id: scenario.id, title: scenario.title, objective: scenario.objective }, + deterministic, + timing, + judge: judge ? { score: judgeScore, result: judge } : undefined, + combined: { + percentage: combined, + criticalPassed: deterministic.criticalPassed, + note: judge + ? "60% deterministic and 40% judge; critical failure caps the score at 59" + : "deterministic score only; no model judge requested", + }, + } + await writeFile(path.join(runDir, "report.json"), `${JSON.stringify(report, null, 2)}\n`) + + const checkRows = deterministic.checks + .map( + (check) => + `| ${check.passed ? "pass" : "fail"} | ${check.critical ? "yes" : "no"} | ${check.label} | ${check.evidence.replaceAll("|", "\\|").replaceAll("\n", " ").slice(0, 240)} |`, + ) + .join("\n") + const judgeRows = judge + ? `\n## Model judge\n\nScore: **${judgeScore?.percentage}%**\n\n${judge.criteria + .map((criterion) => `- ${criterion.id}: ${criterion.score}/2 — ${criterion.reason}`) + .join("\n")}\n\n${judge.summary}\n` + : "\nNo model judge was requested.\n" + const slowest = timing.slowestTurn + const timingSignal = + timing.turnsOverFiveMinutes > 0 || timing.turnsWithSilentMinute > 0 + ? "attention" + : "within turn and visible-update budgets" + const timingRows = timing.turns + .map( + (turn) => + `| ${turn.turn} | ${formatDuration(turn.durationMs)} | ${turn.firstUpdateMs === undefined ? "unknown" : formatDuration(turn.firstUpdateMs)} | ${turn.longestSilentGapMs === undefined ? "unknown" : formatDuration(turn.longestSilentGapMs)} | ${turn.updateCount ?? "unknown"} | ${turn.timedOut ? "yes" : "no"} |`, + ) + .join("\n") + const timingMarkdown = timing.turns.length + ? `\n## Learner wait-time signal\n\nThis signal is reported separately and does not change the quality score.\n\n- Status: **${timingSignal}**\n- Total model time: ${formatDuration(timing.totalDurationMs)}\n- Slowest turn: ${slowest?.turn ?? "none"} (${formatDuration(slowest?.durationMs ?? 0)})\n- Turns over five minutes: ${timing.turnsOverFiveMinutes}\n- Turns with a silent gap over one minute: ${timing.turnsWithSilentMinute}\n- Longest silent turn: ${timing.longestSilentTurn?.turn ?? "none"} (${formatDuration(timing.longestSilentTurn?.longestSilentGapMs ?? 0)})\n\n| Turn | Duration | First visible update | Longest silence | Updates | Timed out |\n| --- | --- | --- | --- | --- | --- |\n${timingRows}\n` + : "\n## Learner wait-time signal\n\nNo turn timing artifacts were found.\n" + const markdown = `# Eval report: ${scenario.title}\n\nCombined score: **${combined}%** \nCritical checks: **${deterministic.criticalPassed ? "pass" : "fail"}**\n\n## Deterministic checks\n\nScore: **${deterministic.percentage}%**\n\n| Result | Critical | Check | Evidence |\n| --- | --- | --- | --- |\n${checkRows}\n${timingMarkdown}${judgeRows}` + await writeFile(path.join(runDir, "report.md"), markdown) + return report +} diff --git a/scripts/path-skill.test.ts b/scripts/path-skill.test.ts index 77beaeb..2a5b087 100644 --- a/scripts/path-skill.test.ts +++ b/scripts/path-skill.test.ts @@ -10,89 +10,119 @@ async function read(relative: string) { } describe("path skill contract", () => { - it("uses the path package and explicit invocation", async () => { + it("pins one machine-readable PathMX compatibility baseline", async () => { + const packageJson = JSON.parse(await read("package.json")) + const baseline = packageJson.pathmxCompatibility.baseline + expect(baseline).toBe(packageJson.devDependencies["@fellowhumans/pathmx"]) + expect(packageJson.pathmxCompatibility.updatePolicy).toBe( + "latest-after-verification", + ) + expect(baseline).toMatch(/^\d+\.\d+\.\d+$/) + }) + + it("supports implicit and explicit invocation", async () => { const skill = await read("skills/path/SKILL.md") const frontmatter = parse(skill.match(/^---\n([\s\S]*?)\n---/)?.[1] ?? "") const interfaceConfig = parse(await read("skills/path/agents/openai.yaml")) + const manifest = JSON.parse(await read("skills/manifest.json")) + const declared = manifest.skills.find((entry: { name: string }) => entry.name === "path") + expect(frontmatter.name).toBe("path") - expect(interfaceConfig.policy.allow_implicit_invocation).toBe(false) + expect(frontmatter.description).toMatch(/automatically/) + expect(interfaceConfig.policy?.allow_implicit_invocation).not.toBe(false) + expect(declared.invocation).toBe("automatic-and-explicit") }) - it("delegates authoring and keeps the learning loop", async () => { + it("uses a buffered module instead of Block-at-a-time generation", async () => { const skill = await read("skills/path/SKILL.md") - expect(skill).toContain("`/pathmx`") for (const term of [ - "Assess position", - "proximal edge", - "retrieval", - "spacing", - "interleaving", - "synthesis", + "3–7 milestones", + "2–4 sessions", + "current module", + "worked example", + "without waiting", + "module checkpoint", + "annotations", ]) { expect(skill).toContain(term) } - expect(skill).toMatch(/capability\s+gap/) + expect(skill).not.toMatch(/one agent turn (?:adds|=) one Block/i) + expect(skill).not.toMatch(/no core concept missed before every/i) }) - it("does not keep legacy HTML workspace rules", async () => { + it("ships a progressive map artifact before the buffered module", async () => { const skill = await read("skills/path/SKILL.md") - expect(skill).not.toMatch(/MISSION\.md|RESOURCES\.md|NOTES\.md|standalone HTML/) + const map = await read("skills/path/assets/path/index.path.md") + expect(skill).toMatch(/before showing it to the\s+learner/) + expect(skill).toContain("Do not create session, review, or checkpoint Sources") + expect(map).toContain("# Proposed learning path") + expect(map.match(/\b(?:ready|planned):?\b/g)?.length ?? 0).toBeGreaterThanOrEqual(3) + expect(map).toContain("Evidence:") }) - it("documents all five worked files", async () => { - const example = await read("skills/path/references/worked-example.md") - for (const file of [ - "path.outcome.md", - "index.path.md", - "index.lesson.md", - "lesson.assessment.md", + it("keeps evidence, progress, and history durable", async () => { + const skill = await read("skills/path/SKILL.md") + for (const term of [ + "Point A", + "Point B", + "evidence targets", "learning.activity.md", + "append-only", + "foreground path", ]) { - expect(example).toContain(file) + expect(skill).toContain(term) } }) - it("points at the adaptive loop fixture", async () => { + it("personalizes presentation without replacing the learning structure", async () => { + const skill = await read("skills/path/SKILL.md") + expect(skill).toContain("visual mood") + expect(skill).toContain("theme tokens") + expect(skill).toContain("reduced motion") + expect(skill).toContain("Keep navigation and learning structure stable") + }) + + it("owns Player uptime, exact routes, and browser fallback", async () => { const skill = await read("skills/path/SKILL.md") - const adaptive = await read("skills/path/references/adaptive-loop-example.md") + const pathmx = await read("skills/pathmx/SKILL.md") + for (const content of [skill, pathmx]) { + expect(content).toContain("pathmx route") + expect(content).toContain("@Browser") + expect(content).toMatch(/system browser|clickable/i) + } + }) + + it("ships self-contained loop and worked references", async () => { + const loop = await read("skills/path/references/buffered-loop.md") const example = await read("skills/path/references/worked-example.md") - expect(skill).toContain("adaptive-loop-example.md") - expect(skill).toContain("onboarding/") - expect(skill).toContain("no core concept missed") - expect(skill).toContain("pathmx play") - expect(skill).toContain("paths/index.path.md") - expect(skill).toContain("theme.css") - expect(skill).toContain("assets/") - expect(adaptive).toContain("chess-opening-principles") - expect(adaptive).toContain("pass = no core miss") - expect(example).toContain("does **not** include onboarding") + expect(loop).toContain("Planning horizon") + expect(loop).toContain("One module") + expect(loop).toContain("Do not use a no-core-miss gate after every session") + expect(example).toContain("Both sessions exist before the learner starts") + expect(example).toContain("Milestone checkpoint") }) - it("keeps assessment evidence and synthesis linked", async () => { - const assessment = await read( - "tests/fixtures/path/paths/sql-foundations/lessons/joins/lesson.assessment.md", - ) - const activity = await read("tests/fixtures/path/paths/learning.activity.md") - expect(assessment).toContain("## Rubric") - expect(assessment).toContain("learning.activity.md") - expect(activity).toContain("lesson.assessment.md") - expect(activity).toContain("**Opens next:**") + it("keeps Tram's contribution visible and specific", async () => { + const readme = await read("README.md") + expect(readme).toContain("Tram Le") + expect(readme).toContain("early hands-on testing") + expect(readme).toContain("math, media, code, tooling") }) - it("keeps the chess adaptive loop fixture wired", async () => { - const pathIndex = await read( - "tests/fixtures/path/paths/chess-opening-principles/index.path.md", - ) - const assessment = await read( - "tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.assessment.md", - ) - const activity = await read("tests/fixtures/path/paths/learning.activity.md") - expect(pathIndex).toContain("onboarding/index.lesson.md") - expect(pathIndex).toContain("lesson.review.md") - expect(assessment).toContain("pass_rule: no_core_miss") - expect(assessment).toContain("Agent rubric") - expect(assessment).toContain("learning.activity.md") - expect(activity).toContain("chess-opening-principles") - expect(activity).toContain("**Opens next:**") + it("provides a one-file bootstrap", async () => { + const bootstrap = await read("bootstrap.md") + for (const term of [ + "bun --version", + "pathmx self-update", + "@fellowhumans/pathmx@latest", + "pathmx-learning-starter", + "git init", + "bun run play", + "bun run check:candidate", + "Player tutorial", + "git restore package.json bun.lock", + ]) { + expect(bootstrap).toContain(term) + } }) }) diff --git a/skills/manifest.json b/skills/manifest.json index 8b5f146..5fc4313 100644 --- a/skills/manifest.json +++ b/skills/manifest.json @@ -12,8 +12,8 @@ { "name": "path", "directory": "skills/path", - "invocation": "explicit", - "purpose": "Create and resume one adaptive learning path.", + "invocation": "automatic-and-explicit", + "purpose": "Create and resume buffered adaptive personal learning paths.", "dependsOn": ["pathmx"] } ] diff --git a/skills/path/SKILL.md b/skills/path/SKILL.md index 16074d6..e15d6ff 100644 --- a/skills/path/SKILL.md +++ b/skills/path/SKILL.md @@ -1,89 +1,214 @@ --- name: path -description: Create and resume an opinionated personal learning path in a PathMX workspace. Use assessment evidence to choose the next lesson for one learner and one learning goal. +description: Start, plan, teach, and resume personal learning in a PathMX learning space. Use automatically when one learner asks to learn, study, practice, build a curriculum, continue a learning path, review progress, or turn a goal into guided lessons, including when a new personal learning repository must be created. --- # Personal Learning Path -Build one durable, adaptive path for one learner. Use `/pathmx` for PathMX -syntax and verification. - -Present learner-facing lessons, reviews, and assessments in the PathMX Player. -Start or reuse `pathmx play` for the active root Source and give the learner -its Player route. Do not use the raw Markdown file as the learning interface. -After creating or updating playable material, open it in the Player again. - -## Learning loop - -Run the same loop at path and lesson level: - -1. **Assess position.** Find Point A from evidence, not vibes: past work, - artifact, sample, picture, or a short diagnostic. -2. **Aim.** Agree on Point B, a small rubric, Lesson 1’s destination, and two - or three Later titles. Confirm before teaching. -3. **Experience.** Build one lesson at the proximal edge. One agent turn adds - one Block with a stable `id` (~40–120 words). -4. **Review.** Ask what the learner can do now and what is still fuzzy. Offer - gap help and short practice until they are ready. -5. **Assess outcome.** Use 3–4 multiple-choice items plus one short answer - grounded in what was taught. Pass = no core concept missed. -6. **Record and advance.** Write a synthesis, update current Point A, and - place the next lesson from evidence. - -Do not build the whole path ahead. Later items stay titles until they are next. - -## Start or resume - -Before teaching, read the nearest repository instructions and root Path -Source. Reuse the workspace's layout when one exists. - -For a new path: - -1. Onboard: goal, why, prior knowledge, stuck points, time/pace, preferred feel. -2. Collect Point A evidence, then write Point A in observable language citing it. -3. Draft Point B, rubric, Lesson 1 destination, and Later titles. -4. Persist the plan and wait for confirm (`roadmap.status: confirmed`). -5. Build only the first lesson shell, then teach one Block at a time. - -For a returning path: - -1. Read `paths/index.path.md`, the learner profile, activity log, active path, - latest assessment, and latest synthesis. -2. Give a short recap and two or three retrieval prompts from earlier work. -3. Continue from the recorded `stage` cursor and current Point A. - -Record only learner-confirmed context. Avoid sensitive data by default. - -## Build one lesson - -1. Set `start` from the current position and latest synthesis. -2. Set `destination` as an observable learner capability. -3. Teach one small win at the proximal edge with retrieval, practice, and - immediate feedback. -4. Persist `stage` in lesson frontmatter after each turn. -5. After teaching, run a short review. If unsure, name the gap and add practice. -6. Write a plain Markdown assessment with agent-facing rationales, concept - tags (core vs peripheral), and a short-answer rubric. -7. Gate the next lesson on evidence. Prefer spaced assessment at the next - session when a break occurs. -8. On a core miss, encourage and retry with variant items. After 2–3 failed - attempts, build a smaller remediation lesson instead of advancing Later. -9. If remediation assessment also fails, renegotiate Point B or pace — do not - loop remediation again. -10. Record a synthesis: evidence, change in perspective, review-queue items, - placement, and what opens next. - -If the installed tools cannot capture suitable evidence, report the capability -gap. Do not weaken the assessment. PathMX does not grade automatically; the -agent scores against the rubric and writes the durable result. +Build a durable personal learning space for one learner. Use the installed +`pathmx` skill for PathMX syntax, Player routes, and verification. + +Keep the learner moving. Prepare a coherent runway of material, then adapt at +meaningful boundaries. Never require an agent turn between ordinary lesson +Blocks. + +## Open or create the space + +If no learning repository exists, carry out the same setup as the hosted +bootstrap: + +1. Ask where to create a new directory. +2. Ensure Bun is installed. +3. Install or update the native command with `pathmx self-update`, or + `bunx @fellowhumans/pathmx@latest self-update` when it is missing. +4. Run + `pathmx init --template pathmx-learning-starter`. +5. Install the lockfile, initialize local Git, and commit the verified scaffold + baseline. +6. Compare the exact project dependency with the freshly updated native + `pathmx --version`. If they match, run the normal check once instead of + reinstalling the same package. If they differ, attempt the latest project + dependency as the version migration described in the `pathmx` tooling + reference. Keep it only after the full build and Player smoke pass; + otherwise restore the baseline version files. +7. Read the new repository instructions and continue there. + +Shell tool calls may not preserve a prior `cd`. Set the new repository as the +working directory, or prefix every project command with an explicit `cd`, so +install, verification, Player, and route commands run in the intended space. + +Never replace an existing directory or create a remote without permission. +Treat the space as private personal data. + +For an existing space, read the root Source, learner profile, activity log, +active path, current module, latest checkpoint, and unresolved annotations. +Read only the history needed to continue. + +## Ask a small onboarding set + +Ask a few questions at a time: + +1. What does the learner want to be able to do, and why? +2. What do they already know or find difficult? +3. What small artifact, example, explanation, or diagnostic can show their + current position? +4. How much time and energy can they usually give a session? +5. What learning feel helps: direct, conversational, exploratory, practical, + visual, or another preference? +6. What visual mood, color direction, light/dark preference, readability need, + or motion sensitivity should shape the Player? + +Save only learner-confirmed, learning-relevant context. Do not solicit private +credentials or unnecessary sensitive information. + +## Map visible progress + +Turn onboarding evidence into: + +- **Point A:** an observable current capability, citing the evidence; +- **Point B:** the capability the learner wants; +- **3–7 milestones:** named capabilities between A and B; +- **evidence targets:** what demonstrates each milestone; +- **current module:** 2–4 sessions with one coherent purpose; +- **later modules:** provisional titles and outcomes. + +Write the proposed map into the learning repository before showing it to the +learner. Use the bundled `assets/path/index.path.md` scaffold so every one of +the 3–7 milestones has one visible `planned`, `ready`, `in progress`, +`demonstrated`, or `paused` status plus an evidence target. Link the proposed +foreground Path from the home Source. This first useful artifact should be +available in Player before the learner waits for the substantial module build. +Update the learner profile and the activity record's current state in the same +change so they agree on the foreground Path. + +In the learning starter, keep `paths/index.path.md` as the single configured +Player root. A learner's individual Path is a nested Source linked from home, +not another `pathmx.config.md` Path entry. Rebuild the home graph after linking +it; do not add Source `handle` frontmatter to make ordinary nested routes work. + +Show the learner that persisted map and confirm it before teaching. Keep +exactly one foreground path while allowing other paths to remain paused or +completed. + +When the learner asks to see the map first, stop after writing and linking the +proposed Path Source. Do not create session, review, or checkpoint Sources in +that turn. Give its exact Player URL, label it as a proposal, and wait for +explicit confirmation of Point B, the milestone map, and the proposed current +module. A request to show a map is not confirmation. + +Do not fully author the entire future curriculum. Fully author the current +module and keep later modules easy to change. + +## Build a learning runway + +Prepare all sessions in the current module before asking the learner to begin. +Keep optional consolidation or retrieval material ready so a slow agent never +blocks learning. Let the learner continue without waiting for another agent +turn. + +Before calling the module ready, check every session for a worked example, an +optional hint or smaller attempt, and an immediate rationale, self-check, or +rubric. Keep focused review and an optional stretch task ready in the module. + +For a standard two-session module, copy the bundled `assets/module/` scaffold +into the new module directory and replace its author notes. Add or remove +session files only when the learner's confirmed rhythm calls for it. + +Work in visible stages when the agent surface supports progress updates: + +1. Create the module index and session skeletons, then report that the runway + exists. +2. Fill the first session, then report the concrete capability now ready. +3. Fill the remaining sessions, review, and checkpoint, then report that the + uninterrupted module is ready for verification. +4. Run one targeted build or route check, then one full check before handoff. + +Keep these updates factual and brief; they are learner-visible progress, not +requests for another reply. Reuse the active Player. Do not repeat installation, +migration, or full compatibility checks within the same healthy task. + +Design each session for roughly 15–30 minutes unless the learner chose another +pace. Give it a complete arc: + +1. **Orient.** State why this matters, the destination, and the session map. +2. **Model.** Explain with a concrete example or worked example. +3. **Guide.** Offer a supported attempt with optional hints. +4. **Apply.** Ask for an independent or transfer attempt. +5. **Check.** Provide immediate self-check, rationale, or a clear rubric. +6. **Reflect.** Invite a short note, question, or annotation. +7. **Complete.** Summarize the capability practiced and what comes next. + +Use Blocks for meaningful phases and Beats for useful reveals. Do not turn +every sentence into a Beat. Make the session readable outside Play mode. + +## Provide immediate and slower feedback + +Build attractive, structured Player support that works without an agent turn: + +- show a hint; +- inspect a worked example; +- try a smaller version; +- reveal a rationale or rubric after an attempt; +- choose an optional stretch task. + +Use existing starter components when they fit. Create bespoke Literate +Components only when they improve the learning experience. + +Treat annotations as an asynchronous curriculum-feedback inbox. Encourage the +learner to mark confusion, disagreement, useful ideas, or requests for more +depth. On return, review open annotations, reply or revise future material, and +resolve threads only when addressed. Preserve learner-authored comments. + +## Adapt at useful boundaries + +Use session checks as low-stakes evidence. Let the learner continue through a +module with small uncertainties when later work does not depend on them. + +At the module checkpoint: + +1. Review work, responses, reflections, and annotations. +2. Compare the evidence with the milestone target. +3. Mark the capability demonstrated, keep it in progress, or offer focused + review. +4. Update current Point A and prepare the next module. +5. Record a short synthesis and the reason for placement. + +Gate progression only when a later capability genuinely depends on a missed +core idea. After repeated difficulty, provide a smaller remediation module or +renegotiate the goal or pace. Do not trap the learner in a remediation loop. + +## Personalize presentation + +Start from the repository's readable default theme. Translate confirmed style +preferences into a small set of theme tokens: color, surface, typography, +measure, contrast, and motion. Keep navigation and learning structure stable. + +Prefer restrained personalization over a bespoke interface. Check contrast, +narrow screens, keyboard use, reduced motion, and both requested color modes. + +## Keep the Player live + +Reuse a healthy Player server that belongs to the repository or start +`bun run play` in a long-lived terminal. Resolve the exact Source route with +`bunx pathmx route` and link to the useful Source, Block, or Beat position. + +Open and review learner-facing work in an integrated browser when available. +In Codex, prefer `@Browser`; in Claude Code, use its Chrome integration when +already configured. Otherwise use the system browser or provide a clickable +URL. Teach first-time learners with the bundled Player tutorial. + +At handoff, give the exact starting URL, say how to enter and navigate Play +mode, name the ready sessions, and tell the learner when to return for review. ## Default layout -This is `/path`'s default, not a general PathMX requirement: +Use the starter's layout when present. The portable default is: ```text paths/ ├── index.path.md +├── getting-started/ +│ └── player.lesson.md ├── learner.profile.md ├── learning.activity.md ├── theme.css @@ -92,42 +217,34 @@ paths/ └── / ├── index.path.md ├── path.outcome.md - ├── onboarding/ - │ ├── index.lesson.md - │ ├── point-a-evidence.md - │ └── confirm-plan.md - ├── lessons/ - │ └── / - │ ├── index.lesson.md - │ ├── lesson.review.md - │ ├── lesson.practice.md - │ └── lesson.assessment.md - └── references/ - └── index.references.md + └── modules/ + └── 01-/ + ├── index.path.md + ├── 01-.lesson.md + ├── 02-.lesson.md + ├── review.practice.md + └── milestone.assessment.md ``` -- `paths/index.path.md` is the learner home / root Source. -- `theme.css` and `assets/` are optional shared styling and components. -- `path.outcome.md` records Point A, Point B, and the path rubric. -- Path `index.path.md` records the outline, gates, and current position. -- `index.lesson.md` records historical `start`, `destination`, and `stage`. -- `lesson.review.md` gates readiness before assessment. -- `lesson.assessment.md` asks for observable evidence; pass = no core miss. -- `learning.activity.md` records reviews, syntheses, placements, and the - review queue. +- `paths/index.path.md` is the personal learning home. +- `learner.profile.md` stores confirmed learning and style preferences. +- `learning.activity.md` is an append-only record of evidence, syntheses, and + placement decisions. +- A path index shows Point A, Point B, milestones, progress, and the current + module. +- A module index links its fully prepared sessions and checkpoint. +- Completed Sources remain history. Do not silently rewrite past evidence. -Renegotiate Point B explicitly when evidence changes the learner's goal. Log -the change; do not rewrite history silently. Complete a path only when its -rubric is met and the learner agrees. +Read the [buffered loop](./references/buffered-loop.md) and the compact +[worked example](./references/worked-example.md) when planning or changing a +path. ## Learning rules -- Use backward design: define evidence before content. -- Teach at the zone of proximal development. -- Use retrieval, spacing, and interleaving to build storage strength. -- Keep lessons small and concrete. -- Prefer real transfer tasks over recall alone. -- Advance on evidence, not fluency or impressions. - -See the compact [worked example](./references/worked-example.md) and the full -[adaptive loop fixture](./references/adaptive-loop-example.md). +- Use backward design: name evidence before detailed content. +- Use the proximal edge to choose difficulty, not generation cadence. +- Prefer worked examples for novices, then fade support. +- Use retrieval, spacing, interleaving, practice, and transfer deliberately. +- Keep sessions concrete and finishable. +- Adapt from evidence without making the learner wait unnecessarily. +- Renegotiate Point B explicitly when the learner's goal changes. diff --git a/skills/path/agents/openai.yaml b/skills/path/agents/openai.yaml index c998d34..7af61ec 100644 --- a/skills/path/agents/openai.yaml +++ b/skills/path/agents/openai.yaml @@ -1,6 +1,4 @@ interface: display_name: "Personal Learning Path" - short_description: "Build one adaptive learning path" - default_prompt: "Use $path to create a learning path for my stated goal." -policy: - allow_implicit_invocation: false + short_description: "Build buffered adaptive learning paths" + default_prompt: "Use $path to help me start or continue learning in my personal PathMX space." diff --git a/skills/path/assets/module/01-session.lesson.md b/skills/path/assets/module/01-session.lesson.md new file mode 100644 index 0000000..c66d508 --- /dev/null +++ b/skills/path/assets/module/01-session.lesson.md @@ -0,0 +1,52 @@ +--- +type: lesson +status: ready +--- + +# Session 1 + + + +## Destination + + + +--- + +## Worked example + + + +--- + +## Guided attempt + + + +### Need a hint? + + + +### Try a smaller version + + + +--- + +## Apply it + + + +## Self-check and rationale + + + +## Optional stretch + + + +--- + +## Finish + + diff --git a/skills/path/assets/module/02-session.lesson.md b/skills/path/assets/module/02-session.lesson.md new file mode 100644 index 0000000..441beba --- /dev/null +++ b/skills/path/assets/module/02-session.lesson.md @@ -0,0 +1,52 @@ +--- +type: lesson +status: ready +--- + +# Session 2 + + + +## Destination + + + +--- + +## Worked example + + + +--- + +## Guided attempt + + + +### Need a hint? + + + +### Try a smaller version + + + +--- + +## Apply it + + + +## Self-check and rationale + + + +## Optional stretch + + + +--- + +## Finish + + diff --git a/skills/path/assets/module/index.path.md b/skills/path/assets/module/index.path.md new file mode 100644 index 0000000..bb162f8 --- /dev/null +++ b/skills/path/assets/module/index.path.md @@ -0,0 +1,18 @@ +--- +type: path +status: ready +--- + +# Current module + + + +## Ready runway + +1. [Session 1](./01-session.lesson.md) +2. [Session 2](./02-session.lesson.md) +3. [Optional focused review](./review.practice.md) +4. [Milestone checkpoint](./milestone.assessment.md) + +The learner can complete both sessions and the optional review without waiting +for another agent turn. Return after the checkpoint for evidence-based review. diff --git a/skills/path/assets/module/milestone.assessment.md b/skills/path/assets/module/milestone.assessment.md new file mode 100644 index 0000000..dfa0210 --- /dev/null +++ b/skills/path/assets/module/milestone.assessment.md @@ -0,0 +1,20 @@ +--- +type: assessment +status: ready +--- + +# Milestone checkpoint + +## Evidence to submit + + + +## Success rubric + + + +## After the checkpoint + +Return to the agent after completing this checkpoint. The agent will review the +evidence, update the visible milestone status, and prepare the next useful +module without rewriting completed work. diff --git a/skills/path/assets/module/review.practice.md b/skills/path/assets/module/review.practice.md new file mode 100644 index 0000000..076ea8a --- /dev/null +++ b/skills/path/assets/module/review.practice.md @@ -0,0 +1,20 @@ +--- +type: practice +status: ready +--- + +# Focused review + +Use this only when the learner wants consolidation before the checkpoint. + +## Quick retrieval + + + +## One smaller practice + + + +## Check and rationale + + diff --git a/skills/path/assets/path/index.path.md b/skills/path/assets/path/index.path.md new file mode 100644 index 0000000..f5ad036 --- /dev/null +++ b/skills/path/assets/path/index.path.md @@ -0,0 +1,38 @@ +--- +type: path +status: planned +--- + +# Proposed learning path + + + +## Point A + + + +## Point B + + + +## Milestone map + +- **First capability:** ready + - Evidence: +- **Second capability:** planned + - Evidence: +- **Third capability:** planned + - Evidence: + + + +## Proposed current module + + + +## Later modules + + + +This map is a proposal. Confirm or revise it before the current module is +authored. diff --git a/skills/path/references/adaptive-loop-example.md b/skills/path/references/adaptive-loop-example.md deleted file mode 100644 index ab9b48b..0000000 --- a/skills/path/references/adaptive-loop-example.md +++ /dev/null @@ -1,102 +0,0 @@ -# Adaptive Loop Example - -The chess fixture is the full `/path` loop: onboard → Point A evidence → -confirm plan → staged lesson → review → optional practice → assessment → -placement or remediation. - -Buildable Sources live under -[`tests/fixtures/path/paths/chess-opening-principles/`](../../../tests/fixtures/path/paths/chess-opening-principles/). -The longer design brief is -[`work-log/2026-07-20-adaptive-learning-loop.brief.md`](../../../work-log/2026-07-20-adaptive-learning-loop.brief.md). - -## Flow - -```mermaid -flowchart TD - Home[Home: New Path] --> Onboard[Onboarding] - Onboard --> Profile[Save answers to learner.profile.md] - Profile --> Evidence[Point A evidence diagnostic] - Evidence --> Draft[Draft Point A from evidence then Point B rubric roadmap] - Draft --> ReviewPlan[Learner reviews plan] - ReviewPlan -->|changes| Draft - ReviewPlan -->|Confirm| Lock[Write path.outcome + index.path] - Lock --> Teach[Lesson N: one Block per turn] - Teach --> PostReview[Short review] - PostReview --> ReadyCheck{Ready or still unsure?} - ReadyCheck -->|unsure| GapHelp[Name gap + help + practice] - GapHelp --> ReadyCheck - ReadyCheck -->|ready| Synth[Synthesis in learning.activity.md] - Synth --> Assess[Assessment of prior lessons] - Assess --> Place[Place learner on roadmap] - Place --> Done{Path rubric met + learner agrees?} - Done -->|yes| Complete[Mark path complete] - Done -->|no| Adapt[Update current Point A + Later drafts] - Adapt --> Teach -``` - -### Between lessons - -```mermaid -flowchart TD - L[Lesson N staged] --> R[Short review] - R --> Ready{Learner feels ready?} - Ready -->|yes| S[Synthesis] - Ready -->|still unsure| Gap[Ask which part feels unclear] - Gap --> Help[Offer help + extra practice on that gap] - Help --> Practice[Short practice Blocks] - Practice --> Ready - S --> A[Assessment: 3-4 MC + 1 short answer] - A --> Score{Any core concept missed?} - Score -->|no| U[Update roadmap] - U --> N[Lesson N+1 toward Point B] - Score -->|yes, attempt under 3| Retry[Encourage + retry a variant] - Retry --> A - Score -->|yes, after 2-3 fails| Remediate[Regenerate next lesson to reteach gaps] - Remediate --> TeachAgain[Staged remediation lesson] - TeachAgain --> R -``` - -### Inside a lesson - -```mermaid -flowchart LR - Shell[Shell: start + destination] --> B1[Block] - B1 --> Play[Play: Beats per click] - Play -->|Continue or Submit| Agent[Agent wake] - Agent -->|one Block only| Bnext[Next Block] - Bnext --> Play -``` - -One agent turn = one Block with a stable `id`. Persist `stage` in lesson -frontmatter after each turn so reload resumes from the Source, not from chat. - -## Journey - -1. [Onboarding](../../../tests/fixtures/path/paths/chess-opening-principles/onboarding/index.lesson.md) -2. [Point A evidence](../../../tests/fixtures/path/paths/chess-opening-principles/onboarding/point-a-evidence.md) -3. [Confirm plan](../../../tests/fixtures/path/paths/chess-opening-principles/onboarding/confirm-plan.md) -4. [Lesson 1](../../../tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/index.lesson.md) -5. [Short review](../../../tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.review.md) -6. [Extra practice](../../../tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.practice.md) (only if unsure) -7. [Assessment](../../../tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.assessment.md) — pass = no core miss -8. [Lesson 2 shell](../../../tests/fixtures/path/paths/chess-opening-principles/lessons/develop-before-queen/index.lesson.md) after a pass -9. [Remediation shell](../../../tests/fixtures/path/paths/chess-opening-principles/lessons/reteach-opening-principles/index.lesson.md) after 2–3 core fails - -## Supporting workspace files - -- [Learner profile](../../../tests/fixtures/path/paths/learner.profile.md) -- [Learning activity](../../../tests/fixtures/path/paths/learning.activity.md) -- [Flashcard component](../../../tests/fixtures/path/paths/assets/learning.components.md) - -## Placement rules shown in the fixture - -| Result | Next | -| --- | --- | -| Pass (no core miss) | Next Later draft toward Point B | -| Pass + peripheral miss | Advance, with a short reteach of the gap | -| Core miss, under 3 attempts | Variant retry from the same rubric | -| Core miss 2–3 times | Remediation lesson, then review → assess | -| Remediation assess also fails | Renegotiate Point B or pace | - -The compact SQL example in [worked-example.md](./worked-example.md) still shows -the five core durable files at one moment in a simpler loop. diff --git a/skills/path/references/buffered-loop.md b/skills/path/references/buffered-loop.md new file mode 100644 index 0000000..938f3af --- /dev/null +++ b/skills/path/references/buffered-loop.md @@ -0,0 +1,94 @@ +# Buffered Adaptive Loop + +Use one visible path map, a fully authored current module, and asynchronous +agent adaptation. The learner should never need an agent turn between ordinary +Blocks. + +## Planning horizon + +| Horizon | Detail | +| --------------- | ----------------------------------------------------------------- | +| Whole path | 3–7 milestone titles, capability outcomes, and evidence targets. | +| Current module | 2–4 complete sessions plus review and checkpoint. | +| Later modules | Provisional titles and outcomes only. | +| Optional runway | Consolidation, retrieval, or stretch work ready without an agent. | + +Point A chooses placement. It does not force tiny content generation. Point B +and the milestone evidence targets give the learner a stable sense of +direction while later details remain adaptable. + +## Flow + +```mermaid +flowchart TD + Goal[Goal and learner context] --> Evidence[Small Point A evidence] + Evidence --> Map[Confirm Point B and milestone map] + Map --> Build[Build current 2-4 session module] + Build --> Learn[Learner completes uninterrupted sessions] + Learn --> Notes[Responses work and annotations] + Notes --> Checkpoint[Module checkpoint] + Checkpoint --> Place{Evidence meets milestone?} + Place -->|yes| Advance[Mark demonstrated and build next module] + Place -->|not yet| Review[Focused review or smaller module] + Review --> Checkpoint +``` + +## One module + +A module has one coherent capability destination. Prepare all its sessions +before opening Session 1. + +```text +Module arrival +├── Session 1: model and guided practice +├── Session 2: varied or independent practice +├── Session 3: transfer or integration, when needed +├── Optional review and stretch work +└── Milestone checkpoint +``` + +Keep the module small enough to revise after new evidence. Do not pre-author +every future module. + +## One session + +Give every session a visible destination and completion point: + +1. Orient the learner and preview the session. +2. Model the idea with a worked example. +3. Guide one attempt with optional hints. +4. Ask for an independent or transfer attempt. +5. Provide immediate rationale, comparison, or rubric. +6. Invite reflection or annotation. +7. Summarize progress and point to the next ready session. + +Use pre-authored help for ordinary friction. Reserve agent review for work that +benefits from judgment or changes placement. + +## Feedback cadence + +| Evidence | Response | +| ----------------------------- | ------------------------------------------------------ | +| Small uncertainty | Continue; add it to review or annotations. | +| Session misconception | Use ready hints, example, or short consolidation. | +| Repeated core difficulty | Agent prepares focused review or a smaller module. | +| Milestone demonstrated | Record evidence and advance. | +| Goal or circumstances changed | Renegotiate Point B, pace, or presentation explicitly. | + +Do not use a no-core-miss gate after every session. Gate only when a later +capability genuinely depends on the missing idea. + +## Durable state + +Record: + +- confirmed learner and style preferences; +- Point A evidence and Point B; +- milestone status and current foreground path; +- completed sessions and linked work; +- checkpoint syntheses and placement decisions; +- open review items and annotations; +- explicit goal or pace changes. + +Keep completed evidence append-only. Revise future plans without rewriting the +learner's history. diff --git a/skills/path/references/worked-example.md b/skills/path/references/worked-example.md index eddd1f3..2f7dfc7 100644 --- a/skills/path/references/worked-example.md +++ b/skills/path/references/worked-example.md @@ -1,154 +1,129 @@ # Worked Example -This SQL path is a compact slice: five durable files at one moment after a -lesson. It does **not** include onboarding, plan confirm, short review, -practice, remediation, or the no-core-miss assessment pattern. +This compact SQL example shows the durable shape of one buffered module. It is +not a required topic or exact file count. -For the full onboard → confirm → review → assess → remediate fixture, see -[adaptive-loop-example.md](./adaptive-loop-example.md). +## Path map -## `paths/sql-foundations/path.outcome.md` +`paths/sql-foundations/index.path.md`: ```md --- -type: outcome +type: path status: active --- -# SQL Foundations Outcome +# SQL Foundations -## Point A +**Point A:** Can filter one table. Joins are unfamiliar. -Can filter one table with `SELECT` and `WHERE`. Joins are unfamiliar. +**Point B:** Can build and explain a small support report across related tables. -## Point B +## Milestones -Can answer support questions that require combining two tables. +- **Choose rows from one table:** demonstrated +- **Combine related tables:** in progress +- **Summarize joined results:** planned +- **Build the support report:** planned -## Rubric +## Current module -- Chooses `INNER JOIN` or `LEFT JOIN` for the required rows. -- Writes the join condition correctly. -- Explains which unmatched rows remain. +[Combine related tables](./modules/01-combine-tables/index.path.md) ``` -## `paths/sql-foundations/index.path.md` +## Current module + +`paths/sql-foundations/modules/01-combine-tables/index.path.md`: ```md --- type: path -status: active +status: ready --- -# SQL Foundations - -[Outcome and rubric](./path.outcome.md) - ---- +# Combine Related Tables -## Current lesson +**Destination:** Choose `INNER JOIN` or `LEFT JOIN`, write the join condition, +and explain which unmatched rows remain. -- [ ] [Joining tables](./lessons/joins/index.lesson.md) +1. [See what a join preserves](./01-rows-that-survive.lesson.md) +2. [Choose the join from the question](./02-choose-the-join.lesson.md) +3. [Optional review](./review.practice.md) +4. [Milestone checkpoint](./milestone.assessment.md) +``` -## Later +Both sessions exist before the learner starts the module. -- Aggregation and `GROUP BY` -- A small support report -``` +## One uninterrupted session -## `paths/sql-foundations/lessons/joins/index.lesson.md` +`01-rows-that-survive.lesson.md`: ````md --- type: lesson -status: active -start: Can filter one table; joins are unfamiliar -destination: Can choose and write INNER or LEFT JOIN and explain which rows remain +status: ready --- -# Joining Tables +# See What a Join Preserves -## Recall - -From memory, what does `WHERE` remove from a result? +By the end, you will predict which ticket rows survive an `INNER JOIN` and a +`LEFT JOIN`. --- -## Two tables + + +## Start with one worked example ```sql SELECT t.id, c.name FROM tickets t -INNER JOIN customers c ON c.id = t.customer_id; +LEFT JOIN customers c ON c.id = t.customer_id; ``` ---- - -## Keep unmatched tickets +A `LEFT JOIN` keeps every ticket. When a customer record is missing, the +customer columns are empty instead of the ticket disappearing. -Change `INNER JOIN` to `LEFT JOIN`. Predict the difference before running it. - -[Take the assessment](./lesson.assessment.md) -```` - -## `paths/sql-foundations/lessons/joins/lesson.assessment.md` - -```md ---- -type: assessment -status: ready --- -# Joining Tables Assessment - -## Evidence target + -Choose and write `INNER JOIN` or `LEFT JOIN`, then explain which unmatched rows -remain. +## Predict before revealing -## Transfer task +What changes if `LEFT JOIN` becomes `INNER JOIN`? -Write a query that lists every ticket, including tickets whose customer record -is missing. Return the ticket ID and customer name. +Write one sentence before continuing. -Save the query and a short explanation in a file, then link that artifact from -the activity log. - -## Rubric +--- -- Uses `LEFT JOIN`. -- Joins the correct keys. -- Keeps all ticket rows. -- Explains why an `INNER JOIN` would drop some tickets. + -The next lesson stays gated until the evidence meets this rubric. -``` +## Compare your prediction -## `paths/learning.activity.md` +`INNER JOIN` keeps only tickets with a matching customer. The important +question is not “Which keyword do I remember?” but “Which rows must survive?” -```md ---- -type: activity --- -# Learning Activity + ---- +## Apply it - +Choose a join for a report that must include every ticket, then explain the +choice. Use the optional review if the row-survival rule is still fuzzy. -## Synthesis: joins +[Continue to Session 2](./02-choose-the-join.lesson.md) +```` -**Evidence:** [Joining tables assessment](./sql-foundations/lessons/joins/lesson.assessment.md) -meets the rubric; the linked work uses `LEFT JOIN` and explains unmatched rows. +The learner can complete the session without another agent turn. They may +annotate any confusing explanation for later review. -**Shift:** The learner now treats join choice as a statement about which rows -must survive, not as syntax to memorize. +## Milestone checkpoint -**Opens next:** Aggregate the joined results by customer segment. -``` +The checkpoint asks for one small support query and explanation. The agent +compares that artifact with the module's evidence target, records a synthesis +in `paths/learning.activity.md`, and either marks the milestone demonstrated or +prepares focused review. -The next lesson starts from the final synthesis, not from a generic sequence. +The next module starts from that recorded evidence, not from a generic course +sequence. diff --git a/skills/pathmx/SKILL.md b/skills/pathmx/SKILL.md index d358ac0..50eb101 100644 --- a/skills/pathmx/SKILL.md +++ b/skills/pathmx/SKILL.md @@ -1,26 +1,53 @@ --- name: pathmx -description: Author, revise, review, and verify PathMX sources. Use for Markdown, Sources, Blocks, Beats, links, directives, questions, components, Play pacing, media, code, math, styling, configuration, and CLI work in a PathMX repository. +description: Author, revise, review, play, and verify PathMX sources. Use automatically for PathMX Markdown, Sources, Blocks, Beats, links, directives, questions, annotations, Literate Components, Play pacing, Player routes, media, code, math, styling, configuration, CLI setup, builds, or browser review in any PathMX repository. --- # PathMX -Create readable Markdown that builds and plays correctly with the repository's -installed PathMX version. +Create readable Markdown that builds correctly and feels coherent in the +PathMX Player. -## Workflow +## Work from the repository -1. If no learning repository location is currently set, ask the user for a location (and suggest one that makes sense for them). -2. Read the nearest repository instructions. -3. Inspect local config, package scripts, PathMX version, entry Source, and - nearby examples. -4. Identify the Source role and audience. -5. Draft ordinary Markdown first. -6. Use `---` Blocks and Beats only when they improve pacing or focus. -7. Reuse local links, directives, components, and styles. Do not invent syntax. -8. Build into scratch output and report the result. +1. Read the nearest repository instructions. +2. Inspect the local config, package scripts, installed PathMX version, entry + Source, and nearby examples relevant to the task. +3. Identify the Source role and audience. +4. Draft ordinary Markdown first. +5. Add Blocks, Beats, directives, components, or custom styling only when they + improve structure, focus, interaction, or comprehension. +6. Preserve relative links and reuse local conventions. Do not invent syntax. +7. Build into scratch output for diagnostics. +8. For playable or visual work, use the active Player and review the exact + route in an available browser. +9. Report changed Sources, verification, and any skipped live review. -Repository instructions and pinned versions override these examples. +Repository instructions and installed versions override these examples. + +## Keep the Player available + +For learning and presentational work, detect a healthy Player that belongs to +the current repository. Reuse it or start the repository's long-lived Play +command. Do not stop an unknown listener. + +Resolve Source routes with the repository's pinned CLI, normally +`bunx pathmx route`; do not guess from filenames. Use a stable Block fragment +or Play Beat position only after reading it from built metadata or the current +Player URL. Link to the narrowest useful position. + +When Sources share a basename such as `index.path.md`, build the current graph +and query `pathmx route` with the full Source path. Never shorten an ambiguous +query or present the root Path result as the exact nested Source route. + +Keep route diagnosis narrow. Do not recursively search or print `.pathmx`; +generated caches can flood the agent context and slow the learner's next turn. +Use `pathmx route --json` and, only when needed, inspect the target Path's +`paths.json`, `serve-routes.json`, `sources.json`, or one source map. + +Use an integrated browser when it is available. In Codex, prefer `@Browser`. +In Claude Code, use its Chrome integration when already configured. Otherwise +open the system browser or give the user a clickable Player URL. ## Terms @@ -36,22 +63,24 @@ Repository instructions and pinned versions override these examples. | Play | Guided traversal of Blocks and Beats. | | Directive | An implemented `@`-labeled Markdown link or definition. | | Literate Component | A custom tag defined in a component Markdown file. | +| Annotation | A durable comment thread anchored in a Source. | -## References +## Route references Read only what the task needs: - [Markdown authoring](./references/pathmx-markdown.md) -- [Player and pacing](./references/pathmx-player.md) +- [Player, Play, and route handoff](./references/pathmx-player.md) - [Directives](./references/pathmx-directives.md) - [Questions and responses](./references/pathmx-questions.md) +- [Annotations](./references/pathmx-annotations.md) - [Literate Components](./references/pathmx-literate-components.md) - [Code](./references/pathmx-code.md) - [Math](./references/pathmx-math.md) - [Media](./references/pathmx-media.md) - [Styling](./references/pathmx-styling.md) - [Configuration](./references/pathmx-config.md) -- [Tooling and verification](./references/pathmx-tooling.md) +- [Tooling, setup, and verification](./references/pathmx-tooling.md) - [Small repository example](./references/pathmx-repo-example/pathmx-repository.md) ## Boundaries @@ -59,8 +88,10 @@ Read only what the task needs: - Keep Sources useful as plain Markdown. - Use relative Source and asset links. - Use source-facing `type`, not a new `kind` field. -- Do not update PathMX or project config unless requested. +- Treat learner responses and annotations as user-owned data. +- Do not update an existing project's dependency or config merely to make an + example work. Follow its installed version and report a real gap. - Do not write diagnostic output into a live `.pathmx` directory. -- Use only the built-in question mappings in the questions reference. +- Use only fixture-backed question mappings and component contracts. - Do not author general actions or spaceholders from this skill yet. -- If a feature is absent or version-mismatched, report the gap. +- If a feature is absent or version-mismatched, state the boundary. diff --git a/skills/pathmx/references/pathmx-annotations.md b/skills/pathmx/references/pathmx-annotations.md new file mode 100644 index 0000000..9e69a7b --- /dev/null +++ b/skills/pathmx/references/pathmx-annotations.md @@ -0,0 +1,66 @@ +# PathMX Annotations + +Use annotations for durable feedback and discussion anchored to a Source, +Block, Beat, or selected text. Prefer the Player's annotation controls or an +installed PathMX annotation tool over hand-editing thread syntax. + +## Source shape + +An annotation is a Markdown footnote family. The anchor is an ordinary +footnote reference and the definitions hold the discussion: + +```md +Photosynthesis turns light into stored chemical energy.[^c1] + +[^c1]: **@learner** (2026-07-20 10:30 -04:00): I understand the words, but not + where the stored energy actually goes. + +[^c1.a]: **@agent** (2026-07-20 10:42 -04:00): I will add a concrete glucose + example to the next session. +``` + +A resolved parent ends with an indented resolution line: + +```md +[^c1]: **@learner** (2026-07-20 10:30 -04:00): I understand the words, but not + where the stored energy actually goes. + + ✓ @learner 2026-07-20 11:05 -04:00 +``` + +Keep the anchor and thread after resolution so the learning history remains +readable. Do not manufacture actor names or timestamps. + +## Text selections + +PathMX may preserve a selected range with a comment marker: + +```md +The energy becomes {==stored in chemical bonds==}[^c2]. + +[^c2]: **@learner** (2026-07-20 10:35 -04:00): Can we make this more concrete? +``` + +Create these through Player or PathMX tooling. Marker placement around code and +tables is version-sensitive and should not be improvised. + +## Learning workflow + +- Invite learners to annotate confusion, disagreement, useful connections, or + requests for more depth. +- Review open threads before changing future curriculum. +- Reply when the discussion itself is useful history. +- Revise future Sources when feedback reveals a curriculum problem. +- Resolve only after the concern is addressed or the learner confirms it. +- Never delete learner-authored threads merely to make a Source look clean. + +Annotations are asynchronous feedback. Do not make the learner wait for an +agent response before continuing a prepared session. + +## Review + +- Confirm the anchor still points at the intended rendered material. +- Check open and resolved state in the Player. +- Preserve attribution and chronological order. +- Build with the installed PathMX version. +- Use a disposable copy when testing actions that write annotation data. diff --git a/skills/pathmx/references/pathmx-config.md b/skills/pathmx/references/pathmx-config.md index e23f152..aefd714 100644 --- a/skills/pathmx/references/pathmx-config.md +++ b/skills/pathmx/references/pathmx-config.md @@ -34,7 +34,7 @@ options. Keep credentials out of config; name environment variables instead. PathMX can build an explicit entry without config: ```sh -pathmx build paths/index.path.md +bunx pathmx build paths/index.path.md ``` When local config defines entries, omit the entry only when you intend to build @@ -42,3 +42,24 @@ those configured Paths. Plugin keys and options are version-specific. Check the repository's installed version and `pathmx --help` before changing them. + +## Multiple Player roots + +Most projects need one configured root whose links discover the rest of its +Source graph. Do not register every nested hub or lesson as another Path entry. + +When a project intentionally configures multiple Player roots, every +non-default root needs a unique authored Source handle in its frontmatter. A +handle starts with `@` and then uses letters, numbers, dots, underscores, or +hyphens: + +```md +--- +handle: "@workshop" +--- + +# Workshop +``` + +Do not invent `handle` values from file paths. The checked fixture uses +`@workshop`; values such as `workshop` or `nested/index.path` are invalid. diff --git a/skills/pathmx/references/pathmx-player.md b/skills/pathmx/references/pathmx-player.md index d3750f1..5e7aa52 100644 --- a/skills/pathmx/references/pathmx-player.md +++ b/skills/pathmx/references/pathmx-player.md @@ -1,34 +1,75 @@ # PathMX Player and Play -The build fixes the Play route. The Player reads built Block and Beat data; it -does not invent structure from the browser DOM. +The Player reads built Source, Block, and Beat data. Use it as the primary +experience for learning material; do not send a learner to raw Markdown when a +Player route is available. ## Model | Level | Use | | --- | --- | -| Block | Coarse stop created by `---`. | -| Browse heading | Free-scroll map stop. | -| Step | Fine Beat such as a paragraph, list item, row, code step, or component state. | +| Source | One page and route. | +| Block | One coherent phase created by `---`. | +| Beat | One focusable step inside a Block. | +| View mode | Free reading and navigation. | +| Play mode | Guided forward and backward traversal. | -Forward and backward move through the route. Skip-out moves up one nesting -level. Active and seen state are projected onto the rendered content. The URL -owns Play position through `?play=`; `#fragment` remains a browse -target. +The URL owns Play position through `?play=`. A fragment remains a +browse target. Never invent a Beat id: copy it from the current Player URL or +read it from built metadata. + +## Keep a server available + +Prefer the repository's script. A direct fallback is: + +```sh +bunx pathmx play --print-url +``` + +Reuse a healthy listener only after confirming that its process and output +belong to the current repository. Do not terminate an unknown server. Keep the +Player running while the learner works. + +Resolve a built Source route instead of guessing it: + +```sh +bunx pathmx route paths/example.lesson.md --base-url http://127.0.0.1:3000 +``` + +If the repository uses another output directory, pass the same `--out` value +to `bunx pathmx route`. + +## Open and hand off routes + +Use this order: + +1. Open the exact route in an available integrated browser and verify it. +2. In Codex, prefer `@Browser`. +3. In Claude Code, use its Chrome integration when already configured. +4. Otherwise open the system browser. +5. If no browser can open, provide a clickable URL. + +Link to the narrowest useful position: + +- Source route for a page or path overview; +- stable Block fragment for one section; +- Play URL with a verified Beat id for an exact learning step. + +State whether the link opens in View or Play mode. For a first-time learner, +explain forward, backward, exit, and free navigation briefly. ## Pacing controls Authors control pacing with: - Block boundaries; -- `play.steps` density for lists, tables, and code; +- ordinary Markdown Beats; +- `play.steps` density for supported content; - code focus steps such as `[1-2|3]`; - Block-local table focus steps; and - ordered Literate Component states. -Use [Markdown authoring](./pathmx-markdown.md) for Blocks and density, -[Code](./pathmx-code.md) for fence steps, and -[Literate Components](./pathmx-literate-components.md) for state. +Use Beats for meaningful reveals. Avoid turning every sentence into a step. ## Table steps @@ -60,17 +101,11 @@ play: Rows and columns are one-based. A step needs `rows`, `columns`, or both. Build warnings fall back to ordinary row Beats. -## Interactive Beats - -Play and direct interaction share a Literate Component's ordered state. The -component must also work outside Play. Use component state for presentation, -not durable learner evidence. - -## Authoring review +## Review -- Make each Block one coherent move. -- Keep meaningful stages addressable. -- Avoid long Blocks with dozens of Beats. -- Do not hide essential steps in an opaque component loop. -- Check forward, backward, and skip-out behavior. -- Check keyboard, pointer, touch, and narrow layouts when interaction changes. +- Check the first arrival in View and Play modes. +- Move forward, backward, and out of nested steps. +- Check direct Source, Block, and Beat links. +- Check keyboard, pointer, touch, and narrow layouts. +- Confirm hints and reveals do not hide essential content permanently. +- Reload once to catch stale route or asset assumptions. diff --git a/skills/pathmx/references/pathmx-styling.md b/skills/pathmx/references/pathmx-styling.md index db38fc9..4941dfa 100644 --- a/skills/pathmx/references/pathmx-styling.md +++ b/skills/pathmx/references/pathmx-styling.md @@ -61,6 +61,24 @@ theme: PathMX maps supported values to `--pmx-*` variables. Use `light` and `dark` branches for mode-specific values. Do not assume a named theme preset exists. +## Learner personalization + +Start from a readable repository theme. Ask for a small set of preferences +instead of asking the learner to design an interface: + +- visual mood, such as calm, warm, playful, editorial, or technical; +- one color direction; +- light, dark, or system preference; +- larger text, stronger contrast, reduced motion, or other access needs. + +Translate confirmed answers into existing theme tokens and a few repository CSS +variables. Keep document structure, navigation, focus states, and component +behavior stable. Prefer one restrained accent and clear surfaces over many +decorative colors. + +Do not record inferred accessibility needs or personal traits. Save only what +the learner chose. + ## Fonts Load a hosted stylesheet or local font, then select the family through theme diff --git a/skills/pathmx/references/pathmx-tooling.md b/skills/pathmx/references/pathmx-tooling.md index fd26adb..154a44c 100644 --- a/skills/pathmx/references/pathmx-tooling.md +++ b/skills/pathmx/references/pathmx-tooling.md @@ -1,30 +1,85 @@ # PathMX Tooling and Verification -Prefer the repository's pinned command or package script. Check the version and -local config before using a recent feature. +Prefer repository scripts and the pinned project dependency. Inspect the +installed version and command help before using a version-sensitive feature. ```sh -pathmx --version -pathmx --help +bunx pathmx --version +bunx pathmx --help ``` -The public package provides `pathmx` and `pmx`: +Use repository scripts first and `bunx pathmx` otherwise so builds, routes, and +graph changes use the exact project dependency. Reserve the native `pathmx` +alias for setup and self-update work outside a project. + +## Install and update + +For a new machine with Bun installed, create or update the native aliases from +the latest published package: + +```sh +bunx @fellowhumans/pathmx@latest self-update +``` + +When the native command already exists: + +```sh +pathmx self-update +pathmx self-update --check +``` + +The repository's `pathmxCompatibility.baseline`, exact dependency, and lockfile +govern authoring. A newer native command does not make newer syntax valid in an +older project. + +Create an official Starter and include current official skills: ```sh -bun add -g @fellowhumans/pathmx@latest -bunx @fellowhumans/pathmx@latest --version +pathmx init --template ``` -Do not install or update PathMX unless the task requests it. +Refresh managed official skills inside a recognizable PathMX Space: + +```sh +pathmx init --skills +``` + +Do not update an existing project's dependency or config merely to make new +syntax work. Use its lockfile unless the user asked for an upgrade. + +## Safe version migration + +For a new learning space, or when the learner asks to stay current, treat an +update as a small transaction: + +1. Start from a verified local commit or back up `package.json` and the + lockfile. +2. Update the project package with + `bun add --exact @fellowhumans/pathmx@latest`. +3. Run the repository's candidate check when it has one; otherwise build every + configured Path into scratch output and run its full check. +4. Smoke-test Player routes plus any questions, annotations, and components + the repository uses. +5. After those checks pass, promote the exact installed dependency to the + repository's compatibility baseline and rerun the normal check. +6. Keep and commit the update only when the normal check passes. +7. On failure, restore only the version files, reinstall the lockfile, and + report the incompatibility. Do not rewrite learner responses to force a + migration. + +After a successful update, inspect release-specific help before adopting new +syntax. Existing Source data and the installed fixtures remain the contract. ## Commands | Command | Purpose | | --- | --- | -| `pathmx init` | Create or migrate config. | +| `pathmx init` | Create config, materialize an official Starter, or update official skills. | +| `pathmx self-update` | Install or update native command aliases. | | `pathmx build` | Build and exit. | | `pathmx dev` | Build, serve, and watch. | -| `pathmx play` | Run the Player authoring loop. | +| `pathmx play` | Build, watch, serve, and open the Player. | +| `pathmx route` | Resolve a built Source path, id, or route. | | `pathmx mv` | Move a Source and update links. | | `pathmx rm` | Remove a Source and update links. | @@ -33,7 +88,7 @@ Do not install or update PathMX unless the task requests it. Never use a live `.pathmx` directory for a diagnostic build. ```sh -pathmx build -o .pathmx-check --clean +bunx pathmx build -o .pathmx-check --clean ``` A passing build has three results: @@ -42,21 +97,43 @@ A passing build has three results: 2. The summary reports at least one built Path. 3. The expected Source appears in a generated `sources.json`. -Review warnings. A startup banner alone is not a completed build. +A startup banner is not a completed build. Review warnings and wait for exit. + +## Route lookup + +Resolve the route from built metadata: + +```sh +bunx pathmx route paths/example.lesson.md +bunx pathmx route paths/example.lesson.md --base-url http://127.0.0.1:3000 +bunx pathmx route example.lesson --json +``` + +Use the full Source path whenever basenames repeat. A query such as +`index.path` can legitimately resolve the root instead of a nested Path; do not +use that result as a verified route for the nested Source. + +Do not run a recursive text search across `.pathmx` to diagnose a route. Its +generated cache can be very large. Prefer `pathmx route --json`, then inspect +only `paths.json`, the relevant `serve-routes.json` or `sources.json`, or the +single target source map. + +Add Block or Beat positioning only from built metadata or a verified Player +URL. ## Graph changes -Preview moves and removals when the installed CLI supports `--dry-run`: +Preview moves and removals when the installed command supports `--dry-run`: ```sh -pathmx mv old.lesson.md new.lesson.md --dry-run -pathmx rm unused.lesson.md --dry-run +bunx pathmx mv old.lesson.md new.lesson.md --dry-run +bunx pathmx rm unused.lesson.md --dry-run ``` Apply the command, inspect rewritten links, then rebuild the affected root. ## Handoff -Report the changed Sources, PathMX version, commands run, build result, and any -skipped Player checks. Include a Player route only when visual or interactive -behavior was reviewed. +Report changed Sources, the PathMX version, commands run, build result, and any +skipped Player checks. Lead with the verified Player route for learner-facing +work. diff --git a/tests/fixtures/path/paths/assets/learning.components.md b/tests/fixtures/path/paths/assets/learning.components.md deleted file mode 100644 index 69a22e5..0000000 --- a/tests/fixtures/path/paths/assets/learning.components.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -componentName: flashcard ---- - -# Flashcard - -A two-sided card for retrieval practice. Front holds a prompt, back holds the -answer. Click (or focus and press Enter) to flip. `states="front | back"` is -an ordered domain, so in Play the forward arrow flips the card — the reveal -is a step Beat. - -Props: `label` — accessible name for the card. - -```html -
-
-
-

Click or press Enter to flip

-
-``` - -```css -:self { - display: grid; - gap: 0.5rem; - padding: 1rem 1.25rem; - border: 1px solid var(--pmx-color-border, currentColor); - border-radius: var(--pmx-radius, 0.75rem); - background: var(--pmx-color-surface, transparent); - cursor: pointer; -} - -:self [data-side="back"], -:self[data-state="back"] [data-side="front"] { - display: none; -} - -:self[data-state="back"] [data-side="back"] { - display: block; -} - -[data-hint] { - font-size: 0.8em; - color: var(--pmx-color-muted, inherit); - margin: 0; -} - -@dark { - :self { - box-shadow: 0 0.25rem 1rem oklch(0 0 0 / 0.4); - } -} -``` - -```js -function flip() { - state.set(state.get() === "back" ? "front" : "back") -} - -on(el, "click", flip) -on(el, "keydown", (event) => { - if (event.key === "Enter" || event.key === " ") { - event.preventDefault() - flip() - } -}) -``` diff --git a/tests/fixtures/path/paths/chess-opening-principles/index.path.md b/tests/fixtures/path/paths/chess-opening-principles/index.path.md deleted file mode 100644 index 02dd40d..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/index.path.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -type: path -status: active -roadmap: - status: proposed ---- - -# Chess Opening Principles - -Full-loop fixture for the `/path` adaptive learning workflow. -[Outcome and rubric](./path.outcome.md) - -Walk **in order**. Live product: one Block per agent turn; here Lesson 1 includes -all stages so you can feel Play pacing. Resume cursor: `stage` in lesson -frontmatter. - ---- - -## Prototype journey - -1. [ ] **→ [Onboarding](./onboarding/index.lesson.md)** *(start here)* -2. [ ] [Point A evidence](./onboarding/point-a-evidence.md) — show level before drafting A -3. [ ] [Confirm your plan](./onboarding/confirm-plan.md) — gate before teaching -4. [ ] [Lesson 1 — Control, development, and king safety](./lessons/control-center-development/index.lesson.md) -5. [ ] [Short review](./lessons/control-center-development/lesson.review.md) — ready or name gap -6. [ ] [Extra practice](./lessons/control-center-development/lesson.practice.md) — only if unsure -7. [ ] [Assessment before Lesson 2](./lessons/control-center-development/lesson.assessment.md) — pass = no core miss -8. [ ] [Lesson 2 shell](./lessons/develop-before-queen/index.lesson.md) — after a pass -9. [ ] [Remediation shell](./lessons/reteach-opening-principles/index.lesson.md) — only after 2–3 core fails - ---- - -## Current lesson - -- [ ] **→ [Control, development, and king safety](./lessons/control-center-development/index.lesson.md)** - -## Completed - -*(none yet)* - -## Later, adapted from your evidence - -- Contested center — when to push vs develop *(titles only until next)* -- Early mistakes — queen raids and neglected development -- Mini-game — first 10 moves with a short postmortem - -## Gates - -| Before… | Require… | -| --- | --- | -| Draft Point A / roadmap | Onboarding + **Point A evidence** | -| Lesson 1 teaching | Plan **Confirm** | -| Mark Lesson 1 complete | Short **review** | -| Open assessment | **Ready** (or finished gap practice) | -| Lesson 2 toward Point B | Assessment with **no core miss** + placement | -| Ambitious Later draft | Not after 2–3 core fails — use remediation first | -| After failed remediation assess | **Renegotiate** Point B / pace | - -[Path references](./references/index.references.md) diff --git a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/index.lesson.md b/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/index.lesson.md deleted file mode 100644 index cf80e24..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/index.lesson.md +++ /dev/null @@ -1,195 +0,0 @@ ---- -type: lesson -status: active -start: Knows piece moves; opening choices feel arbitrary -destination: Can state three opening principles, pick the better of two moves, and justify with a principle -stage: lesson-frame ---- - -# Control, development, and king safety - - - ---- - - - -## This lesson - -You will learn three opening principles and use them to choose between two -moves. In Play, advance one Beat at a time. In the live product, each **Stage** -below would be generated as its own agent turn after you continue or submit. - ---- - - - -## Stage 1 — Three principles - -In the opening, most good plans lean on three ideas: - -- **Center** — fight for d4, d5, e4, and e5 so your pieces have room -- **Development** — bring knights and bishops out to useful squares early -- **King safety** — get the king secure (often by castling) before starting - adventures - -You do not need a named opening yet — only these lenses. - ---- - - - -## Stage 2 — What “center” looks like - -Imagine the board after `1. e4`. White’s pawn sits on **e4**, claiming space -in the center. - -| Idea | On the board | -| --- | --- | -| Center claim | Pawn on e4 influences d5 and f5 | -| Next useful idea | Develop a knight (e.g. Nf3) or challenge with …e5 / …c5 later | -| Not the priority yet | Chasing the opponent’s queen around the board | - - - After 1. e4, which goal did White invest in first? - Center — the e-pawn takes space and opens lines for the bishop and queen. - - -[@learning]: ../../../assets/learning.components.md - ---- - - - -## Stage 3 — Quick check - -White plays `1. e4` and Black answers `1… e5`. White then plays `2. Qh5`, -attacking the e5 pawn and eyeing f7. - -Which principle is White **neglecting** most clearly? - -- Center control -- Development of minor pieces -- King safety by castling immediately on move two - ---- - - - -## Stage 4 — Check your thinking - -*(Live loop: the agent appends this Block after your submit, using your -answer.)* - -The best fit is **development of minor pieces**. `2. Qh5` brings the queen out -very early while the knights and bishops stay home. Center was already being -contested with e4/e5; castling on move two was never realistic. Early queen -trips often waste time if Black develops and kicks the queen. - -If you missed it — good signal. The next stages practice choosing with -principles, not memorizing “never move the queen.” - ---- - - - -## Stage 5 — Two candidate moves - -Same position: after `1. e4 e5`, it is White’s second move. - -| Move | What it does | -| --- | --- | -| **A. Nf3** | Develops a knight, attacks e5, prepares castling | -| **B. Qg4** | Brings the queen out early, eyes g7, delays minor pieces | - -Which move better follows the three principles? - ---- - - - -## Stage 6 — Choose - -Which move is the better opening choice here? - -- A. Nf3 — develop, pressure the center, prepare king safety -- B. Qg4 — early queen attack on g7 - ---- - - - -## Stage 7 — Feedback - -*(Live loop: generated from your choice.)* - -**A. Nf3** fits development, keeps pressure on the center, and prepares -castling. **B. Qg4** is an early queen outing that Black can often meet by -developing and later harassing the queen — you fall behind in development. - -If you picked B, you are not “bad at chess” — you are learning to prefer -principled development over early attacks. That is exactly what Lesson 2 can -reteach if assessment shows the same gap. - ---- - - - -## Stage 8 — Justify in one sentence - -Why is Nf3 a better opening move than Qg4 here? Name at least one principle. - ---- - - - -## Stage 9 — Wrap - -You should now be able to name the three principles and use them to prefer a -developing move over an early queen raid in a simple position. - -Next (required by the loop): - -1. [Short review](./lesson.review.md) — ready or name the gap -2. [Extra practice](./lesson.practice.md) — only if still unsure -3. [Assessment before Lesson 2](./lesson.assessment.md) — when ready; pass = no core miss - -Agent: after each generated Stage, set frontmatter `stage` to that Block `id` -so reload resumes deterministically. diff --git a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.assessment.md b/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.assessment.md deleted file mode 100644 index fcb496c..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.assessment.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -type: assessment -status: ready -attempts: 0 -pass_rule: no_core_miss ---- - -# Assessment before Lesson 2 - -Take this when the [short review](./lesson.review.md) says you are **ready** -(after [extra practice](./lesson.practice.md) if you were unsure). - -Fair check — not a trick exam. Prefer at the **start of your next session** -(spaced retrieval); same session is fine if you want to keep going. - -**Format:** 4 multiple choice + 1 short answer + 1 optional stretch. -**Pass:** **no core concept missed.** Peripheral misses still pass with a -carry-forward note. No percentage score. - -Your agent scores with the rubric below, notes **confidence** (especially -confident-and-wrong), and persists result / attempts in -[learning.activity.md](../../../learning.activity.md). Core miss → encourage + -**variant** retry (not the same MC set). After 2–3 core-fail attempts → -[remediation lesson](../reteach-opening-principles/index.lesson.md). If -remediation assessment also fails → renegotiate Point B / pace with the learner -(no second remediation loop). - ---- - - - -## 1. Center *(core: center)* - -Which squares are the main “center” you want to influence in the opening? - -- a1, a8, h1, and h8 -- d4, d5, e4, and e5 -- Only the square the king starts on - ---- - - - -## Confidence — question 1 - -- High — I am sure -- Medium — somewhat sure -- Low — mostly guessing - ---- - - - -## 2. Development *(core: development)* - -Which move best shows **development** early in the opening? - -- Bringing a knight to f3 -- Moving the same pawn three times in a row for no reason -- Bringing the queen to h5 on move two before any minor piece is out - ---- - - - -## Confidence — question 2 - -- High — I am sure -- Medium — somewhat sure -- Low — mostly guessing - ---- - - - -## 3. Early queen *(core: development / early queen)* - -After `1. e4 e5`, why is `2. Nf3` usually better than `2. Qg4`? - -- Nf3 develops a minor piece, pressures the center, and prepares castling -- Queens always must move before knights -- Qg4 forces checkmate in two moves in every game - ---- - - - -## Confidence — question 3 - -- High — I am sure -- Medium — somewhat sure -- Low — mostly guessing - ---- - - - -## 4. King safety *(core: king safety)* - -Which idea most clearly supports **king safety** in the opening? - -- Leaving the king in the center while launching a queen raid -- Preparing to castle after developing a few pieces -- Capturing every available pawn as fast as possible - ---- - - - -## Confidence — question 4 - -- High — I am sure -- Medium — somewhat sure -- Low — mostly guessing - ---- - - - -## 5. Short answer *(core: three principles + apply)* - -In one or two sentences: name the three opening principles from this lesson, -and say which one `2. Nf3` helps most after `1. e4 e5`. - ---- - - - -## Confidence — short answer - -- High — I am sure -- Medium — somewhat sure -- Low — mostly guessing - ---- - - - -## 6. Stretch *(peripheral — does not affect pass/fail)* - -Drawn from the Later title “Contested center.” Optional warm-up for what is -next. - -After `1. e4 e5 2. Nf3`, Black plays `2… Nc6`. What is White’s most principle- -aligned idea among these? - -- Keep developing (e.g. Bb5 or Bc4) while the center stays contested -- Immediately move the queen to h5 every time -- Ignore development and push every remaining pawn - ---- - -## Agent rubric - -| Item | Concept | Tier | Correct / full credit | Why others are weaker | -| --- | --- | --- | --- | --- | -| 1 | center | **core** | d4, d5, e4, and e5 | Corners are not the opening center; king’s start square alone is too narrow | -| 2 | development | **core** | Bringing a knight to f3 | Pawn thrashing wastes time; early queen before minors neglects development | -| 3 | development / early queen | **core** | Nf3 develops, pressures center, prepares castling | Not a law that queens move first; Qg4 is not forced mate | -| 4 | king safety | **core** | Preparing to castle after developing | Queen raids / pawn-grabbing with a central king ignore safety | -| 5 | three principles + apply | **core** | Names center, development, king safety; links Nf3 mainly to **development** (center / preparing safety OK if clear) | Vague “play good moves” without principles → miss or partial | -| 6 | contested center | peripheral | Keep developing while center stays contested | Early queen every time / pawn storm ignores principles | - -**Pass = no core miss** on items 1–5. Item 6 never fails the assessment. - -**Confident-and-wrong** (High + incorrect) = top remediation targets. - -### After scoring - -| Result | Agent move | -| --- | --- | -| Pass, on track | Update Point A; open [Lesson 2](../develop-before-queen/index.lesson.md) toward next Later gap | -| Pass + peripheral miss | Advance, but Lesson 2 opens with a short reteach of that gap | -| Core miss, under 3 attempts | Praise what was right; regenerate **variant** items from same criteria; increment `attempts` | -| Core miss 2–3 times | [Remediation lesson](../reteach-opening-principles/index.lesson.md) on missed concepts (confident-and-wrong first); then review → assess again | -| Remediation assessment also fails | Renegotiate plan (smaller Point B or slower pace) — do not loop remediation again | -| Stretch strong twice in a row (across assessments) | Offer to compress/skip a Later title (learner confirms) | - -### Tone - -Warm, specific praise first. Gaps = next practice move — never shame. diff --git a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.practice.md b/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.practice.md deleted file mode 100644 index c9371e4..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.practice.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -type: lesson -status: active -start: Finished Lesson 1 teaching but still unsure about one idea -destination: Feels clearer on the chosen gap and ready to try the assessment ---- - -# Extra practice before assessment - -Use this only if the [short review](./lesson.review.md) said you are still -unsure. Matches the brief: **1–3 short Blocks** on the named gap (re-explain, -example, tiny check). Live loop: agent generates only the matching Blocks. - -When clearer, return to the review, choose **Ready**, then take the -[assessment](./lesson.assessment.md) (prefer next session for spacing). - ---- - - - -## Gap: the three principles - -Say them out loud once: - -- **Center** — fight for d4, d5, e4, e5 -- **Development** — bring knights and bishops out early -- **King safety** — usually castle before big adventures - - - What are the three opening principles from this lesson? - Center, development, and king safety. - - -[@learning]: ../../../assets/learning.components.md - ---- - - - -## Gap: what “center” looks like - -After `1. e4`, White’s pawn sits on e4. - -| Question | Simple answer | -| --- | --- | -| What did White invest in? | Center space | -| What is a natural follow-up? | Develop a piece (e.g. Nf3) or keep fighting for the middle | -| What can wait? | Hunting the other side’s queen on move two | - ---- - - - -## Gap: early queen vs development - -`2. Qh5` or `2. Qg4` is not always illegal — it is often **slow**. The queen -moves twice later when kicked, while knights and bishops stay home. - -Prefer a move like **Nf3**: one new piece works, e5 is under pressure, castling -gets closer. - ---- - - - -## Tiny check - -After `1. e4 e5`, which move better follows the principles? - -- Nf3 -- Qg4 - ---- - - - -## Done with practice? - -Nice work taking a breath before the check. Go back to the -[short review](./lesson.review.md), mark yourself **Ready**, then open the -[assessment](./lesson.assessment.md). - -If it still feels unclear, tell your agent which line above was least helpful — -they can add one more short Block on that line only. diff --git a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.review.md b/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.review.md deleted file mode 100644 index 8f754a1..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/lessons/control-center-development/lesson.review.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -type: assessment -status: ready ---- - -# Lesson 1 — Short review - -Close the lesson with a quick reflection. If you still feel unsure, we find -**which part** is fuzzy and do help + practice **before** assessment. This does -**not** unlock Lesson 2. - -Agent writes a synthesis in -[learning.activity.md](../../../learning.activity.md) (quote your words), adds -1–2 concepts to the **review queue**, and may harvest one profile field if -still “to be discovered.” - ---- - - - -## 1. What can you do now that you could not at the start of this lesson? - -One or two sentences is enough. - ---- - - - -## 2. Ready for a quick check, or still unsure? - -- Ready — I want the assessment next -- Still unsure — help me find the gap first - ---- - - - -## 3. If still unsure — which part feels least clear? - -*(Skip if Ready. Pick the closest match.)* - -- The three principles (center, development, king safety) — what they mean -- What “center” looks like on the board after 1. e4 -- Why an early queen move (like Qh5 / Qg4) neglects development -- Choosing between Nf3 and an early queen move -- Explaining my choice in a sentence -- Something else — I will say in the next box - ---- - - - -## 4. In your own words, what feels unclear? - -If Ready, write “ready.” If unsure, name the gap. - ---- - - - -## 5. Optional profile harvest - -Only if your [profile](../../../learner.profile.md) still says formats / pace -are “to be discovered.” Otherwise skip mentally and pick any option to continue. - -Which format helps you most right now? - -- Short Blocks with a board or table each time -- More words, fewer positions -- Mix — brief explain, then practice -- Skip — already recorded in my profile - ---- - - - -## Help and extra practice (when still unsure) - -*(Live loop: agent appends 1–3 short Blocks for your gap only — re-explain, -example, tiny check. Warm tone.)* - -Prototype strip: - -**→ [Extra practice before assessment](./lesson.practice.md)** - -Then return here and choose **Ready**. - ---- - - - -## When you are ready - -Prefer assessment at the **start of your next session** (spaced retrieval). -Same session is fine if you want to continue: - -**→ [Assessment before Lesson 2](./lesson.assessment.md)** - -### Agent moves - -| Signal | Do | -| --- | --- | -| Ready | Synthesis + review-queue items → open assessment (or next session) | -| Still unsure + gap | Help + practice; keep assessment locked | -| After practice, still unsure | Narrower gap question; one more short practice | -| After practice, ready | Synthesis → assessment | - -Synthesis fields: Evidence (quotes), Shift, Still fuzzy, Practice, Review queue -(+ due-by lesson), Opens next (assessment when ready). diff --git a/tests/fixtures/path/paths/chess-opening-principles/lessons/develop-before-queen/index.lesson.md b/tests/fixtures/path/paths/chess-opening-principles/lessons/develop-before-queen/index.lesson.md deleted file mode 100644 index 5d25247..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/lessons/develop-before-queen/index.lesson.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -type: lesson -status: active -start: Can name three opening principles; may still be tempted by early queen trips -destination: Prefers developing minor pieces before early queen adventures and can explain why with a principle -stage: lesson-frame ---- - -# Develop before the queen (Lesson 2 shell) - -Instant shell after Assessment 1 **passes** (no core miss). Teaching Stages are -not generated until the agent appends the next Block. Persist `stage` in -frontmatter after each turn so reload resumes here. - -**Default after a clean pass:** prefer developing knights/bishops before early -queen raids. - -**If a peripheral miss or carry-forward gap:** open Stage 1 with a short reteach -of that gap, then continue toward this destination. - -**If assessment had core misses 2–3 times:** use -[Reteach: opening principles](../reteach-opening-principles/index.lesson.md) -instead of this shell. - ---- - - - -## Waiting for the next Block - -Ask your agent to generate **Stage 1 only** from your assessment placement and -[learning.activity.md](../../../learning.activity.md) synthesis. - -Until then: [path index](../../index.path.md) or retry a -[variant assessment](../control-center-development/lesson.assessment.md) only -if you have not passed yet. diff --git a/tests/fixtures/path/paths/chess-opening-principles/lessons/reteach-opening-principles/index.lesson.md b/tests/fixtures/path/paths/chess-opening-principles/lessons/reteach-opening-principles/index.lesson.md deleted file mode 100644 index 072bcff..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/lessons/reteach-opening-principles/index.lesson.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -type: lesson -status: active -start: Missed one or more core Lesson 1 concepts on assessment after 2–3 attempts -destination: Can restate the missed principle(s) and apply them on one simple choice -stage: lesson-frame ---- - -# Reteach: opening principles (remediation) - -Shell for the **remediation** branch — used only after 2–3 assessment -attempts with a **core miss**. Do not advance the ambitious Later draft -(“Contested center”) yet. - -In the live loop, the agent rewrites `start` / `destination` from the missed -concepts and **confident-and-wrong** items, then appends Stages one Block at a -time. - -If the assessment after this remediation also fails → **renegotiate** Point B -or pace with the learner (no second remediation loop). - ---- - - - -## What this lesson is for - -We rebuild only the ideas that slipped — usually one of: center, development, -king safety, or why an early queen move is often too slow. - -Tell your agent which assessment items missed (and which felt “High” -confidence but wrong). Then ask for **Stage 1 only**. - -When staging is done: [short review](../control-center-development/lesson.review.md) -pattern → ready/practice → a **variant** assessment — not the identical MC set. - -Return to the [path index](../../index.path.md) if you are not on this branch. diff --git a/tests/fixtures/path/paths/chess-opening-principles/onboarding/confirm-plan.md b/tests/fixtures/path/paths/chess-opening-principles/onboarding/confirm-plan.md deleted file mode 100644 index 9d44315..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/onboarding/confirm-plan.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -type: assessment -status: ready -roadmap: - status: proposed ---- - -# Confirm your plan - -The agent drafted this plan from your onboarding answers **and** -[Point A evidence](./point-a-evidence.md). Demo defaults below — replace with -evidence-based Point A before treating as final. Request changes with your -agent, or confirm to unlock Lesson 1. - -Full outcome text: [path.outcome.md](../path.outcome.md). - ---- - - - -## Draft Point A *(must cite diagnostic evidence)* - -**Based on your evidence** *(agent fills: mode + artifact/experience summary -and/or question pattern, e.g. “repo … + chose Nf3” or “questions only: unsure”)*: - -You can move pieces legally and know check exists, but opening moves still feel -arbitrary. You cannot yet use **center**, **development**, and **king safety** -to choose between two sensible-looking moves. - -*(If evidence showed stronger play, raise Point A and shorten Lesson 1. If -weaker, keep Lesson 1 at naming principles before move choice.)* - -## Draft Point B - -In a new game, you can play the first 8–10 moves by those three principles, -compare two candidate moves, reject one common early mistake, and explain your -plan without a long memorized line. - -## Rubric (path) - -- Name the three principles and what each looks like on the board -- Choose between two moves and justify with a principle -- Spot one early mistake (e.g. premature queen outing) -- Explain a short opening plan in plain language - -## Proposed roadmap - -1. **Lesson 1 — Control, development, and king safety** *(first)* - Destination: state the three principles; pick the better of two moves; justify. -2. **Later — Contested center** — when to push vs develop *(title only)* -3. **Later — Early mistakes** — queen raids and neglected development *(title only)* -4. **Later — Mini-game** — first 10 moves with a short postmortem *(title only)* - ---- - - - -## Are you satisfied with this plan? - -- Yes — confirm and start Lesson 1 -- Not yet — I want changes before we continue - ---- - - - -## After you confirm - -If you chose **Yes**, tell your agent (or continue this prototype) so they can -set `roadmap.status: confirmed` on this file, [path.outcome.md](../path.outcome.md), -and [index.path.md](../index.path.md), then open: - -**→ [Lesson 1: Control, development, and king safety](../lessons/control-center-development/index.lesson.md)** - -(Start with the lesson **shell** only in the live product; this prototype -includes all Stages for pacing.) - -If you chose **Not yet**, list the change you want (narrower Point B, different -Later titles, more/less time). The agent revises only the draft outcome and -roadmap — no Lesson 1 teaching Blocks until you confirm. - -### Agent note - -Confirm is binary for the gate. “Yes” → `roadmap.status: confirmed` and unlock -Lesson 1. “Not yet” keeps `proposed`. diff --git a/tests/fixtures/path/paths/chess-opening-principles/onboarding/index.lesson.md b/tests/fixtures/path/paths/chess-opening-principles/onboarding/index.lesson.md deleted file mode 100644 index 771b536..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/onboarding/index.lesson.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -type: lesson -status: active -start: Starting a new path; goal and constraints not yet recorded -destination: Has answered onboarding so the agent can run a Point A evidence diagnostic and draft the plan ---- - -# New Path — Onboarding - -This is the **New Path** onboarding flow for `/path`. -Answer each question. Your agent copies confirmed answers into -[learner.profile.md](../../learner.profile.md), then asks for **evidence** -before drafting Point A. - -Play step by step → **Point A evidence** → **Confirm your plan**. - ---- - - - -## What do you want to be able to do? - -One concrete capability — for this demo, something like: play sensible opening -moves in the first 8–10 moves without memorizing a long line. - ---- - - - -## Why does that matter to you right now? - ---- - - - -## What do you already know about chess openings? - -Mention piece moves, any openings you have tried, or what still feels random. - ---- - - - -## Where do you tend to get stuck? - ---- - - - -## Time and pace - -I can spend about **___** minutes per session, about **___** times per week. - ---- - - - -## How should this path feel? - -- Calm and stepwise — short Blocks, clear checks -- Brisk — more positions, fewer words -- Mix — explain briefly, then practice on the board - ---- - - - -## Next: show where you are (Point A evidence) - -Self-description alone is not enough. Open the short diagnostic so the agent -can scaffold Lesson 1 from what you can **do**: - -**→ [Point A evidence](./point-a-evidence.md)** - -Then review and confirm the draft plan. Teaching Blocks stay locked until -confirm. diff --git a/tests/fixtures/path/paths/chess-opening-principles/onboarding/point-a-evidence.md b/tests/fixtures/path/paths/chess-opening-principles/onboarding/point-a-evidence.md deleted file mode 100644 index 677f4cd..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/onboarding/point-a-evidence.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -type: assessment -status: ready ---- - -# Point A evidence - -Show where you are — so Lesson 1 is scaffolded from proof, not a guess. Pick -**how** you want to show it, then give that evidence. You can use questions -only if you have nothing to attach. - -After this → [Confirm your plan](./confirm-plan.md). - -Privacy: share only what you are comfortable making durable in this workspace. -No passwords, private invites, or sensitive personal data. - ---- - - - -## How do you want to show your current level? - -- Past experience — what I’ve tried and how far I got -- Artifact link — repo, gist, notebook, portfolio, or game history URL -- Code or work sample — I’ll paste a small snippet or describe a file -- Picture / screenshot — link or path to an image (board, UI, notes, etc.) -- Quick questions only — I don’t have an artifact; just ask me - ---- - - - -## Past experience *(if you chose that)* - -What have you already done in this area? Mention courses, games, projects, -tutors, or apps — and where you usually get stuck. If you chose another mode, -write “n/a”. - ---- - - - -## Artifact link *(if you chose that)* - -Paste a **public** URL (or a path inside this workspace) to a repo, gist, -notebook, portfolio page, or game history. If not using a link, write “n/a”. - ---- - - - -## Code or work sample *(if you chose that)* - -Paste a **small** snippet (about 5–20 lines) or describe exactly which file / -position the agent should look at. If not using a sample, write “n/a”. - ---- - - - -## Picture / screenshot *(if you chose that)* - -Paste a public image URL, or a relative path to an image in this workspace -(e.g. `./assets/my-board.png`). If not using a picture, write “n/a”. - ---- - - - -## Quick questions *(everyone can answer these — required if you chose questions only)* - -Even if you shared an artifact, these help the agent double-check level. - ---- - - - -## 1. What would you play? - -You are White. The game started `1. e4 e5`. It is your second move. Which -choice best matches how you would actually play today? - -- Nf3 — develop a knight and look at the center -- Qh5 or Qg4 — bring the queen out early to attack -- I’m not sure — I usually pick something random or copy a video move - ---- - - - -## 2. Why that choice? - -In one or two sentences. If unsure, say what feels confusing. - ---- - - - -## 3. Quick meaning check - -Which best matches what **development** means in the opening? - -- Bringing knights and bishops to useful squares early -- Moving the queen as far as possible on move two -- Capturing every pawn you can see -- I don’t know that word in chess yet - ---- - - - -## What the agent does next - -1. Read the mode you chose + any artifact / experience / sample. -2. Use the quick questions to confirm or adjust. -3. Write Point A in observable language **citing that evidence**. -4. Draft Point B and the roadmap at your proximal edge. - -| Signal | Likely Point A sketch | -| --- | --- | -| Thin evidence + unsure answers | Legal moves; openings feel arbitrary; principles not yet usable | -| Early-queen habits / weak “why” | Knows pieces; overweights early attacks | -| Solid Nf3 + clear why, or strong game/repo evidence | Near Lesson 1 destination — shorten or skip basics | - -**→ [Confirm your plan](./confirm-plan.md)** diff --git a/tests/fixtures/path/paths/chess-opening-principles/path.outcome.md b/tests/fixtures/path/paths/chess-opening-principles/path.outcome.md deleted file mode 100644 index 4d3ba94..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/path.outcome.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -type: outcome -status: active -roadmap: - status: proposed ---- - -# Chess Opening Principles — Outcome - -## Point A - -The learner can move pieces legally and knows check/checkmate exist, but -chooses opening moves by habit or memorized first moves. They cannot yet -explain why a move helps in the first 8–10 moves using center, development, -and king safety. - -## Point B - -In a new game as White or Black, the learner can play the first 8–10 moves by -applying three opening principles (control/contest the center, develop minor -pieces, safeguard the king), compare two candidate moves on a board, reject -one common early mistake, and explain their plan in plain language without -relying on a long memorized line. - -## Rubric - -- Names the three principles and what each looks like on the board. -- Chooses between two candidate moves and justifies with a principle. -- Spots one early mistake (e.g. premature queen outing / neglecting development). -- Plays or annotates a short opening that shows center + development + king - safety. -- Explains the plan in 2–4 sentences without a memorized ECO line. - -Point B changes only if **renegotiated** with the learner (including after a -failed remediation assessment). Current Point A advances after review + a -passed assessment (**no core concept missed**). diff --git a/tests/fixtures/path/paths/chess-opening-principles/references/index.references.md b/tests/fixtures/path/paths/chess-opening-principles/references/index.references.md deleted file mode 100644 index 2fff9bd..0000000 --- a/tests/fixtures/path/paths/chess-opening-principles/references/index.references.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -type: references -status: active ---- - -# Chess Opening Principles — References - -Small vocabulary for this path. Expand only when a lesson needs it. - -## Principles - -- **Center** — the d4, d5, e4, e5 squares (and influence over them) -- **Development** — bringing minor pieces (knights and bishops) into useful - squares early -- **King safety** — usually castling and not leaving the king in a drafty center - -## Fixture note - -This fixture uses Markdown boards and flashcards. A maintained chess board -component can replace the text boards later without changing the learning loop. diff --git a/tests/fixtures/path/paths/learner.profile.md b/tests/fixtures/path/paths/learner.profile.md index 96a7cb1..ced5b13 100644 --- a/tests/fixtures/path/paths/learner.profile.md +++ b/tests/fixtures/path/paths/learner.profile.md @@ -1,54 +1,21 @@ --- -type: profile +type: learner-profile status: active --- # Learner Profile -Confirmed learner goals and preferences. Record only what the learner chooses -to share. For each new path, collect onboarding answers and Point A evidence -before drafting Point A, Point B, and the roadmap. +This fictional fixture records only confirmed, learning-relevant context. ---- - -## New path onboarding - -- Goal — what concrete capability does the learner want? -- Why — why does it matter now? -- Prior knowledge — what can the learner already do? *(claim only — not enough)* -- Stuck points — where does progress break down? -- Time and pace — how long and how often can they practice? -- Preferred feel — calm and stepwise, brisk, or a mix? -- **Point A evidence** — proof of level: past experience, artifact link, - small sample, picture/screenshot, and/or quick placement questions - -Copy only confirmed answers here. Write Point A from that evidence, then draft -Point B and the roadmap. Learner must confirm before `roadmap.status: -confirmed`. - -Fixture entry: -[Onboarding](./chess-opening-principles/onboarding/index.lesson.md) → -[Point A evidence](./chess-opening-principles/onboarding/point-a-evidence.md). - ---- +## Current direction -## Goals - -- Chess opening principles: play sensible opening moves in the first 8–10 moves - without memorizing a long line - - Complete onboarding before treating demo defaults as learner-confirmed - ---- - -## Preferences - -- Formats that help: to be discovered -- Useful constraints or accessibility needs: to be discovered -- Chess fixture feel: calm and stepwise *(demo default; update from onboarding)* - ---- +- **Foreground path:** [SQL Foundations](./sql-foundations/index.path.md) +- **Point B:** Build and explain a small support report across related tables. +- **Point A evidence:** Correctly filters one table; has not used joins. +- **Available rhythm:** Two 20-minute sessions each week. -## Milestones +## Learning preferences -Link durable wins here as they accumulate. The detailed record lives in -[learning activity](./learning.activity.md). +- Practical examples before terminology. +- Calm, direct feedback with an optional stretch task. +- Neutral theme with a blue accent and system appearance. diff --git a/tests/fixtures/path/paths/learning.activity.md b/tests/fixtures/path/paths/learning.activity.md index 6e9d28f..996996c 100644 --- a/tests/fixtures/path/paths/learning.activity.md +++ b/tests/fixtures/path/paths/learning.activity.md @@ -1,83 +1,23 @@ --- -type: activity +type: learning-activity +status: active --- # Learning Activity -Global record across paths. After each lesson’s short review (and any gap -practice), write a **synthesis** before assessment. Assessment does not use a -percentage — **pass = no core concept missed.** Persist placements, review-queue -items, and renegotiations here. Do not rewrite old records. +Keep completed evidence append-only. This fixture begins before the learner's +first module session. ---- - -## Record each lesson loop - -One Block with stable `id` and date: - -- **Evidence:** quote review / in-lesson responses (learner’s phrasing) -- **Shift:** what they understand or can do differently -- **Still fuzzy / practice:** gap named + practice done (or none) -- **Review queue:** 1–2 retrieval concepts with due-by lesson number -- **Assessment:** pass/fail by core miss, attempt number, missed concepts, - confident-and-wrong items, optional stretch note -- **Placement:** advance, advance + peripheral reteach, variant retry, - remediation, or renegotiate after failed remediation -- **Point A now:** capability used to design the next lesson -- **Opens next:** next lesson destination or remediation / renegotiation target - ---- - -## Review queue - -Oldest due item is pulled into assessments when lesson N ≥ 2. - -| Concept | Due by lesson | Added | -| --- | --- | --- | -| *(e.g. three principles)* | 2 | after Lesson 1 synthesis | - ---- - - - -## Synthesis: joins - -**Evidence:** [Joining tables assessment](./sql-foundations/lessons/joins/lesson.assessment.md) -meets the rubric; the linked work uses `LEFT JOIN` and explains unmatched rows. - -**Shift:** The learner now treats join choice as a statement about which rows -must survive, not as syntax to memorize. - -**Opens next:** Aggregate the joined results by customer segment. - ---- - - - -## Placeholder: chess Lesson 1 synthesis - -*(Fill after [short review](./chess-opening-principles/lessons/control-center-development/lesson.review.md) when ready for assessment.)* - -**Evidence:** *(quotes)* - -**Shift:** *(how openings look different)* - -**Still fuzzy / practice:** *(gap + [practice](./chess-opening-principles/lessons/control-center-development/lesson.practice.md) or none)* - -**Review queue:** *(1–2 concepts; due-by lesson)* +## Current state -**Assessment:** *(after ready; core pass/fail; attempts; confident-and-wrong)* +- **Foreground path:** [SQL Foundations](./sql-foundations/index.path.md) +- **Current milestone:** Combine related tables — `ready` +- **Next session:** [See what a join preserves](./sql-foundations/modules/01-combine-tables/01-rows-that-survive.lesson.md) +- **Open review items:** None. -**Placement:** Locked until ready → then no-core-miss advance, variant retry, -[remediation](./chess-opening-principles/lessons/reteach-opening-principles/index.lesson.md) -after 2–3 core fails, or renegotiate if remediation assess also fails. +## Activity log -**Point A now:** *(assessed reality)* +### Module prepared -**Opens next:** *(Lesson 2 [shell](./chess-opening-principles/lessons/develop-before-queen/index.lesson.md) or remediation)* +Point A evidence, Point B, the milestone map, and both current-module sessions +are ready. No milestone has been marked demonstrated yet. diff --git a/tests/fixtures/path/paths/sql-foundations/index.path.md b/tests/fixtures/path/paths/sql-foundations/index.path.md index c281ee9..8763bab 100644 --- a/tests/fixtures/path/paths/sql-foundations/index.path.md +++ b/tests/fixtures/path/paths/sql-foundations/index.path.md @@ -5,14 +5,26 @@ status: active # SQL Foundations -[Outcome](./path.outcome.md) +**Point A:** Can filter one table. Joins are unfamiliar. + +**Point B:** Can build and explain a small support report across related tables. + +[Outcome and evidence targets](./path.outcome.md) --- -## Current lesson +## Milestone map + +- **Choose rows from one table:** demonstrated +- **Combine related tables:** ready +- **Summarize joined results:** planned +- **Build the support report:** planned + +## Current module -- [Joining tables](./lessons/joins/index.lesson.md) +- [Combine related tables](./modules/01-combine-tables/index.path.md) -## Activity +## Learning record -- [Learning record](../learning.activity.md) +- [Learner profile](../learner.profile.md) +- [Learning activity](../learning.activity.md) diff --git a/tests/fixtures/path/paths/sql-foundations/lessons/joins/index.lesson.md b/tests/fixtures/path/paths/sql-foundations/lessons/joins/index.lesson.md deleted file mode 100644 index 45df1fd..0000000 --- a/tests/fixtures/path/paths/sql-foundations/lessons/joins/index.lesson.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -type: lesson -status: active -start: Can filter one table; joins are unfamiliar -destination: Can choose and write INNER or LEFT JOIN and explain which rows remain ---- - -# Joining Tables - -## Recall - -What does `WHERE` remove? - ---- - -## Compare joins - -`INNER JOIN` keeps matches. `LEFT JOIN` also keeps unmatched left rows. - -[Assessment](./lesson.assessment.md) diff --git a/tests/fixtures/path/paths/sql-foundations/lessons/joins/lesson.assessment.md b/tests/fixtures/path/paths/sql-foundations/lessons/joins/lesson.assessment.md deleted file mode 100644 index 1374abb..0000000 --- a/tests/fixtures/path/paths/sql-foundations/lessons/joins/lesson.assessment.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -type: assessment -status: ready ---- - -# Joining Tables Assessment - -Write a query that keeps every ticket when some customer records are missing. -Explain why the other join would drop rows. - -## Rubric - -- Uses `LEFT JOIN`. -- Joins the correct keys. -- Explains unmatched rows. - -Record the reviewed result in the [activity log](../../../learning.activity.md). diff --git a/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/01-rows-that-survive.lesson.md b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/01-rows-that-survive.lesson.md new file mode 100644 index 0000000..f2253d2 --- /dev/null +++ b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/01-rows-that-survive.lesson.md @@ -0,0 +1,53 @@ +--- +type: lesson +status: ready +estimatedMinutes: 20 +--- + +# See What a Join Preserves + +By the end, predict which ticket rows survive an `INNER JOIN` and a +`LEFT JOIN`. + +--- + + + +## Start with one worked example + +```sql +SELECT t.id, c.name +FROM tickets t +LEFT JOIN customers c ON c.id = t.customer_id; +``` + +A `LEFT JOIN` keeps every ticket. When a customer record is missing, the +customer columns are empty instead of the ticket disappearing. + +--- + + + +## Predict before revealing + +Write one sentence predicting what changes if `LEFT JOIN` becomes +`INNER JOIN`. + +--- + + + +## Compare your prediction + +`INNER JOIN` keeps only tickets with a matching customer. Ask “Which rows must +survive?” before choosing the keyword. + +--- + + + +## Apply it + +Choose a join for a report that must include every ticket, then explain the +choice. Continue to [Session 2](./02-choose-the-join.lesson.md) without waiting +for an agent. diff --git a/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/02-choose-the-join.lesson.md b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/02-choose-the-join.lesson.md new file mode 100644 index 0000000..b3768b0 --- /dev/null +++ b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/02-choose-the-join.lesson.md @@ -0,0 +1,41 @@ +--- +type: lesson +status: ready +estimatedMinutes: 20 +--- + +# Choose the Join From the Question + +By the end, translate a reporting requirement into a join choice and condition. + +--- + + + +## Model the decision + +“Show every customer, including customers with no tickets” means customers are +the left table and the query needs a `LEFT JOIN`. + +--- + + + +## Try two cases + +1. Show only tickets with a valid customer. +2. Show every ticket, even when its customer record is missing. + +For each case, name the left table, join type, and matching keys. + +--- + + + +## Check your reasoning + +Case 1 may use `INNER JOIN`. Case 2 needs tickets on the left with a +`LEFT JOIN`. In both cases, match `customers.id` to `tickets.customer_id`. + +Use the [focused review](./review.practice.md) if row survival is still fuzzy; +otherwise continue to the [milestone checkpoint](./milestone.assessment.md). diff --git a/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/index.path.md b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/index.path.md new file mode 100644 index 0000000..4e3eb91 --- /dev/null +++ b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/index.path.md @@ -0,0 +1,18 @@ +--- +type: path +status: ready +--- + +# Combine Related Tables + +**Destination:** Choose `INNER JOIN` or `LEFT JOIN`, write the join condition, +and explain which unmatched rows remain. + +Both sessions are ready before the learner begins: + +1. [See what a join preserves](./01-rows-that-survive.lesson.md) +2. [Choose the join from the question](./02-choose-the-join.lesson.md) +3. [Optional focused review](./review.practice.md) +4. [Milestone checkpoint](./milestone.assessment.md) + +[Return to the path map](../../index.path.md) diff --git a/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/milestone.assessment.md b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/milestone.assessment.md new file mode 100644 index 0000000..3553aed --- /dev/null +++ b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/milestone.assessment.md @@ -0,0 +1,36 @@ +--- +type: assessment +status: ready +--- + +# Combine Related Tables Checkpoint + +Write a query that keeps every ticket when some customer records are missing. + +--- + + + +## Explain your choice + +Explain why the other join would drop rows and identify the matching keys. + +--- + + + +## Evidence target + +- Uses `LEFT JOIN` with tickets on the left. +- Joins `customers.id` to `tickets.customer_id`. +- Explains what happens to unmatched ticket rows. + +After agent review, record the evidence and placement decision in the +[activity log](../../../learning.activity.md). diff --git a/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/review.practice.md b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/review.practice.md new file mode 100644 index 0000000..db54af3 --- /dev/null +++ b/tests/fixtures/path/paths/sql-foundations/modules/01-combine-tables/review.practice.md @@ -0,0 +1,19 @@ +--- +type: practice +status: ready +--- + +# Optional Focused Review + +Use this only when the join choice is still uncertain. + +## Smaller version + +Imagine three tickets. Two have matching customers and one does not. + +- `INNER JOIN` returns two ticket rows. +- `LEFT JOIN` with tickets on the left returns all three. + +Now explain which join answers “How many tickets were opened?” without losing +the unmatched ticket. Return to the [checkpoint](./milestone.assessment.md) +when the row-survival rule is clear. diff --git a/tests/fixtures/path/paths/sql-foundations/path.outcome.md b/tests/fixtures/path/paths/sql-foundations/path.outcome.md index 2708222..29a5a76 100644 --- a/tests/fixtures/path/paths/sql-foundations/path.outcome.md +++ b/tests/fixtures/path/paths/sql-foundations/path.outcome.md @@ -13,8 +13,11 @@ Can filter one table. Joins are unfamiliar. Can combine two tables and explain which rows remain. -## Rubric +## Milestone evidence -- Chooses the right join. -- Writes the join condition. -- Explains unmatched rows. +| Milestone | Evidence | Status | +| --- | --- | --- | +| Choose rows from one table | Existing filter query | demonstrated | +| Combine related tables | Query plus row-survival explanation | ready | +| Summarize joined results | Grouped support totals | planned | +| Build the support report | Complete report and explanation | planned | diff --git a/tests/fixtures/pathmx/annotations/index.path.md b/tests/fixtures/pathmx/annotations/index.path.md new file mode 100644 index 0000000..1fff09a --- /dev/null +++ b/tests/fixtures/pathmx/annotations/index.path.md @@ -0,0 +1,18 @@ +--- +type: path +status: active +--- + +# Feedback Practice + +Use this sentence to test a durable annotation.[^c1] + +The key idea is {==worth revisiting later==}[^c2]. + +[^c1]: **@learner** (2026-07-20 10:30 -04:00): Please add another example. + + ✓ @learner 2026-07-20 11:05 -04:00 + +[^c1.a]: **@agent** (2026-07-20 10:42 -04:00): I added one to the next session. + +[^c2]: **@learner** (2026-07-20 10:35 -04:00): Connect this to the milestone. diff --git a/tests/fixtures/pathmx/config/paths/workshop.path.md b/tests/fixtures/pathmx/config/paths/workshop.path.md index 3804b8a..7c9425b 100644 --- a/tests/fixtures/pathmx/config/paths/workshop.path.md +++ b/tests/fixtures/pathmx/config/paths/workshop.path.md @@ -1,3 +1,7 @@ +--- +handle: "@workshop" +--- + # Workshop This is the second configured root. diff --git a/work-log/2026-07-20-adaptive-learning-loop.brief.md b/work-log/2026-07-20-adaptive-learning-loop.brief.md index 75dea5e..233dee9 100644 --- a/work-log/2026-07-20-adaptive-learning-loop.brief.md +++ b/work-log/2026-07-20-adaptive-learning-loop.brief.md @@ -1,13 +1,13 @@ --- -status: proposed +status: superseded date: 2026-07-20 related: - https://build-week.pathmx.net/work-log/2026-07-18-player-native-learning-reshape.brief - ../skills/path/SKILL.md - - ../skills/path/references/adaptive-loop-example.md + - ../skills/path/references/buffered-loop.md - ../tests/fixtures/path/paths/learner.profile.md - ../tests/fixtures/path/paths/learning.activity.md - - ../tests/fixtures/path/paths/chess-opening-principles/index.path.md + - ../tests/fixtures/path/paths/sql-foundations/index.path.md --- # Adaptive Point A → Point B Learning Loop @@ -17,11 +17,11 @@ Extends the confirm a plan, teach in small stages, review after each lesson, assess before the next, then update the roadmap from evidence. -Kept here as the design brief behind `/path`. The skill summary and charts live -in -[adaptive-loop-example.md](../skills/path/references/adaptive-loop-example.md); -the playable fixture is -[chess-opening-principles](../tests/fixtures/path/paths/chess-opening-principles/index.path.md). +Kept here as the first design brief behind `/path` and as a record of Tram Le +and Mark Johnson's early testing. That testing exposed the waiting and weak +progress structure in the one-Block-at-a-time loop described below. The current +contract is the [buffered loop](../skills/path/references/buffered-loop.md), with +a compact [SQL fixture](../tests/fixtures/path/paths/sql-foundations/index.path.md). | | | | --- | --- | @@ -281,9 +281,9 @@ paths/ └── lesson.assessment.md ``` -In this repository the playable fixture lives under -`tests/fixtures/path/paths/`. Workflow briefs live in `work-log/` (not playable -learner Sources). +The retired Chess prototype remains recoverable from Git history. The current +playable fixture lives under `tests/fixtures/path/paths/sql-foundations/`. +Workflow briefs live in `work-log/`, not in synced learner Sources. --- diff --git a/work-log/index.md b/work-log/index.md index 6871c8c..7403b57 100644 --- a/work-log/index.md +++ b/work-log/index.md @@ -7,7 +7,6 @@ PathMX Sources and not part of the synced skill packages. The skill contract is ## Entries - [Adaptive Point A → Point B Learning Loop](./2026-07-20-adaptive-learning-loop.brief.md) - — fixture: - [Chess Opening Principles](../tests/fixtures/path/paths/chess-opening-principles/index.path.md); - skill summary: - [adaptive-loop-example.md](../skills/path/references/adaptive-loop-example.md) + — the superseded first design. Its testing led to the current + [buffered loop](../skills/path/references/buffered-loop.md) and compact + [SQL fixture](../tests/fixtures/path/paths/sql-foundations/index.path.md).