diff --git a/ROADMAP.md b/ROADMAP.md index 0cc31b3..786710d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -10,6 +10,34 @@ The long-term idea is inspired by reinforcement learning, but the product path s Evals → Experiments → Optimization → Environment → RL ``` +## Current status after PR #44 + +Completed: + +- Public alpha package surface under `@ignitionai/agent-trainer-*`. +- Local experiments, reports, report bundles, regression gates and CI validation. +- RAG presets, strategy presets and IgnitionRAG adapter/bridge contracts. +- Policy, trajectory, contextual bandit and offline policy evaluation foundations. +- Tested environment episodes with trajectory reports. +- Dedicated core package tests. +- npm alpha publication at `0.1.0-alpha.1`. + +Next open-source work: + +1. Improve CLI ergonomics for history, baseline selection and regression checks. +2. Add deeper mocked examples for ecosystem adapters. +3. Dogfood the alpha in IgnitionRAG and collect real trajectory findings. +4. Add a lightweight policy optimization loop only after useful dogfood trajectories exist. + +Still out of scope: + +- PPO implementation, +- GRPO training, +- model fine-tuning, +- production routing, +- real provider calls in core examples, +- hosted SaaS runtime. + --- ## Phase 0 — Research and design diff --git a/docs/ALPHA_READINESS.md b/docs/ALPHA_READINESS.md index d436ecc..b4e7468 100644 --- a/docs/ALPHA_READINESS.md +++ b/docs/ALPHA_READINESS.md @@ -55,10 +55,11 @@ All packages declare `license: MIT`, matching the root `LICENSE` file. ## Known Work After Internal Alpha -- Add CLI history/baseline/regression ergonomics after report bundles and CI examples. -- Decide package publishing policy before any npm publication. -- Add deeper examples for ecosystem adapters. +- Add CLI history, baseline selection and regression command ergonomics after report bundles and CI examples. +- Add deeper mocked examples for ecosystem adapters. +- Dogfood the alpha inside IgnitionRAG and collect trajectory/reward evidence. - Add a lightweight policy optimization loop after real dogfood produces trajectory data. +- Decide any next-step publishing automation policy before a future public release workflow. ## Explicit Non-goals diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 099db0f..84aa14a 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -2732,7 +2732,7 @@ Next PR: Status: -- current +- completed Branch: @@ -2785,9 +2785,255 @@ Definition of done: - package readiness no longer says core tests are missing, - docs keep runtime schema validation out of scope. +Next PR: + +- PR #45 - `docs: sync roadmap after core coverage` + +### PR #45 - `docs: sync roadmap after core coverage` + +Status: + +- current + +Branch: + +```txt +docs/sync-roadmap-after-core-tests +``` + +Goal: + +Bring roadmap, milestones and backlog back in sync after PR #43 and PR #44, then document the next open-source work in a usable order. + +Scope: + +- mark PR #44 as completed, +- update roadmap status after tested environment episodes and core tests, +- document the next planned PRs with scope, non-goals and acceptance, +- keep PPO, GRPO training and provider calls out of near-term scope. + +Out of scope: + +- runtime source code changes, +- package API changes, +- test implementation changes, +- npm publication, +- IgnitionRAG repo work. + +Required APIs / files: + +- `ROADMAP.md`, +- `docs/POST_20_ROADMAP.md`, +- `docs/MILESTONES.md`, +- `docs/ALPHA_READINESS.md`, +- `docs/BACKLOG.md`. + +Acceptance: + +```bash +bun run lint +git diff --check +``` + +Definition of done: + +- roadmap states the repo is complete through PR #44, +- backlog has a concrete next sequence, +- next work is phrased as developer ergonomics and dogfood-driven policy work, +- no runtime files are modified. + +Next PR: + +- PR #46 - `feat: add CLI history baseline and regression ergonomics` + +### PR #46 - `feat: add CLI history baseline and regression ergonomics` + +Status: + +- planned + +Branch: + +```txt +feat/cli-history-baseline-regression +``` + +Goal: + +Make the CLI easier to use for repeated local and CI evaluation loops. + +Scope: + +- expose a history-oriented CLI workflow over existing experiment history helpers, +- add baseline selection ergonomics for report comparisons, +- add a regression-check command or flags that reuse existing regression gate primitives, +- document exact commands for local and CI usage, +- add focused CLI tests for success and failure paths. + +Out of scope: + +- database or hosted history, +- SaaS dashboard, +- remote execution, +- provider calls, +- new reward algorithms, +- PPO or policy optimization. + +Required APIs / files: + +- `packages/cli`, +- `packages/experiments`, +- `examples/context-engineering`, +- CLI README/docs, +- alpha readiness and project audit docs if status changes. + +Acceptance: + +```bash +bun run lint +bun run typecheck +bun run test +bun run build +bun run pack:check +``` + +Definition of done: + +- a developer can list or inspect local experiment history from the CLI, +- a developer can choose a baseline without hand-editing files, +- a regression check can fail clearly with actionable output, +- docs include copy-pastable commands. + +Next PR: + +- PR #47 - `examples: add deeper ecosystem adapter examples` + +### PR #47 - `examples: add deeper ecosystem adapter examples` + +Status: + +- planned + +Branch: + +```txt +examples/deeper-ecosystem-adapters +``` + +Goal: + +Prove the adapter packages are usable without relying only on structural unit tests. + +Scope: + +- add deterministic mocked examples for LangChain, LangGraph, Mastra and Vercel AI SDK adapters, +- show how each adapter plugs into `createExperiment`, +- include trace, usage and metadata mapping where each framework shape supports it, +- document adapter limits honestly. + +Out of scope: + +- live provider credentials, +- real network calls, +- deep framework feature coverage, +- hosted demos, +- IgnitionRAG integration. + +Required APIs / files: + +- `examples/*adapter*` or one grouped adapter example, +- adapter package READMEs, +- root README example list, +- alpha readiness and project audit docs. + +Acceptance: + +```bash +bun run lint +bun run typecheck +bun run test +bun run build +bun run pack:check +``` + +Definition of done: + +- every ecosystem adapter has at least one runnable mocked example, +- examples use public package APIs, +- examples do not require secrets or network access, +- docs make unsupported framework behavior clear. + +Next PR: + +- PR #48 - `feat: add dogfood-driven policy optimization loop` + +### PR #48 - `feat: add dogfood-driven policy optimization loop` + +Status: + +- planned, blocked until dogfood trajectories exist + +Branch: + +```txt +feat/dogfood-policy-optimization-loop +``` + +Goal: + +Turn real or representative dogfood trajectories into a lightweight policy comparison and selection loop. + +Precondition: + +- IgnitionRAG dogfood must produce useful trajectory/reward data, or this repo must get committed representative fixtures derived from that dogfood. + +Scope: + +- load or define trajectory fixtures, +- replay trajectories offline, +- compare simple policies against offline records, +- select a policy with contextual bandit or fixed-strategy bandit primitives, +- produce a deterministic policy selection report, +- optionally expose the workflow through CLI only if the CLI contract is already ready. + +Out of scope: + +- PPO implementation, +- GRPO training, +- neural policy training, +- model fine-tuning, +- live traffic routing, +- production serving, +- real provider calls. + +Required APIs / files: + +- `packages/rl`, +- `packages/environment` only if episode contracts need small fixes, +- `packages/cli` only if CLI integration is included, +- dogfood-derived fixtures or examples, +- docs explaining offline evaluation limits. + +Acceptance: + +```bash +bun run lint +bun run typecheck +bun run test +bun run build +bun run pack:check +``` + +Definition of done: + +- policy comparison works offline from deterministic trajectory data, +- selection output explains why one policy wins, +- no model training is introduced, +- docs explicitly say this is lightweight policy optimization, not PPO. + ## Dogfood phase - IgnitionRAG -Next work should happen in IgnitionRAG, not by adding more framework abstractions here. +Default next product work should happen in IgnitionRAG. Open-source work should stay limited to concrete developer ergonomics or framework gaps found while dogfooding. Target product slices: diff --git a/docs/MILESTONES.md b/docs/MILESTONES.md index 5e40be4..d34d2a6 100644 --- a/docs/MILESTONES.md +++ b/docs/MILESTONES.md @@ -296,13 +296,13 @@ docs/ALPHA_VALIDATION_PLAN.md Current status: ```txt -PR #36 through PR #43 are complete. The internal v0.1.0-alpha.0 tag, npm v0.1.0-alpha.1 publication, and tested environment episode foundation are complete. +PR #36 through PR #44 are complete. The internal v0.1.0-alpha.0 tag, npm v0.1.0-alpha.1 publication, tested environment episode foundation, trajectory reports and dedicated core tests are complete. ``` Next phase: ```txt -Close the remaining core test coverage gap, then dogfood the alpha inside IgnitionRAG and return to this repository only for concrete framework gaps found during use. +Keep open-source work focused on developer ergonomics: CLI history/baseline/regression workflows, deeper adapter examples, then dogfood the alpha inside IgnitionRAG and return for concrete framework gaps found during use. ``` Explicit non-goals: diff --git a/docs/POST_20_ROADMAP.md b/docs/POST_20_ROADMAP.md index 31cb9e8..d6b9bc4 100644 --- a/docs/POST_20_ROADMAP.md +++ b/docs/POST_20_ROADMAP.md @@ -111,6 +111,20 @@ PR #35 must add interfaces only. It must not implement PPO optimization, neural ## Completion Status -The Post-#20 sequence is complete through PR #35 once the PPO interface skeletons PR is merged. +The Post-#20 sequence is complete through PR #35, and the follow-up alpha hardening work is complete through PR #44. -Remaining RL work belongs to a future phase and must be scoped separately. +Completed follow-up work: + +- npm alpha publishing readiness, +- tested environment episodes and trajectory rewards, +- deterministic trajectory reports, +- offline policy evaluation compatibility from environment episodes, +- dedicated core package tests. + +Next open-source work is practical developer ergonomics first: + +1. CLI history, baseline and regression workflows. +2. Deeper mocked examples for ecosystem adapters. +3. Dogfood-driven policy optimization once IgnitionRAG produces useful trajectories. + +Remaining RL work still belongs to a future phase and must be scoped from real dogfood evidence, not from PPO-first assumptions.