Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,34 @@ The long-term idea is inspired by reinforcement learning, but the product path s
Evals → Experiments → Optimization → Environment → RL
```

## Current status after PR #44

Completed:

- Public alpha package surface under `@ignitionai/agent-trainer-*`.
- Local experiments, reports, report bundles, regression gates and CI validation.
- RAG presets, strategy presets and IgnitionRAG adapter/bridge contracts.
- Policy, trajectory, contextual bandit and offline policy evaluation foundations.
- Tested environment episodes with trajectory reports.
- Dedicated core package tests.
- npm alpha publication at `0.1.0-alpha.1`.

Next open-source work:

1. Improve CLI ergonomics for history, baseline selection and regression checks.
2. Add deeper mocked examples for ecosystem adapters.
3. Dogfood the alpha in IgnitionRAG and collect real trajectory findings.
4. Add a lightweight policy optimization loop only after useful dogfood trajectories exist.

Still out of scope:

- PPO implementation,
- GRPO training,
- model fine-tuning,
- production routing,
- real provider calls in core examples,
- hosted SaaS runtime.

---

## Phase 0 — Research and design
Expand Down
7 changes: 4 additions & 3 deletions docs/ALPHA_READINESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,11 @@ All packages declare `license: MIT`, matching the root `LICENSE` file.

## Known Work After Internal Alpha

- Add CLI history/baseline/regression ergonomics after report bundles and CI examples.
- Decide package publishing policy before any npm publication.
- Add deeper examples for ecosystem adapters.
- Add CLI history, baseline selection and regression command ergonomics after report bundles and CI examples.
- Add deeper mocked examples for ecosystem adapters.
- Dogfood the alpha inside IgnitionRAG and collect trajectory/reward evidence.
- Add a lightweight policy optimization loop after real dogfood produces trajectory data.
- Decide any next-step publishing automation policy before a future public release workflow.

## Explicit Non-goals

Expand Down
250 changes: 248 additions & 2 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2732,7 +2732,7 @@ Next PR:

Status:

- current
- completed

Branch:

Expand Down Expand Up @@ -2785,9 +2785,255 @@ Definition of done:
- package readiness no longer says core tests are missing,
- docs keep runtime schema validation out of scope.

Next PR:

- PR #45 - `docs: sync roadmap after core coverage`

### PR #45 - `docs: sync roadmap after core coverage`

Status:

- current

Branch:

```txt
docs/sync-roadmap-after-core-tests
```

Goal:

Bring roadmap, milestones and backlog back in sync after PR #43 and PR #44, then document the next open-source work in a usable order.

Scope:

- mark PR #44 as completed,
- update roadmap status after tested environment episodes and core tests,
- document the next planned PRs with scope, non-goals and acceptance,
- keep PPO, GRPO training and provider calls out of near-term scope.

Out of scope:

- runtime source code changes,
- package API changes,
- test implementation changes,
- npm publication,
- IgnitionRAG repo work.

Required APIs / files:

- `ROADMAP.md`,
- `docs/POST_20_ROADMAP.md`,
- `docs/MILESTONES.md`,
- `docs/ALPHA_READINESS.md`,
- `docs/BACKLOG.md`.

Acceptance:

```bash
bun run lint
git diff --check
```

Definition of done:

- roadmap states the repo is complete through PR #44,
- backlog has a concrete next sequence,
- next work is phrased as developer ergonomics and dogfood-driven policy work,
- no runtime files are modified.

Next PR:

- PR #46 - `feat: add CLI history baseline and regression ergonomics`

### PR #46 - `feat: add CLI history baseline and regression ergonomics`

Status:

- planned

Branch:

```txt
feat/cli-history-baseline-regression
```

Goal:

Make the CLI easier to use for repeated local and CI evaluation loops.

Scope:

- expose a history-oriented CLI workflow over existing experiment history helpers,
- add baseline selection ergonomics for report comparisons,
- add a regression-check command or flags that reuse existing regression gate primitives,
- document exact commands for local and CI usage,
- add focused CLI tests for success and failure paths.

Out of scope:

- database or hosted history,
- SaaS dashboard,
- remote execution,
- provider calls,
- new reward algorithms,
- PPO or policy optimization.

Required APIs / files:

- `packages/cli`,
- `packages/experiments`,
- `examples/context-engineering`,
- CLI README/docs,
- alpha readiness and project audit docs if status changes.

Acceptance:

```bash
bun run lint
bun run typecheck
bun run test
bun run build
bun run pack:check
```

Definition of done:

- a developer can list or inspect local experiment history from the CLI,
- a developer can choose a baseline without hand-editing files,
- a regression check can fail clearly with actionable output,
- docs include copy-pastable commands.

Next PR:

- PR #47 - `examples: add deeper ecosystem adapter examples`

### PR #47 - `examples: add deeper ecosystem adapter examples`

Status:

- planned

Branch:

```txt
examples/deeper-ecosystem-adapters
```

Goal:

Prove the adapter packages are usable without relying only on structural unit tests.

Scope:

- add deterministic mocked examples for LangChain, LangGraph, Mastra and Vercel AI SDK adapters,
- show how each adapter plugs into `createExperiment`,
- include trace, usage and metadata mapping where each framework shape supports it,
- document adapter limits honestly.

Out of scope:

- live provider credentials,
- real network calls,
- deep framework feature coverage,
- hosted demos,
- IgnitionRAG integration.

Required APIs / files:

- `examples/*adapter*` or one grouped adapter example,
- adapter package READMEs,
- root README example list,
- alpha readiness and project audit docs.

Acceptance:

```bash
bun run lint
bun run typecheck
bun run test
bun run build
bun run pack:check
```

Definition of done:

- every ecosystem adapter has at least one runnable mocked example,
- examples use public package APIs,
- examples do not require secrets or network access,
- docs make unsupported framework behavior clear.

Next PR:

- PR #48 - `feat: add dogfood-driven policy optimization loop`

### PR #48 - `feat: add dogfood-driven policy optimization loop`

Status:

- planned, blocked until dogfood trajectories exist

Branch:

```txt
feat/dogfood-policy-optimization-loop
```

Goal:

Turn real or representative dogfood trajectories into a lightweight policy comparison and selection loop.

Precondition:

- IgnitionRAG dogfood must produce useful trajectory/reward data, or this repo must get committed representative fixtures derived from that dogfood.

Scope:

- load or define trajectory fixtures,
- replay trajectories offline,
- compare simple policies against offline records,
- select a policy with contextual bandit or fixed-strategy bandit primitives,
- produce a deterministic policy selection report,
- optionally expose the workflow through CLI only if the CLI contract is already ready.

Out of scope:

- PPO implementation,
- GRPO training,
- neural policy training,
- model fine-tuning,
- live traffic routing,
- production serving,
- real provider calls.

Required APIs / files:

- `packages/rl`,
- `packages/environment` only if episode contracts need small fixes,
- `packages/cli` only if CLI integration is included,
- dogfood-derived fixtures or examples,
- docs explaining offline evaluation limits.

Acceptance:

```bash
bun run lint
bun run typecheck
bun run test
bun run build
bun run pack:check
```

Definition of done:

- policy comparison works offline from deterministic trajectory data,
- selection output explains why one policy wins,
- no model training is introduced,
- docs explicitly say this is lightweight policy optimization, not PPO.

## Dogfood phase - IgnitionRAG

Next work should happen in IgnitionRAG, not by adding more framework abstractions here.
Default next product work should happen in IgnitionRAG. Open-source work should stay limited to concrete developer ergonomics or framework gaps found while dogfooding.

Target product slices:

Expand Down
4 changes: 2 additions & 2 deletions docs/MILESTONES.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,13 +296,13 @@ docs/ALPHA_VALIDATION_PLAN.md
Current status:

```txt
PR #36 through PR #43 are complete. The internal v0.1.0-alpha.0 tag, npm v0.1.0-alpha.1 publication, and tested environment episode foundation are complete.
PR #36 through PR #44 are complete. The internal v0.1.0-alpha.0 tag, npm v0.1.0-alpha.1 publication, tested environment episode foundation, trajectory reports and dedicated core tests are complete.
```

Next phase:

```txt
Close the remaining core test coverage gap, then dogfood the alpha inside IgnitionRAG and return to this repository only for concrete framework gaps found during use.
Keep open-source work focused on developer ergonomics: CLI history/baseline/regression workflows, deeper adapter examples, then dogfood the alpha inside IgnitionRAG and return for concrete framework gaps found during use.
```

Explicit non-goals:
Expand Down
18 changes: 16 additions & 2 deletions docs/POST_20_ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,20 @@ PR #35 must add interfaces only. It must not implement PPO optimization, neural

## Completion Status

The Post-#20 sequence is complete through PR #35 once the PPO interface skeletons PR is merged.
The Post-#20 sequence is complete through PR #35, and the follow-up alpha hardening work is complete through PR #44.

Remaining RL work belongs to a future phase and must be scoped separately.
Completed follow-up work:

- npm alpha publishing readiness,
- tested environment episodes and trajectory rewards,
- deterministic trajectory reports,
- offline policy evaluation compatibility from environment episodes,
- dedicated core package tests.

Next open-source work is practical developer ergonomics first:

1. CLI history, baseline and regression workflows.
2. Deeper mocked examples for ecosystem adapters.
3. Dogfood-driven policy optimization once IgnitionRAG produces useful trajectories.

Remaining RL work still belongs to a future phase and must be scoped from real dogfood evidence, not from PPO-first assumptions.
Loading