feat: add tested environment episodes and trajectory rewards

README.md

@@ -132,6 +132,12 @@ Run the IgnitionRAG evaluation bridge prototype:
 bun run --filter './examples/ignitionrag-evaluation-bridge' dev
 ```
 
+Run a deterministic RAG environment episode and trajectory report:
+
+```bash
+bun run --filter './examples/rag-environment-episode' dev
+```
+
 ---
 
 ## Minimal usage

docs/ALPHA_READINESS.md

@@ -29,7 +29,7 @@ All packages declare `license: MIT`, matching the root `LICENSE` file.
 | `@ignitionai/agent-trainer-adapter-vercel-ai` | ready | ready | ready | ready | partial | Structural adapter only; no streaming, tools or live provider calls. |
 | `@ignitionai/agent-trainer-cli` | ready | ready | ready | ready | partial | Runs typed experiments and writes standalone reports or timestamped bundles; no history/baseline/regression flags yet. |
 | `@ignitionai/agent-trainer-core` | ready | ready | missing | ready | partial | Foundational types and helpers need dedicated tests before alpha-stable status. |
-| `@ignitionai/agent-trainer-environment` | ready | ready | missing | ready | prototype | Early environment loop only; no recorder, policy evaluation or tests yet. |
+| `@ignitionai/agent-trainer-environment` | ready | ready | ready | ready | partial | Tested episode runner with safety guards and a deterministic RAG episode example; no production runtime or optimization loop. |
 | `@ignitionai/agent-trainer-evals` | ready | ready | ready | ready | partial | Current rewards are tested; RAG presets and richer scoring are still missing. |
 | `@ignitionai/agent-trainer-experiments` | ready | ready | ready | ready | ready | Local runner, definitions, gates and JSONL history are tested and documented. |
 | `@ignitionai/agent-trainer-exporters` | ready | ready | ready | ready | ready | JSON/Markdown report shape and local report bundles are tested. |
@@ -55,10 +55,10 @@ All packages declare `license: MIT`, matching the root `LICENSE` file.
 ## Known Work After Internal Alpha
 
 - Add dedicated tests for `@ignitionai/agent-trainer-core`.
-- Add dedicated tests for `@ignitionai/agent-trainer-environment` or keep it explicitly prototype-only.
 - Add CLI history/baseline/regression ergonomics after report bundles and CI examples.
 - Decide package publishing policy before any npm publication.
 - Add deeper examples for ecosystem adapters.
+- Add a lightweight policy optimization loop after real dogfood produces trajectory data.
 
 ## Explicit Non-goals
 

docs/BACKLOG.md

@@ -2596,7 +2596,7 @@ Next PR:
 
 Status:
 
-- current
+- completed
 
 Branch:
 
@@ -2658,6 +2658,72 @@ Definition of done:
 - docs show manual alpha publish order and external smoke test,
 - no npm publish happens in this PR.
 
+Next PR:
+
+- PR #43 - `feat: add tested environment episodes and trajectory rewards`
+
+### PR #43 - `feat: add tested environment episodes and trajectory rewards`
+
+Status:
+
+- current
+
+Branch:
+
+```txt
+feat/tested-environment-episodes
+```
+
+Goal:
+
+Make `@ignitionai/agent-trainer-environment` usable as a tested RL foundation before adding policy optimization or PPO.
+
+Scope:
+
+- enrich `runEpisode` with options, final state, average reward, metadata and safety guards,
+- add dedicated environment tests,
+- convert environment episodes to RL trajectories without making `environment` depend on `rl`,
+- export deterministic trajectory JSON/Markdown reports,
+- prove compatibility with offline policy evaluation,
+- add a deterministic RAG environment episode example.
+
+Out of scope:
+
+- PPO implementation,
+- GRPO training,
+- fine-tuning,
+- real provider calls,
+- production routing,
+- CLI optimization loop,
+- IgnitionRAG app integration.
+
+Required APIs / files:
+
+- `packages/environment`,
+- `packages/rl`,
+- `examples/rag-environment-episode`,
+- readiness and audit docs.
+
+Acceptance:
+
+```bash
+bun run lint
+bun run typecheck
+bun run test
+bun run build
+bun run pack:check
+bun run --filter './examples/rag-environment-episode' dev
+```
+
+Definition of done:
+
+- `runEpisode(environment, policy)` remains compatible,
+- environment episodes have dedicated tests,
+- episode trajectories can become offline policy records,
+- trajectory report output is deterministic,
+- example documents `search -> rerank -> verify -> answer`,
+- docs state that PPO and training remain out of scope.
+
 ## Dogfood phase - IgnitionRAG
 
 Next work should happen in IgnitionRAG, not by adding more framework abstractions here.

docs/MILESTONES.md

@@ -296,13 +296,13 @@ docs/ALPHA_VALIDATION_PLAN.md
 Current status:
 
 ```txt
-PR #36 through PR #41 are complete. The internal v0.1.0-alpha.0 tag is pushed on the merged bridge prototype. PR #42 is the current npm alpha publishing readiness branch.
+PR #36 through PR #42 are complete. The internal v0.1.0-alpha.0 tag and npm v0.1.0-alpha.1 publication are complete.
 ```
 
 Next phase:
 
 ```txt
-Prepare npm alpha publishing, tag v0.1.0-alpha.1 after PR #42 merges, publish with dist-tag alpha, then dogfood the alpha inside IgnitionRAG.
+Dogfood the alpha inside IgnitionRAG, then return to this repository for concrete framework gaps such as tested environment episodes, trajectory reporting and policy-evaluation ergonomics.
 ```
 
 Explicit non-goals:

docs/PROJECT_AUDIT.md

@@ -94,11 +94,11 @@ If a package exists but is intentionally narrow, minimal or untested, it is part
 ### `@ignitionai/agent-trainer-environment`
 
 - Purpose: early state/action/reward/policy environment loop primitives.
-- Main exports: `runEpisode`, `AgentEnvironment`, `EnvironmentState`, `EnvironmentAction`, `Policy`, `EpisodeResult`.
-- Stability level: prototype.
-- Tests present: no.
-- Example present: no.
-- Known limitations: no tests, no rollout recorder, no production policy evaluation.
+- Main exports: `runEpisode`, `RunEpisodeOptions`, `AgentEnvironment`, `EnvironmentState`, `EnvironmentAction`, `Policy`, `EpisodeStep`, `EpisodeResult`.
+- Stability level: partial.
+- Tests present: yes.
+- Example present: yes, `examples/rag-environment-episode`.
+- Known limitations: no production environment implementation, no durable rollout store, no automatic policy optimization loop and no CLI integration for environment episodes.
 
 ### `@ignitionai/agent-trainer-evals`
 
@@ -148,10 +148,10 @@ If a package exists but is intentionally narrow, minimal or untested, it is part
 ### `@ignitionai/agent-trainer-rl`
 
 - Purpose: experimental RL-inspired utilities, deterministic policy selection helpers, fixed-strategy/contextual bandit prototypes, offline policy evaluation, GRPO-style candidate selection and PPO interface skeletons.
-- Main exports: `Policy`, `PolicyContext`, `PolicyDecision`, `createStaticPolicy`, `createScoreBasedPolicy`, `Trajectory`, `TrajectoryStep`, `recordTrajectory`, `summarizeTrajectory`, `EpsilonGreedyBandit`, `RandomPolicy`, `ExperimentalBanditStrategySelector`, `ContextualBanditStrategySelector`, `ContextFeatures`, `scoreContextMatch`, `evaluatePolicyOffline`, `PolicyEvaluationResult`, `selectGroupRelativeBest`, `rankCandidateGroup`, `GroupRelativeSelectionResult`, `PPOConfig`, `PPOTrainer`, `PPOTrainingBatch`, `PPOTrainingResult`, `UnimplementedPPOTrainer`, `createUnimplementedPPOTrainer`.
+- Main exports: `Policy`, `PolicyContext`, `PolicyDecision`, `createStaticPolicy`, `createScoreBasedPolicy`, `Trajectory`, `TrajectoryStep`, `recordTrajectory`, `summarizeTrajectory`, `recordEpisodeTrajectory`, `exportTrajectoryReport`, `toMarkdownTrajectoryReport`, `EpsilonGreedyBandit`, `RandomPolicy`, `ExperimentalBanditStrategySelector`, `ContextualBanditStrategySelector`, `ContextFeatures`, `scoreContextMatch`, `evaluatePolicyOffline`, `PolicyEvaluationResult`, `selectGroupRelativeBest`, `rankCandidateGroup`, `GroupRelativeSelectionResult`, `PPOConfig`, `PPOTrainer`, `PPOTrainingBatch`, `PPOTrainingResult`, `UnimplementedPPOTrainer`, `createUnimplementedPPOTrainer`.
 - Stability level: prototype.
-- Tests present: yes for policy helpers, trajectories, fixed-strategy bandit, contextual bandit, offline policy evaluation, GRPO-style selection and PPO skeletons.
-- Example present: no.
+- Tests present: yes for policy helpers, trajectories, environment episode trajectory conversion, fixed-strategy bandit, contextual bandit, offline policy evaluation, GRPO-style selection and PPO skeletons.
+- Example present: yes, `examples/rag-environment-episode`.
 - Known limitations: no PPO optimization, no GRPO model training, no production routing.
 
 ### `@ignitionai/agent-trainer`
@@ -210,6 +210,13 @@ If a package exists but is intentionally narrow, minimal or untested, it is part
 - Mocked or live: deterministic mocked records and adapters.
 - Product concept: bridge prototype proving `Dataset` and `AgentVariant` mapping without IgnitionRAG app code.
 
+### `examples/rag-environment-episode`
+
+- Demonstrates: modeling a RAG workflow as an environment episode with `search`, `rerank`, `verify` and `answer` actions.
+- Command: `bun run --filter './examples/rag-environment-episode' dev`.
+- Mocked or live: deterministic mocked environment.
+- Product concept: episode rewards, trajectory reporting and offline policy record creation before any training loop.
+
 ## Current Capabilities Matrix
 
 | Capability | Status | Package/File | Stable? | Notes |
@@ -235,7 +242,9 @@ If a package exists but is intentionally narrow, minimal or untested, it is part
 | IgnitionRAG evaluation bridge prototype | prototype | `examples/ignitionrag-evaluation-bridge` | No | Deterministic record mapping only; no database, hosted worker, auth or real provider calls. |
 | file-based history | done | `packages/experiments/src/history.ts` | Yes | JSONL local history helpers, no CLI flag yet. |
 | policy abstraction | partial | `packages/rl/src/policy.ts` | No | Deterministic static and score-based selection only; no training loop. |
+| environment episodes | partial | `packages/environment` | No | Tested episode runner with max-step safety, seed forwarding, final state and metadata. |
 | trajectory recorder | partial | `packages/rl/src/trajectory.ts` | No | Local state/action/reward/outcome records with deterministic summaries. |
+| trajectory reports | partial | `packages/rl/src/episode-trajectory.ts` | No | Environment episodes can become trajectories, Markdown reports and offline policy records. |
 | bandit prototype | prototype | `packages/rl/src/strategy-bandit.ts` | No | Clearly experimental, fixed arms only, no PPO. |
 | contextual bandit prototype | prototype | `packages/rl/src/contextual-bandit.ts` | No | Deterministic fixed-feature scoring over task type, citation need, cost sensitivity, latency sensitivity and risk level. |
 | offline policy evaluation | prototype | `packages/rl/src/offline-policy-evaluation.ts` | No | Deterministic replay over local records or observed trajectory steps; no live traffic path. |
@@ -250,7 +259,8 @@ If a package exists but is intentionally narrow, minimal or untested, it is part
 - No real LLM calls by default.
 - No hosted IgnitionRAG integration code.
 - IgnitionRAG integration is design-only.
-- Core and environment need dedicated tests before alpha-stable status.
+- Core needs dedicated tests before alpha-stable status.
+- Environment episodes still need real dogfood data and CLI ergonomics before stable status.
 - Ecosystem adapters are minimal and structural.
 - CLI does not yet support history, baseline selection or fail-on-regression flags.
 - Bandit support is prototype-only.

examples/rag-environment-episode/README.md

@@ -0,0 +1,19 @@
+# RAG Environment Episode
+
+This example models a deterministic RAG workflow as a lightweight environment episode.
+
+The scripted policy chooses:
+
+```txt
+search -> rerank -> verify -> answer
+```
+
+Each action returns a reward. The completed episode is converted into a trajectory, summarized, exported as Markdown and converted into offline policy evaluation records.
+
+## Run
+
+```bash
+bun run --filter './examples/rag-environment-episode' dev
+```
+
+The example is fully mocked. It does not call LLM providers, vector databases or external tools.

examples/rag-environment-episode/package.json

@@ -0,0 +1,13 @@
+{
+  "name": "rag-environment-episode",
+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "bun src/index.ts"
+  },
+  "dependencies": {
+    "@ignitionai/agent-trainer-environment": "workspace:*",
+    "@ignitionai/agent-trainer-rl": "workspace:*"
+  }
+}