diff --git a/docs/ALPHA_READINESS.md b/docs/ALPHA_READINESS.md index df03060..d436ecc 100644 --- a/docs/ALPHA_READINESS.md +++ b/docs/ALPHA_READINESS.md @@ -28,7 +28,7 @@ All packages declare `license: MIT`, matching the root `LICENSE` file. | `@ignitionai/agent-trainer-adapter-mastra` | ready | ready | ready | ready | partial | Structural adapter only; no memory, tool or full Mastra coverage. | | `@ignitionai/agent-trainer-adapter-vercel-ai` | ready | ready | ready | ready | partial | Structural adapter only; no streaming, tools or live provider calls. | | `@ignitionai/agent-trainer-cli` | ready | ready | ready | ready | partial | Runs typed experiments and writes standalone reports or timestamped bundles; no history/baseline/regression flags yet. | -| `@ignitionai/agent-trainer-core` | ready | ready | missing | ready | partial | Foundational types and helpers need dedicated tests before alpha-stable status. | +| `@ignitionai/agent-trainer-core` | ready | ready | ready | ready | partial | Foundational helpers have dedicated tests; runtime schema validation remains outside the current helper surface. | | `@ignitionai/agent-trainer-environment` | ready | ready | ready | ready | partial | Tested episode runner with safety guards and a deterministic RAG episode example; no production runtime or optimization loop. | | `@ignitionai/agent-trainer-evals` | ready | ready | ready | ready | partial | Current rewards are tested; RAG presets and richer scoring are still missing. | | `@ignitionai/agent-trainer-experiments` | ready | ready | ready | ready | ready | Local runner, definitions, gates and JSONL history are tested and documented. | @@ -38,7 +38,7 @@ All packages declare `license: MIT`, matching the root `LICENSE` file. | `@ignitionai/agent-trainer-rl` | ready | ready | partial | ready | prototype | Deterministic policy helpers, trajectory recorder, fixed-strategy bandits, offline policy evaluation, GRPO-style selection and PPO interface skeletons are tested; deeper RL is intentionally absent. | | `@ignitionai/agent-trainer` | ready | ready | ready | ready | ready | Deterministic recommendation, candidate evaluation and grid search are tested. | -## Verified Through PR #38 +## Verified Through PR #44 - Package names use the `@ignitionai/*` scope. - Root and workspace package versions are aligned on `0.1.0-alpha.0`. @@ -51,10 +51,10 @@ All packages declare `license: MIT`, matching the root `LICENSE` file. - `bun install`, `bun run lint`, `bun run typecheck`, `bun run test` and `bun run build` pass locally. - The alpha dogfood experiment runs locally and through the CLI. - The alpha dogfood regression gate passes against its committed baseline. +- `@ignitionai/agent-trainer-core` has dedicated package-level tests for dataset helpers, adapter helpers and score helpers. ## Known Work After Internal Alpha -- Add dedicated tests for `@ignitionai/agent-trainer-core`. - Add CLI history/baseline/regression ergonomics after report bundles and CI examples. - Decide package publishing policy before any npm publication. - Add deeper examples for ecosystem adapters. diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 9031334..099db0f 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -2666,7 +2666,7 @@ Next PR: Status: -- current +- completed Branch: @@ -2724,6 +2724,67 @@ Definition of done: - example documents `search -> rerank -> verify -> answer`, - docs state that PPO and training remain out of scope. +Next PR: + +- PR #44 - `test: add core package coverage` + +### PR #44 - `test: add core package coverage` + +Status: + +- current + +Branch: + +```txt +test/core-package-coverage +``` + +Goal: + +Add dedicated package-level tests for `@ignitionai/agent-trainer-core` before relying on the public alpha surface in dogfood work. + +Scope: + +- test dataset creation and validation through public exports, +- test agent input conversion and mock adapter behavior through public exports, +- test run-result normalization, +- test score clamping and weighted averages, +- update readiness and audit docs. + +Out of scope: + +- public API changes, +- runtime schema validation, +- report serialization, +- provider calls, +- CLI changes, +- IgnitionRAG integration. + +Required APIs / files: + +- `packages/core/src/index.test.ts`, +- `packages/core/README.md`, +- readiness and audit docs. + +Acceptance: + +```bash +bun test packages/core/src/index.test.ts +bun run lint +bun run typecheck +bun run test +bun run build +bun run pack:check +``` + +Definition of done: + +- core helpers are covered through the public barrel export, +- no implementation detail mocks are introduced, +- package readiness no longer says core tests are missing, +- docs keep runtime schema validation out of scope. + ## Dogfood phase - IgnitionRAG Next work should happen in IgnitionRAG, not by adding more framework abstractions here. diff --git a/docs/MILESTONES.md b/docs/MILESTONES.md index 778a20a..5e40be4 100644 --- a/docs/MILESTONES.md +++ b/docs/MILESTONES.md @@ -296,13 +296,13 @@ docs/ALPHA_VALIDATION_PLAN.md Current status: ```txt -PR #36 through PR #42 are complete. The internal v0.1.0-alpha.0 tag and npm v0.1.0-alpha.1 publication are complete. +PR #36 through PR #43 are complete. The internal v0.1.0-alpha.0 tag, npm v0.1.0-alpha.1 publication, and tested environment episode foundation are complete. ``` Next phase: ```txt -Dogfood the alpha inside IgnitionRAG, then return to this repository for concrete framework gaps such as tested environment episodes, trajectory reporting and policy-evaluation ergonomics. +Close the remaining core test coverage gap, then dogfood the alpha inside IgnitionRAG and return to this repository only for concrete framework gaps found during use. ``` Explicit non-goals: diff --git a/docs/PROJECT_AUDIT.md b/docs/PROJECT_AUDIT.md index 592e9ba..d223620 100644 --- a/docs/PROJECT_AUDIT.md +++ b/docs/PROJECT_AUDIT.md @@ -87,9 +87,9 @@ If a package exists but is intentionally narrow, minimal or untested, it is part - Purpose: shared dataset, adapter, trace, reward, run and experiment result types plus small helpers. - Main exports: `createDataset`, `createMockAdapter`, `normalizeRunResult`, `toAgentInput`, `clampScore`, `weightedAverage`, shared interfaces. - Stability level: partial. -- Tests present: no dedicated package tests. +- Tests present: yes. - Example present: yes, used by all current examples. -- Known limitations: no dedicated package tests, no schema serialization helpers beyond TypeScript types. +- Known limitations: no schema serialization helpers beyond TypeScript types. ### `@ignitionai/agent-trainer-environment` diff --git a/packages/core/README.md b/packages/core/README.md index 38e1616..389555b 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -25,11 +25,10 @@ Main exports: ## Alpha Readiness Status -This package is foundational but still partial for alpha readiness. +This package is foundational and covered by dedicated package-level tests for the current helper surface. Known gaps: -- no dedicated package-level tests yet, - no runtime schema validation for serialized reports, - no compatibility policy beyond the current monorepo usage. diff --git a/packages/core/src/index.test.ts b/packages/core/src/index.test.ts new file mode 100644 index 0000000..20e2feb --- /dev/null +++ b/packages/core/src/index.test.ts @@ -0,0 +1,178 @@ +import { describe, expect, it } from "vitest"; +import { + assertDatasetItem, + clampScore, + createDataset, + createMockAdapter, + normalizeRunResult, + toAgentInput, + weightedAverage, +} from "./index"; + +describe("@ignitionai/agent-trainer-core", () => { + describe("datasets", () => { + it("creates a default named dataset from items and preserves item fields", () => { + const dataset = createDataset([ + { + id: "case-1", + input: "Summarize the contract.", + expected: { contains: ["termination"] }, + metadata: { source: "contract.pdf" }, + }, + ]); + + expect(dataset).toEqual({ + name: "dataset", + items: [ + { + id: "case-1", + input: "Summarize the contract.", + expected: { contains: ["termination"] }, + metadata: { source: "contract.pdf" }, + }, + ], + }); + }); + + it("accepts a named dataset object with metadata", () => { + const dataset = createDataset({ + name: "rag-eval", + description: "RAG evaluation set", + metadata: { owner: "evals" }, + items: [{ id: "case-1", input: "Find the citation." }], + }); + + expect(dataset.name).toBe("rag-eval"); + expect(dataset.description).toBe("RAG evaluation set"); + expect(dataset.metadata).toEqual({ owner: "evals" }); + }); + + it("rejects unsafe dataset definitions", () => { + expect(() => createDataset({ name: " ", items: [] })).toThrow("Dataset name is required."); + expect(() => + createDataset([ + { id: "case-1", input: "A" }, + { id: "case-1", input: "B" }, + ]), + ).toThrow("Duplicate dataset item id: case-1"); + expect(() => assertDatasetItem({ id: " ", input: "A" })).toThrow( + "Dataset item id is required.", + ); + expect(() => assertDatasetItem({ id: "case-1", input: " " })).toThrow( + "Dataset item case-1 input is required.", + ); + }); + }); + + describe("agent adapters", () => { + it("converts dataset items into agent inputs without dropping expected output or metadata", () => { + const input = toAgentInput({ + id: "case-1", + input: "Answer with JSON.", + expected: { json: { ok: true } }, + metadata: { priority: "high" }, + }); + + expect(input).toEqual({ + id: "case-1", + input: "Answer with JSON.", + expected: { json: { ok: true } }, + metadata: { priority: "high" }, + }); + }); + + it("normalizes raw adapter outputs into run results with an empty trace", () => { + expect(normalizeRunResult("plain output")).toEqual({ + output: "plain output", + trace: { steps: [] }, + }); + }); + + it("normalizes structured adapter outputs while preserving trace, usage and metadata", () => { + const result = normalizeRunResult({ + output: { answer: "A" }, + trace: { + steps: [{ type: "decision", action: "answer", confidence: 0.9 }], + metadata: { policy: "scripted" }, + }, + usage: { totalTokens: 12 }, + metadata: { model: "mock" }, + }); + + expect(result).toEqual({ + output: { answer: "A" }, + trace: { + steps: [{ type: "decision", action: "answer", confidence: 0.9 }], + metadata: { policy: "scripted" }, + }, + usage: { totalTokens: 12 }, + metadata: { model: "mock" }, + }); + }); + + it("runs static mock adapters with fallback trace and default usage", async () => { + const adapter = createMockAdapter("answer", { + name: "static-agent", + trace: { steps: [{ type: "message", role: "assistant", content: "answer" }] }, + usage: { inputTokens: 5, totalTokens: 7 }, + }); + + const result = await adapter.run({ id: "case-1", input: "Question?" }, {}); + + expect(adapter.name).toBe("static-agent"); + expect(result).toEqual({ + output: "answer", + trace: { steps: [{ type: "message", role: "assistant", content: "answer" }] }, + usage: { inputTokens: 5, totalTokens: 7 }, + }); + }); + + it("runs functional mock adapters with caller input/context and preserves explicit traces", async () => { + const adapter = createMockAdapter( + (input, context) => ({ + output: `${context.variantId}:${input.input}`, + trace: { steps: [{ type: "decision", action: "answer", reason: "scripted" }] }, + usage: { outputTokens: 3 }, + }), + { + trace: { steps: [{ type: "message", role: "system", content: "fallback" }] }, + usage: { inputTokens: 4, outputTokens: 1 }, + }, + ); + + const result = await adapter.run( + { id: "case-1", input: "Question?" }, + { variantId: "agent-v1" }, + ); + + expect(result).toEqual({ + output: "agent-v1:Question?", + trace: { steps: [{ type: "decision", action: "answer", reason: "scripted" }] }, + usage: { inputTokens: 4, outputTokens: 3 }, + }); + }); + }); + + describe("scores", () => { + it("clamps invalid and out-of-range scores to the normalized reward range", () => { + expect(clampScore(Number.NaN)).toBe(0); + expect(clampScore(-0.25)).toBe(0); + expect(clampScore(0.75)).toBe(0.75); + expect(clampScore(1.5)).toBe(1); + }); + + it("computes a clamped weighted average and treats missing weights as one", () => { + const score = weightedAverage([ + { name: "quality", score: 1, weight: 2 }, + { name: "latency", score: 0.25 }, + { name: "cost", score: 2, weight: 1 }, + ]); + + expect(score).toBe(0.8125); + }); + + it("returns zero when total score weight is zero", () => { + expect(weightedAverage([{ name: "disabled", score: 1, weight: 0 }])).toBe(0); + }); + }); +});