From 466bb6ab8a2157ae026ef71d7ed96ffc740d863a Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Wed, 12 Aug 2026 19:19:46 -0300 Subject: [PATCH] docs: document implemented AgentOps features Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f4c30cd9-3ff3-4761-a01f-7b772df77f6e --- .../checklists/requirements.md | 38 +++++ specs/001-evaluation-configuration/spec.md | 142 ++++++++++++++++++ .../checklists/requirements.md | 38 +++++ specs/002-evaluation-execution/spec.md | 131 ++++++++++++++++ .../checklists/requirements.md | 38 +++++ .../spec.md | 134 +++++++++++++++++ .../checklists/requirements.md | 37 +++++ specs/004-cicd-workflow-generation/spec.md | 134 +++++++++++++++++ .../checklists/requirements.md | 37 +++++ specs/005-release-evidence/spec.md | 122 +++++++++++++++ .../checklists/requirements.md | 37 +++++ specs/006-doctor-readiness/spec.md | 132 ++++++++++++++++ .../checklists/requirements.md | 37 +++++ specs/007-read-only-cockpit/spec.md | 127 ++++++++++++++++ .../checklists/requirements.md | 37 +++++ .../spec.md | 134 +++++++++++++++++ .../checklists/requirements.md | 37 +++++ specs/009-trace-to-regression/spec.md | 126 ++++++++++++++++ 18 files changed, 1518 insertions(+) create mode 100644 specs/001-evaluation-configuration/checklists/requirements.md create mode 100644 specs/001-evaluation-configuration/spec.md create mode 100644 specs/002-evaluation-execution/checklists/requirements.md create mode 100644 specs/002-evaluation-execution/spec.md create mode 100644 specs/003-evaluation-results-and-regression/checklists/requirements.md create mode 100644 specs/003-evaluation-results-and-regression/spec.md create mode 100644 specs/004-cicd-workflow-generation/checklists/requirements.md create mode 100644 specs/004-cicd-workflow-generation/spec.md create mode 100644 specs/005-release-evidence/checklists/requirements.md create mode 100644 specs/005-release-evidence/spec.md create mode 100644 specs/006-doctor-readiness/checklists/requirements.md create mode 100644 specs/006-doctor-readiness/spec.md create mode 100644 specs/007-read-only-cockpit/checklists/requirements.md create mode 100644 specs/007-read-only-cockpit/spec.md create mode 100644 specs/008-foundry-operations-observability/checklists/requirements.md create mode 100644 specs/008-foundry-operations-observability/spec.md create mode 100644 specs/009-trace-to-regression/checklists/requirements.md create mode 100644 specs/009-trace-to-regression/spec.md diff --git a/specs/001-evaluation-configuration/checklists/requirements.md b/specs/001-evaluation-configuration/checklists/requirements.md new file mode 100644 index 0000000..c2a92c2 --- /dev/null +++ b/specs/001-evaluation-configuration/checklists/requirements.md @@ -0,0 +1,38 @@ +# Specification Quality Checklist: Evaluation Configuration + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/001-evaluation-configuration/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/core/agentops_config.py`, `config_loader.py`, and `evaluators.py`, and against `tests/unit/test_agentops_config.py` and `tests/unit/test_evaluators.py`, not against a future proposal. +- Revalidation pass: corrected the spec to match source exactly on three boundaries that were previously conflated - (1) configuration loading (`load_agentops_config`) validates only YAML/schema fields and never opens the dataset file; (2) dataset existence/emptiness is checked later by `detect_dataset_shape` during evaluation preparation, which raises on a missing or empty dataset; (3) a row's missing `input` field is caught only at per-row invocation time (`invocations.py`), not by config loading or dataset-shape detection. Also broadened the `execution: cloud` restriction from "Foundry `name:version` prompt agent only" to "Foundry prompt or Foundry hosted agent with a derivable name/version," and clarified that this restriction is enforced at evaluation-run time, not configuration-load time (only `execution: azd` is rejected by the config model itself). +- Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (classification correctness, rejection timing/ordering, evaluator-set stability, threshold isolation) instead of referring to automated test coverage. +- Future or unrelated capabilities (new evaluator types, multi-file configuration composition, execution/network behavior) are explicitly listed under "Out of Scope" in the spec and are not treated as gaps in this checklist. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/001-evaluation-configuration/spec.md b/specs/001-evaluation-configuration/spec.md new file mode 100644 index 0000000..88e20b2 --- /dev/null +++ b/specs/001-evaluation-configuration/spec.md @@ -0,0 +1,142 @@ +# Feature Specification: Evaluation Configuration + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing `agentops.yaml` configuration surface as-built, not a new proposal. Sources reviewed include `src/agentops/core/agentops_config.py`, `src/agentops/core/config_loader.py`, `src/agentops/core/evaluators.py`, `tests/unit/test_agentops_config.py`, `tests/unit/test_evaluators.py`, and `docs/concepts.md`. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Describe an agent and dataset in one flat file (Priority: P1) + +A release engineer wants to declare which agent to evaluate and which dataset to use for a Foundry prompt agent, a Foundry hosted endpoint, a generic HTTP/JSON agent, or a raw model deployment, using a single `agentops.yaml` file at the project root. + +**Why this priority**: Without a valid, loadable configuration there is nothing to evaluate. This is the entry point for every other AgentOps capability. + +**Independent Test**: Can be fully tested by writing a minimal `agentops.yaml` (`version`, `agent`, `dataset`) for each of the four supported agent value shapes and confirming the loader accepts it and correctly classifies the target kind, without executing any evaluation. + +**Acceptance Scenarios**: + +1. **Given** an `agentops.yaml` with `agent: "my-agent:3"`, **When** the configuration is loaded, **Then** the system classifies the target as a Foundry prompt agent with name `my-agent` and version `3`. +2. **Given** an `agentops.yaml` with `agent: "https://.services.ai.azure.com/api/projects//agents/..."`, **When** the configuration is loaded, **Then** the system classifies the target as a Foundry hosted agent reachable over REST. +3. **Given** an `agentops.yaml` with `agent: "https://api.example.com/chat"` and no Foundry-specific path shape, **When** the configuration is loaded, **Then** the system classifies the target as a generic HTTP/JSON agent. +4. **Given** an `agentops.yaml` with `agent: "model:gpt-4o"`, **When** the configuration is loaded, **Then** the system classifies the target as a raw model deployment. +5. **Given** a `dataset` path naming a file that has not yet been inspected, **When** the configuration is loaded, **Then** loading succeeds on the strength of the YAML fields alone (`version`, `agent`, `dataset` present and well-formed); whether that file exists, parses as JSONL, and contains usable rows is determined afterward, during evaluation preparation, not during configuration loading. + +--- + +### User Story 2 - Let the system infer evaluators and thresholds, or override them (Priority: P2) + +A release engineer wants sensible evaluators and pass/fail thresholds chosen automatically based on the agent type and dataset columns, but wants the ability to override the evaluator list or specific threshold expressions when the defaults do not fit. + +**Why this priority**: Automatic inference removes configuration burden for the common case; overrides are needed once teams have specific quality bars, so this is the second most critical capability after basic loading. + +**Independent Test**: Can be fully tested by loading configurations that vary only in dataset columns present (e.g., with/without `context`, with/without `tool_calls`) and confirming the evaluator set changes accordingly, then adding an explicit `evaluators:` list and confirming inference is bypassed, and adding a `thresholds:` map and confirming a user-specified expression overrides the default for that metric while other metrics keep their defaults. + +**Acceptance Scenarios**: + +1. **Given** a dataset with `input` and `expected` columns only, **When** evaluators are resolved, **Then** the system selects the default quality evaluator set for that agent type without requiring an explicit `evaluators:` entry. +2. **Given** a dataset that additionally has a `context` column, **When** evaluators are resolved, **Then** retrieval/grounding-oriented evaluators are added to the selected set. +3. **Given** a dataset that additionally has `tool_definitions` and `tool_calls` columns, **When** evaluators are resolved, **Then** agent-workflow evaluators (tool call accuracy, task completion, and related) are added to the selected set. +4. **Given** an `agentops.yaml` with an explicit `evaluators:` list, **When** evaluators are resolved, **Then** the explicit list is used verbatim and automatic inference is skipped. +5. **Given** an `agentops.yaml` with `thresholds: {coherence: ">=4"}` and no other threshold entries, **When** thresholds are resolved, **Then** the `coherence` threshold uses the user value `>=4` while every other metric keeps its inferred default. + +--- + +### User Story 3 - Configure protocol-specific and safety-relevant fields for URL-based agents (Priority: P3) + +A release engineer integrating a generic HTTP/JSON or Foundry hosted agent wants to configure how requests are shaped and how responses are parsed (protocol, request/response field paths, headers, authentication, streaming aggregation), and wants invalid configuration caught before any network call is made. + +**Why this priority**: This unlocks evaluation of the broadest class of custom agents (LangGraph, LangChain, ACA, AKS, custom REST) but is only needed once the basic agent/dataset declaration and evaluator inference already work, so it ranks below them. + +**Independent Test**: Can be fully tested by constructing configurations with different `protocol`, `response_mode`, `request_field`, `response_field`, `tool_calls_field`, `headers`, `auth_header_env`, `auth_header_name`, `auth_value_template`, and `stream` values, and verifying the loader accepts well-formed combinations and rejects malformed ones (e.g., unsupported protocol name) with a clear, actionable error, without invoking the agent endpoint. + +**Acceptance Scenarios**: + +1. **Given** a URL-based agent configuration with `protocol: http-json`, `request_field: message`, and `response_field: text`, **When** the configuration is loaded, **Then** it validates successfully and the resolved settings are available for the execution layer to consume. +2. **Given** a URL-based agent configuration with `response_mode: sse` and a `stream` block specifying `text_field` and `done_marker`, **When** the configuration is loaded, **Then** the streaming aggregation settings are accepted and preserved. +3. **Given** a configuration with an unsupported `protocol` value, **When** the configuration is loaded, **Then** loading fails with a validation error naming the offending field, and no evaluation run is attempted. +4. **Given** a configuration with `auth_header_env` set to a variable name, **When** the configuration is loaded, **Then** the loader accepts the field as a reference to be resolved at execution time and does not require the variable to be already set in the environment at load time. + +--- + +### Edge Cases + +- What happens when `agentops.yaml` is missing required fields (`version`, `agent`, or `dataset`)? Loading MUST fail with a validation error identifying the missing field(s) rather than raising an unhandled exception. +- What happens when `version` is present but not the supported value `1`? Loading MUST fail with a clear, actionable error rather than silently coercing the schema. +- How does the system handle a `dataset` path that does not exist on disk? Configuration loading itself does not open or inspect the dataset file, so a missing file does not fail at load time; the missing-file condition is caught during the later evaluation-preparation step (dataset shape detection), which fails with a file-not-found style error referencing the configured path before any row is invoked. +- How does the system handle a JSONL dataset row that is missing the required `input` field? Neither configuration loading nor evaluation-preparation's dataset-shape detection reads every row for an `input` field; the missing field is caught when that specific row is invoked, which fails that row while the run continues with the remaining rows. +- What happens when both an explicit `evaluators:` override and dataset columns that would normally trigger additional evaluators are both present? The explicit override MUST win; inference MUST NOT silently add evaluators on top of the override. +- What happens when `execution: cloud` is combined with an `agent:` value that is not a Foundry agent (prompt or hosted)? Configuration loading does not reject this combination by itself; the restriction is enforced when the evaluation run starts, which fails with a clear error before the target or any evaluator is invoked, and before any Azure/Foundry call is made. +- What happens when both `project_endpoint` in `agentops.yaml` and the `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT` environment variable are set to different values? The `agentops.yaml` value MUST take precedence. +- How does the system handle a `thresholds:` map containing a metric name that does not correspond to any selected evaluator? The unused threshold entry MUST NOT crash configuration loading; unmatched threshold keys are simply not applied to any evaluator. +- How does the system handle a dataset file that exists but contains zero usable rows? Evaluation preparation MUST reject it with a clear "dataset is empty" error before any row is invoked; a zero-row run is not a supported outcome. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST load a single flat `agentops.yaml` file at the project root as the sole source of evaluation configuration, with no separate bundle, scenario, or per-dataset configuration files required. +- **FR-002**: The system MUST require `version`, `agent`, and `dataset` fields, and MUST reject configurations missing any of these with a validation error identifying the missing field. +- **FR-003**: The system MUST classify the `agent` field into one of four target kinds based on its value shape: Foundry prompt agent (`name:version`), Foundry hosted agent (a Foundry-shaped `https://` URL), generic HTTP/JSON agent (any other `https://` URL), or raw model deployment (`model:`). +- **FR-004**: The system MUST record the `dataset` field as a file path at configuration-load time without opening, parsing, or validating the referenced file's existence, JSONL structure, or per-row contents; those checks happen later, during evaluation preparation and per-row invocation, not during configuration loading. +- **FR-005**: The system MUST recognize optional dataset columns (`expected`, `context`, `tool_definitions`, `tool_calls`) and use their presence to drive automatic evaluator selection. +- **FR-006**: The system MUST automatically select an evaluator set based on the classified agent target kind and the dataset columns present, without requiring the user to enumerate evaluators. +- **FR-007**: The system MUST allow an explicit `evaluators:` list in `agentops.yaml` to fully override automatic evaluator selection for that run. +- **FR-008**: The system MUST provide default threshold expressions for every selected evaluator metric and MUST allow a user-supplied `thresholds:` map to override the default expression for any subset of metrics while leaving the rest at their defaults. +- **FR-009**: The system MUST support an `execution` field with values `local` (the default), `cloud`, `azd`, and `auto`. At configuration-load time, the system MUST reject `execution: azd` when the classified agent target is not a Foundry prompt or Foundry hosted agent. The system MUST also restrict `execution: cloud` to Foundry prompt or Foundry hosted agent targets with a derivable name and version, but this restriction is enforced when the evaluation run starts rather than at configuration-load time. +- **FR-010**: The system MUST support an `eval_recipe` field usable when `execution: azd`, and MUST be able to proceed without it by auto-discovering a single eval recipe when exactly one is present. +- **FR-011**: The system MUST support a `publish` boolean field that, combined with `execution: local`, controls whether results are additionally uploaded to the Classic Foundry Evaluations panel. +- **FR-012**: The system MUST support a `project_endpoint` field in `agentops.yaml` that takes precedence over the `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT` environment variable when both are set. +- **FR-013**: The system MUST support URL-based agent configuration fields `protocol` (`responses`, `invocations`, `http-json`), `request_field`, `response_field`, `tool_calls_field`, `headers`, `auth_header_env`, `auth_header_name`, and `auth_value_template`, applying documented defaults when any is omitted. +- **FR-014**: The system MUST support a `response_mode` field (`json`, `sse`, `text`) and, when `response_mode` is `sse` or `text`, MUST accept a `stream` block describing streaming aggregation (`text_field`, `done_marker`, `strip_leading_token`). +- **FR-015**: The system MUST reject an unsupported `protocol` or `response_mode` value at configuration-load time with a validation error naming the offending field, before any network call is attempted. +- **FR-016**: The system MUST support optional evidence-reference fields `assert_path`, `acs_path`, and `redteam_path` that record paths to external policy/results artifacts without executing those external tools itself. +- **FR-017**: The system MUST treat configuration validation as a purely local, offline operation that does not require network access or Azure credentials to succeed or fail. + +### Key Entities + +- **AgentOpsConfig**: The root configuration object parsed from `agentops.yaml`, holding `version`, `agent`, `dataset`, `thresholds`, `evaluators`, `project_endpoint`, `execution`, `eval_recipe`, `publish`, protocol/streaming fields, and evidence-reference path fields. +- **TargetResolution**: The classified interpretation of the `agent` field, capturing the resolved kind (Foundry prompt agent, Foundry hosted agent, generic HTTP/JSON agent, or model deployment) plus any parsed name/version/url/deployment components. +- **Dataset Row**: A single JSONL record consumed at invocation time; it must carry an `input` field to be invoked successfully, and may carry optional `expected`, `context`, `tool_definitions`, and `tool_calls` fields that influence evaluator inference during evaluation preparation. +- **Evaluator Preset**: A named, cataloged evaluator definition (score key, input mapping, default threshold expression, applicable agent-kind/category) used both for automatic inference and for validating explicit `evaluators:` overrides. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: 100% of the four supported `agent` value shapes (Foundry prompt, Foundry hosted, generic HTTP/JSON, raw model deployment) resolve to the correct, single target kind from `version` + `agent` + `dataset` alone, with no additional configuration required. +- **SC-002**: Every documented configuration rejection condition (a missing required field, an unsupported `version`, an unsupported `protocol`, an unsupported `response_mode`, `execution: azd` on a non-Foundry target) is reported before any dataset row is read or any network call is made, and each rejection names the offending field. +- **SC-003**: For the same agent target, configuration fields, and dataset column shape, the automatically selected evaluator set and default thresholds are stable across repeated evaluation preparation; changing the agent kind or relevant dataset columns may change selection. +- **SC-004**: When an explicit `evaluators:` list is present, the resolved evaluator set is always exactly that list - never the list plus any evaluator that column-based inference would have added. +- **SC-005**: A user-supplied threshold for one metric changes the resolved threshold for that metric only; every other metric's resolved threshold is unchanged from its documented default. +- **SC-006**: `execution: cloud` is rejected for every target kind other than a Foundry prompt or Foundry hosted agent, and accepted for both of those kinds when a name and version can be derived; the rejection surfaces before the agent or any evaluator runs. + +## Assumptions + +- Teams have exactly one `agentops.yaml` per evaluated project/workspace; multi-agent or multi-dataset scenarios are handled by maintaining separate project directories rather than a single richer schema. +- The evaluator catalog and its default thresholds are maintained inside AgentOps and are considered part of this baseline; changes to individual evaluator defaults are out of scope for this specification. +- Users are expected to run `agentops init` or otherwise scaffold the workspace before hand-editing `agentops.yaml`, so this specification assumes a syntactically valid YAML file is the starting point for validation. +- Configuration loading validates only the YAML/schema fields themselves; dataset existence, JSONL well-formedness, row count, and per-row field presence are validated later, during evaluation preparation and per-row invocation (see the Evaluation Execution specification), not during configuration loading. +- Network/Azure credential validity for `project_endpoint`, `auth_header_env`, and similar fields is checked at execution time (see the Evaluation Execution specification), not at configuration-load time. + +## Out of Scope + +- Actually invoking the agent or running evaluators against it (covered by the Evaluation Execution specification). +- Interpreting or executing `assert_path`, `acs_path`, or `redteam_path` contents; this specification only covers accepting and recording these paths. +- Adding new evaluator types or changing default threshold values for existing evaluators. +- Multi-file or hierarchical configuration composition; the schema is intentionally a single flat file. +- CI/CD workflow generation, release evidence, and Doctor readiness checks, which consume this configuration but are specified separately. + +## Implementation Evidence + +- `src/agentops/core/agentops_config.py` - Pydantic model for the flat `agentops.yaml` v1 schema, including `version`, `agent`, `dataset`, `thresholds`, `evaluators`, `project_endpoint`, `execution`, `eval_recipe`, `publish`, `protocol`, `request_field`, `response_field`, `tool_calls_field`, `headers`, `auth_header_env`, `auth_header_name`, `auth_value_template`, `response_mode`, `stream`, `assert_path`, `acs_path`, `redteam_path`, and the agent-target classification logic. +- `src/agentops/core/config_loader.py` - loads and validates `agentops.yaml` into the config model, raising validation errors for malformed configuration. +- `src/agentops/core/evaluators.py` - dataset shape detection, evaluator catalog, automatic evaluator selection, and threshold default/override merging. +- `tests/unit/test_agentops_config.py` - unit tests for schema parsing, agent-kind classification, and field validation. +- `tests/unit/test_evaluators.py` - unit tests for evaluator inference by agent kind and dataset shape, explicit `evaluators:` override behavior, and threshold merging. +- `tests/unit/test_agentops_config_identity.py` - additional configuration/identity-adjacent field validation coverage. +- `docs/concepts.md` and `docs/how-it-works.md` - narrative documentation of the configuration model and evaluation scenario matrix consistent with this baseline. diff --git a/specs/002-evaluation-execution/checklists/requirements.md b/specs/002-evaluation-execution/checklists/requirements.md new file mode 100644 index 0000000..e2597bd --- /dev/null +++ b/specs/002-evaluation-execution/checklists/requirements.md @@ -0,0 +1,38 @@ +# Specification Quality Checklist: Evaluation Execution + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/002-evaluation-execution/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/pipeline/orchestrator.py`, `runtime.py`, `invocations.py`, `cloud_runner.py`, `cloud_results.py`, `publisher.py`, `azd_runner.py`, and `src/agentops/core/azd_eval.py`, and against `tests/unit/test_invocations.py`, `test_cloud_runner.py`, `test_pipeline_publisher.py`, `test_azd_runner.py`, and `tests/integration/test_pipeline_smoke.py`, not against a future proposal. +- Revalidation pass: corrected two factual errors against `orchestrator.py`. First, `detect_dataset_shape()` is called unconditionally before both local and cloud execution and raises on an empty dataset, so a zero-row dataset is rejected before any row is invoked - it does not complete with "zero rows evaluated" as the spec previously claimed. Second, `_run_evaluation_cloud()` accepts both `foundry_prompt` (`name:version`) and `foundry_hosted` (URL with a derivable name/version) target kinds, not only the `name:version` form, so the cloud-execution restriction was broadened accordingly. +- Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (result-shape equivalence, rejection ordering, isolation guarantees, artifact durability) instead of referring to automated test coverage. +- Future or unrelated capabilities (a standalone `agentops eval compare` command, results/regression schema details, and CI/CD or Doctor consumption of results) are explicitly listed under "Out of Scope" and are not treated as gaps in this checklist. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/002-evaluation-execution/spec.md b/specs/002-evaluation-execution/spec.md new file mode 100644 index 0000000..22f5a2d --- /dev/null +++ b/specs/002-evaluation-execution/spec.md @@ -0,0 +1,131 @@ +# Feature Specification: Evaluation Execution + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing evaluation-run execution behavior as-built, not a new proposal. Sources reviewed include `src/agentops/pipeline/orchestrator.py`, `runtime.py`, `invocations.py`, `cloud_runner.py`, `cloud_results.py`, `publisher.py`, `azd_runner.py`, and their associated unit/integration tests. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Run an evaluation locally against any agent type (Priority: P1) + +A release engineer runs `agentops eval run` against a Foundry hosted agent, a generic HTTP/JSON agent, or a raw model deployment, and gets a normalized run of the agent invoked once per dataset row with local evaluators scoring each response. + +**Why this priority**: Local execution is the default and most universally applicable execution path; it works for every agent kind and requires no cloud evaluation service, so it is the foundational capability the rest of execution builds on. + +**Independent Test**: Can be fully tested end-to-end by pointing `agentops eval run` at a small local HTTP echo agent and a JSONL dataset, then confirming each row is invoked exactly once, evaluators run against the response, and a normalized result set is produced without any cloud evaluation dependency. + +**Acceptance Scenarios**: + +1. **Given** a valid `agentops.yaml` with `execution: local` (or the field omitted, since local is the default) and a reachable HTTP/JSON agent, **When** `agentops eval run` is executed, **Then** the agent is invoked once per dataset row and each response is scored by the evaluators selected for that configuration. +2. **Given** the same setup but the agent target is a raw model deployment (`model:`), **When** `agentops eval run` is executed, **Then** the model deployment is called directly per row and evaluated the same way as any other local target. +3. **Given** a URL-based agent configured with `response_mode: sse` and a `stream` aggregation block, **When** `agentops eval run` is executed, **Then** the streamed response is reassembled into a single response text before being scored. +4. **Given** an agent invocation that fails for one dataset row (for example, a network error), **When** the run completes, **Then** the failure is captured and attributed to that row rather than silently aborting the entire run or crashing without diagnostic information. + +--- + +### User Story 2 - Run a Foundry prompt agent through the cloud evaluation service (Priority: P2) + +A release engineer with a Foundry prompt agent (`name:version`) wants the dataset submitted to the OpenAI Evals API through Foundry so the agent and evaluators run server-side, with results normalized back into the same result shape as a local run. + +**Why this priority**: Cloud evaluation offloads compute and centralizes evaluation history in Foundry, which matters once teams adopt Foundry prompt agents, but it is a narrower, agent-kind-specific path layered on top of the local execution foundation. + +**Independent Test**: Can be fully tested by configuring `execution: cloud` with a Foundry prompt agent target, running `agentops eval run`, and confirming the dataset is synced, a cloud run/eval identifier is returned, and the resulting normalized output items match the same `RunResult` shape produced by local execution. + +**Acceptance Scenarios**: + +1. **Given** an `agentops.yaml` with `execution: cloud` and `agent: "my-agent:3"`, **When** `agentops eval run` is executed, **Then** the dataset is synced to the cloud evaluation service and a cloud evaluation/run identifier is captured. +2. **Given** a completed cloud run, **When** results are retrieved, **Then** cloud output items are normalized into the same row/metric structure used by local runs, so downstream reporting does not need to distinguish execution mode. +3. **Given** `execution: cloud` is combined with an agent target that is not a Foundry prompt or Foundry hosted agent, **When** `agentops eval run` is executed, **Then** the run is rejected before any cloud submission occurs. + +--- + +### User Story 3 - Delegate execution to azd and normalize its output (Priority: P3) + +A team already using Azure Developer CLI (`azd`) evaluation recipes wants `agentops eval run` to delegate execution to `azd ai agent eval` and fold the emitted metrics into the standard AgentOps result contract. + +**Why this priority**: This path serves teams with an existing azd-centric workflow; it is valuable but narrower in audience than local or cloud execution, so it is prioritized last among the three execution modes. + +**Independent Test**: Can be fully tested by configuring `execution: azd` with a discoverable `eval.yaml` recipe (or an explicit `eval_recipe` path), running `agentops eval run`, and confirming the azd command is invoked and its emitted metrics are normalized into the standard result schema. + +**Acceptance Scenarios**: + +1. **Given** `execution: azd` and exactly one discoverable `eval.yaml` recipe in the workspace, **When** `agentops eval run` is executed, **Then** the recipe is auto-discovered without requiring an explicit `eval_recipe` path. +2. **Given** `execution: azd` with an explicit `eval_recipe` path, **When** `agentops eval run` is executed, **Then** that recipe is used instead of auto-discovery. +3. **Given** an azd delegated run completes, **When** results are retrieved, **Then** the emitted azd metrics are normalized into the same result schema produced by local and cloud execution. + +--- + +### Edge Cases + +- What happens when the configured agent endpoint is unreachable for the entire run (not just one row)? The run MUST surface a runtime error distinct from a threshold failure, rather than reporting a false pass or false threshold-fail. +- How does the system handle a dataset with zero rows? The run MUST reject it before any row is invoked, with a clear "dataset is empty" error; a zero-row run is not treated as a completed pass. +- What happens when `publish: true` is set but the local run's authentication to Foundry is invalid? Publishing MUST fail without discarding or corrupting the already-computed local `results.json`/`report.md`. +- What happens when `execution: cloud` is used and the cloud service returns a subset of the submitted rows (e.g., partial completion)? The normalization step MUST reflect the actual number of returned rows rather than assuming full completion. +- How does the system behave when both `--baseline` (see Evaluation Results and Regression) and `execution: cloud` are used together? The comparison MUST operate on the normalized result shape identically regardless of which execution mode produced the current run. +- What happens when an `eval_recipe` path is given for `execution: azd` but more than one recipe exists and no explicit path narrows the choice unambiguously? Auto-discovery MUST only apply when exactly one recipe is discoverable; otherwise the run must not guess. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST dispatch each evaluation run to exactly one of three execution modes -- local, cloud, or azd -- based on the `execution` field resolved from the loaded configuration, defaulting to local when the field is omitted. +- **FR-002**: For local execution, the system MUST invoke the target agent once per dataset row and run the selected evaluators against each response, regardless of whether the target is a Foundry hosted agent, a generic HTTP/JSON agent, or a raw model deployment. +- **FR-003**: For local execution against URL-based agents, the system MUST support the `responses`, `invocations`, and `http-json` protocols and the `json`, `sse`, and `text` response modes, reassembling streamed responses per the configured `stream` aggregation settings before evaluation. +- **FR-004**: For cloud execution, the system MUST only be used with a Foundry prompt (`name:version`) or Foundry hosted agent target with a derivable name and version, MUST sync the dataset to the cloud evaluation service, and MUST capture a cloud run/evaluation identifier for the submitted run. +- **FR-005**: For cloud execution, the system MUST normalize the cloud service's returned output items into the same row/metric result structure produced by local execution. +- **FR-006**: For azd execution, the system MUST delegate to the `azd ai agent eval` command, MUST auto-discover a single `eval.yaml` recipe when `eval_recipe` is not explicitly set and exactly one recipe is discoverable, and MUST normalize the emitted azd metrics into the standard result schema. +- **FR-007**: The system MUST attribute a per-row invocation failure to that specific row's result rather than aborting the entire run, when the failure is isolated to one row. +- **FR-008**: The system MUST persist run artifacts under a timestamped directory within the workspace results area and MUST maintain a pointer to the most recent run's artifacts for downstream tools to consume without knowing the timestamp. +- **FR-009**: The system MUST support an optional publishing step, controlled by the `publish` configuration field for local execution (and implicitly for cloud execution), that uploads results to the appropriate Foundry Evaluations panel without altering the already-persisted local result artifacts if publishing fails. +- **FR-010**: The system MUST reject a dataset with zero rows before invoking any target, for both local and cloud execution, with a clear error rather than completing a run that reports zero rows evaluated. +- **FR-011**: The system MUST keep the resulting run's row/metric schema identical across local, cloud, and azd execution modes so downstream reporting and comparison logic do not need to branch on execution mode. + +### Key Entities + +- **Execution Mode**: One of `local`, `cloud`, or `azd`, selected from configuration and determining which invocation/evaluation path a run takes. +- **Row Invocation**: The act of calling the target agent (or model deployment) for a single dataset row and capturing its raw response (including streamed text reassembly when applicable). +- **Cloud Run/Evaluation Reference**: The identifier(s) returned by the cloud evaluation service (Foundry via the OpenAI Evals API) that allow a submitted run to be tracked and its output items retrieved. +- **AZD Eval Recipe**: A discoverable or explicitly referenced `eval.yaml`-style recipe consumed by `azd ai agent eval` when `execution: azd` is used. +- **Publish Target**: The Foundry Evaluations panel (Classic or New, depending on execution mode and the `publish` field) that an already-computed run's results may optionally be uploaded to. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Each of the three execution modes (local, cloud, azd) produces a completed run whose row/metric result structure is identical in shape and field names, independent of which mode produced it. +- **SC-002**: 100% of supported local agent kinds (Foundry hosted, generic HTTP/JSON, model deployment) across every supported protocol/response-mode combination are invoked and scored using the same row/metric result shape, with no kind- or protocol-specific fields required by downstream reporting. +- **SC-003**: A single failing row invocation never prevents the remaining rows in the same dataset from being invoked and evaluated; every non-failing row in that run still produces a result. +- **SC-004**: A dataset with zero rows is always rejected with a clear error before any row is invoked, for both local and cloud execution; it never produces a completed run and never reports a false pass. +- **SC-005**: A failed publish step never corrupts or removes the already-persisted local `results.json`/`report.md` for that run; the local artifacts remain exactly as computed before the publish attempt. +- **SC-006**: `execution: cloud` succeeds only for a Foundry prompt or Foundry hosted agent target with a derivable name and version; every other target kind is rejected before any cloud submission. + +## Assumptions + +- The target agent endpoint (for local execution) is reachable over the network the AgentOps process runs on; network policy and connectivity are the operator's responsibility. +- Cloud execution requires valid Foundry/Azure credentials configured in the environment (see the Evaluation Configuration specification for `project_endpoint` resolution); this specification assumes those credentials, when required, are already available. +- The azd execution path assumes the Azure Developer CLI is installed and its `ai agent eval` capability is available in the environment; installing or configuring azd itself is out of scope. +- Exactly one dataset drives a single evaluation run; concurrent multi-dataset runs from a single `agentops eval run` invocation are not part of this baseline. + +## Out of Scope + +- The content and structure of the configuration file that selects execution mode (see the Evaluation Configuration specification). +- The schema, thresholds, exit codes, and reporting of the produced results (see the Evaluation Results and Regression specification). +- A standalone `agentops eval compare` command; it is not implemented. Regression comparison during execution is limited to the `--baseline` flag on `agentops eval run`, covered by the Evaluation Results and Regression specification. +- Doctor readiness analysis, release evidence composition, and CI/CD workflow generation, which consume execution results but are specified separately. +- Installing, configuring, or authenticating the Azure Developer CLI (`azd`) itself. + +## Implementation Evidence + +- `src/agentops/pipeline/orchestrator.py` - end-to-end `eval run` orchestration, including execution-mode dispatch (local/cloud/azd), dataset resolution, and the `exit_code_from()` translation of a run's outcome. +- `src/agentops/pipeline/runtime.py` - local per-row invocation and evaluator execution engine, including evaluator loading and model configuration binding. +- `src/agentops/pipeline/invocations.py` - protocol (`responses`, `invocations`, `http-json`) and response-mode (`json`, `sse`, `text`) handling for URL-based agents, including streaming aggregation. +- `src/agentops/pipeline/cloud_runner.py` and `src/agentops/pipeline/cloud_results.py` - cloud dataset submission via the OpenAI Evals API and normalization of cloud output items back into the standard result schema. +- `src/agentops/pipeline/publisher.py` - Classic/New Foundry Evaluations publishing logic gated by the `publish` configuration field and execution mode. +- `src/agentops/pipeline/azd_runner.py` and `src/agentops/core/azd_eval.py` - azd delegation, `eval.yaml` recipe discovery, and azd metric normalization. +- `tests/unit/test_invocations.py`, `test_cloud_runner.py`, `test_cloud_results.py`, `test_pipeline_publisher.py`, `test_azd_runner.py`, `test_azd_eval.py`, `test_azd_eval_init.py`, `test_runtime_conversation.py`, `test_runtime_dataset_response_source.py`, `test_runtime_model_config.py`, `test_runtime_response_fields.py` - unit coverage of the corresponding execution behaviors above. +- `tests/integration/test_pipeline_smoke.py` (`test_http_pipeline_end_to_end`, `test_http_pipeline_with_baseline`) - end-to-end local HTTP agent execution coverage. +- `docs/how-it-works.md`, `docs/tutorial-hosted-agent.md`, `docs/tutorial-prompt-agent.md` - narrative documentation of the local/cloud/azd execution paths consistent with this baseline. diff --git a/specs/003-evaluation-results-and-regression/checklists/requirements.md b/specs/003-evaluation-results-and-regression/checklists/requirements.md new file mode 100644 index 0000000..ef55fa6 --- /dev/null +++ b/specs/003-evaluation-results-and-regression/checklists/requirements.md @@ -0,0 +1,38 @@ +# Specification Quality Checklist: Evaluation Results and Regression + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/003-evaluation-results-and-regression/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/core/results.py`, `src/agentops/pipeline/thresholds.py`, `comparison.py`, `reporter.py`, and `orchestrator.py`, and against `tests/unit/test_cli_commands.py::test_eval_help_does_not_expose_compare_subcommand`, `tests/unit/test_pipeline_reporter.py`, and `tests/integration/test_pipeline_smoke.py`, not against a future proposal. +- The explicit absence of an `agentops eval compare` subcommand is documented as a factual finding (confirmed by an existing test), not a gap; future addition of such a command is out of scope for this baseline. +- Revalidation pass: corrected the zero-row dataset claim against `orchestrator.py`, which calls `detect_dataset_shape()` unconditionally before any row is invoked; `detect_dataset_shape()` raises on an empty dataset, so a zero-row run is rejected before execution and never reaches the point of writing a `results.json` - it does not "still write a valid `results.json` reflecting zero evaluated rows" as the spec previously claimed. +- Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (structural equivalence, exit-code distinctness, reproducibility without network access, comparison completeness) instead of referring to automated test coverage. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/003-evaluation-results-and-regression/spec.md b/specs/003-evaluation-results-and-regression/spec.md new file mode 100644 index 0000000..bd3438b --- /dev/null +++ b/specs/003-evaluation-results-and-regression/spec.md @@ -0,0 +1,134 @@ +# Feature Specification: Evaluation Results and Regression + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing results, reporting, and baseline-comparison behavior as-built, not a new proposal. Sources reviewed include `src/agentops/core/results.py`, `src/agentops/pipeline/thresholds.py`, `comparison.py`, `reporter.py`, `orchestrator.py`, `src/agentops/cli/app.py` (`eval run`, `report generate`), and associated tests. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Get a normalized, machine-readable result and a human-readable report from every run (Priority: P1) + +A release engineer runs an evaluation and needs both a machine-readable artifact for automation (CI gating, dashboards) and a human-readable summary suitable for a pull request review, produced consistently regardless of which agent kind or execution mode was used. + +**Why this priority**: Every other capability in this feature (thresholds, exit codes, comparison, reporting) depends on there being a single, stable, normalized result shape to operate on. + +**Independent Test**: Can be fully tested by running any supported evaluation configuration and confirming a `results.json` and `report.md` are written to the run's output directory, with `results.json` containing run-level summary metrics, per-row metrics, and threshold outcomes, and `report.md` rendering the same information in Markdown. + +**Acceptance Scenarios**: + +1. **Given** a completed evaluation run, **When** the run finishes, **Then** a `results.json` file is written containing run metrics, row-level metrics, and per-threshold pass/fail outcomes. +2. **Given** the same completed run, **When** the run finishes, **Then** a `report.md` file is written that summarizes the same run in human-readable Markdown, including a per-threshold summary table. +3. **Given** an existing `results.json` from a prior run, **When** `agentops report generate --in ` is executed, **Then** a `report.md` is regenerated from that file without re-running the evaluation. +4. **Given** a cloud-executed run, **When** the run finishes, **Then** an additional `cloud_evaluation.json` is written containing the cloud evaluation/run identifiers and a deep-link to the corresponding Foundry Evaluations page, alongside the same `results.json`/`report.md` shape used by local runs. + +--- + +### User Story 2 - Get a stable, automatable pass/fail signal from every run (Priority: P1) + +A CI pipeline author wants a run's outcome to be expressed as one of exactly three exit codes so that a pipeline step can gate merges or deployments without parsing free-form text. + +**Why this priority**: A stable exit-code contract is what makes AgentOps usable as an automated release gate; without it, results are informative but not actionable in CI. + +**Independent Test**: Can be fully tested by running configurations engineered to succeed, to fail a threshold, and to hit a runtime/configuration error, and confirming the process exit code is `0`, `2`, and `1` respectively. + +**Acceptance Scenarios**: + +1. **Given** a run in which every threshold passes, **When** `agentops eval run` completes, **Then** the process exits with code `0`. +2. **Given** a run in which the agent was invoked successfully but at least one threshold fails, **When** `agentops eval run` completes, **Then** the process exits with code `2`. +3. **Given** a run that cannot proceed due to a configuration or runtime error (for example, an invalid `agentops.yaml` or an unreachable required resource before any row is evaluated), **When** `agentops eval run` is executed, **Then** the process exits with code `1` and the error is described in the output. + +--- + +### User Story 3 - Compare a run against a previous baseline (Priority: P2) + +A release engineer wants to see whether a new run's metrics have improved, regressed, or stayed the same relative to a previously saved baseline result, as part of the same `eval run` invocation. + +**Why this priority**: Regression comparison against a baseline is valuable for catching drift over time, but it builds on top of (and is optional relative to) the core result/threshold/exit-code contract, so it ranks below the P1 stories. + +**Independent Test**: Can be fully tested by running `agentops eval run --baseline ` and confirming the resulting `results.json`/`report.md` include a comparison section showing, per metric, the current value, the baseline value, and the direction of change (improved, regressed, or unchanged). + +**Acceptance Scenarios**: + +1. **Given** a prior run's `results.json` supplied via `--baseline`, **When** the new run completes, **Then** the result includes a comparison entry for every aggregate metric present in either run, showing the available current and baseline values without inventing a missing-side value, plus the computed change direction. +2. **Given** a baseline file that does not exist or cannot be parsed as a valid prior result, **When** `agentops eval run --baseline ` is executed, **Then** the run fails with a clear error rather than silently skipping the comparison. +3. **Given** a `--baseline` comparison is requested, **When** the run completes, **Then** the presence of a baseline comparison does not change the threshold pass/fail outcome or exit code of the current run; comparison is informational. + +--- + +### Edge Cases + +- What happens when a run's dataset is empty (zero rows)? The run MUST be rejected with a clear "dataset is empty" error before any row is invoked and before `results.json` is written for that attempt; a zero-row run is not a supported outcome. +- What happens when a threshold expression references a metric that no evaluator produced for a given row? That row's threshold evaluation for the missing metric MUST be represented explicitly (for example, as not evaluated) rather than silently counted as a pass. +- How does `agentops report generate` behave when pointed at a `results.json` that does not match the expected schema version? Report generation MUST fail with a clear error rather than producing a malformed or misleading `report.md`. +- What happens when `--baseline` is combined with a current run whose metric set differs from the baseline's (for example, a threshold was added or removed between runs)? Metrics present in only one of the two runs MUST be reported without fabricating a value for the side missing that metric. +- What happens when a run's agent invocation succeeds for every row but every row fails every threshold? The exit code MUST still be `2` (a threshold failure), not `1`, since the failure is a quality gate outcome rather than a runtime error. +- How is `agentops eval compare` handled as a standalone subcommand? It does not exist; only the `--baseline` flag on `agentops eval run` provides comparison, and the CLI help surface must not expose a `compare` subcommand. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST produce a normalized `results.json` for every completed run, containing a run-level summary, per-row metrics, and per-threshold pass/fail outcomes, regardless of execution mode or agent kind. +- **FR-002**: The system MUST produce a human-readable `report.md` for every completed run, rendered from the same `results.json` content, including a per-threshold summary. +- **FR-003**: The system MUST support regenerating `report.md` from an existing `results.json` via `agentops report generate --in ` without re-running the evaluation. +- **FR-004**: For cloud-executed runs, the system MUST additionally produce a `cloud_evaluation.json` containing cloud evaluation/run identifiers and a deep-link to the corresponding Foundry Evaluations page. +- **FR-005**: The system MUST translate a completed run's outcome into exactly one of three process exit codes: `0` when all thresholds pass, `2` when the run completed but at least one threshold failed, and `1` when a runtime or configuration error prevented the run from completing. +- **FR-006**: The system MUST keep the exit-code contract independent of execution mode; a threshold failure under local, cloud, or azd execution MUST all yield exit code `2`. +- **FR-007**: The system MUST support an optional `--baseline ` flag on `agentops eval run` that loads a prior run's `results.json` and produces a comparison across the union of current and baseline aggregate metrics (available current value, available baseline value, and direction of change) alongside the current run's own results. +- **FR-008**: The system MUST fail clearly when a `--baseline` path does not exist or does not parse as a valid prior result, rather than silently proceeding without a comparison. +- **FR-009**: The system MUST NOT let the presence or outcome of a baseline comparison change the current run's threshold pass/fail outcome or resulting exit code; comparison is informational only. +- **FR-010**: The system MUST NOT expose a standalone `agentops eval compare` subcommand; comparison functionality is only reachable through the `--baseline` flag on `agentops eval run`. +- **FR-011**: The system MUST persist run results under the workspace's results history directory and maintain a "latest" pointer to the most recent run for tools that need the current result without knowing its timestamp. +- **FR-012**: The system MUST reject a run whose dataset contains zero rows before any row is invoked, with a clear error, rather than producing a `results.json` that reports zero evaluated rows. + +### Key Entities + +- **RunResult**: The root normalized result object persisted as `results.json`, containing run summary, target information, row results, threshold evaluations, and optional cloud/comparison sections. +- **RowResult / RowMetric**: Per-dataset-row outcome and per-metric score captured for that row. +- **ThresholdEvaluation**: The pass/fail outcome, expression, and observed aggregate value for a single configured threshold. +- **RunSummary**: Aggregate run-level statistics (for example, overall pass/fail, item counts, per-metric averages) used to derive the exit code. +- **ComparisonInfo / ComparisonRow / ComparisonMetric**: The baseline-comparison structure capturing, per metric, the current value, baseline value, and computed direction of change. +- **TargetInfo**: Metadata identifying which agent target and execution mode produced a given `RunResult`. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Every completed run, regardless of agent kind or execution mode, produces a `results.json` and `report.md` pair with the same top-level structure (run summary, row results, threshold outcomes). +- **SC-002**: Each of the three exit-code outcomes is distinguishable by its exit code alone: `0` when every threshold passes, `2` when the run completed but at least one threshold failed, `1` when a runtime or configuration error prevented completion - the three codes never overlap for the same outcome. +- **SC-003**: `agentops report generate --in ` reproduces `report.md` from a saved `results.json` alone, with no network access and no re-invocation of the evaluated agent. +- **SC-004**: When `--baseline` is supplied, the resulting comparison section includes exactly one entry for every aggregate metric present in either the current or baseline run; a metric present on only one side retains a missing value on the other side rather than receiving a fabricated score. +- **SC-005**: `agentops eval --help` never lists a `compare` subcommand; comparison is reachable only through `--baseline` on `agentops eval run`. +- **SC-006**: A dataset with zero rows never reaches a completed run; every zero-row attempt is rejected before any row is invoked. + +## Assumptions + +- Downstream consumers (CI systems, Doctor, Cockpit, release evidence) read `results.json` as the canonical machine-readable artifact and treat `report.md` as a rendering of it, not an independent source of truth. +- A `--baseline` file is itself a `results.json` produced by a prior AgentOps run; comparing against externally produced or hand-authored result files is not part of this baseline. +- The exit-code contract (`0`/`1`/`2`) is a stable public contract that downstream automation depends on and is not expected to change as part of this feature area. +- Cloud evaluation identifiers and Foundry deep-links in `cloud_evaluation.json` depend on the cloud evaluation service being reachable at run time; this specification assumes the run has already reached completion when that artifact is produced. + +## Out of Scope + +- A standalone `agentops eval compare` command; it is not implemented, and this specification explicitly documents its absence rather than describing it as a capability. +- The mechanics of how a run is executed (agent invocation, per-row scoring) which are covered by the Evaluation Execution specification. +- Doctor readiness analysis and release evidence composition, which consume `results.json` history but are specified separately. +- Publishing results to Foundry Evaluations panels, which is covered by the Evaluation Execution specification's publishing behavior. +- Defining or changing individual evaluator scoring logic; this specification concerns the result container and reporting layer, not evaluator internals. + +## Implementation Evidence + +- `src/agentops/core/results.py` - `RunResult`, `RowResult`, `RowMetric`, `ThresholdEvaluation`, `RunSummary`, `TargetInfo`, `ComparisonInfo`, `ComparisonRow`, and `ComparisonMetric` schema definitions. +- `src/agentops/pipeline/thresholds.py` - `evaluate()` threshold evaluation logic that produces `ThresholdEvaluation` entries and drives `RunSummary.overall_passed`. +- `src/agentops/pipeline/orchestrator.py` - `exit_code_from(result)` translating `result.summary.overall_passed` into the `0`/`2` contract, with runtime errors raised as exceptions surfaced as exit code `1` by the CLI layer. +- `src/agentops/pipeline/comparison.py` - `load_baseline()` and `build_comparison()` implementing the `--baseline` comparison logic. +- `src/agentops/pipeline/reporter.py` - `render(result)` rendering `RunResult` into `report.md`, including threshold, comparison, cloud evaluation, and azd aggregate sections. +- `src/agentops/cli/app.py` - `eval run` command wiring `--baseline`, exit-code translation, and the `report generate` command regenerating `report.md` from a saved `results.json`. +- `tests/unit/test_cli_commands.py::test_eval_help_does_not_expose_compare_subcommand` - direct evidence that `agentops eval compare` is not implemented. +- `tests/integration/test_pipeline_smoke.py` (`test_http_pipeline_end_to_end`, `test_http_pipeline_with_baseline`) - end-to-end coverage of result production and baseline comparison. +- `tests/unit/test_pipeline_reporter.py` - unit coverage of `report.md` rendering from `results.json`. +- `docs/how-it-works.md` - narrative documentation of the results/report/exit-code contract consistent with this baseline. diff --git a/specs/004-cicd-workflow-generation/checklists/requirements.md b/specs/004-cicd-workflow-generation/checklists/requirements.md new file mode 100644 index 0000000..3aa3f1b --- /dev/null +++ b/specs/004-cicd-workflow-generation/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: CI/CD Workflow Generation + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/004-cicd-workflow-generation/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/services/cicd.py` and `workflow_analysis.py`, the packaged templates under `src/agentops/templates/workflows/` and `src/agentops/templates/pipelines/azuredevops/`, and `tests/unit/test_cicd.py` and `test_workflow_analysis.py`, not against a future proposal. +- The `doctor`/`watchdog` kind-naming nuance (source template historically named "watchdog", current CLI kind name "doctor", "watchdog" retained as a legacy alias) was directly verified in `src/agentops/services/cicd.py` and is documented as a factual finding, not treated as an open question. +- Revalidation pass: no factual corrections against source were required for this spec; Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (file-write completeness, skip/overwrite behavior, distinct template selection, exit-code/zero-file guarantees) instead of referring to automated test coverage. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/004-cicd-workflow-generation/spec.md b/specs/004-cicd-workflow-generation/spec.md new file mode 100644 index 0000000..0df1995 --- /dev/null +++ b/specs/004-cicd-workflow-generation/spec.md @@ -0,0 +1,134 @@ +# Feature Specification: CI/CD Workflow Generation + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing `agentops workflow analyze`/`agentops workflow generate` behavior as-built, not a new proposal. Sources reviewed include `src/agentops/services/workflow_analysis.py`, `src/agentops/services/cicd.py`, the packaged templates under `src/agentops/templates/workflows/` and `src/agentops/templates/pipelines/azuredevops/`, and `tests/unit/test_cicd.py` and `test_workflow_analysis.py`. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Understand what CI/CD setup is recommended before generating anything (Priority: P1) + +A release engineer runs `agentops workflow analyze` to see, without writing any files, which deploy mode and evaluation runner AgentOps recommends for the current repository, and which workflow files already exist versus would need to be created. + +**Why this priority**: Teams need a safe, read-only way to preview what generation would do before committing to writing files into their repository; this is the natural first step in adopting CI/CD generation. + +**Independent Test**: Can be fully tested by running `agentops workflow analyze` against a repository in various states (no workflows present, some workflows present, azd project detected, no azd project detected) and confirming the reported recommended deploy mode and file-existence findings match the repository's actual state, with zero files written. + +**Acceptance Scenarios**: + +1. **Given** a repository with no existing AgentOps workflow files, **When** `agentops workflow analyze` is run, **Then** the output reports which workflow files would be created, the recommended deploy mode, and the recommended evaluation runner, without writing any file. +2. **Given** a repository that already contains AgentOps-managed workflow files, **When** `agentops workflow analyze` is run, **Then** the output distinguishes files that already exist from files that are missing. +3. **Given** a repository recognized as an Azure Developer CLI (azd) project, **When** `agentops workflow analyze` is run, **Then** the recommended deploy mode reflects azd-aware provisioning rather than the generic placeholder mode. + +--- + +### User Story 2 - Generate PR and environment-promotion workflows for GitHub Actions or Azure DevOps (Priority: P1) + +A release engineer runs `agentops workflow generate` to create a consistent set of workflow/pipeline files covering pull-request evaluation gating and dev/qa/prod deployment promotion, for either GitHub Actions or Azure DevOps, without overwriting files the team has already customized. + +**Why this priority**: Generating the actual CI/CD templates is the core value of this feature; it turns the recommendation from User Story 1 into concrete, runnable automation. + +**Independent Test**: Can be fully tested by running `agentops workflow generate --platform github` and `--platform azure-devops` against a clean directory and confirming the expected set of files is written under `.github/workflows/` or the Azure DevOps pipelines location respectively, then re-running generation without `--force` and confirming existing files are left untouched and reported as skipped. + +**Acceptance Scenarios**: + +1. **Given** a clean repository and `--platform github` (the default), **When** `agentops workflow generate` is run with the default kinds, **Then** GitHub Actions workflow files for pull-request gating and dev/qa/prod deployment are written under `.github/workflows/`. +2. **Given** the same command with `--platform azure-devops`, **When** run, **Then** equivalent Azure DevOps pipeline YAML files are written to the Azure DevOps pipelines location instead. +3. **Given** a repository where a previously generated workflow file has since been manually edited, **When** `agentops workflow generate` is run again without `--force`, **Then** that file is left unmodified and reported as skipped rather than overwritten. +4. **Given** the same scenario, **When** `agentops workflow generate --force` is run, **Then** the previously edited file is overwritten with the current template content. +5. **Given** `--kinds pr,dev` is passed explicitly, **When** generation runs, **Then** only the pull-request and dev-environment templates are generated; qa, prod, and doctor templates are not written. + +--- + +### User Story 3 - Choose a deploy mode and a Doctor readiness gate for generated workflows (Priority: P2) + +A release engineer wants generated deployment workflows to match their infrastructure approach (a stack-agnostic placeholder, an azd-driven provision/deploy flow, or a Foundry prompt-agent candidate/eval/deploy flow), and wants the pull-request workflow to enforce a configurable Doctor readiness severity gate before merge. + +**Why this priority**: This tailors the generated automation to a team's actual deployment stack and desired readiness bar; it depends on the base generation capability in User Story 2 already working. + +**Independent Test**: Can be fully tested by running generation with each of the four `--deploy-mode` values and each of the supported `--doctor-gate` severities, and confirming the generated PR/deploy templates reference the corresponding deploy approach and gate severity. + +**Acceptance Scenarios**: + +1. **Given** `--deploy-mode auto` and a repository recognized as an azd project, **When** generation runs, **Then** the azd-specific deploy templates are selected automatically. +2. **Given** `--deploy-mode placeholder` is passed explicitly, **When** generation runs, **Then** the stack-agnostic placeholder deploy templates are written regardless of what `auto` would have selected. +3. **Given** `--deploy-mode prompt-agent`, **When** generation runs, **Then** Foundry prompt-agent candidate/eval/deploy templates are written instead of the generic deploy templates. +4. **Given** `--doctor-gate warning` is passed, **When** the pull-request workflow template is generated, **Then** the generated template's Doctor gate severity threshold reflects `warning` rather than the default `critical`. +5. **Given** an unsupported value is passed to `--deploy-mode` or `--doctor-gate`, **When** generation is invoked, **Then** the command fails with a validation error listing the valid values, and no files are written. + +--- + +### Edge Cases + +- What happens when `--kinds` includes an unrecognized kind name? Unknown kinds are ignored rather than causing the whole generation command to fail. +- What happens when the `doctor` workflow kind is requested? It is generated from a scheduled readiness-check template (historically named "watchdog" in the source template path) and written out under a Doctor-named output file; the legacy kind name `watchdog` is still accepted as an alias for `doctor`. +- How does the system handle generating for `--platform azure-devops` when the target directory has no existing Azure DevOps pipelines folder? The folder structure is created as part of writing the generated files. +- What happens when both an azd-specific template and a non-azd template exist for the same kind and `--deploy-mode azd` is selected? The azd-specific template is used for that kind instead of the generic one. +- What happens when `agentops workflow generate` is run twice in a row with identical flags and no manual edits in between? The second run reports every pre-existing output file as skipped because `--force` was not supplied; it does not need to compare file contents. +- What happens when only some of the requested `--kinds` have azd-specific templates available under `--deploy-mode azd`? Kinds without an azd-specific template fall back to their standard template rather than failing the whole command. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST provide a read-only `agentops workflow analyze` command that reports the recommended deploy mode, the recommended evaluation runner, and which workflow files currently exist versus would be created, without writing any file. +- **FR-002**: The system MUST provide an `agentops workflow generate` command that writes workflow/pipeline files for one or more of the kinds `pr`, `dev`, `qa`, `prod`, and `doctor`. +- **FR-003**: The system MUST support `--platform github` (default) writing GitHub Actions workflow YAML under `.github/workflows/`, and `--platform azure-devops` writing equivalent Azure DevOps pipeline YAML to the Azure DevOps pipelines location. +- **FR-004**: The system MUST support `--deploy-mode` values `auto`, `placeholder`, `azd`, and `prompt-agent`, where `auto` resolves to a concrete mode based on repository detection (for example, recognizing an azd project) and the other three are explicit selections. +- **FR-005**: The system MUST support `--doctor-gate` values that set the Doctor readiness severity floor enforced by the generated pull-request workflow before allowing a merge-gating step to pass. +- **FR-006**: The system MUST support `--kinds` as a comma-separated subset of the five supported kinds, generating only the requested kinds and silently ignoring unrecognized kind names. +- **FR-007**: The system MUST NOT overwrite an existing generated file unless `--force` is passed, and MUST report untouched pre-existing files as skipped. +- **FR-008**: The system MUST overwrite existing generated files with current template content when `--force` is passed. +- **FR-009**: The system MUST accept the legacy kind name `watchdog` as an alias for the `doctor` kind for backward compatibility. +- **FR-010**: The system MUST reject an unsupported `--deploy-mode` or `--doctor-gate` value with a validation error that lists the valid values, without writing any file. +- **FR-011**: The system MUST select azd-specific templates for a given kind when `--deploy-mode azd` (or `auto` resolving to azd) is active and an azd-specific template exists for that kind, falling back to the standard template for kinds without an azd-specific variant. +- **FR-012**: The system MUST only generate static template files (workflow/pipeline YAML); it MUST NOT execute, trigger, or validate any CI/CD run as part of generation. + +### Key Entities + +- **CicdResult**: The outcome of a generation invocation, capturing the effective deploy mode, the effective evaluation runner, and the lists of written versus skipped files. +- **Workflow Kind**: One of `pr`, `dev`, `qa`, `prod`, or `doctor`, each mapping to one or more source template files and output file locations per platform. +- **Deploy Mode**: One of `auto`, `placeholder`, `azd`, or `prompt-agent`, determining which template variant is used for deployment-oriented kinds. +- **Doctor Gate**: The configured Doctor severity floor (for example, `critical` or `warning`) embedded into the generated pull-request workflow's readiness-check step. +- **Platform Template Map**: The per-platform (GitHub Actions vs Azure DevOps) mapping from kind and deploy mode to a source template path and output path. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: `agentops workflow analyze` never writes, modifies, or deletes any file in the target repository, regardless of the repository's state. +- **SC-002**: For every one of the five workflow kinds on every one of the two platforms, running generation produces the corresponding output file at its documented location - no requested kind/platform combination is silently skipped. +- **SC-003**: Re-running `agentops workflow generate` without `--force` after a manual edit never alters the manually edited file's content; the file is reported as skipped, not overwritten. +- **SC-004**: Each explicit `--deploy-mode` selects its documented template strategy, `auto` resolves to one supported concrete strategy from repository signals, and each supported `--doctor-gate` value is reflected in the generated pull-request gate where applicable. +- **SC-005**: Passing an unsupported `--deploy-mode` or `--doctor-gate` value always results in a non-zero exit and zero files written to the target repository. +- **SC-006**: Every recognized requested workflow kind is either written to disk or explicitly reported as skipped because its output file already exists; unrecognized kind names are silently ignored as documented, while invalid deploy-mode or Doctor-gate values reject the command before files are written. + +## Assumptions + +- Generated workflow/pipeline files are intended to be committed by the user after review; this feature does not commit, push, or open pull requests on the user's behalf. +- The `doctor` workflow kind's underlying template historically used the file name "watchdog"; this baseline treats `doctor` as the current, supported kind name and `watchdog` as a backward-compatible alias, not a separate capability. +- Azure DevOps generation assumes the user will place the generated pipeline YAML into their organization's pipeline definitions; this feature does not call the Azure DevOps REST API to register a pipeline. +- Deploy-mode `auto` detection (for example, recognizing an azd project) relies on local file-system signals in the target repository, not on any network call. + +## Out of Scope + +- Executing, triggering, or validating a CI/CD run; this feature only produces static template files. +- Creating or managing Azure DevOps pipeline definitions via the Azure DevOps REST API or `az pipelines` CLI. +- Provisioning or deploying actual Azure infrastructure; azd-mode templates reference `azd` commands but this feature does not invoke them. +- Doctor's own readiness analysis logic, which is specified separately; this feature only wires a configured severity gate into generated templates. +- Release evidence composition, which is specified separately and may be invoked from within a generated deployment workflow but is not generated by this feature. + +## Implementation Evidence + +- `src/agentops/services/cicd.py` - `generate_cicd_workflows()`, `DEPLOY_MODES`, `ALL_KINDS`, `LEGACY_KIND_ALIASES` (`{"watchdog": "doctor"}`), `DOCTOR_GATES`, per-platform template maps, and the force/skip file-write logic. +- `src/agentops/services/workflow_analysis.py` - read-only `agentops workflow analyze` logic, including `recommended_deploy_mode()` and `recommended_eval_runner()` detection helpers. +- `src/agentops/templates/workflows/` - GitHub Actions templates including `agentops-pr.yml`, `agentops-pr-prompt-agent.yml`, `agentops-deploy-dev.yml`, `agentops-deploy-dev-azd.yml`, `agentops-deploy-qa.yml`, `agentops-deploy-qa-azd.yml`, `agentops-deploy-prod.yml`, `agentops-deploy-prod-azd.yml`, `agentops-deploy-prompt-agent.yml`, and `agentops-watchdog.yml` (source template for the `doctor` kind). +- `src/agentops/templates/pipelines/azuredevops/` - the equivalent Azure DevOps pipeline YAML set mirroring the GitHub Actions templates above. +- `src/agentops/cli/app.py` - `workflow analyze` and `workflow generate` Typer commands wiring `--platform`, `--deploy-mode`, `--doctor-gate`, `--kinds`, and `--force`. +- `tests/unit/test_cicd.py` - unit coverage of template generation, force/skip behavior, deploy-mode selection, and doctor-gate substitution. +- `tests/unit/test_workflow_analysis.py` - unit coverage of the read-only analyze command's recommendations and file-existence reporting. +- `docs/ci-github-actions.md` - narrative documentation of the generated GitHub Actions workflow set consistent with this baseline. diff --git a/specs/005-release-evidence/checklists/requirements.md b/specs/005-release-evidence/checklists/requirements.md new file mode 100644 index 0000000..d968642 --- /dev/null +++ b/specs/005-release-evidence/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Release Evidence + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/005-release-evidence/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/core/release_evidence.py` and `src/agentops/services/evidence_pack.py`, and against `tests/unit/test_release_evidence.py`, not against a future proposal. +- Future or unrelated capabilities (running new evaluations, remediating findings, comprehensive secret scanning, external publishing) are explicitly listed under "Out of Scope" and are not treated as gaps in this checklist. +- Revalidation pass: no factual corrections against source were required for this spec; Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (status-condition equivalence, redaction guarantees, artifact validity, output-location control, non-mutation of existing files) instead of referring to automated test coverage. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/005-release-evidence/spec.md b/specs/005-release-evidence/spec.md new file mode 100644 index 0000000..b40a2f9 --- /dev/null +++ b/specs/005-release-evidence/spec.md @@ -0,0 +1,122 @@ +# Feature Specification: Release Evidence + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing `agentops doctor --evidence-pack` behavior as-built, not a new proposal. Sources reviewed include `src/agentops/core/release_evidence.py`, `src/agentops/services/evidence_pack.py`, `src/agentops/cli/app.py`, and `tests/unit/test_release_evidence.py`. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Get a single evidence artifact answering "can we ship it" (Priority: P1) + +A release manager preparing to promote a build wants one composed artifact that pulls together evaluation results, Doctor readiness, workflow presence, and related signals into a single ready/ready-with-warnings/blocked verdict, without running any new evaluation or remediation. + +**Why this priority**: This is the entire purpose of the feature -- a single, trustworthy release-readiness artifact assembled from signals that already exist in the workspace. + +**Independent Test**: Can be fully tested by running `agentops doctor --evidence-pack` against a workspace with controlled eval, Doctor, workflow, Foundry, and governance signals, then confirming the resulting evidence artifact's overall status and per-check statuses match those inputs without executing a new evaluation or deployment. + +**Acceptance Scenarios**: + +1. **Given** every composed evidence check is ready and none produces a warning or blocker, **When** `agentops doctor --evidence-pack` is run, **Then** the composed evidence reports an overall status of `ready`. +2. **Given** no composed check is blocked but at least one check produces a non-blocking advisory (for example, a `warning`-severity Doctor finding or a missing baseline), **When** the evidence pack is composed, **Then** the overall status is `ready_with_warnings` rather than `ready` or `blocked`. +3. **Given** any composed check produces a blocking condition (for example, the latest eval run failed its thresholds), **When** the evidence pack is composed, **Then** the overall status is `blocked`. +4. **Given** the evidence pack composition completes, **When** the output is inspected, **Then** both a machine-readable `evidence.json` and a human-readable `evidence.md` are written, and every individual check inside them carries its own status (`ready`, `warning`, `blocked`, or `unknown`). + +--- + +### User Story 2 - Trust that evidence never leaks secrets (Priority: P1) + +A security-conscious release manager wants any credential-shaped values (for example, connection strings, keys, tokens) that might appear inside underlying signal data to be redacted before they are written into the evidence artifacts. + +**Why this priority**: Evidence artifacts are meant to be shared broadly (attached to a release, reviewed by stakeholders); leaking a secret through this artifact would be a serious regression, so redaction is as critical as composing the evidence itself. + +**Independent Test**: Can be fully tested by seeding underlying signal data (for example, a results file or config value) with a recognizable secret-shaped string and confirming the resulting `evidence.json`/`evidence.md` contain a redaction marker in place of that value rather than the raw value. + +**Acceptance Scenarios**: + +1. **Given** an underlying signal contains a connection-string-shaped or key-shaped value, **When** the evidence pack is composed, **Then** the value is replaced with a redaction marker in both `evidence.json` and `evidence.md`. +2. **Given** the redaction has been applied, **When** the resulting JSON is parsed back, **Then** it remains valid JSON conforming to the `ReleaseEvidence` schema, confirming redaction does not corrupt the artifact's structure. + +--- + +### User Story 3 - Control where evidence is written (Priority: P3) + +A release engineer integrating evidence composition into an automated deployment workflow wants to specify a custom output location for the evidence artifacts, or otherwise rely on a sensible default location within the workspace. + +**Why this priority**: Custom output paths matter for pipeline integration but are a secondary convenience relative to the correctness of the composed evidence itself. + +**Independent Test**: Can be fully tested by running `agentops doctor --evidence-pack --evidence-out ` and confirming the artifacts are written to the custom path, then running without `--evidence-out` and confirming they are written to the documented default location. + +**Acceptance Scenarios**: + +1. **Given** `--evidence-out ` is supplied, **When** the evidence pack is composed, **Then** `evidence.json` and `evidence.md` are written under that path. +2. **Given** `--evidence-out` is omitted, **When** the evidence pack is composed, **Then** the artifacts are written under `.agentops/release/latest`. + +--- + +### Edge Cases + +- What happens when no prior eval run exists in the workspace? The "Latest eval gate" check MUST report a `blocked` or `unknown` status with an explanatory summary rather than silently omitting the check. +- What happens when no Doctor analysis is supplied to the evidence composer? The composer MUST still produce valid `evidence.json`/`evidence.md` artifacts with Doctor readiness represented as unavailable or warning, rather than claiming a successful Doctor result. +- How does composition handle a workspace with CI/CD workflows partially generated (for example, PR gate present, deploy workflows missing)? The "PR gate" and "Deploy workflows" checks MUST be reported independently, each reflecting only what is actually present. +- What happens when the trace-to-dataset promotion flywheel has never been used in the workspace? The "Trace-to-dataset flywheel" check MUST report its status based on the absence of promotion manifests rather than erroring. +- What happens when a `--baseline` regression comparison was configured for the latest eval run? The "Regression baseline" check MUST reflect that a baseline comparison was present for that run. +- What happens when redaction encounters a value that only partially resembles a secret pattern (for example, a UUID that is not actually a credential)? Redaction rules MUST be scoped narrowly enough that this does not corrupt non-secret evidence content beyond what the documented redaction patterns are designed to match. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST provide `agentops doctor --evidence-pack` to compose a single release-evidence artifact from the current Doctor analysis and existing workspace signals (eval results, generated CI/CD workflows, trace-to-dataset promotion state, and related checks) without invoking any new evaluation. +- **FR-002**: The system MUST compute an overall readiness status of exactly one of `ready`, `ready_with_warnings`, or `blocked`, derived from the presence of blocking versus warning-level findings across the composed checks. +- **FR-003**: The system MUST represent each individual composed check with its own status of `ready`, `warning`, `blocked`, or `unknown`, a human-readable summary, and supporting evidence detail. +- **FR-004**: The system MUST write a machine-readable `evidence.json` conforming to a versioned `ReleaseEvidence` schema and a human-readable `evidence.md` rendering of the same content. +- **FR-005**: The system MUST redact credential-shaped values (for example, connection strings, keys, tokens) from both `evidence.json` and `evidence.md` before they are written to disk. +- **FR-006**: The system MUST support a custom output location via `--evidence-out ` and MUST fall back to a documented default location within the workspace when the flag is omitted. +- **FR-007**: The system MUST NOT execute, remediate, or modify any evaluation, Doctor finding, or CI/CD workflow as a side effect of composing evidence; composition is read-only over already-existing signals. +- **FR-008**: The system MUST produce valid `evidence.json`/`evidence.md` artifacts even when one or more underlying signals are unavailable (for example, no eval history), representing the affected checks as `unknown` or `blocked` rather than aborting composition entirely. +- **FR-009**: The system MUST include links to relevant external resources (for example, the latest Foundry evaluation or Azure Monitor dashboard) in the composed evidence when the corresponding signal identifies such a resource. + +### Key Entities + +- **ReleaseEvidence**: The root composed artifact, containing an overall `ReadinessStatus`, a list of `ReleaseEvidenceCheck` entries, and a list of `ReleaseEvidenceLink` entries. +- **ReleaseEvidenceCheck**: A single named check (for example, "Latest eval gate", "Doctor readiness", "PR gate", "Deploy workflows", "Regression baseline", "Trace-to-dataset flywheel", "Runtime monitoring", "AI Landing Zone readiness") with its own `CheckStatus`, summary, and evidence payload. +- **ReleaseEvidenceLink**: A named external link (for example, to a Foundry evaluation run or an Azure Monitor workbook) surfaced alongside the composed evidence. +- **ReadinessStatus**: The overall composed verdict, one of `ready`, `ready_with_warnings`, or `blocked`. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Each of the three overall readiness statuses (`ready`, `ready_with_warnings`, `blocked`) is produced exactly when its documented underlying-signal condition holds - a `blocked`-triggering condition never composes as `ready` or `ready_with_warnings`, and vice versa. +- **SC-002**: A credential-shaped value present in any underlying signal never appears verbatim in the composed `evidence.json` or `evidence.md`; it is always replaced with a redaction marker. +- **SC-003**: Composing evidence in a workspace with no prior eval history always produces a valid, schema-conforming `evidence.json` with the affected check reported as `unknown` or `blocked`, never an unhandled error and never a missing artifact. +- **SC-004**: `--evidence-out ` always controls the artifacts' output location; omitting it always falls back to the same documented default location. +- **SC-005**: Composing evidence never changes the content of any existing eval result, Doctor history entry, or CI/CD workflow file already present in the workspace. +- **SC-006**: Every individual composed check reports exactly one of `ready`, `warning`, `blocked`, or `unknown` - no check is left without a status. + +## Assumptions + +- Release evidence composition depends on the current Doctor analysis plus workspace signals also used by Doctor and Cockpit (results history, generated workflows, trace promotion manifests); this feature does not introduce a new monitoring or evaluation source of its own. +- The redaction patterns implemented are pattern-based (matching common credential-shaped strings) rather than an exhaustive secret-scanning solution; teams with additional secret formats are expected to avoid placing them in evidence-adjacent files. +- Evidence composition itself does not execute a new cloud query or mutation. The surrounding Doctor run may still read configured cloud sources using the credentials normally required by Doctor. + +## Out of Scope + +- Running a new evaluation as part of evidence composition; only already-completed eval history is consumed. +- Remediating any blocking or warning finding surfaced in the evidence; this feature only reports status. +- Comprehensive secret scanning beyond the implemented pattern-based redaction rules. +- Doctor's own readiness analysis logic, which is specified separately (see Doctor Readiness); this feature only consumes Doctor's output. +- Publishing or distributing the evidence artifact externally (for example, attaching it to a release or uploading it to a portal); this feature only writes local files. + +## Implementation Evidence + +- `src/agentops/core/release_evidence.py` - `ReleaseEvidence`, `ReleaseEvidenceCheck`, `ReleaseEvidenceLink` schema models, `ReadinessStatus` (`ready`/`ready_with_warnings`/`blocked`), and `CheckStatus` (`ready`/`warning`/`blocked`/`unknown`) literal types. +- `src/agentops/services/evidence_pack.py` - `build_release_evidence()` composing the overall status from per-check results, `write_release_evidence()` writing `evidence.json`/`evidence.md`, `render_release_evidence_markdown()` for Markdown rendering, and `_redact_text()`/`_redact_obj()` redaction logic applied before writing. +- `src/agentops/services/evidence_pack.py` (`_add_eval_check`, `_add_doctor_check`, `_add_workflow_checks`, `_add_baseline_check`, `_add_trace_dataset_check`, `_add_monitoring_check`, `_add_ailz_check`, `_add_governance_check`, `_add_agent_identity_check`, `_add_foundry_check`) - the individual composed checks referenced in the Key Entities and User Story acceptance scenarios above. +- `src/agentops/cli/app.py` - `doctor` command wiring of `--evidence-pack` and `--evidence-out`. +- `tests/unit/test_release_evidence.py` - unit coverage of evidence composition, status derivation, and redaction behavior. +- `docs/doctor-explained.md` - narrative documentation of `--evidence-pack` as a readiness projection over existing signals, consistent with this baseline. diff --git a/specs/006-doctor-readiness/checklists/requirements.md b/specs/006-doctor-readiness/checklists/requirements.md new file mode 100644 index 0000000..a21ca17 --- /dev/null +++ b/specs/006-doctor-readiness/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Doctor Readiness + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/006-doctor-readiness/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/agent/analyzer.py`, `findings.py`, `report.py`, `history.py`, `checks/`, and `sources/`, and against the corresponding `tests/unit/test_agent_*.py` and `test_doctor_*.py` files, not against a future proposal. +- The claim that Doctor produces zero findings for an "empty" workspace is explicitly avoided per instruction; User Story 1's third scenario and the Out of Scope section state findings depend on what implemented checks actually detect, not a blanket zero-findings guarantee. +- Revalidation pass: no factual corrections against source were required for this spec; Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (category/severity completeness, exit-code/severity-floor consistency, non-mutation, history durability, graceful degradation) instead of referring to automated test coverage. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/006-doctor-readiness/spec.md b/specs/006-doctor-readiness/spec.md new file mode 100644 index 0000000..ffce199 --- /dev/null +++ b/specs/006-doctor-readiness/spec.md @@ -0,0 +1,132 @@ +# Feature Specification: Doctor Readiness + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing `agentops doctor` readiness analysis behavior as-built, not a new proposal. Sources reviewed include `src/agentops/agent/analyzer.py`, `findings.py`, `report.py`, `history.py`, `checks/`, `sources/`, `docs/doctor-checks.md`, `docs/doctor-explained.md`, and associated tests. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Get a severity-ranked list of readiness findings from multiple local signals (Priority: P1) + +An operator runs `agentops doctor` and receives a categorized, severity-ranked list of findings drawn from multiple signal sources (eval result history, Azure resource configuration, Azure Monitor telemetry, Foundry configuration), without Doctor altering project source, project configuration, generated workflows, or cloud resources. + +**Why this priority**: The severity-ranked findings list is the core output of Doctor; every other capability (history, recommendations, evidence integration) exists to make this list more useful over time. + +**Independent Test**: Can be fully tested by running `agentops doctor` against a workspace with a known mix of quality, performance, reliability, operational-excellence, security, and responsible-AI conditions, and confirming each finding is reported with the correct category and severity (`info`, `warning`, or `critical`), with no source or configuration file modified. + +**Acceptance Scenarios**: + +1. **Given** a workspace with a recent eval run that regressed against its prior baseline, **When** `agentops doctor` runs, **Then** a finding in the quality or reliability category is reported reflecting the regression. +2. **Given** a workspace whose generated CI/CD workflows are missing a Doctor readiness gate, **When** `agentops doctor` runs, **Then** an operational-excellence finding is reported describing the gap. +3. **Given** a workspace with no readiness issues that the currently implemented checks can detect, **When** `agentops doctor` runs, **Then** the run completes successfully and reports whatever findings the implemented checks actually produce for that state, without fabricating findings that do not correspond to an implemented check. +4. **Given** any of the above runs, **When** completed, **Then** no project source file, project configuration file, generated workflow, or Azure resource is modified; only documented local report, history, and explicitly requested evidence artifacts may be written. + +--- + +### User Story 2 - Gate a pipeline on a configurable severity floor (Priority: P1) + +A CI pipeline author wants `agentops doctor` to exit with code `2` when findings at or above a configured severity floor are present, so that a pipeline step can block promotion on readiness regressions while distinguishing a finding-gate failure from a runtime/configuration error. + +**Why this priority**: Turning findings into an automatable gate is what allows Doctor to function inside CI/CD, matching the same automation value proposition as the eval exit-code contract. + +**Independent Test**: Can be fully tested by running `agentops doctor --severity-fail critical` against a workspace with only warning-level findings (expecting `0`) and then against a workspace with a critical finding (expecting `2`), and by varying `--severity-fail` to confirm the floor is respected. + +**Acceptance Scenarios**: + +1. **Given** `--severity-fail critical` and a workspace whose worst finding is `warning`, **When** `agentops doctor` runs, **Then** the command succeeds (exit code `0`). +2. **Given** the same flag and a workspace with at least one `critical` finding, **When** `agentops doctor` runs, **Then** the command exits with code `2`. +3. **Given** `--severity-fail warning` and a workspace whose worst finding is `warning`, **When** `agentops doctor` runs, **Then** the command exits with code `2`, since `warning` now meets the configured floor. + +--- + +### User Story 3 - Preserve Doctor analyses for later operator and Cockpit review (Priority: P2) + +An operator wants each Doctor run's findings recorded to local history so that recurring, new, or resolved findings can be reviewed later by an operator or through Cockpit. + +**Why this priority**: Historical tracking increases the value of each individual run and supports Cockpit review, but it is secondary to the core single-run analysis and gating capabilities. + +**Independent Test**: Can be fully tested by running `agentops doctor` multiple times against an evolving writable workspace, confirming each completed analysis appends an entry to local history, and confirming the records are returned by Cockpit's Doctor history endpoint without being rewritten. + +**Acceptance Scenarios**: + +1. **Given** a workspace with no prior Doctor history, **When** `agentops doctor` runs for the first time, **Then** a new local history entry is created recording that run's findings. +2. **Given** a workspace with existing Doctor history, **When** `agentops doctor` runs again, **Then** a new history entry is appended without discarding prior entries. +3. **Given** accumulated Doctor history, **When** Cockpit's Doctor history endpoint is requested, **Then** it returns the persisted analyses for read-only review; a later Doctor analysis does not consume those prior Doctor records as input signals. + +--- + +### Edge Cases + +- What happens when one of Doctor's underlying data sources (for example, Azure Monitor) is unreachable? The affected check(s) MUST report a degraded/unknown outcome rather than causing the entire `agentops doctor` run to fail. +- What happens when the workspace has no `.agentops/agent/history.jsonl` yet? Doctor MUST create it on the first run rather than requiring it to pre-exist. +- What happens when `--severity-fail` is given a value that is not one of the supported severities? The command MUST reject it with a clear validation error rather than silently defaulting to a different floor. +- What happens when `--lookback-days` excludes all eval-history or telemetry records used by a time-bounded source? Doctor MUST still run and report current findings from other sources; persisted Doctor analysis records are not themselves filtered or consumed as input to the new analysis. +- What happens when the local Doctor history file cannot be written? The completed analysis and report remain valid, the append failure is logged for debugging, and the command does not claim that a new history record was persisted. +- How does Doctor behave when it is invoked from within a generated CI/CD workflow versus interactively? The severity-fail exit-code contract MUST behave identically in both contexts, since this is what allows the same command to be reused as a CI gate. +- What happens when two findings from different checks describe the same underlying condition? Each check's finding MUST still be reported independently; Doctor does not silently deduplicate findings across unrelated checks. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST analyze multiple local signal sources -- including eval result history, generated CI/CD workflow presence, Azure resource configuration, Azure Monitor telemetry, and Foundry configuration -- and produce a combined list of findings from a single `agentops doctor` invocation. +- **FR-002**: The system MUST assign every finding a category from a fixed set (quality, performance, reliability, operational excellence, security, responsible AI) and a severity from a fixed set (info, warning, critical). +- **FR-003**: The system MUST NOT modify project source, project configuration, generated workflows, or Azure resources as a result of running `agentops doctor`; it MAY write only its documented local report, best-effort history record, and explicitly requested evidence artifacts. +- **FR-004**: The system MUST support a `--severity-fail ` option that causes the command to exit with code `2` when at least one finding at or above the configured severity is present, and with code `0` when the analysis succeeds below that floor; runtime or configuration errors MUST use code `1`. +- **FR-005**: The system MUST reject an unsupported `--severity-fail` value with a validation error rather than silently substituting a default. +- **FR-006**: After a completed analysis, the system MUST attempt to append its findings to local history (`.agentops/agent/history.jsonl` or equivalent), MUST create the history store automatically on first successful write, and MUST leave prior records unchanged; a history-write failure MUST NOT invalidate the completed analysis. +- **FR-007**: The system MUST support a positive `--lookback-days` option that scopes time-bounded eval-history and telemetry source queries, without treating prior persisted Doctor analyses as inputs to the current run. +- **FR-008**: The system MUST degrade gracefully when an individual data source is unreachable, reporting that specific check as degraded/unknown rather than aborting the entire `agentops doctor` run. +- **FR-009**: The system MUST produce a human-readable report (in addition to the exit code and history entry) summarizing the run's findings, organized by category and severity. +- **FR-010**: The system MUST support an `agentops doctor explain` command that provides long-form documentation of Doctor's checks and behavior, reachable without running an analysis. +- **FR-011**: The system MUST expose its composed analysis result in a form that the Release Evidence feature can consume (see the Release Evidence specification) without Doctor itself performing evidence composition. + +### Key Entities + +- **Finding**: A single reported condition with a `Category`, a `Severity`, and a human-readable description, produced by exactly one check. +- **Category**: One of `quality`, `performance`, `reliability`, `operational_excellence`, `security`, or `responsible_ai`. +- **Severity**: One of `info`, `warning`, or `critical`, ordered for comparison against `--severity-fail`. +- **AnalysisResult**: The aggregate output of a single `agentops doctor` run, containing the full list of findings across all checks and sources. +- **History Entry**: A persisted record of a single past `agentops doctor` run's findings, appended to local history and available to Cockpit and operator review without becoming an input to later Doctor analyses. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Every finding reported by any check always carries exactly one category from the fixed set and exactly one severity from the fixed set - no finding is ever reported without both. +- **SC-002**: For every severity floor value, `--severity-fail` produces exit code `0` when the worst reported finding is below the floor and exit code `2` when it is at or above the floor; runtime or configuration errors use exit code `1`. +- **SC-003**: Running `agentops doctor` never modifies project source, project configuration, generated workflows, or Azure resources; its writes are limited to documented report/history artifacts and explicitly requested evidence artifacts. +- **SC-004**: In a writable workspace, a first completed analysis creates local history when none exists and each subsequent completed analysis appends one record without discarding prior entries. +- **SC-005**: An unreachable or unavailable individual data source always degrades only that check's outcome (reported as unknown/degraded), never causing the overall `agentops doctor` invocation to abort. +- **SC-006**: `agentops doctor explain` is reachable and returns Doctor documentation without performing any analysis or touching any file. + +## Assumptions + +- Doctor's checks operate over signals already available locally or via read-only Azure/Foundry queries that the operator's existing credentials can already access; Doctor does not provision new credentials or resources. +- Doctor history is a local read-only review surface for operators and Cockpit; the current implementation does not feed prior Doctor analysis records into subsequent Doctor analyses or release-evidence composition. +- The fixed category and severity sets are considered stable for this baseline; adding new categories or severities is a larger change outside this specification's scope. +- Doctor is expected to be run both interactively by an operator and non-interactively inside a generated CI/CD workflow, and both invocation styles are covered by the same command and flag surface. + +## Out of Scope + +- Remediating, auto-fixing, or opening pull requests for any finding Doctor reports. +- Real-time or continuous monitoring; Doctor performs a single point-in-time analysis per invocation. +- Composing the cross-feature release-evidence artifact itself, which is specified separately (see Release Evidence) even though it consumes Doctor's `AnalysisResult`. +- Defining new finding categories or severities beyond the fixed sets already implemented. +- Claiming that an empty or minimal workspace always yields zero findings; the actual outcome depends on which checks are implemented and what conditions they detect, and is not asserted as a blanket guarantee in this baseline. + +## Implementation Evidence + +- `src/agentops/agent/analyzer.py` - orchestration of multiple checks into a combined `AnalysisResult`. +- `src/agentops/agent/findings.py` - `Finding`, `Category` (`quality`, `performance`, `reliability`, `operational_excellence`, `security`, `responsible_ai`), and `Severity` (`info`, `warning`, `critical`) definitions, including severity ordering used for `--severity-fail` comparisons. +- `src/agentops/agent/report.py` - human-readable Doctor report rendering grouped by category and severity. +- `src/agentops/agent/history.py` - `.agentops/agent/history.jsonl` read/write logic, including first-run creation and lookback-window filtering. +- `src/agentops/agent/checks/` - individual check implementations (for example, Foundry configuration, regression, operational-excellence, and posture-rule checks) each producing `Finding` entries. +- `src/agentops/agent/sources/` - data source adapters (results history, Azure resources, Azure Monitor, Foundry control plane) each degrading independently when unavailable. +- `src/agentops/cli/app.py` - `doctor` and `doctor explain` command wiring, including `--severity-fail` and `--lookback-days`. +- `tests/unit/test_agent_analyzer.py`, `test_agent_history.py`, `test_doctor_catalog.py`, `test_doctor_cli_explain.py`, `test_agent_checks_foundry_config.py`, `test_agent_checks_observability.py` - unit coverage of analysis composition, history persistence, and individual checks. +- `docs/doctor-checks.md` and `docs/doctor-explained.md` - narrative documentation of Doctor's checks, categories, severities, and operations model consistent with this baseline. diff --git a/specs/007-read-only-cockpit/checklists/requirements.md b/specs/007-read-only-cockpit/checklists/requirements.md new file mode 100644 index 0000000..2cdedbd --- /dev/null +++ b/specs/007-read-only-cockpit/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Read-Only Cockpit + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/007-read-only-cockpit/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every route claim was directly verified against the `@app.get(...)` decorators in `src/agentops/agent/cockpit.py` (confirming zero mutating routes), and every other checklist item was checked against `src/agentops/agent/production_telemetry.py`, the CLI `cockpit` command in `src/agentops/cli/app.py`, and `tests/unit/test_cockpit.py`, `test_cli_cockpit_connection_summary.py`, and `test_cli_cockpit_port_conflict.py`, not against a future proposal. +- Future or unrelated capabilities (mutation endpoints, real-time alerting, embedded external UIs, multi-user auth) are explicitly listed under "Out of Scope" and are not treated as gaps in this checklist. +- Revalidation pass: no factual corrections against source were required for this spec; Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (read-only route completeness, state fidelity, fallback behavior, deferred loading, not-found handling) instead of referring to automated test coverage. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/007-read-only-cockpit/spec.md b/specs/007-read-only-cockpit/spec.md new file mode 100644 index 0000000..4c0f886 --- /dev/null +++ b/specs/007-read-only-cockpit/spec.md @@ -0,0 +1,127 @@ +# Feature Specification: Read-Only Cockpit + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing `agentops cockpit` local dashboard behavior as-built, not a new proposal. Sources reviewed include `src/agentops/agent/cockpit.py`, `production_telemetry.py`, the `cockpit` command in `src/agentops/cli/app.py`, and `tests/unit/test_cockpit.py` and related CLI cockpit tests. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - See workspace readiness at a glance from local history (Priority: P1) + +An operator runs `agentops cockpit` and opens a local web page that summarizes the workspace's eval and Doctor readiness status, built entirely from local result and history files already produced by prior `agentops eval run` and `agentops doctor` invocations. + +**Why this priority**: Presenting a consolidated readiness view from history the operator already has is the entire value of Cockpit; every other capability (links, telemetry, next actions) is layered on top of this core view. + +**Independent Test**: Can be fully tested by starting Cockpit against a workspace with known eval and Doctor history, confirming `GET /` returns the loading shell, then requesting `/?_partial=1` and confirming the hydrated page reflects the known readiness state, run count, and latest results without triggering a new eval or Doctor run. + +**Acceptance Scenarios**: + +1. **Given** a workspace with existing eval run history, **When** Cockpit's loading shell requests `/?_partial=1`, **Then** the hydrated response reflects the latest known eval readiness state derived from that history. +2. **Given** the same workspace, **When** `/api/eval-runs` is requested, **Then** it returns eval run summaries derived from results already present under `.agentops/results/`. +3. **Given** a workspace with existing Doctor history, **When** `/api/history` is requested, **Then** it returns the persisted Doctor analysis records, and the hydrated readiness page reflects the latest Doctor state. +4. **Given** a workspace with no eval or Doctor history at all, **When** Cockpit is started and `/?_partial=1` is requested, **Then** the page renders a defined empty/fallback state rather than raising an error. + +--- + +### User Story 2 - Get contextual next actions and external links without leaving the local view (Priority: P2) + +An operator viewing Cockpit wants to see recommended next actions (for example, "run an eval", "check Doctor findings") along with direct links out to the relevant Foundry, Azure Monitor, or workbook resource, so they can act without Cockpit itself performing any remote action. + +**Why this priority**: Contextual guidance and external links make the readiness summary actionable, but they build on top of the core summary in User Story 1 rather than being independently central. + +**Independent Test**: Can be fully tested by inspecting the rendered page or its API response for a workspace in a known state and confirming the next-action guidance and external links match what that state implies (for example, a workspace with a failed threshold surfaces a "review report" action and a link to the relevant run's report). + +**Acceptance Scenarios**: + +1. **Given** a workspace whose latest eval run failed a threshold, **When** the partial Cockpit page is rendered, **Then** it surfaces a next-action pointing the operator toward the failing run's report. +2. **Given** a workspace with a configured Foundry project endpoint, **When** the partial Cockpit page is rendered, **Then** it includes an outbound link to the corresponding Foundry resource rather than embedding Foundry's own UI. +3. **Given** a workspace with an Azure Monitor Workbook deployed via `agentops telemetry dashboard deploy`, **When** the partial Cockpit page is rendered, **Then** it includes a tile or link to that workbook. + +--- + +### User Story 3 - View a specific run's report and deferred production telemetry (Priority: P3) + +An operator wants to open a specific historical run's rendered report from Cockpit, and separately view production telemetry (for example, recent Application Insights activity) that is fetched only when requested rather than blocking the initial page load. + +**Why this priority**: Drilling into a specific run and loading production telemetry are secondary, on-demand views that depend on the core summary already being available. + +**Independent Test**: Can be fully tested by requesting `/api/runs/{run_id}/report` for a known run id and confirming the corresponding report is returned, and separately requesting `/api/production` and `/api/production/html` and confirming production telemetry is fetched only in response to that request rather than during the initial root page load. + +**Acceptance Scenarios**: + +1. **Given** a known historical run id, **When** `/api/runs/{run_id}/report` is requested, **Then** the rendered report for that specific run is returned. +2. **Given** an unknown or missing run id, **When** the same endpoint is requested, **Then** a defined not-found response is returned rather than an unhandled error. +3. **Given** production telemetry configuration is present, **When** `/api/production` is requested, **Then** telemetry is fetched at that time (deferred), not eagerly during Cockpit startup, loading-shell rendering, or partial readiness-page rendering. + +--- + +### Edge Cases + +- What happens when Cockpit is started in a workspace with no `agentops.yaml` at all? Cockpit MUST still start and serve its defined fallback/empty state rather than failing to start. +- What happens when production telemetry configuration (for example, an Application Insights connection string) is absent? The `/api/production` and `/api/production/html` endpoints MUST report that telemetry is unavailable rather than raising an unhandled error. +- What happens when a requested run id does not exist in history? The report endpoint MUST return a defined not-found response. +- What happens when history files are malformed or partially written? Cockpit MUST render its defined fallback state for the affected section rather than crashing the whole page. +- Does any Cockpit endpoint accept a request that mutates workspace state, cloud resources, or history? No; every exposed endpoint is a read (`GET`) operation over already-existing local data or external metadata. +- What happens when `--no-preflight` is passed to `agentops cockpit`? Cockpit MUST start without performing its startup preflight checks, while still serving the same read-only endpoints once running. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST provide `agentops cockpit` to start a local web server whose `GET /` response is a loading shell and whose `GET /?_partial=1` response presents a consolidated readiness view built from existing local eval and Doctor history, without requiring or triggering a new eval or Doctor run. +- **FR-002**: Every route exposed by Cockpit MUST be a read-only (`GET`) operation; the system MUST NOT expose any endpoint that creates, modifies, or deletes workspace files, Azure resources, or Foundry configuration. +- **FR-003**: The system MUST expose Doctor analysis history through `/api/history` and eval run summaries through the separate `/api/eval-runs` endpoint, each reflecting the corresponding local records already written by Doctor or eval execution. +- **FR-004**: The system MUST expose a per-run report endpoint (`/api/runs/{run_id}/report`) that renders the stored report for a specific historical run and MUST return a defined not-found response for an unknown run id. +- **FR-005**: The system MUST surface contextual next-action guidance on its hydrated partial page derived from the current readiness state (for example, pointing to a failing run's report or to Doctor findings). +- **FR-006**: The system MUST surface outbound links to external resources (Foundry project, Azure Monitor Workbook, and related dashboards) when the corresponding configuration or deployed resource is present, without embedding or proxying those external systems' own UIs. +- **FR-007**: The system MUST defer fetching production telemetry (`/api/production`, `/api/production/html`) until those endpoints are explicitly requested, rather than fetching it during Cockpit startup, loading-shell rendering, or partial readiness-page rendering. +- **FR-008**: The system MUST render a defined empty/fallback state for any section (eval history, Doctor history, production telemetry) when the corresponding local data or configuration is absent or unreadable, rather than raising an unhandled error. +- **FR-009**: The system MUST support `--host`, `--port`, `--workspace`, and `--no-preflight` options to control how and where Cockpit is served. +- **FR-010**: The system MUST provide a health-check endpoint (`/healthz`) suitable for verifying the local server is responsive. + +### Key Entities + +- **Readiness Summary**: The consolidated view of the workspace's latest eval and Doctor status, computed from local history rather than a live query. +- **Doctor Analysis Record**: A persisted Doctor run exposed through `/api/history` and used in the readiness view. +- **Evaluation Run Summary**: A persisted eval result (per `.agentops/results//`) surfaced separately through `/api/eval-runs` and the readiness view. +- **Next Action**: A contextual recommendation (for example, "review failing report", "run Doctor") derived from the current readiness state. +- **External Link**: An outbound reference to a Foundry project, Azure Monitor Workbook, or similar external resource, rendered as a link rather than embedded content. +- **Production Telemetry Snapshot**: The deferred, on-demand result of querying production telemetry (for example, Application Insights) for the `/api/production` family of endpoints. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: 100% of routes registered by Cockpit are `GET` routes - no mutating endpoint (create, update, delete) is exposed anywhere in the served API. +- **SC-002**: Starting Cockpit against a workspace with existing eval and Doctor history returns a loading shell from `GET /`, while `GET /?_partial=1`, `/api/history`, and `/api/eval-runs` reflect their corresponding local records without fabricating state. +- **SC-003**: Starting Cockpit against a workspace with no eval or Doctor history still returns the loading shell and a defined hydrated empty/fallback state, never an error. +- **SC-004**: Requesting either the loading shell or the partial readiness page never triggers a production-telemetry fetch as a side effect; telemetry is retrieved only when `/api/production` or `/api/production/html` is explicitly requested. +- **SC-005**: Requesting a report for an unknown run id always returns a defined not-found response, never an unhandled exception. +- **SC-006**: No request to any Cockpit endpoint ever creates, modifies, or deletes a workspace file, Azure resource, or Foundry configuration. + +## Assumptions + +- Cockpit is intended for local, single-operator use (typically `localhost`) rather than as a multi-tenant hosted service; authentication/authorization beyond host/port binding is out of scope for this baseline. +- Cockpit's readiness summary is only as current as the local history it reads; it does not poll Foundry or Azure Monitor continuously, consistent with its read-only, on-demand design. +- The external links Cockpit renders assume the operator already has appropriate access to the linked Foundry/Azure resources; Cockpit does not manage or verify that access itself. + +## Out of Scope + +- Any endpoint or capability that mutates workspace files, Azure resources, or Foundry configuration; Cockpit is read-only by design. +- Real-time or continuously polling alerting; production telemetry is fetched on demand, not streamed or pushed. +- Triggering a new `agentops eval run` or `agentops doctor` invocation from within Cockpit itself. +- Embedding or proxying the full Foundry, Azure Monitor, or Application Insights UI; Cockpit only links out to those systems. +- Multi-user authentication, authorization, or remote-hosting concerns beyond local `--host`/`--port` binding. + +## Implementation Evidence + +- `src/agentops/agent/cockpit.py` - FastAPI route registrations: `GET /`, `GET /favicon.ico`, `GET /api/history`, `GET /api/eval-runs`, `GET /api/runs/{run_id}/report`, `GET /api/telemetry`, `GET /api/production`, `GET /api/production/html`, `GET /healthz` -- all `GET`-only, confirming no mutating endpoint exists. +- `src/agentops/agent/production_telemetry.py` - deferred production telemetry snapshot logic backing `/api/production` and `/api/production/html`. +- `src/agentops/cli/app.py` - `cockpit` command wiring `--host`, `--port`, `--workspace`, and `--no-preflight`. +- `src/agentops/services/preflight.py` - startup preflight checks shared by Cockpit and Doctor, skippable via `--no-preflight`. +- `tests/unit/test_cockpit.py` - unit coverage of Cockpit routes, history rendering, and fallback states. +- Additional CLI-level cockpit tests under `tests/unit/` covering the `cockpit` command surface and option handling. diff --git a/specs/008-foundry-operations-observability/checklists/requirements.md b/specs/008-foundry-operations-observability/checklists/requirements.md new file mode 100644 index 0000000..1a1bcbe --- /dev/null +++ b/specs/008-foundry-operations-observability/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Foundry Operations Observability + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/008-foundry-operations-observability/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against `src/agentops/services/dashboard.py`, the packaged `src/agentops/templates/workbooks/foundry-ops.workbook.json` and its KQL query assets, the Doctor posture rule `src/agentops/agent/checks/posture_rules/aoai_diagnostic_categories.py`, and `tests/unit/test_dashboard.py`, `test_cli_dashboard.py`, `test_agent_checks_observability.py`, and `test_agent_posture_rules.py`, not against a future proposal. +- Future or unrelated capabilities (alert-rule creation, a general telemetry monitor CLI family, automatic remediation, live trace streaming) are explicitly listed under "Out of Scope" and are not treated as gaps in this checklist. +- Revalidation pass corrected three factual issues against `dashboard.py`: (1) `--dry-run` was overstated as making zero Azure calls of any kind - corrected to describe it as resolving local/azd target metadata and emitting the ARM template, with no Azure resource created or modified; (2) RBAC preflight was overstated as always blocking on any missing prerequisite - corrected to block only on a conclusive missing-role finding and fail open with a warning otherwise; (3) missing diagnostic categories were conflated with RBAC-blocking - corrected to a non-fatal advisory in `dashboard deploy` (prints the fix command, still deploys), distinct from Doctor's separate posture-rule finding. Success Criteria were also rewritten to remove "automated test" language. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/008-foundry-operations-observability/spec.md b/specs/008-foundry-operations-observability/spec.md new file mode 100644 index 0000000..5208d85 --- /dev/null +++ b/specs/008-foundry-operations-observability/spec.md @@ -0,0 +1,134 @@ +# Feature Specification: Foundry Operations Observability + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing Foundry operations observability workbook and Doctor posture rule behavior as-built, not a new proposal. Sources reviewed include `src/agentops/services/dashboard.py`, `src/agentops/templates/workbooks/`, `src/agentops/agent/checks/posture_rules/aoai_diagnostic_categories.py`, `src/agentops/agent/cockpit.py`, `docs/foundry-ops-workbook*.md`, and associated tests. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Deploy a ready-made Azure OpenAI operations workbook (Priority: P1) + +An operator wants a single command to deploy a pre-built Azure Monitor Workbook (`foundry-ops`) covering Azure OpenAI capacity, traffic, latency, and error signals into their own subscription, without hand-authoring KQL queries or workbook JSON. + +**Why this priority**: Deploying the ready-made workbook is the primary deliverable; every other capability (opening, exporting, gating) depends on this workbook existing. + +**Independent Test**: Can be fully tested by running `agentops telemetry dashboard deploy` against a target subscription/resource group/workspace and confirming a workbook resource is created whose content matches the packaged `foundry-ops` template, with the expected parameters (subscription, workspace, Azure OpenAI resource, deployment, model, time range, streaming) applied. + +**Acceptance Scenarios**: + +1. **Given** a target subscription, resource group, Log Analytics workspace, and Azure OpenAI resource are supplied, **When** `agentops telemetry dashboard deploy` is run, **Then** an Azure Monitor Workbook resource is created or updated using the packaged `foundry-ops` template content. +2. **Given** the deployed workbook, **When** it is opened, **Then** it exposes parameters for subscription, workspace, Azure OpenAI resource, deployment, model, time range, and a streaming toggle. +3. **Given** the deployed workbook, **When** its sections are reviewed, **Then** it presents Capacity, Traffic and tokens, Latency, and Errors and throttling sections backed by the packaged KQL query assets (`capacity_ptu_spillover`, `traffic_tokens`, `latency_percentiles`, `errors_throttling`). +4. **Given** a target environment where RBAC preflight conclusively finds a missing required role, **When** `agentops telemetry dashboard deploy` is run, **Then** the command reports the specific missing permission and stops before deploying the workbook. +5. **Given** a target environment where the Azure OpenAI resource is missing one or more required diagnostic log categories, **When** `agentops telemetry dashboard deploy` is run, **Then** the command prints the exact command needed to enable the missing category as a non-fatal advisory and still deploys the workbook. + +--- + +### User Story 2 - Verify diagnostic settings are configured for observability (Priority: P2) + +An operator wants Doctor to flag when an Azure OpenAI resource's diagnostic settings are missing the log categories the `foundry-ops` workbook depends on, so the gap is caught before the workbook is deployed or relied upon. + +**Why this priority**: The workbook is only useful if the underlying diagnostic data exists; this check closes the loop between deployment and data availability, but is secondary to the deployment capability itself. + +**Independent Test**: Can be fully tested by running `agentops doctor` against a workspace whose discovered Azure OpenAI resource has diagnostic settings missing one or more required log categories, and confirming a finding for rule `waf.observability.aoai_diagnostic_categories` is reported naming the missing categories. + +**Acceptance Scenarios**: + +1. **Given** an Azure OpenAI resource with diagnostic settings missing a required log category, **When** `agentops doctor` runs, **Then** a finding for `waf.observability.aoai_diagnostic_categories` is reported identifying the missing category or categories. +2. **Given** an Azure OpenAI resource with all required diagnostic log categories enabled, **When** `agentops doctor` runs, **Then** no finding for that rule is reported. +3. **Given** a missing-category finding, **When** its detail is inspected, **Then** it includes a runnable command suggestion for enabling the missing diagnostic category. + +--- + +### User Story 3 - Open or export the workbook definition without leaving the CLI (Priority: P3) + +An operator wants to open the deployed workbook directly in the Azure portal from the command line, or export the workbook's JSON definition locally (for example, to version it or adapt it), without needing to locate it manually in the portal. + +**Why this priority**: Convenience access to an already-deployed or packaged workbook is useful but secondary to deployment and diagnostic verification. + +**Independent Test**: Can be fully tested by running `agentops telemetry dashboard open --print-url` and confirming a portal URL for the deployed workbook resource is produced, and by running `agentops telemetry dashboard export --out ` and confirming the packaged workbook JSON is written to that path. + +**Acceptance Scenarios**: + +1. **Given** a previously deployed workbook, **When** `agentops telemetry dashboard open --print-url` is run, **Then** a valid Azure portal URL referencing that workbook resource is printed. +2. **Given** no prior deployment, **When** `agentops telemetry dashboard export --out ` is run, **Then** the packaged `foundry-ops` workbook JSON is written to the given path unchanged. +3. **Given** the Cockpit dashboard is rendered, **When** a workbook has been deployed for the workspace, **Then** Cockpit surfaces a tile or link to that workbook (see Read-Only Cockpit specification). + +--- + +### Edge Cases + +- What happens when `agentops telemetry dashboard deploy` is run without the Azure CLI (`az`) available? The command MUST fail with a clear error identifying the missing prerequisite rather than a raw stack trace. +- What happens when the target resource group already contains a workbook with the same name? The deploy command MUST update the existing resource using an idempotent ARM template rather than creating a duplicate. +- What happens when the caller lacks RBAC permission to read the Azure OpenAI resource's diagnostic settings and this can be conclusively determined? The `check_rbac` preflight MUST report the specific missing role/permission and MUST block deployment. +- What happens when RBAC status cannot be conclusively determined (for example, the identity-listing call itself fails)? The preflight MUST fail open with a warning rather than blocking deployment, since ARM's own authorization check still applies at deploy time. +- What happens when `--dry-run` is passed to `deploy`? The command MUST resolve the target's local/azd metadata and emit the same ARM template that would be deployed, without making any Azure API call that creates or modifies a resource. +- What happens when the Doctor posture rule cannot discover any Azure OpenAI resource in scope? The rule MUST report an appropriately scoped outcome (no applicable resource) rather than a false-positive missing-category finding. +- What happens when one or more required diagnostic log categories are missing at deploy time? `dashboard deploy` MUST treat this as a non-fatal advisory: it prints the exact command to enable the missing category and still completes the deployment, rather than refusing to proceed. +- What happens when `dashboard export` is run with no `--out` value? The command MUST fall back to a documented default output path. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST package a `foundry-ops` Azure Monitor Workbook template covering Capacity, Traffic and tokens, Latency, and Errors and throttling sections for Azure OpenAI resources. +- **FR-002**: The packaged workbook MUST expose parameters for subscription, Log Analytics workspace, Azure OpenAI resource, deployment, model, time range, and a streaming toggle. +- **FR-003**: The system MUST provide `agentops telemetry dashboard deploy` to create or update the workbook resource in a target subscription/resource group using an idempotent ARM template built from the packaged workbook content. +- **FR-004**: The system MUST perform an RBAC preflight check before deployment and MUST block deployment with a specific missing-permission error only when the check conclusively finds a missing required role; when the check cannot conclusively determine RBAC status, it MUST fail open with a warning rather than blocking. +- **FR-005**: The system MUST perform a diagnostic-settings preflight check before deployment and MUST treat a missing required diagnostic log category as a non-fatal advisory - printing the exact command to enable it while still attempting the deployment. +- **FR-006**: The system MUST provide a `--dry-run` mode for `dashboard deploy` that resolves target metadata and emits the intended ARM template without creating or modifying any Azure resource. +- **FR-007**: The system MUST provide `agentops telemetry dashboard open` to produce a portal URL for the deployed workbook, including a `--print-url` mode. +- **FR-008**: The system MUST provide `agentops telemetry dashboard export` to write the packaged workbook JSON to a local path unchanged, with a documented default path when `--out` is omitted. +- **FR-009**: The system MUST implement a Doctor posture rule (`waf.observability.aoai_diagnostic_categories`) that reports a finding identifying any Azure OpenAI diagnostic log category required by the workbook queries that is not enabled. +- **FR-010**: The Doctor posture rule MUST include an actionable fix-command suggestion in its finding detail for enabling a missing diagnostic category. +- **FR-011**: The Cockpit dashboard view MUST surface a tile or link to a deployed `foundry-ops` workbook when one exists for the workspace, consistent with the Read-Only Cockpit specification. + +### Key Entities + +- **DashboardTarget**: The resolved subscription, resource group, workspace, and Azure OpenAI resource identifiers used for workbook deployment. +- **RBAC Preflight Result**: The conclusive allowed/blocked or inconclusive outcome of the workbook-writer role check; only a conclusive missing-role result blocks before deployment. +- **Diagnostic-Settings Advisory**: The non-blocking result of inspecting required Azure OpenAI diagnostic categories, including the exact remediation command when a category is absent. +- **Workbook Template**: The packaged `foundry-ops.workbook.json` content defining the workbook's parameters and sections. +- **KQL Query Asset**: One of the packaged query files (`capacity_ptu_spillover`, `traffic_tokens`, `latency_percentiles`, `errors_throttling`) backing a workbook section. +- **Posture Finding**: A Doctor finding produced by the `waf.observability.aoai_diagnostic_categories` rule identifying a missing diagnostic log category. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: The ARM template produced by `dashboard deploy` (including its `--dry-run` mode) always embeds the exact packaged `foundry-ops` workbook content - the rendered workbook a user sees always matches the packaged template. +- **SC-002**: When the RBAC preflight conclusively finds a missing required role, deployment always stops before any workbook is created or updated; when RBAC status cannot be conclusively determined, deployment always proceeds with a warning rather than blocking. +- **SC-003**: The Doctor posture rule reports a finding naming the missing category whenever an Azure OpenAI resource's diagnostic settings omit a required category, and reports no finding when all required categories are present - the two outcomes never overlap for the same resource state. +- **SC-004**: `dashboard export` always writes the packaged workbook JSON unchanged to the requested path, or to the documented default path when `--out` is omitted. +- **SC-005**: `dashboard open --print-url` always produces a well-formed Azure portal URL referencing the deployed workbook resource. +- **SC-006**: When deployment is otherwise permitted, a missing required diagnostic category does not block the workbook deployment attempt and the exact remediation command is printed; diagnostic-category absence alone never produces a blocking preflight result. +- **SC-007**: `dashboard deploy --dry-run` never creates or modifies an Azure resource, regardless of RBAC or diagnostic-category state. + +## Assumptions + +- The operator has (or can obtain) the Azure RBAC permissions the preflight check identifies as required; this feature surfaces missing permissions but does not grant them. +- The packaged KQL queries assume Azure OpenAI diagnostic logs are (or will be) sent to the target Log Analytics workspace; the workbook itself does not configure diagnostic settings, only reports on whether they are correctly enabled (via the Doctor posture rule). +- This feature is scoped to Azure OpenAI operational telemetry; it does not cover telemetry for other Foundry model types beyond what the packaged queries already target. + +## Out of Scope + +- Creating Azure Monitor alert rules or action groups; the workbook and CLI surface visualization and readiness signals only, not alerting. +- A general-purpose `agentops telemetry monitor setup|show|configure` command family; only `dashboard deploy|open|export` are implemented. +- Automatically remediating a missing diagnostic category; the posture rule surfaces a fix-command suggestion but does not execute it. +- Live or streaming trace visualization; the workbook renders Azure Monitor Log Analytics query results, not a live trace feed. +- Deployment or observability support for non-Azure-OpenAI Foundry model types beyond what the packaged `foundry-ops` queries already cover. + +## Implementation Evidence + +- `src/agentops/services/dashboard.py` - `DashboardTarget`, `PreflightResult`, `load_workbook_template()`, `load_workbook_content()`, `build_arm_template()`, `deploy_workbook()`, `check_rbac()`, `missing_diagnostic_categories()`, `build_diagnostic_settings_command()`, `build_workbook_portal_url()`, `discover_target()`. +- `src/agentops/templates/workbooks/foundry-ops.workbook.json` - packaged workbook definition with Capacity, Traffic and tokens, Latency, and Errors and throttling sections and parameters for subscription/workspace/resource/deployment/model/time range/streaming. +- `src/agentops/templates/workbooks/queries/capacity_ptu_spillover.kql`, `traffic_tokens.kql`, `latency_percentiles.kql`, `errors_throttling.kql` - packaged KQL query assets backing the workbook sections. +- `src/agentops/agent/checks/posture_rules/aoai_diagnostic_categories.py` - `RULE_ID = "waf.observability.aoai_diagnostic_categories"`, `evaluate()` producing findings for missing diagnostic categories, and `_fix_command()` producing the actionable remediation suggestion. +- `src/agentops/agent/cockpit.py` - dashboard/workbook tile rendering in the Cockpit view. +- `src/agentops/cli/app.py` - `telemetry dashboard deploy|open|export` command wiring, including `--dry-run` and `--print-url`. +- `tests/unit/test_dashboard.py`, `test_cli_dashboard.py`, `test_agent_checks_observability.py`, `test_agent_posture_rules.py` - unit coverage of workbook deployment, preflight checks, and the diagnostic-categories posture rule. +- `docs/foundry-ops-workbook.md` and related `docs/foundry-ops-workbook*.md` - narrative documentation of the workbook's sections and deployment flow consistent with this baseline. diff --git a/specs/009-trace-to-regression/checklists/requirements.md b/specs/009-trace-to-regression/checklists/requirements.md new file mode 100644 index 0000000..e3665fd --- /dev/null +++ b/specs/009-trace-to-regression/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Trace-to-Regression Promotion + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-12 +**Feature**: [specs/009-trace-to-regression/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is a retrospective baseline: every checklist item was checked against `src/agentops/services/trace_promotion.py`, the `eval promote-traces` command in `src/agentops/cli/app.py`, the `_trace_dataset_status`/`_add_trace_dataset_check` consumers in `src/agentops/services/evidence_pack.py`, the `_read_trace_regression_manifest` consumer in `src/agentops/agent/cockpit.py`, and `tests/unit/test_trace_promotion.py`, not against a future proposal. +- Future or unrelated capabilities (trace export itself, live/streaming ingestion, automatic labeling, bidirectional sync, scheduled or event-triggered promotion) are explicitly listed under "Out of Scope" and are not treated as gaps in this checklist. +- Revalidation pass corrected five factual issues against `trace_promotion.py`: (1) de-duplication was overstated as persisting across repeated `--apply` runs - corrected to an in-memory, single-run-only `seen` set, since `_write_trace_dataset` overwrites rather than merges with any existing output; (2) the manifest was described as per-row lineage - corrected to describe `_lineage_from_rows`'s aggregate arrays/counts (trace_ids, replay_urls, evaluation_urls, source systems, agents, agent versions, sampling policies, multi-turn-row count); (3) added an explicit statement that `_trace_to_row` sets `metadata.needs_review: true` unconditionally in both label modes; (4) the preview was overstated as rendering every field of every row - corrected to the actual up-to-3-sample-plus-summary rendering; (5) invalid JSON (which aborts the whole load) and well-formed-but-unusable records (which are skipped) are now kept as two distinct, separately named failure modes. Success Criteria were also rewritten to remove "automated test" language. +- All items pass; no unresolved issues remain after this revalidation pass. diff --git a/specs/009-trace-to-regression/spec.md b/specs/009-trace-to-regression/spec.md new file mode 100644 index 0000000..21e8f2a --- /dev/null +++ b/specs/009-trace-to-regression/spec.md @@ -0,0 +1,126 @@ +# Feature Specification: Trace-to-Regression Promotion + +**Feature Branch**: `placerda-spec-kit-feature` + +**Created**: 2026-08-12 + +**Status**: Implemented Baseline + +**Input**: This specification was reverse-engineered from the current implementation, tests, and public documentation of AgentOps Toolkit. It documents the existing `agentops eval promote-traces` behavior as-built, not a new proposal. Sources reviewed include `src/agentops/services/trace_promotion.py`, the `promote-traces` command in `src/agentops/cli/app.py`, `src/agentops/services/evidence_pack.py`, `src/agentops/agent/cockpit.py`, and `tests/unit/test_trace_promotion.py`. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Preview candidate dataset rows from a production trace export before writing anything (Priority: P1) + +An operator has a JSON or JSONL export of production traces and wants to preview what a promoted regression-dataset candidate would look like -- how many rows, what fields, what labels -- before any file is written or modified. + +**Why this priority**: A safe, review-first preview is the foundation of the whole feature; every other capability (applying, labeling, deduplication, lineage) only matters once a trustworthy preview exists. + +**Independent Test**: Can be fully tested by running `agentops eval promote-traces --source ` without `--apply` and confirming a preview of candidate rows is printed and that no dataset or manifest file is created or modified as a result. + +**Acceptance Scenarios**: + +1. **Given** a JSON or JSONL trace export file, **When** `agentops eval promote-traces --source ` is run without `--apply`, **Then** a preview of the candidate dataset rows is displayed and no output file is written. +2. **Given** the same export, **When** the preview is inspected, **Then** it shows a truncated sample of up to 3 candidate rows' input text alongside a summary of the total candidate count, skipped-record count, and any warnings, so the operator can judge quality before applying without needing every row rendered. +3. **Given** a `--max-rows` value smaller than the number of traces in the export, **When** the preview is generated, **Then** it contains at most that many rows. + +--- + +### User Story 2 - Apply a reviewed promotion to create a candidate dataset with lineage (Priority: P1) + +An operator who has reviewed a preview and is satisfied wants to run the same command with `--apply` to actually write the candidate dataset file and an accompanying manifest recording aggregate lineage for the run. + +**Why this priority**: Writing the reviewed output with traceable lineage is what turns a preview into a usable dataset candidate for future eval runs, Doctor, Cockpit, and release evidence. + +**Independent Test**: Can be fully tested by running `agentops eval promote-traces --source --apply --out ` and confirming the candidate dataset file and a `trace-regression-manifest.json` are written, with the manifest recording aggregate lineage information (for example, distinct source trace identifiers) for the run as a whole. + +**Acceptance Scenarios**: + +1. **Given** a reviewed trace export, **When** `agentops eval promote-traces --source --apply --out ` is run, **Then** the candidate dataset file is written to `` and a `trace-regression-manifest.json` is written alongside it. +2. **Given** the written manifest, **When** it is inspected, **Then** it records aggregate lineage available across the promoted rows - such as distinct source trace identifiers, replay URLs, evaluation URLs, source systems, agents, agent versions, and sampling policies when those values exist - plus a count of multi-turn rows, rather than a per-row source mapping. +3. **Given** the same export is promoted twice with `--apply` targeting the same `--out` path, **When** the second run completes, **Then** duplicate rows within that second run's own candidate set are not written twice, but the written output reflects only that run's candidates - the second run's write overwrites the first run's output file rather than merging or deduplicating against it. + +--- + +### User Story 3 - Choose a labeling mode appropriate to review maturity (Priority: P2) + +An operator wants to choose between `self-similarity` labeling (storing the production response as the `expected` value, to catch future behavior drift against a known production answer) and `pending` labeling (leaving `expected` blank, marking rows as awaiting human review), depending on how much manual review has already happened. + +**Why this priority**: Label-mode choice affects how the resulting dataset can be used (drift detection versus a to-be-reviewed backlog), but it is a refinement on top of the core preview/apply flow. + +**Independent Test**: Can be fully tested by running promotion once with `--label-mode self-similarity` and once with `--label-mode pending` against the same export, and confirming the `expected` field is populated from the production response in the first case and left blank with a pending marker in the second. + +**Acceptance Scenarios**: + +1. **Given** `--label-mode self-similarity`, **When** promotion runs, **Then** each candidate row's `expected` field is populated with the corresponding production response text, and the row's metadata still marks it as needing review. +2. **Given** `--label-mode pending`, **When** promotion runs, **Then** each candidate row's `expected` field is left blank and the output communicates that rows are pending human labeling before being used as a blocking gate. +3. **Given** either label mode, **When** a candidate row is produced, **Then** its metadata always marks `needs_review` as true - `self-similarity` labeling never marks a row as already reviewed. +4. **Given** an unsupported `--label-mode` value, **When** promotion is invoked, **Then** the command rejects it with a validation error rather than silently defaulting to a different mode. + +--- + +### Edge Cases + +- What happens when `--max-rows` is zero or negative? The command MUST reject it with a validation error rather than silently promoting zero or a default number of rows. +- What happens when a trace record's JSON cannot be parsed at all (invalid JSON on a line)? The command MUST abort the entire load with a clear error rather than silently skipping the malformed line. +- What happens when a trace record parses as valid JSON but lacks a usable input or response? That record MUST be skipped (not counted as a candidate row) rather than causing the entire promotion to fail. +- What happens when multiple source records produce the same candidate `(input, expected)` pair? The de-duplication key MUST prevent that pair from producing duplicate candidate rows within that one promotion run, even if the source records have different trace identifiers. +- What happens when the same export (or an overlapping export) is promoted with `--apply` more than once against the same `--out` path? De-duplication applies only within each individual run's own candidate set; the write MUST overwrite the previous output file rather than merging with or deduplicating against rows already written by an earlier run. +- What happens when `--apply` is used without `--out`? The command MUST fall back to a documented default output location rather than requiring the flag. +- What happens when self-similarity-labeled rows are later consumed by an eval run? The dataset and its documentation MUST make clear that a self-similarity label only supports drift detection against a prior production response, not a verified-correct answer, and every row (in either label mode) MUST still be marked as needing review. +- How does release evidence or Cockpit reflect a workspace that has never run promotion? The corresponding check MUST report the absence of a promotion manifest rather than erroring (see Release Evidence specification). + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST provide `agentops eval promote-traces --source ` to read a JSON or JSONL production trace export and derive candidate regression-dataset rows from it. +- **FR-002**: The system MUST default to a preview-only mode that displays a summary and a truncated sample of at most three candidate inputs without writing any file, requiring an explicit `--apply` flag to write output. +- **FR-003**: The system MUST support `--max-rows ` to cap the number of candidate rows produced, and MUST reject a non-positive value with a validation error. +- **FR-004**: The system MUST support `--label-mode self-similarity|pending`, where `self-similarity` populates each row's `expected` field from the production response and `pending` leaves it blank pending human review, and MUST reject any other value. +- **FR-005**: The system MUST mark every candidate row's metadata as needing review (`needs_review: true`) regardless of which label mode produced it - `self-similarity` labeling never marks a row as already reviewed. +- **FR-006**: When `--apply` is used, the system MUST write the candidate dataset to the requested (or default) `--out` path and MUST also write an accompanying `trace-regression-manifest.json` recording aggregate lineage available for the run (distinct source trace identifiers, replay URLs, evaluation URLs, source systems, agents, agent versions, sampling policies, and a multi-turn-row count when those values exist), rather than a per-row source mapping. +- **FR-007**: The system MUST de-duplicate candidate rows within a single promotion invocation using the `(input, expected)` pair. This de-duplication does not extend across separate invocations: each `--apply` run's write to `--out` overwrites any prior output rather than merging with or deduplicating against rows from an earlier run. +- **FR-008**: The system MUST abort the entire promotion with a clear error when a trace-export line cannot be parsed as valid JSON, and MUST separately skip (without aborting) any well-formed record that lacks a usable input or response. +- **FR-009**: The system MUST clearly communicate, in both `pending`-mode output and accompanying documentation, that promoted rows always require human review before being trusted as a blocking regression gate, and that `self-similarity` labels support drift detection rather than verified correctness. +- **FR-010**: The manifest written on `--apply` MUST be consumable by the Release Evidence and Cockpit features to report the state of the trace-to-dataset promotion flywheel from its aggregate lineage fields (see the Release Evidence and Read-Only Cockpit specifications) without those features re-implementing promotion logic themselves. + +### Key Entities + +- **TracePromotionPreview**: The in-memory result of a promotion run, containing candidate rows, the resolved output path, and the resolved manifest path, used both for preview rendering and for `--apply` writing. +- **Candidate Row**: A single derived dataset row (input, expected/context/tool fields per the current label mode) produced from one source trace, always carrying `metadata.needs_review: true` regardless of label mode. +- **LabelMode**: One of `self-similarity` or `pending`, controlling how the `expected` field is populated. +- **trace-regression-manifest.json**: The lineage manifest written alongside the candidate dataset on `--apply`, recording aggregate lineage values available for the whole run (distinct trace identifiers, replay URLs, evaluation URLs, source systems, agents, agent versions, sampling policies, and a multi-turn-row count) - not a per-row source mapping - and overwritten (not merged) on each `--apply` invocation. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Running promotion without `--apply` always displays a preview (a sample of up to 3 candidate rows' input text plus a summary) and never creates or modifies a dataset or manifest file. +- **SC-002**: Running promotion with `--apply` writes both the candidate dataset and `trace-regression-manifest.json`, and the manifest aggregates every trace identifier, agent, system, URL, version, and sampling policy that is available in that run's candidate metadata without inventing absent lineage values. +- **SC-003**: `self-similarity` mode always populates `expected` from the production response and `pending` mode always leaves it blank; in both modes, every candidate row is always marked as needing review. +- **SC-004**: Within a single promotion run, no two candidate rows share the same `(input, expected)` pair; across separate `--apply` invocations targeting the same output path, each run's write reflects only that run's own candidates, never a merge with a prior run's output. +- **SC-005**: A non-positive `--max-rows` value and an unsupported `--label-mode` value are always rejected with a validation error before any row is processed. +- **SC-006**: An invalid JSON line in the trace export always aborts the entire load with a clear error, while a well-formed record lacking usable input/response is always skipped without aborting the run - the two failure modes are never conflated. + +## Assumptions + +- The trace export consumed by `promote-traces` is assumed to already exist (produced by an external export mechanism); this feature does not perform the export itself. +- Operators are expected to review promoted rows -- especially `pending`-mode rows -- before relying on the resulting dataset as a release-blocking eval gate. +- The de-duplication key is the candidate's `(input, expected)` pair and is scoped to a single in-memory promotion run only; it is not persisted or checked against a previously written output file, so operators who run `--apply` repeatedly against overlapping exports are expected to treat each run's output as authoritative and complete for that run, not as an incremental merge. + +## Out of Scope + +- Exporting traces from a running agent or Application Insights instance; this feature only consumes an already-produced export file. +- Live or streaming trace ingestion; promotion operates on a static export file per invocation. +- Automatic labeling or correctness judgment beyond the `self-similarity` (prior production response) and `pending` (blank) modes; no model-graded labeling is performed. +- Bidirectional synchronization between the promoted dataset and the original production trace source. +- Triggering promotion automatically (for example, on a schedule or CI event); the command is invoked explicitly by an operator. + +## Implementation Evidence + +- `src/agentops/services/trace_promotion.py` - `TracePromotionPreview`, `promote_traces()` (with `max_rows`, `apply`, and `label_mode` parameters and validation), `LabelMode = Literal["self-similarity", "pending"]`, `_trace_to_row()` populating `expected` conditionally on label mode, `_write_trace_dataset()` and the `trace-regression-manifest.json` output path, `_lineage_from_rows()` for manifest lineage, and the `seen` de-duplication set keyed per trace. +- `src/agentops/cli/app.py` - `eval promote-traces` command wiring `--source`, `--out`, `--max-rows`, `--label-mode`, and `--apply`. +- `src/agentops/services/evidence_pack.py` (`_trace_dataset_status`, `_add_trace_dataset_check`) - consumption of the promotion manifest for the "Trace-to-dataset flywheel" release-evidence check. +- `src/agentops/agent/cockpit.py` - Cockpit surfacing of trace-to-dataset promotion state consistent with the Read-Only Cockpit specification. +- `tests/unit/test_trace_promotion.py` - unit coverage of preview rendering, `--apply` writing, label-mode behavior, de-duplication, and validation errors.