docs: document implemented AgentOps feature baselines

specs/001-evaluation-configuration/checklists/requirements.md

@@ -0,0 +1,38 @@
+# Specification Quality Checklist: Evaluation Configuration
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-08-12
+**Feature**: [specs/001-evaluation-configuration/spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/core/agentops_config.py`, `config_loader.py`, and `evaluators.py`, and against `tests/unit/test_agentops_config.py` and `tests/unit/test_evaluators.py`, not against a future proposal.
+- Revalidation pass: corrected the spec to match source exactly on three boundaries that were previously conflated - (1) configuration loading (`load_agentops_config`) validates only YAML/schema fields and never opens the dataset file; (2) dataset existence/emptiness is checked later by `detect_dataset_shape` during evaluation preparation, which raises on a missing or empty dataset; (3) a row's missing `input` field is caught only at per-row invocation time (`invocations.py`), not by config loading or dataset-shape detection. Also broadened the `execution: cloud` restriction from "Foundry `name:version` prompt agent only" to "Foundry prompt or Foundry hosted agent with a derivable name/version," and clarified that this restriction is enforced at evaluation-run time, not configuration-load time (only `execution: azd` is rejected by the config model itself).
+- Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (classification correctness, rejection timing/ordering, evaluator-set stability, threshold isolation) instead of referring to automated test coverage.
+- Future or unrelated capabilities (new evaluator types, multi-file configuration composition, execution/network behavior) are explicitly listed under "Out of Scope" in the spec and are not treated as gaps in this checklist.
+- All items pass; no unresolved issues remain after this revalidation pass.

specs/002-evaluation-execution/checklists/requirements.md

@@ -0,0 +1,38 @@
+# Specification Quality Checklist: Evaluation Execution
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-08-12
+**Feature**: [specs/002-evaluation-execution/spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- This is a retrospective baseline: every checklist item was checked against the current source in `src/agentops/pipeline/orchestrator.py`, `runtime.py`, `invocations.py`, `cloud_runner.py`, `cloud_results.py`, `publisher.py`, `azd_runner.py`, and `src/agentops/core/azd_eval.py`, and against `tests/unit/test_invocations.py`, `test_cloud_runner.py`, `test_pipeline_publisher.py`, `test_azd_runner.py`, and `tests/integration/test_pipeline_smoke.py`, not against a future proposal.
+- Revalidation pass: corrected two factual errors against `orchestrator.py`. First, `detect_dataset_shape()` is called unconditionally before both local and cloud execution and raises on an empty dataset, so a zero-row dataset is rejected before any row is invoked - it does not complete with "zero rows evaluated" as the spec previously claimed. Second, `_run_evaluation_cloud()` accepts both `foundry_prompt` (`name:version`) and `foundry_hosted` (URL with a derivable name/version) target kinds, not only the `name:version` form, so the cloud-execution restriction was broadened accordingly.
+- Success Criteria were rewritten to describe measurable, technology-agnostic outcomes (result-shape equivalence, rejection ordering, isolation guarantees, artifact durability) instead of referring to automated test coverage.
+- Future or unrelated capabilities (a standalone `agentops eval compare` command, results/regression schema details, and CI/CD or Doctor consumption of results) are explicitly listed under "Out of Scope" and are not treated as gaps in this checklist.
+- All items pass; no unresolved issues remain after this revalidation pass.