diff --git a/Docs/ADR_Skill_Evaluation_Pipeline_and_Harbor_Execution_Strategy.txt b/Docs/ADR_Skill_Evaluation_Pipeline_and_Harbor_Execution_Strategy.txt index 3e6fdcd2..58c42630 100644 --- a/Docs/ADR_Skill_Evaluation_Pipeline_and_Harbor_Execution_Strategy.txt +++ b/Docs/ADR_Skill_Evaluation_Pipeline_and_Harbor_Execution_Strategy.txt @@ -10,7 +10,7 @@ The objective is to establish an automated pipeline for evaluating skill submiss 2. Architectural Details (Skill Evaluation Pipeline Workflow) The pipeline is an 8-step process orchestrated by Tekton. -Step 1 — Submission +Step 1 -- Submission * Trigger: Git + Tekton EventListener. A user pushes a skill directory to a branch in a dedicated repository (e.g., skill-eval-submissions). * Submission Contract: The directory follows this structure: my-skill-name/ @@ -23,7 +23,7 @@ Step 1 — Submission ├── supportive/ # optional: mock MCPs, data files └── metadata.yaml # user-provided: description, persona, etc. * Tools: GitHub, Tekton Triggers (EventListener + TriggerTemplate), and Tekton Interceptors (to filter on pushes to submissions/ path). -Step 2 — Validation (Tekton Task) +Step 2 -- Validation (Tekton Task) A container runs validation checks on the submitted files: @@ -33,7 +33,7 @@ A container runs validation checks on the submitted files: * Validate metadata.yaml schema (pydantic or jsonschema). * Check supportive files are a reasonable size (<50MB total). * Tools: Custom Tekton Task with a Python image, using ubi9/python-311 as the base image. -Step 3 — Scaffold (Tekton Task) +Step 3 -- Scaffold (Tekton Task) Generates the files the user shouldn't have to write, including a templated Dockerfile (based on supportive files) and creating needed files like test.sh and task.toml. @@ -41,13 +41,13 @@ Generates the files the user shouldn't have to write, including a templated Dock * A skilled variant (tasks/my-skill-name/) with the Dockerfile configured to COPY skills, docs. * An unskilled variant (tasks-no-skills/my-skill-name/) with a Dockerfile that excludes skills/docs but includes supportive infra. * Tools: Custom Tekton Task with Python/Jinja2 for templating. Store Dockerfile/test.sh templates in a ConfigMap or in the pipeline repo. -Step 4 & 5 — Build & Push Images (Tekton Tasks) +Step 4 & 5 -- Build & Push Images (Tekton Tasks) Build both the skilled and unskilled container images and push them to a registry. * Tools: docker buildx or Buildah. * Registry Options: Quay.io. -Step 6 — Run Harbor Evaluation (Tekton Task) +Step 6 -- Run Harbor Evaluation (Tekton Task) This is the core evaluation step. Harbor must run N attempts per variant (skilled + unskilled), where each attempt spins up a container from the pre-built image, runs the agent and the verifier inside it, and aggregates results. There are two viable approaches for how Harbor executes these trials on OpenShift. @@ -61,9 +61,9 @@ Approach Requires privileged access or nested containers to function; introduces security concerns. OpenShift Backend for Harbor Harbor uses the Kubernetes API to create, manage, and clean up lightweight, unprivileged Pods for each trial, replacing Google-specific calls[c][d]. - Creates lightweight, regular Pods—no nesting, no privileges; standard RBAC; OpenShift can see, schedule, and resource-limit each trial Pod independently; trial Pod logs are visible in the OpenShift Console like any other workload. + Creates lightweight, regular Pods--no nesting, no privileges; standard RBAC; OpenShift can see, schedule, and resource-limit each trial Pod independently; trial Pod logs are visible in the OpenShift Console like any other workload. Requires writing one new file in Harbor (the OpenShift environment backend) to replace the three Google-specific functions. - Step 7 — Analyze & Report (Tekton Task) + Step 7 -- Analyze & Report (Tekton Task) Runs existing analysis and plot scripts to generate a summary report (markdown or HTML) that includes: @@ -72,7 +72,7 @@ Runs existing analysis and plot scripts to generate a summary report (markdown o * Statistical significance (p-value). * Heatmap figures embedded. * Links to detailed logs. -Step 8 — Publish & Store (Tekton Task) +Step 8 -- Publish & Store (Tekton Task) Finalizes the publishing and storage of evaluation results. @@ -92,7 +92,7 @@ The consequences and required dependencies depend on the chosen approach: * Cons: Requires privileged access or nested containers to function; introduces security concerns. * Dependencies: Podman, OpenShift Secrets for credentials/volume mounting. * If adopting "OpenShift Backend for Harbor": - * Pros: Creates lightweight, regular Pods—no nesting, no privileges; utilizes standard RBAC; individual trial Pods are fully visible, schedulable, and resource-limited by OpenShift. + * Pros: Creates lightweight, regular Pods--no nesting, no privileges; utilizes standard RBAC; individual trial Pods are fully visible, schedulable, and resource-limited by OpenShift. * Cons: Requires writing one new file in Harbor (the OpenShift environment backend) to replace three Google-specific functions. * Dependencies: Custom Harbor Backend implementation (a new file). 4. Operational and Security Considerations @@ -119,7 +119,7 @@ The consequences and required dependencies depend on the chosen approach: Notes and Comments Following the ADR meeting, here is a concise summary of the Tessl investigation: * Harbor (The Local Tool): Harbor is an open-source CLI that lets you run and evaluate agents locally using Podman containers and "behind the scenes" Tessl and Harbor run the same backend (by Tessl). -* Tessl Cloud (The Automated Tool): This cloud service creates test scenarios and runs your agent twice—once as a baseline and once with context—to show you the performance difference (the "delta") as with Harbor. +* Tessl Cloud (The Automated Tool): This cloud service creates test scenarios and runs your agent twice--once as a baseline and once with context--to show you the performance difference (the "delta") as with Harbor. * No Local Tessl: There is no local version of Tessl, it functions as a cloud wrapper where the CLI handles the connection, but the server does all the work. * Infrastructure Limits: You cannot self-host Tessl or use your own cloud infrastructure for these runs. * Model Restrictions: The platform currently only supports Claude models (via api key so it rules out as i understand the vertex solution we use) and does not allow you to bring outside models like Qwen. @@ -145,9 +145,9 @@ continuous performance monitoring From the Harbor GKE backend : -_init_client — authenticates to the cluster using gcloud -_build_and_push_image — builds and pushes the container image using gcloud -_image_exists — checks if the image already exists in the registry using gcloud +_init_client -- authenticates to the cluster using gcloud +_build_and_push_image -- builds and pushes the container image using gcloud +_image_exists -- checks if the image already exists in the registry using gcloud Everything else from what i saw is ok and no need to fix. diff --git a/Docs/SOTA_demands.md b/Docs/SOTA_demands.md index 021e379f..2534d547 100644 --- a/Docs/SOTA_demands.md +++ b/Docs/SOTA_demands.md @@ -1,6 +1,6 @@ # SOTA Demands: CI and Monitoring Pipeline -Assessment of ABEvalFlow's CI (`abevalflow-pipeline`) and monitoring (`abevalflow-monitoring-pipeline`) against state-of-the-art expectations for agent, tool, skill, and MCP evaluation platforms. +Assessment of Agentic Eval Flow's CI (`abevalflow-pipeline`) and monitoring (`abevalflow-monitoring-pipeline`) against state-of-the-art expectations for agent, tool, skill, and MCP evaluation platforms. **Scope:** Gap analysis and recommended roadmap. Does not replace existing ADRs or implementation plans. @@ -8,7 +8,7 @@ Assessment of ABEvalFlow's CI (`abevalflow-pipeline`) and monitoring (`abevalflo ## Executive Summary -ABEvalFlow is **SOTA on breadth**: multi-engine evaluation (Harbor, ASE, MCPChecker, A2A) on OpenShift with persistence, statistical A/B analysis, and a separate monitoring path. +Agentic Eval Flow is **SOTA on breadth**: multi-engine evaluation (Harbor, ASE, MCPChecker, A2A) on OpenShift with persistence, statistical A/B analysis, and a separate monitoring path. It is **not yet SOTA on enforcement, observability depth, or lifecycle closure**: @@ -92,9 +92,9 @@ This combination is ahead of most approaches that only lint or unit-test skill f | **Test phase** | Skipped in monitoring pipeline (no security re-scan on canary runs) | | **Degradation logic** | Last-run vs previous-run ratio (`monitor.py`, default 0.85) | | **Statistics** | No CUSUM, rolling baseline, or seasonality (noted as post-MVP in APPENG-4911) | -| **Schedule** | Cron every 10 days — coarse | +| **Schedule** | Cron every 10 days -- coarse | | **Triggers** | No webhook on model upgrade, prompt change, or agent deploy | -| **Canary config** | ConfigMap — not a versioned baseline registry or dedicated canary repo | +| **Canary config** | ConfigMap -- not a versioned baseline registry or dedicated canary repo | | **CI linkage** | CI has `enable-degradation-check: false`; monitoring and CI use disconnected policies | **SOTA expectation:** Event-driven canaries (model/prompt/infra change) + statistical process control + fleet dashboards + auto-ticket on regression. @@ -105,16 +105,16 @@ This combination is ahead of most approaches that only lint or unit-test skill f | Gap | Detail | |-----|--------| -| **Static security only** | Cisco scan on skill content — no dynamic red-team or adversarial eval during agent runs | +| **Static security only** | Cisco scan on skill content -- no dynamic red-team or adversarial eval during agent runs | | **No skills quality linter in CI** | skillsaw-style structural lint not integrated | | **Snyk / dependency scan** | Designed (APPENG-5305) but not wired like Cisco in the current test phase | | **Single model per run** | No matrix (model × agent × engine) | | **Flake handling** | No pass@k reporting, retry-aware stats, or infra-error exclusion in marketplace gate | | **Multi-task benchmarks** | One submission ≈ one task; no curated benchmark suite execution | | **MCPChecker bypasses test phase** | MCP submissions skip security and quality gates | -| **A2A pass criteria** | Heuristic threshold — not comparable to Harbor A/B semantics | +| **A2A pass criteria** | Heuristic threshold -- not comparable to Harbor A/B semantics | -**SOTA expectation:** Layered eval — lint → static security → functional eval → adversarial probe → monitoring canary. +**SOTA expectation:** Layered eval -- lint → static security → functional eval → adversarial probe → monitoring canary. --- @@ -145,28 +145,28 @@ The pipeline narrative includes GEPA, SkillOps, and skillberry-style authoring, ## Recommended Roadmap -### P0 — Marketplace Trust (3–6 weeks) +### P0 -- Marketplace Trust (3–6 weeks) -1. **Enforced gate matrix** — Skip `evaluate` when `tests-passed=false`. Add `marketplace-mode` with stricter thresholds (uplift + p-value + security block). -2. **GitHub commit status / required check** — Not only PR comments. -3. ✅ **Unified scorecard schema** — `scorecard.json` with `gates[]` breakdown combining engine, security, and quality gates. Configurable via `gate_policy` in `metadata.yaml`. *(Implemented 2026-06-16)* -4. ✅ **Quality review blocking mode** — Gates support `disabled`, `warn`, or `block` modes. *(Implemented 2026-06-16)* +1. **Enforced gate matrix** -- Skip `evaluate` when `tests-passed=false`. Add `marketplace-mode` with stricter thresholds (uplift + p-value + security block). +2. **GitHub commit status / required check** -- Not only PR comments. +3. ✅ **Unified scorecard schema** -- `scorecard.json` with `gates[]` breakdown combining engine, security, and quality gates. Configurable via `gate_policy` in `metadata.yaml`. *(Implemented 2026-06-16)* +4. ✅ **Quality review blocking mode** -- Gates support `disabled`, `warn`, or `block` modes. *(Implemented 2026-06-16)* -### P1 — Observability and Monitoring (6–10 weeks) +### P1 -- Observability and Monitoring (6–10 weeks) 5. **Grafana dashboards** on PostgreSQL (pass rate, uplift, p-value, degradation, cost). 6. **Token and time tracking** per run and per trial. -7. **Monitoring v2** — Rolling baseline, CUSUM, triggers on model/prompt/agent change, Jira or Slack ticket on regression. -8. **MLflow or Langfuse observer** — Implement at least one observer behind `ResultsObserver`. +7. **Monitoring v2** -- Rolling baseline, CUSUM, triggers on model/prompt/agent change, Jira or Slack ticket on regression. +8. **MLflow or Langfuse observer** -- Implement at least one observer behind `ResultsObserver`. -### P2 — SOTA Eval Depth (ongoing) +### P2 -- SOTA Eval Depth (ongoing) -9. **ODH `eval.yaml` engine adapter** — Shared contract without replacing the Tekton shell. -10. **Benchmark registry** — Versioned datasets (skills, MCP tasks, agent scenarios), not only ad hoc submissions. +9. **ODH `eval.yaml` engine adapter** -- Shared contract without replacing the Tekton shell. +10. **Benchmark registry** -- Versioned datasets (skills, MCP tasks, agent scenarios), not only ad hoc submissions. 11. **skillsaw + Snyk** in test phase alongside Cisco. -12. **Adversarial eval pack** — Runtime prompt-injection probes, not only static scan. -13. **Multi-model matrix** — Optional PipelineRun matrix parameter. -14. **skillberry / GEPA hook** — Optional post-fail optimization pipeline branch. +12. **Adversarial eval pack** -- Runtime prompt-injection probes, not only static scan. +13. **Multi-model matrix** -- Optional PipelineRun matrix parameter. +14. **skillberry / GEPA hook** -- Optional post-fail optimization pipeline branch. --- @@ -188,10 +188,10 @@ The pipeline narrative includes GEPA, SkillOps, and skillberry-style authoring, ## Related Documentation -- [implementation_plan.md](implementation_plan.md) — Phased pipeline build-out -- [results_persistence_and_observability_plan.md](results_persistence_and_observability_plan.md) — DB, observers, Grafana follow-ups -- [trigger_models_and_experiment_types.md](trigger_models_and_experiment_types.md) — PR vs submissions-repo models -- [failure_handling.md](failure_handling.md) — Pipeline failure behavior +- [implementation_plan.md](implementation_plan.md) -- Phased pipeline build-out +- [results_persistence_and_observability_plan.md](results_persistence_and_observability_plan.md) -- DB, observers, Grafana follow-ups +- [trigger_models_and_experiment_types.md](trigger_models_and_experiment_types.md) -- PR vs submissions-repo models +- [failure_handling.md](failure_handling.md) -- Pipeline failure behavior --- diff --git a/Docs/certification_and_checks.md b/Docs/certification_and_checks.md index 23130dad..fe597aa7 100644 --- a/Docs/certification_and_checks.md +++ b/Docs/certification_and_checks.md @@ -1,6 +1,6 @@ # Certification, Scorecards, Checks, and Facts -This document describes ABEvalFlow's certification system, which evaluates AI artifacts (skills, MCP servers, agents) against a comprehensive set of checks organized into three certification levels. +This document describes Agentic Eval Flow's certification system, which evaluates AI artifacts (skills, MCP servers, agents) against a comprehensive set of checks organized into three certification levels. ## Overview @@ -118,9 +118,9 @@ certification_policy: ### Priority Order -1. **Submission's `metadata.yaml`** — if `certification_policy` is present, use it -2. **Pipeline's `--certification-profile`** — if set, load profile from config -3. **Hardcoded defaults** — fallback to `FOUNDATIONAL_CHECKS`, `TRUSTED_CHECKS`, `CERTIFIED_CHECKS` in Python +1. **Submission's `metadata.yaml`** -- if `certification_policy` is present, use it +2. **Pipeline's `--certification-profile`** -- if set, load profile from config +3. **Hardcoded defaults** -- fallback to `FOUNDATIONAL_CHECKS`, `TRUSTED_CHECKS`, `CERTIFIED_CHECKS` in Python ### Code Location @@ -688,9 +688,9 @@ profiles: | Change fact payload structure | `compass_facts.py` | **No changes needed in:** -- `compass_facts.py` — automatically picks up new checks -- `scorecard.py` — automatically includes new checks -- Pipeline YAML — just select the profile +- `compass_facts.py` -- automatically picks up new checks +- `scorecard.py` -- automatically includes new checks +- Pipeline YAML -- just select the profile --- diff --git a/Docs/compass_facts_integration.md b/Docs/compass_facts_integration.md index db1bf790..dc21f837 100644 --- a/Docs/compass_facts_integration.md +++ b/Docs/compass_facts_integration.md @@ -1,11 +1,11 @@ # Compass Facts API Integration -ABEvalFlow can push gate evaluation results to Red Hat Compass as Soundcheck facts. +Agentic Eval Flow can push gate evaluation results to Red Hat Compass as Soundcheck facts. This enables visibility of skill evaluation metrics directly in the Compass developer portal. ## Overview -When configured, ABEvalFlow will POST gate results to the Compass Facts API after each +When configured, Agentic Eval Flow will POST gate results to the Compass Facts API after each gate evaluation completes. This provides real-time visibility into: - Engine gate results (Harbor, ASE, A2A, MCPChecker) @@ -110,7 +110,7 @@ Each gate result is pushed as a Soundcheck fact with this structure: ## Certification Levels -ABEvalFlow automatically computes certification levels based on gate results and pushes +Agentic Eval Flow automatically computes certification levels based on gate results and pushes them to Compass. Three certification levels are supported: | Level | Description | Requirements | @@ -121,7 +121,7 @@ them to Compass. Three certification levels are supported: ### Certification Facts -When `push_facts` is configured, ABEvalFlow automatically pushes 4 certification facts: +When `push_facts` is configured, Agentic Eval Flow automatically pushes 4 certification facts: | Fact Reference | Description | |----------------|-------------| @@ -299,7 +299,7 @@ will raise a `ValueError` at runtime. ## Validation Warning If `push_facts.endpoint` is configured but no gates have `push_fact: true`, -ABEvalFlow logs a warning: +Agentic Eval Flow logs a warning: ``` WARNING: push_facts.endpoint is configured but no gates have push_fact=True. @@ -429,7 +429,7 @@ kubectl run dns-test --rm -it --restart=Never --image=busybox -- nslookup compas ## Error Handling -- If a fact push fails, ABEvalFlow logs a warning but continues processing +- If a fact push fails, Agentic Eval Flow logs a warning but continues processing - Gate evaluation results are not affected by fact push failures - Timeouts default to 30 seconds per push request - Authentication failures (401/403) are logged with the specific error diff --git a/Docs/extensibility.md b/Docs/extensibility.md new file mode 100644 index 00000000..0e6d6344 --- /dev/null +++ b/Docs/extensibility.md @@ -0,0 +1,149 @@ +# Extensibility + +## Adding a New Engine + +1. Create a new file in `abevalflow/engines/`: + +```python +# abevalflow/engines/my_engine.py +from abevalflow.engines import register_engine +from abevalflow.engines.base import EvalEngine +from abevalflow.gates.base import GateResult, GateType + +@register_engine("my-engine") +class MyEngine(EvalEngine): + name = "my-engine" + + def read_result(self, reports_dir: Path) -> dict | None: + """Read engine results from reports directory.""" + result_path = reports_dir / "my-engine-report.json" + if not result_path.exists(): + return None + return json.loads(result_path.read_text()) + + def to_gate_result(self, raw_result: dict, policy: GatePolicy) -> GateResult: + """Convert engine result to standardized GateResult.""" + score = raw_result.get("score", 0.0) + threshold = policy.get_gate_policy(self.name).threshold or 0.0 + + return GateResult( + gate_type=GateType.ENGINE, + gate_name="evaluation", + policy_key=self.name, + passed=score >= threshold, + score=score, + mode=policy.get_gate_policy(self.name).mode, + message=f"MyEngine: score={score:.2f}", + ) +``` + +2. Import in `abevalflow/engines/__init__.py`: + +```python +from abevalflow.engines.my_engine import MyEngine +``` + +## Adding a New Security Gate + +1. Create a new file in `abevalflow/gates/security/`: + +```python +# abevalflow/gates/security/snyk.py +from abevalflow.gates.security import register_security_gate +from abevalflow.gates.security.base import SecurityGate +from abevalflow.gates.base import GateResult, GateType + +@register_security_gate("snyk") +class SnykGate(SecurityGate): + name = "snyk" + + def evaluate(self, reports_dir: Path, policy: GatePolicy) -> GateResult: + """Evaluate Snyk security scan results.""" + # Read snyk-report.json and produce GateResult + ... +``` + +2. Import in `abevalflow/gates/security/__init__.py`: + +```python +from abevalflow.gates.security.snyk import SnykGate +``` + +## Adding a New Quality Gate + +1. Create a new file in `abevalflow/gates/quality/`: + +```python +# abevalflow/gates/quality/custom_review.py +from abevalflow.gates.quality import register_quality_gate +from abevalflow.gates.quality.base import QualityGate +from abevalflow.gates.base import GateResult, GateType + +@register_quality_gate("custom-review") +class CustomReviewGate(QualityGate): + name = "custom-review" + + def evaluate(self, workspace_root: Path, policy: GatePolicy) -> GateResult: + """Evaluate custom quality review results.""" + # Read review artifacts and produce GateResult + ... +``` + +2. Import in `abevalflow/gates/quality/__init__.py`: + +```python +from abevalflow.gates.quality.custom_review import CustomReviewGate +``` + +## Adding a New Gate Category + +To add an entirely new gate category (e.g., "compliance", "performance"): + +1. **Add the GateType enum** in `abevalflow/gates/base.py`: + +```python +class GateType(str, Enum): + ENGINE = "engine" + SECURITY = "security" + QUALITY = "quality" + COMPLIANCE = "compliance" # New category +``` + +2. **Create the gate directory** at `abevalflow/gates/compliance/`: + +``` +abevalflow/gates/compliance/ +├── __init__.py # Registry and exports +├── base.py # ComplianceGate base class +└── my_checker.py # First implementation +``` + +3. **Create the base class** in `abevalflow/gates/compliance/base.py`: + +```python +from abc import abstractmethod +from abevalflow.gates.base import GateResult, GateType + +class ComplianceGate: + name: str + + @abstractmethod + def evaluate(self, reports_dir: Path, policy: GatePolicy) -> GateResult: + """Evaluate compliance and return standardized GateResult.""" + pass +``` + +4. **Update the scorecard aggregation** in `scripts/aggregate_scorecard.py`: + +```python +from abevalflow.gates.compliance import get_all_compliance_gates + +# In aggregate_scorecard(): +for compliance_gate in get_all_compliance_gates(): + if not policy.is_enabled(compliance_gate.name): + continue + gate_result = compliance_gate.evaluate(reports_dir, policy) + gates.append(gate_result) +``` + +5. **Add the category to policy schema** in `abevalflow/schemas.py` (documentation only, the schema is flexible) diff --git a/Docs/failure_handling.md b/Docs/failure_handling.md index ed20b3ee..a080b4a6 100644 --- a/Docs/failure_handling.md +++ b/Docs/failure_handling.md @@ -1,6 +1,6 @@ # Failure Handling, Retries, and Idempotency -## Retry Policy (Target — Not Yet Applied) +## Retry Policy (Target -- Not Yet Applied) > **Note:** The per-task retry values below are the target policy. They > will be added to `pipeline.yaml` once the pipeline assembly PR merges @@ -13,11 +13,11 @@ | `validate` | 1 | Read-only, deterministic | | `scaffold` | 1 | Deterministic template rendering | | `build-push` | 2 | Transient registry/network errors; Buildah is idempotent with layer caching | -| `harbor-eval` | 0 | Long-running (up to 3h), not idempotent — partial trial results would conflict with a fresh run | +| `harbor-eval` | 0 | Long-running (up to 3h), not idempotent -- partial trial results would conflict with a fresh run | | `analyze` | 1 | Reads from workspace, deterministic computation | | `store-results` | 2 | Database transient errors; upsert logic ensures idempotency via `pipeline_run_id` uniqueness | -## Timeouts (Target — Not Yet Applied) +## Timeouts (Target -- Not Yet Applied) > **Note:** The per-task timeouts below are the target policy. They will > be added to `pipeline.yaml` alongside the retry values. Currently only @@ -39,8 +39,8 @@ Certain failure categories should not be retried because they indicate a problem that will not resolve on its own: -- **Validation failures** — malformed submission, missing required files -- **Schema violations** — invalid `metadata.yaml` +- **Validation failures** -- malformed submission, missing required files +- **Schema violations** -- invalid `metadata.yaml` - **Build failures** from syntax errors in user code - **Harbor evaluation failures** from test assertion errors (the skill genuinely fails) @@ -64,7 +64,7 @@ When a PipelineRun fails after exhausting retries: 1. **Artifacts are retained** on the workspace PVC (not cleaned up) 2. The `abevalflow-dead-letter` PVC is provisioned and reserved for failed-run artifact storage. Automatic copy logic is **not yet - implemented** — operators can manually copy artifacts from the + implemented** -- operators can manually copy artifacts from the workspace PVC for post-mortem analysis. 3. PipelineRun metadata remains queryable via `tkn pipelinerun describe` until the cleanup CronJob prunes it (keeps the 7 most recent by @@ -75,25 +75,25 @@ When a PipelineRun fails after exhausting retries: Tekton does not natively support resuming a pipeline from a specific task. The recovery strategy is: -1. **Workspace snapshot** — the PVC retains all intermediate artifacts +1. **Workspace snapshot** -- the PVC retains all intermediate artifacts from completed tasks. A re-run with the same submission will overwrite these, effectively starting fresh. -2. **Harbor checkpointing** — the Harbor fork persists individual trial +2. **Harbor checkpointing** -- the Harbor fork persists individual trial results to the workspace as they complete. If `harbor-eval` fails mid-way (e.g., after 15 of 20 trials), the partial `result.json` files are available for inspection. However, the analysis step expects a complete set, so a re-run of `harbor-eval` is needed. -3. **Manual re-trigger** — use `tkn pipeline start` with the same +3. **Manual re-trigger** -- use `tkn pipeline start` with the same parameters to re-run the full pipeline. Since all tasks before the failure point are idempotent, they will complete quickly using cached layers (builds) or deterministic outputs (scaffold). ## Concurrency -- **PipelineRuns** — no built-in Tekton limit; use `ResourceQuota` on +- **PipelineRuns** -- no built-in Tekton limit; use `ResourceQuota` on the namespace (`config/security/resource_quota.yaml`) to cap total pods, which indirectly limits concurrent runs. -- **Trial Pods** — Harbor's `OpenShiftEnvironment` controls concurrency +- **Trial Pods** -- Harbor's `OpenShiftEnvironment` controls concurrency via its `max_concurrent` parameter in the job config. diff --git a/Docs/gates-architecture.md b/Docs/gates-architecture.md new file mode 100644 index 00000000..bc6db7de --- /dev/null +++ b/Docs/gates-architecture.md @@ -0,0 +1,164 @@ +# Gates Architecture + +Gates are evaluation checkpoints that produce standardized results. The unified scorecard aggregates all gate results to produce a final recommendation. + +## Gate Types + +| Category | Policy Key | Purpose | Implementation | +|----------|------------|---------|----------------| +| **evaluation** | `evaluation` | Results from the selected eval engine | Harbor, ASE, A2A, MCPChecker, or AEH | +| **security** | `security` | Security scanning results | Cisco AI Defense scanner, [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval) deterministic scanner | +| **quality** | `quality` | Quality review results | LLM-powered review, [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval) deterministic quality checks | + +## Gate Modes + +Each gate operates in one of three modes: + +| Mode | Behavior | +|------|----------| +| `disabled` | Gate is skipped entirely | +| `warn` | Gate runs; failures produce warnings but don't block | +| `block` | Gate runs; failures cause the scorecard to fail | + +## GateResult Schema + +All gates produce a standardized `GateResult`: + +```python +class GateResult: + gate_type: GateType # engine, security, or quality + gate_name: str # Category name: "evaluation", "security", or "quality" + policy_key: str # Implementation: "harbor", "cisco", "llm-review", etc. + passed: bool # Whether the gate passed + score: float # Normalized score (0.0 to 1.0) + mode: GateMode # Mode that was applied (disabled/warn/block) + threshold: float | None # Threshold used for pass/fail + findings: list[Finding] # Issues discovered (security/quality gates) + details: dict # Implementation-specific data (e.g., {"engine": "harbor"}) + message: str # Human-readable summary +``` + +The `gate_name` is the category used in policy configuration, while `policy_key` identifies the specific implementation. + +## Existing Gates + +### Evaluation Gate (`evaluation`) + +The primary gate that wraps the selected evaluation engine's results. + +- **Location:** `abevalflow/engines/*.py` (each engine produces evaluation gate results) +- **Input:** Engine-specific report from `reports/{submission}/` +- **Engines:** Harbor, ASE, A2A, MCPChecker (selected via `eval_engine` in metadata.yaml) +- **Pass criteria:** + - Harbor/ASE/A2A: `treatment_score - control_score >= threshold` (default threshold: 0.0) + - MCPChecker: All tasks pass verification +- **Score:** Mean reward or pass rate depending on engine + +### Security Gate (`security`) + +Two scanners feed into the security gate: + +**Cisco AI Defense** (`CiscoGate`): +- **Location:** `abevalflow/gates/security/cisco.py` +- **Input:** `reports/{submission}/security-scan.json` +- **Scanner:** Cisco AI Defense model-based detection + +**harness-eval** (`SkillMdScannerGate`): +- **Location:** `abevalflow/gates/security/skillmd_scanner.py` +- **Input:** `reports/{submission}/skillmd-security-scan.json` +- **Scanner:** [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval) `skill-submission-scan` CLI (27 rule categories, 97 deterministic rules) +- **Includes:** Optional LLM semantic security review (anti-jailbreak, semantic attacks, description-behavior mismatch) + +Both gates use the same pass criteria: +- `warn` mode: Always passes (findings are advisory) +- `block` mode: Fails if any HIGH or CRITICAL findings exist +- **Score:** Weighted average based on finding severities + +### Quality Gate (`quality`) + +Two sources feed into quality gates: + +**LLM Quality Review** (`LLMReviewGate`): +- **Location:** `abevalflow/gates/quality/llm_review.py` +- **Input:** `{workspace}/_ai_review.json` +- **Dimensions evaluated:** coherence, coverage, clarity, feasibility, robustness +- **Default threshold:** 0.6 + +**harness-eval Quality** (`SkillMdQualityGate`): +- **Location:** `abevalflow/gates/quality/skillmd_quality.py` +- **Input:** `{workspace}/skillmd-quality-scan.json` +- **Checks:** description quality, broken references, imprecise instructions, unfinished content, stale references, scope overreach, token budget, and more + +## Scorecard + +The scorecard is the single source of truth for submission evaluation, aggregating all gate results with configurable policy. + +### Scorecard Schema + +```python +class Scorecard: + submission_name: str # Name of the evaluated submission + pipeline_run_id: str # Tekton PipelineRun ID + eval_engine: str # Primary evaluation engine used + gates: list[GateResult] # All gate results + policy: GatePolicy # Policy that was applied + recommendation: Recommendation # pass, warn, or fail + recommendation_reason: str # Human-readable explanation + gates_passed: int # Count of passed gates + gates_failed: int # Count of failed gates + blocking_gates_passed: int # Count of passed blocking gates + blocking_gates_failed: int # Count of failed blocking gates +``` + +### Combination Modes + +| Mode | Logic | +|------|-------| +| `all_pass` | All blocking gates must pass; failing warn gates produce warnings | +| `any_pass` | At least one blocking gate must pass | +| `weighted` | Weighted average of gate scores determines outcome | + +### Output + +The scorecard is written to `reports/{submission}/scorecard.json` and includes: +- All gate results with scores and findings +- Final recommendation with reasoning +- Provenance metadata (commit SHA, branch, pipeline run ID) + +## Gate Policy Configuration + +Gate policies are configured in `metadata.yaml` under the `gate_policy` key: + +```yaml +# metadata.yaml +name: my-skill +eval_engine: harbor + +gate_policy: + default_mode: warn # Default mode for all gates + combination: all_pass # How to combine gate results + + gates: + # Security gate configuration + security: + mode: block # Fail the scorecard on security issues + threshold: 0.8 # Minimum score to pass + + # Quality gate configuration + quality: + mode: warn # Advisory only + threshold: 0.6 # Threshold for pass/fail + + # Engine gate configuration (uses eval_engine automatically) + evaluation: + mode: block + threshold: 0.0 # Any positive uplift passes +``` + +### GatePolicyItem Options + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `mode` | `disabled`/`warn`/`block` | `warn` | Enforcement mode | +| `threshold` | `float` | Gate-specific | Score threshold for pass/fail | +| `weight` | `float` | `1.0` | Weight for weighted combination mode | diff --git a/Docs/harbor_fork_requirements.md b/Docs/harbor_fork_requirements.md index 79bfb3d4..577332d4 100644 --- a/Docs/harbor_fork_requirements.md +++ b/Docs/harbor_fork_requirements.md @@ -1,14 +1,14 @@ -# Harbor Fork — Integration Requirements for ABEvalFlow +# Harbor Fork -- Integration Requirements for Agentic Eval Flow > **Target repo:** [RHEcosystemAppEng/skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) -> **Open PR:** [#1 — feat: add OpenShift environment backend](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/1) -> **ABEvalFlow branch:** `APPENG-4906/harbor-eval-task` +> **Open PR:** [#1 -- feat: add OpenShift environment backend](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/1) +> **Agentic Eval Flow branch:** `APPENG-4906/harbor-eval-task` --- -## ABEvalFlow-Side Implementation (current state) +## Agentic Eval Flow-Side Implementation (current state) -### How ABEvalFlow invokes Harbor +### How Agentic Eval Flow invokes Harbor The `harbor-eval` Tekton task (`pipeline/tasks/harbor-eval.yaml`) runs a single step that: @@ -62,7 +62,7 @@ eval-results// ... (N trials) ``` -### What ABEvalFlow reads from metadata.yaml +### What Agentic Eval Flow reads from metadata.yaml The config generator extracts these fields from `SubmissionMetadata`: @@ -103,7 +103,7 @@ The config generator extracts these fields from `SubmissionMetadata`: The fork's `OpenShiftEnvironment` must handle the full trial lifecycle when invoked with `--env openshift` (or `environment.type: openshift` in config): -- Accept `image_ref` via `environment.kwargs` — verify the image is pullable, +- Accept `image_ref` via `environment.kwargs` -- verify the image is pullable, skip building - Create trial Pods with the pre-built image - Execute agent + verifier inside the Pod via `exec` @@ -125,7 +125,7 @@ the config path should behave identically but needs confirmation. **3. `result.json` output format** -ABEvalFlow's pass-rate parser expects: +Agentic Eval Flow's pass-rate parser expects: ```json { @@ -135,7 +135,7 @@ ABEvalFlow's pass-rate parser expects: } ``` -Where `reward > 0.0` means pass. This is Harbor's standard format — no change +Where `reward > 0.0` means pass. This is Harbor's standard format -- no change needed, but any deviation would break the parser. ### Implemented (fork PR #2) @@ -146,7 +146,7 @@ Implemented in [PR #2](https://github.com/RHEcosystemAppEng/skills_eval_correcti `TaskConfig` now has an `environment_kwargs: dict[str, Any]` field. `Job._env_config_for_task` merges per-task kwargs into the global `EnvironmentConfig.kwargs` (task-level overrides global). -ABEvalFlow currently runs each variant as a separate Harbor job, which works +Agentic Eval Flow currently runs each variant as a separate Harbor job, which works without this feature. Per-task kwargs enables a single-job alternative for sweep-based workflows: @@ -172,7 +172,7 @@ the actual implementation in PR #1. These items should be updated: |---------|-------------------|---------| | File path | `openshift_environment.py` | `openshift.py` | | Build modes | `_build_and_push_image` is no-op only | Supports pre-built (`--ek image_ref=`) AND local podman build | -| Pod security | `readOnlyRootFilesystem: true` | Intentionally unset — many agent workloads need writes; `HOME=/tmp` is injected instead | +| Pod security | `readOnlyRootFilesystem: true` | Intentionally unset -- many agent workloads need writes; `HOME=/tmp` is injected instead | | RBAC table | ConfigMaps, Secrets, PVCs, ImageStreams | Only Pods + exec + Secrets used in practice | | Naming | "skilled / unskilled" | "treatment / control" | | Trial count | "20 skilled + 20 unskilled" | "20 treatment + 20 control" | @@ -182,6 +182,6 @@ the actual implementation in PR #1. These items should be updated: ### Additional Notes - The OpenShift backend supports a `cpu_request` kwarg (`--ek cpu_request=`) - for clusters with tight resource constraints — not documented in the handoff doc. + for clusters with tight resource constraints -- not documented in the handoff doc. - The `--ek registry=` kwarg enables local podman build+push to a specified - registry — also undocumented. + registry -- also undocumented. diff --git a/Docs/harbor_openshift_backend.md b/Docs/harbor_openshift_backend.md index 8f60dcad..20ce6de0 100644 --- a/Docs/harbor_openshift_backend.md +++ b/Docs/harbor_openshift_backend.md @@ -1,8 +1,8 @@ -# Harbor OpenShift Backend — Handoff Document +# Harbor OpenShift Backend -- Handoff Document -> **Jira:** APPENG-4906 (Phase 4 — Harbor OpenShift Backend) +> **Jira:** APPENG-4906 (Phase 4 -- Harbor OpenShift Backend) > **Target repo:** [RHEcosystemAppEng/skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) (Harbor fork) -> **Fork PRs:** [#1 — OpenShift environment backend](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/1), [#2 — per-task environment_kwargs](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/2) +> **Fork PRs:** [#1 -- OpenShift environment backend](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/1), [#2 -- per-task environment_kwargs](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/2) > **Full spec:** See [implementation_plan.md](./implementation_plan.md), Phase 4 --- @@ -41,11 +41,11 @@ The GKE backend (`src/harbor/environments/gke.py`) implements the full ### `_build_and_push_image` - GKE: `gcloud builds submit` (Cloud Build) -- **OpenShift — two modes:** +- **OpenShift -- two modes:** | Mode | Behavior | When to use | |------|----------|-------------| -| **Prebuilt** (`image_ref` kwarg set) | Verify image is pullable, skip building | Default pipeline flow — Tekton builds with Buildah | +| **Prebuilt** (`image_ref` kwarg set) | Verify image is pullable, skip building | Default pipeline flow -- Tekton builds with Buildah | | **Local build** (`force_build: true`) | Build with podman, push to specified registry | Local dev, skipping the build-push Tekton step | **Prebuilt contract:** @@ -60,7 +60,7 @@ The GKE backend (`src/harbor/environments/gke.py`) implements the full - **Kwargs:** `registry` (push target URL), `tls_verify` (default `"true"`) ### `start`, `stop`, `exec`, `upload_file/dir`, `download_file/dir` -- Same K8s API patterns as GKE — the `kubernetes` Python client is identical +- Same K8s API patterns as GKE -- the `kubernetes` Python client is identical ## Environment kwargs @@ -71,9 +71,9 @@ global kwargs when set. | Kwarg | Description | Default | |-------|-------------|---------| | `namespace` | OpenShift namespace for trial Pods | Required | -| `image_ref` | Digest-based image ref (prebuilt mode) | — | -| `registry` | Push target URL (local-build mode) | — | -| `cpu_request` | CPU request for trial Pods (e.g. `"250m"`) | — | +| `image_ref` | Digest-based image ref (prebuilt mode) | -- | +| `registry` | Push target URL (local-build mode) | -- | +| `cpu_request` | CPU request for trial Pods (e.g. `"250m"`) | -- | | `tls_verify` | TLS verification for registry (e.g. `"false"`) | `"true"` | ### Per-task environment_kwargs (PR #2) @@ -93,14 +93,14 @@ tasks: image_ref: "registry/ns/my-submission@sha256:def..." ``` -ABEvalFlow currently runs each variant as a separate Harbor job (two +Agentic Eval Flow currently runs each variant as a separate Harbor job (two `harbor run` invocations). Per-task kwargs enables a single-job alternative if needed in the future. ## Pod Security Requirements Trial Pods run with OpenShift's default security constraints. The -`readOnlyRootFilesystem` is intentionally **not** set — many agent +`readOnlyRootFilesystem` is intentionally **not** set -- many agent workloads need filesystem writes. Instead, `HOME=/tmp` is injected to direct writes to a writable location. @@ -110,7 +110,7 @@ Verify the target cluster uses `restricted-v2` SCC (OpenShift 4.11+). - **N = 20** attempts per variant (treatment + control = 40 total sessions) - Image refs come as params from the build-push Tekton task (digest-based) -- LLM endpoint via environment variable — backend is agnostic to LLM access mode +- LLM endpoint via environment variable -- backend is agnostic to LLM access mode - Configurable per-trial timeout and global evaluation timeout - Resource requests/limits per trial Pod @@ -138,12 +138,12 @@ from interfering with regular REST calls in other coroutines. - **Unit tests:** Mock K8s API with `pytest` + `unittest.mock`. No live cluster needed. - **Integration tests:** Test against OpenShift developer sandbox (ROSA/OSD). - Do **not** use Kind/Minikube — they won't catch SCC/Routes differences. + Do **not** use Kind/Minikube -- they won't catch SCC/Routes differences. -## Tekton Task (in ABEvalFlow repo) +## Tekton Task (in Agentic Eval Flow repo) The `harbor-eval` Tekton task (`pipeline/tasks/harbor-eval.yaml`) invokes -Harbor from the ABEvalFlow pipeline: +Harbor from the Agentic Eval Flow pipeline: - Installs Harbor from the fork via `pip install git+@` - Generates two per-variant job configs using `scripts/generate_eval_config.py` @@ -159,13 +159,13 @@ Harbor from the ABEvalFlow pipeline: - [x] Config examples for prebuilt, local-build, and per-task modes (PR #2) - [x] Unit tests with mocked K8s API - [ ] 40 trial Pods complete (20 treatment + 20 control) on live cluster -- [ ] Cleanup verified — no stale Pods after evaluation +- [ ] Cleanup verified -- no stale Pods after evaluation - [ ] Retry behavior validated for transient failures - [ ] Integration test passes on OpenShift sandbox ## LLM Access Modes (for reference) -The backend doesn't need to know which mode is used — it just passes +The backend doesn't need to know which mode is used -- it just passes env vars to trial Pods: | Mode | Env Var | Infrastructure | diff --git a/Docs/implementation_plan.md b/Docs/implementation_plan.md index 86dfbdec..a2cbfa25 100644 --- a/Docs/implementation_plan.md +++ b/Docs/implementation_plan.md @@ -10,7 +10,7 @@ Build an automated, Tekton-orchestrated pipeline on OpenShift that accepts skill ### Non-Goals -- Tessl Cloud integration — Tessl has no self-hosted option, only supports Claude via API key (not Vertex), and pricing is uncertain. Harbor (open-source, self-hostable) is the chosen evaluation engine. +- Tessl Cloud integration -- Tessl has no self-hosted option, only supports Claude via API key (not Vertex), and pricing is uncertain. Harbor (open-source, self-hostable) is the chosen evaluation engine. - Tessl-driven evaluation runs are out of scope for this pipeline; any future integration would require a separate ADR. ### Two-Repository Model (ADR Decision #1) @@ -19,10 +19,10 @@ This pipeline spans two repositories: | Repository | Purpose | Contents | |---|---|---| -| **[ABEvalFlow](https://github.com/RHEcosystemAppEng/agentic_eval_flow)** (this repo) | Pipeline definitions, scripts, templates, config | Tekton YAML, Python scripts, Jinja2 templates, Harbor backend | +| **[Agentic Eval Flow](https://github.com/RHEcosystemAppEng/agentic_eval_flow)** (this repo) | Pipeline definitions, scripts, templates, config | Tekton YAML, Python scripts, Jinja2 templates, Harbor backend | | **[agentic-collections](https://github.com/RHEcosystemAppEng/agentic-collections)** | Skills, tasks, tests (post-evaluation) | Persona-based plugins (`rh-sre`, `rh-developer`, `ocp-admin`, etc.), 100+ skills | -The `tasks-treatment/` and `tasks-control/` directories generated during scaffolding are **ephemeral workspace artifacts** — they exist only during a pipeline run, not as permanent directories in either repo. +The `tasks-treatment/` and `tasks-control/` directories generated during scaffolding are **ephemeral workspace artifacts** -- they exist only during a pipeline run, not as permanent directories in either repo. ### Harbor Fork @@ -38,16 +38,16 @@ The ADR originally specified Google Vertex AI + LiteLLM proxy. Since then, two a | **opencode + self-hosted** | opencode wrapper | Local/self-hosted model (e.g., vLLM, Ollama) | No | | **Vertex AI + proxy** | Claude Code | Google Vertex AI via LiteLLM proxy | Yes | -LiteLLM is **optional infrastructure**, needed only for the Vertex AI path. The pipeline and Harbor backend should be agnostic to the LLM access mode — the agent inside the trial container is configured via environment variables. +LiteLLM is **optional infrastructure**, needed only for the Vertex AI path. The pipeline and Harbor backend should be agnostic to the LLM access mode -- the agent inside the trial container is configured via environment variables. --- -## Phase 0 — Project Bootstrap +## Phase 0 -- Project Bootstrap ### 0.1 Repository Structure ``` -ABEvalFlow/ +agentic_eval_flow/ ├── Docs/ # ADR, plans, design docs ├── pipeline/ # Tekton pipeline definitions │ ├── pipeline.yaml # Main Pipeline resource @@ -95,7 +95,7 @@ ABEvalFlow/ ### 0.2 Python Project Setup - Initialize `pyproject.toml` with dependencies: `pydantic`, `jinja2`, `pyyaml`, `matplotlib`, `scipy`, `kubernetes` (Python client), `tenacity`. -- Existing `.venv` (Python 3.13) is compatible — no need to recreate. Pipeline base image uses `ubi9/python-311` (3.11+); local dev uses the existing venv. +- Existing `.venv` (Python 3.13) is compatible -- no need to recreate. Pipeline base image uses `ubi9/python-311` (3.11+); local dev uses the existing venv. - Use `uv` for dependency management (consistent with Harbor fork and agentic-collections). ### 0.3 Pre-Requisites Checklist @@ -122,11 +122,11 @@ ABEvalFlow/ --- -## Phase 1 — Submission & Validation (Steps 1-2) +## Phase 1 -- Submission & Validation (Steps 1-2) ### 1.0 Submission Contract -Canonical filename is **`instruction.md`** (not `instructions.md` — the ADR body uses the singular form; this is normative). +Canonical filename is **`instruction.md`** (not `instructions.md` -- the ADR body uses the singular form; this is normative). A skill submission directory must follow this structure: @@ -147,7 +147,7 @@ my-skill-name/ **Goal:** Define and enforce the skill submission metadata. - [ ] Create a **Pydantic model** for `metadata.yaml` schema validation. - - Fields: `schema_version` (for forward compatibility), `name`, `description`, `persona`, `version`, `author`, `tags` (optional), `generation_mode` (`manual` | `ai` — source of truth for dual path). + - Fields: `schema_version` (for forward compatibility), `name`, `description`, `persona`, `version`, `author`, `tags` (optional), `generation_mode` (`manual` | `ai` -- source of truth for dual path). - [ ] Document the submission contract in a dedicated reference doc. ### 1.2 Validation Script (`scripts/validate.py`) @@ -189,15 +189,15 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out --- -## Phase 2 — Scaffolding (Step 3) +## Phase 2 -- Scaffolding (Step 3) ### 2.1 Jinja2 Templates **Goal:** Create templates that generate the correct Dockerfiles and supporting files. -- [x] `Dockerfile.j2` — Unified template using `copy_pairs` loop; COPYs strategy-determined directories plus common files (`tests/`, `supportive/`, `instruction.md`). -- [ ] `test.sh.j2` — Entry script that runs the agent, then executes `test_outputs.py` and optional `llm_judge.py`. -- [ ] `task.toml.j2` — Harbor task configuration. +- [x] `Dockerfile.j2` -- Unified template using `copy_pairs` loop; COPYs strategy-determined directories plus common files (`tests/`, `supportive/`, `instruction.md`). +- [ ] `test.sh.j2` -- Entry script that runs the agent, then executes `test_outputs.py` and optional `llm_judge.py`. +- [ ] `task.toml.j2` -- Harbor task configuration. ### 2.2 Scaffold Script (`scripts/scaffold.py`) @@ -205,8 +205,8 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out - Input: path to validated submission directory. - Output (ephemeral workspace artifacts, not permanent repo dirs): - - `tasks-treatment//` — treatment variant with rendered Dockerfile, test.sh, task.toml. - - `tasks-control//` — control variant (baseline). + - `tasks-treatment//` -- treatment variant with rendered Dockerfile, test.sh, task.toml. + - `tasks-control//` -- control variant (baseline). - Renders templates with context from `metadata.yaml`, directory inspection, and experiment strategy (which determines copy specs per variant). ### 2.3 Scaffold Tekton Task (`pipeline/tasks/scaffold.yaml`) @@ -223,13 +223,13 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out --- -## Phase 3 — Build & Push Images (Steps 4-5) +## Phase 3 -- Build & Push Images (Steps 4-5) ### 3.1 Build Task (`pipeline/tasks/build-push.yaml`) **Goal:** Build both treatment and control images and push to registry. -- **Build tool constraint:** ADR Decision #5 specifies `docker buildx`. However, OpenShift clusters run CRI-O (not Docker) and do not provide a Docker daemon in pods. Using `docker buildx` inside unprivileged Tekton steps requires a Docker-in-Docker sidecar or socket mount, both of which require privileged access and contradict the security posture. **Buildah** (`buildah bud` + `buildah push`) is the standard rootless, daemonless alternative on OpenShift and runs in `ubi9` base images without privilege escalation. This constraint must be reconciled with ADR Decision #5 before implementation — likely by adopting Buildah for OpenShift. +- **Build tool constraint:** ADR Decision #5 specifies `docker buildx`. However, OpenShift clusters run CRI-O (not Docker) and do not provide a Docker daemon in pods. Using `docker buildx` inside unprivileged Tekton steps requires a Docker-in-Docker sidecar or socket mount, both of which require privileged access and contradict the security posture. **Buildah** (`buildah bud` + `buildah push`) is the standard rootless, daemonless alternative on OpenShift and runs in `ubi9` base images without privilege escalation. This constraint must be reconciled with ADR Decision #5 before implementation -- likely by adopting Buildah for OpenShift. - Builds from the scaffolded directories. - Tags: `//:treatment-` and `//:control-`. - Push to **internal OpenShift registry** for evaluation (per ADR decision #6). @@ -246,8 +246,8 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out The `build-push` Tekton task must emit two **results** for downstream consumption: -- `treatment-image-ref` — full digest-based reference (e.g., `registry/ns/skill@sha256:...`) -- `control-image-ref` — same format +- `treatment-image-ref` -- full digest-based reference (e.g., `registry/ns/skill@sha256:...`) +- `control-image-ref` -- same format The `pipeline.yaml` wires these to the `harbor-eval` task: @@ -269,7 +269,7 @@ Use digest-based references (not mutable tags) between tasks to avoid tag mutati --- -## Phase 4 — Harbor OpenShift Backend (Step 6) +## Phase 4 -- Harbor OpenShift Backend (Step 6) ### 4.1 OpenShift Environment Backend (in Harbor fork) @@ -280,12 +280,12 @@ The GKE backend (`src/harbor/environments/gke.py`, ~1044 lines) serves as the re | Method | GKE Implementation | OpenShift Replacement | |---|---|---| | `_init_client` | `gcloud container clusters get-credentials` + `load_kube_config()` | `load_incluster_config()` (in-cluster SA token) or `load_kube_config()` (local dev) | -| `_build_and_push_image` | `gcloud builds submit` (Cloud Build) | **No-op — see contract below** | +| `_build_and_push_image` | `gcloud builds submit` (Cloud Build) | **No-op -- see contract below** | | `_image_exists` | `gcloud artifacts docker images describe` | Query OpenShift internal registry API or Quay API via `skopeo inspect` or registry HTTP API | | `start` | Creates Pod spec, waits for ready | Same pattern, using `load_incluster_config()` instead of gcloud auth | | `stop` | Deletes Pod, waits for termination | Same pattern | -| `exec` | `kubectl exec` via K8s stream API | Same — the `kubernetes` Python client is identical | -| `upload_file` / `upload_dir` | tar + K8s stream stdin | Same — portable across K8s distributions | +| `exec` | `kubectl exec` via K8s stream API | Same -- the `kubernetes` Python client is identical | +| `upload_file` / `upload_dir` | tar + K8s stream stdin | Same -- portable across K8s distributions | | `download_file` / `download_dir` | tar + K8s stream stdout | Same | **`_build_and_push_image` contract:** In this pipeline, image build/push is owned by Tekton Steps 4-5. The Harbor OpenShift backend only orchestrates trial Pod lifecycle using pre-built images. @@ -320,7 +320,7 @@ Additional requirements: - [x] `harbor-eval` Tekton task accepts `treatment-image-ref` and `control-image-ref` as params wired from Phase 3 results. - [x] N = configurable attempts per variant (default 20, treatment + control = 40 total sessions) via `n-trials` from `metadata.yaml`. - [x] Resource requests/limits per trial Pod (from `metadata.yaml`: `cpus`, `memory_mb`, `storage_mb`). -- [ ] LLM endpoint configured via environment variable — backend is agnostic to whether it points to LiteLLM, a direct API, or a self-hosted model. +- [ ] LLM endpoint configured via environment variable -- backend is agnostic to whether it points to LiteLLM, a direct API, or a self-hosted model. - [x] Trial Pod timeout: configurable via timeout multipliers derived from `metadata.yaml`. - [x] Eval config generation script (`scripts/generate_eval_config.py`) reads metadata and produces Harbor job config YAML. - [x] Supports two modes: `prebuilt` (digest image refs) and `local-build` (Harbor builds from Dockerfiles). @@ -336,21 +336,21 @@ The pipeline ServiceAccount needs (prefer named Secrets for least-privilege wher | Pods, Pods/exec, Pods/log | create, get, list, watch, delete | Trial Pod lifecycle | Done | | Secrets | get | LLM credentials injection via `envFrom` | Done | | Events | get, list | Diagnosing hung/failed trial Pods | Done | -| ConfigMaps | get, list | Trial configuration | Deferred — not used by current backend | -| PVCs | get, list, create | Pipeline workspaces and artifacts | Deferred — handled by Tekton | -| ImageStreams (OpenShift) | get, list | Registry access | Deferred — not used by current backend | +| ConfigMaps | get, list | Trial configuration | Deferred -- not used by current backend | +| PVCs | get, list, create | Pipeline workspaces and artifacts | Deferred -- handled by Tekton | +| ImageStreams (OpenShift) | get, list | Registry access | Deferred -- not used by current backend | ### 4.6 Definition of Done - [ ] Trial Pods complete (N per variant × 2 variants, default 40 total). -- [ ] Cleanup verified — no stale Pods after evaluation. +- [ ] Cleanup verified -- no stale Pods after evaluation. - [ ] Retry behavior validated for transient failures. - [ ] Unit tests pass with mocked K8s API. - [ ] Integration test passes on OpenShift sandbox. --- -## Phase 5 — Analysis & Reporting (Step 7) +## Phase 5 -- Analysis & Reporting (Step 7) ### 5.1 Analysis Script (`scripts/analyze.py`) @@ -369,7 +369,7 @@ Output: Markdown (or HTML) report with: - Links to detailed trial logs. - Pass/fail recommendation based on configurable threshold. - LLM judge summary (if applicable). -- Estimated vs. actual token spend (when available from LLM provider — not all modes expose usage). +- Estimated vs. actual token spend (when available from LLM provider -- not all modes expose usage). - **Run provenance block:** commit SHA(s), Harbor fork SHA, image digest(s), model identifier/version, pipeline run ID, timestamp. ### 5.2 Analyze Tekton Task (`pipeline/tasks/analyze-report.yaml`) @@ -381,13 +381,13 @@ Output: Markdown (or HTML) report with: ### 5.3 Definition of Done - [x] Report includes uplift + p-value + run provenance (artifact links deferred to Phase 6 publish step). -- [ ] Heatmaps generated and embedded (deferred — not applicable to single A/B comparison). -- [ ] LLM judge scores aggregated when present (deferred — `llm_judge.py` not yet implemented). +- [ ] Heatmaps generated and embedded (deferred -- not applicable to single A/B comparison). +- [ ] LLM judge scores aggregated when present (deferred -- `llm_judge.py` not yet implemented). - [x] Unit tests pass for `analyze.py`. --- -## Phase 6 — Publish & Store (Step 8) +## Phase 6 -- Publish & Store (Step 8) ### 6.1 Publish Script (`scripts/publish.py`) @@ -397,7 +397,7 @@ Actions: - Upload final report to artifact storage (PVC/S3). - Redact PII/secrets from published logs and reports. - If evaluation passed thresholds: - - Re-tag and push images to **Quay.io** with TTL metadata (single promotion point — not in Phase 3). + - Re-tag and push images to **Quay.io** with TTL metadata (single promotion point -- not in Phase 3). - **Commit to agentic-collections repo** (ADR Decision #4): open a PR to `agentic-collections` with the submission files (`instruction.md`, `tests/`, `metadata.yaml`, `skills/`). Target directory derived from `persona` field + normalized skill slug + version. PR includes a standardized title and body with evaluation summary and report link. Requires deploy key/token stored as OpenShift Secret. - Post summary as a commit status or PR comment on the submissions repo. - Clean up ephemeral resources (temporary PVCs, intermediate images). @@ -416,7 +416,7 @@ Actions: --- -## Phase 7 — AI-Assisted Skill Evaluation (Decision #2 & #3) +## Phase 7 -- AI-Assisted Skill Evaluation (Decision #2 & #3) > This phase is **optional for MVP**. Use feature flags `ENABLE_AI_TEST_GENERATION` and `ENABLE_AI_QUALITY_REVIEW` to toggle. @@ -441,13 +441,13 @@ Pipeline task order when AI features are enabled: - [x] Create `scripts/test_quality_review.py` that uses the LLM to assess coherence, coverage, clarity, feasibility, and robustness. - [x] Add as a pipeline task (`test-quality-review`) between validation and scaffolding. - [x] Support both Harbor (`instruction.md` + `test_outputs.py`) and ASE (`evals.json`) formats. -- [x] Review is **advisory** (non-blocking) — always exits 0 regardless of recommendation. +- [x] Review is **advisory** (non-blocking) -- always exits 0 regardless of recommendation. ### 7.3 ASE (Agent Skills Eval) Integration > **Jira:** APPENG-5312 -ABEvalFlow supports two evaluation engines: +Agentic Eval Flow supports two evaluation engines: | Engine | Use Case | Eval Format | |--------|----------|-------------| @@ -456,7 +456,7 @@ ABEvalFlow supports two evaluation engines: ASE integration includes: - [x] `eval-engine` pipeline parameter (`harbor` / `ase` / `both`) -- [x] `scripts/generate_ase_evals.py` — generates `evals.json` from `SKILL.md` if not provided +- [x] `scripts/generate_ase_evals.py` -- generates `evals.json` from `SKILL.md` if not provided - [x] Semantic review of generated `evals.json` (skill specificity, prompt quality, assertion alignment) - [x] Upload generated files to MinIO under `generated/` folder - [x] ASE results aggregated into unified `report.json` format @@ -469,7 +469,7 @@ Optional security scan step using Cisco AI Defense `skill-scanner` to detect: - Malicious code patterns Implementation: -- [x] `pipeline/tasks/security-scan.yaml` — Tekton task with configurable modes +- [x] `pipeline/tasks/security-scan.yaml` -- Tekton task with configurable modes - [x] Three modes: `disabled` (skip), `warn` (report only), `block` (fail on HIGH/CRITICAL) - [x] Submission-level control via `metadata.yaml` (`security_scan: warn|block|disabled`) - [x] LLM semantic analysis using pipeline's configured model @@ -502,9 +502,9 @@ Database schema (`security_scans` table): --- -## Phase 8 — Infrastructure & Operations +## Phase 8 -- Infrastructure & Operations -### 8.1 LiteLLM Deployment (Optional — Vertex AI mode only) +### 8.1 LiteLLM Deployment (Optional -- Vertex AI mode only) > **Ordering note:** If `LLM_MODE=vertex`, Phase 8.1 must complete before Phase 4 (Harbor evaluation) can run trials. For other modes, this phase is skipped entirely. @@ -540,17 +540,17 @@ Database schema (`security_scans` table): - Expired evaluation reports. - OpenShift internal registry images (after Quay promotion or on failure). -### 8.4 LLM Model Strategy (Decision #8 — Updated) +### 8.4 LLM Model Strategy (Decision #8 -- Updated) Three supported modes, configured per pipeline run: | Mode | Configuration | Infrastructure | |---|---|---| -| Direct API key | Set `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` in trial Pod env | None — agent calls provider directly | +| Direct API key | Set `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` in trial Pod env | None -- agent calls provider directly | | opencode + self-hosted | Set model endpoint in trial Pod env, use opencode as agent wrapper | Self-hosted model (vLLM, Ollama, etc.) | | Vertex AI + LiteLLM | Set `LITELLM_BASE_URL` in trial Pod env | LiteLLM Deployment + Vertex AI credentials | -The pipeline and Harbor backend are agnostic — they pass LLM config as environment variables to trial Pods. +The pipeline and Harbor backend are agnostic -- they pass LLM config as environment variables to trial Pods. ### 8.5 Cost Controls & Observability @@ -558,7 +558,7 @@ A single evaluation run consumes N × 2 LLM sessions (default N=20, 40 total). C - [ ] Configure LiteLLM per-key budget limits (when using Vertex mode). - [ ] Implement pre-flight cost estimate: before launching Harbor, estimate token usage based on skill complexity and configured N. Log the estimate to the run summary to flag potential runaway cost before spend happens. -- [ ] Implement per-run token usage tracking — surface estimated vs. actual spend in the evaluation report. +- [ ] Implement per-run token usage tracking -- surface estimated vs. actual spend in the evaluation report. - [ ] Set up concurrency limits: max parallel `PipelineRun`s and max parallel trial Pods per namespace. - [ ] Track metrics: pass/fail rate by skill and model, mean/95p trial duration, failure categories (validation, build, runtime, LLM, infra). - [ ] Alert thresholds for cost spikes and error-rate spikes. @@ -569,27 +569,27 @@ A single evaluation run consumes N × 2 LLM sessions (default N=20, 40 total). C - [ ] Partial-run recovery: Tekton does not natively support "resume from task T." Options: (a) manual re-run with workspace snapshot from PVC, (b) split into smaller chained Pipelines, or (c) application-level checkpointing in Harbor (persist partial results per trial). Document chosen mechanism before implementation. - [ ] Timeouts: per-trial Pod timeout and global evaluation timeout. - [ ] Dead-letter path: failed runs retain artifacts for debugging. -- [ ] Persist partial results to PVC after each trial — prevents full re-run on LiteLLM HA failure. +- [ ] Persist partial results to PVC after each trial -- prevents full re-run on LiteLLM HA failure. --- -## Phase 9 — Continuous Performance Monitoring +## Phase 9 -- Continuous Performance Monitoring ### 9.1 Platform Update Regression Testing - [ ] Define a dedicated test set for mission-critical Insights API calls and tool-mapping expectations. - [ ] Create a separate Tekton pipeline/trigger for upstream platform/vendor update events (e.g., Gemini/Insights stack). - [ ] Reuse the evaluation infrastructure (containerized runs) for regression tests. -- [ ] **MCP LightSpeed regression pack** (ADR footnote [e]): Identify the source location of MCP LightSpeed tests (specific repo/path — to be inventoried during this phase). Bundle as committed fixtures in `ABEvalFlow/tests/canary/` or reference via config-driven pointer. The monitoring pipeline task should accept a `canary-test-set` param that defaults to the MCP LightSpeed pack but can be overridden. +- [ ] **MCP LightSpeed regression pack** (ADR footnote [e]): Identify the source location of MCP LightSpeed tests (specific repo/path -- to be inventoried during this phase). Bundle as committed fixtures in `agentic_eval_flow/tests/canary/` or reference via config-driven pointer. The monitoring pipeline task should accept a `canary-test-set` param that defaults to the MCP LightSpeed pack but can be overridden. ### 9.2 Degradation Detection -**9.2a — Simple thresholds (MVP):** +**9.2a -- Simple thresholds (MVP):** - [ ] Store historical pass rates per skill in PVC/S3. - [ ] Configure hard-coded alerting thresholds for performance drops (e.g., >10% pass-rate decline). - [ ] Wire alerts into the notification channel from Phase 6 (Slack/GitHub status). -**9.2b — CUSUM (post-MVP hardening):** +**9.2b -- CUSUM (post-MVP hardening):** - [ ] Implement CUSUM in `scripts/monitor.py` using configurable drift threshold (starting defaults: `k=0.5`, `h=5`). - [ ] Input: historical pass-rate time series per skill. - [ ] Output: boolean alert flag + CUSUM statistic at time of detection. @@ -601,20 +601,20 @@ A single evaluation run consumes N × 2 LLM sessions (default N=20, 40 total). C | Priority | Phase | Estimated Effort | Dependencies | |---|---|---|---| -| 1 | Phase 0 — Bootstrap | 1-2 days | None | -| 2 | Phase 8.2 — RBAC & Security | 1-2 days | OpenShift cluster (parallel with Phases 1-3) | -| 3 | Phase 1 — Validation | 2-3 days | Phase 0 | -| 4 | Phase 2 — Scaffolding | 2-3 days | Phase 1 | -| 5 | Phase 3 — Build & Push | 2-3 days | Phase 2, Registry access | -| 6 | Phase 8.1 — LiteLLM (if Vertex mode) | 1-2 days | Vertex AI credentials (must complete before Phase 4 trials if Vertex mode) | -| 7 | Phase 4 — Harbor Backend | 3-5 days | Harbor fork, Phase 3 | -| 8 | Phase 5 — Analysis | 2-3 days | Phase 4 | -| 9 | Phase 6 — Publish | 1-2 days | Phase 5, agentic-collections deploy key | -| 10 | Phase 7 — AI Assist (optional) | 3-4 days | LLM access configured | -| 11 | Phase 8.3-8.6 — Ops & Cost | 2-3 days | Phases 5-6 | -| 12 | Phase 9 — Monitoring | 2-3 days | Phases 5-6 | - -**Total estimated effort: ~22-33 days** (parallelizable — RBAC/Security work runs alongside Phases 1-3; LiteLLM setup slots in before Harbor eval if using Vertex mode). +| 1 | Phase 0 -- Bootstrap | 1-2 days | None | +| 2 | Phase 8.2 -- RBAC & Security | 1-2 days | OpenShift cluster (parallel with Phases 1-3) | +| 3 | Phase 1 -- Validation | 2-3 days | Phase 0 | +| 4 | Phase 2 -- Scaffolding | 2-3 days | Phase 1 | +| 5 | Phase 3 -- Build & Push | 2-3 days | Phase 2, Registry access | +| 6 | Phase 8.1 -- LiteLLM (if Vertex mode) | 1-2 days | Vertex AI credentials (must complete before Phase 4 trials if Vertex mode) | +| 7 | Phase 4 -- Harbor Backend | 3-5 days | Harbor fork, Phase 3 | +| 8 | Phase 5 -- Analysis | 2-3 days | Phase 4 | +| 9 | Phase 6 -- Publish | 1-2 days | Phase 5, agentic-collections deploy key | +| 10 | Phase 7 -- AI Assist (optional) | 3-4 days | LLM access configured | +| 11 | Phase 8.3-8.6 -- Ops & Cost | 2-3 days | Phases 5-6 | +| 12 | Phase 9 -- Monitoring | 2-3 days | Phases 5-6 | + +**Total estimated effort: ~22-33 days** (parallelizable -- RBAC/Security work runs alongside Phases 1-3; LiteLLM setup slots in before Harbor eval if using Vertex mode). --- @@ -642,13 +642,13 @@ A single evaluation run consumes N × 2 LLM sessions (default N=20, 40 total). C Consolidated evaluation results from all engines, security gates, and quality gates into a single `scorecard.json` with configurable policy. **Components:** -- `abevalflow/gates/base.py` — `GateResult` schema with normalized scores -- `abevalflow/scorecard.py` — `Scorecard` model and combination logic -- `abevalflow/engines/` — Registry + adapters for Harbor, ASE, A2A, MCPChecker -- `abevalflow/gates/security/` — Registry + CiscoGate adapter -- `abevalflow/gates/quality/` — Registry + LLMReviewGate adapter -- `scripts/aggregate_scorecard.py` — Aggregation script -- `pipeline/tasks/post/analyze-and-check-degradation.yaml` — `aggregate-scorecard` step +- `abevalflow/gates/base.py` -- `GateResult` schema with normalized scores +- `abevalflow/scorecard.py` -- `Scorecard` model and combination logic +- `abevalflow/engines/` -- Registry + adapters for Harbor, ASE, A2A, MCPChecker +- `abevalflow/gates/security/` -- Registry + CiscoGate adapter +- `abevalflow/gates/quality/` -- Registry + LLMReviewGate adapter +- `scripts/aggregate_scorecard.py` -- Aggregation script +- `pipeline/tasks/post/analyze-and-check-degradation.yaml` -- `aggregate-scorecard` step **Configuration:** `gate_policy` in `metadata.yaml`: ```yaml diff --git a/Docs/infrastructure_ops.md b/Docs/infrastructure_ops.md index e0216e75..dd07ca52 100644 --- a/Docs/infrastructure_ops.md +++ b/Docs/infrastructure_ops.md @@ -1,6 +1,6 @@ # Infrastructure & Operations Guide -Deployment and operations reference for running ABEvalFlow on OpenShift. +Deployment and operations reference for running Agentic Eval Flow on OpenShift. ## Prerequisites @@ -11,7 +11,7 @@ Deployment and operations reference for running ABEvalFlow on OpenShift. ## Namespace Setup ```bash -oc new-project ab-eval-flow --description="ABEvalFlow A/B evaluation pipeline" +oc new-project ab-eval-flow --description="Agentic Eval Flow A/B evaluation pipeline" ``` ## Deployment Order @@ -19,21 +19,21 @@ oc new-project ab-eval-flow --description="ABEvalFlow A/B evaluation pipeline" Apply manifests in this order to satisfy dependencies: ```bash -# 1. RBAC — ServiceAccount, Roles, RoleBindings +# 1. RBAC -- ServiceAccount, Roles, RoleBindings oc apply -f config/rbac.yaml -# 2. Security — resource quotas +# 2. Security -- resource quotas oc apply -f config/security/resource_quota.yaml -# 3. Network policies — choose ONE based on LLM mode (see below) +# 3. Network policies -- choose ONE based on LLM mode (see below) oc apply -f config/security/network_policy_default_deny.yaml oc apply -f config/security/network_policy_.yaml -# 4. Storage — workspace and dead-letter PVCs +# 4. Storage -- workspace and dead-letter PVCs oc apply -f config/storage/workspace_pvc.yaml oc apply -f config/storage/dead_letter_pvc.yaml -# 5. Cleanup — create ConfigMap from script, then apply CronJob +# 5. Cleanup -- create ConfigMap from script, then apply CronJob oc create configmap cleanup-script \ --from-file=cleanup.sh=scripts/cleanup.sh \ -n ab-eval-flow --dry-run=client -o yaml | oc apply -f - @@ -50,7 +50,7 @@ oc create route edge el-submission-listener \ --service=el-submission-listener \ --port=http-listener -# 9. (Optional) LiteLLM — only for Vertex AI mode +# 9. (Optional) LiteLLM -- only for Vertex AI mode # Creates a dedicated litellm ServiceAccount, Deployment, Service, and ConfigMap. # Requires the litellm-credentials Secret (see LiteLLM Setup below). oc apply -f config/litellm/ diff --git a/Docs/investigations/aeh_downloadverifiererror.md b/Docs/investigations/aeh_downloadverifiererror.md index b05b4fd7..9352f951 100644 --- a/Docs/investigations/aeh_downloadverifiererror.md +++ b/Docs/investigations/aeh_downloadverifiererror.md @@ -25,7 +25,7 @@ DownloadVerifierDirError: Failed to download verifier directory from environment - **OpenShift SCC**: restricted-v2 (enforces read-only root filesystem) - **Local Development**: - agent-eval-harness cloned at `/Users/gziv/Dev/agent-eval-harness` - - ABEvalFlow pipeline repo at `/Users/gziv/Dev/ABEvalFlow` + - Agentic Eval Flow pipeline repo at `/Users/gziv/Dev/agentic_eval_flow` ## Attempts and Results @@ -129,7 +129,7 @@ Confirmed from `case-001__XpJxhiR/exception.txt` / `result.json`: `tar cf - -C /logs/verifier . | base64 -w0` - Host `verifier/` directory in the job artifact is **empty** - Related: `download_file /logs/artifacts: No such file or directory` (best-effort artifact download) -- Agent did reach `/workspace` and teed to `/logs/agent/claude-code.txt` (agent then failed with `Unknown command: /aeh-hello-world-single` — separate from verifier FS) +- Agent did reach `/workspace` and teed to `/logs/agent/claude-code.txt` (agent then failed with `Unknown command: /aeh-hello-world-single` -- separate from verifier FS) Interpretation: download of `/logs/verifier` returned empty stdout (typical when `tar` fails on a missing/unreadable directory but `base64` still exits 0). Current `OpenShiftEnvironment` mounts only `/workspace` and `/tmp`, not Harbor paths `/logs`, `/tests`, `/solution`. @@ -140,7 +140,7 @@ Trial pod `aeh-case-001-bmkubbm-env`: | Check | Result | |---|---| | `OpenShiftEnvironment` mounts present | Yes: emptyDir `/workspace`, `/tmp` | -| `readOnlyRootFilesystem` | **Not set** — root overlay is `rw` | +| `readOnlyRootFilesystem` | **Not set** -- root overlay is `rw` | | Writability `/logs|/tests|/solution|/tmp|/workspace` | **All writable** without extra mounts | | `/logs/verifier` after failed trial | **Missing** | | `tar … \| base64` on missing dir | `out_len=0` (base64 masks tar failure) | @@ -160,7 +160,7 @@ Reproduced on the kept pod: ## Fix In `abevalflow/harbor_extensions/openshift_environment.py`: -1. After `start()`, `mkdir -p` Harbor paths including `/logs/verifier` (no chmod — SCC UIDs get EPERM) +1. After `start()`, `mkdir -p` Harbor paths including `/logs/verifier` (no chmod -- SCC UIDs get EPERM) 2. Override `download_dir` with `set -o pipefail` so missing dirs fail clearly 3. Keep `/workspace`+`/tmp` emptyDirs; log injected mounts 4. `aggregate_aeh.py` emits `AnalysisResult`-compatible fields so analyze does not crash @@ -173,7 +173,7 @@ Blocked initially by PVC quota (`persistentvolumeclaims=10`); freed by deleting **Result:** `DownloadVerifierDirError` **gone**. Trial completed with rewards: - Exceptions: 0 - Exit_Success: 1.000 -- File_Created: 0.000 / Reward: 0.000 (real eval miss — agent did not create `output/greeting.txt`) +- File_Created: 0.000 / Reward: 0.000 (real eval miss -- agent did not create `output/greeting.txt`) - `step-aeh-eval` exits 1 due to AEH `REGRESSIONS` detection (not infra FS) Remaining non-blocker: evaluate task fails on regression exit code; analyze skipped because evaluate failed. diff --git a/Docs/konflux-integration-guide.md b/Docs/konflux-integration-guide.md index 4da1861f..6491504d 100644 --- a/Docs/konflux-integration-guide.md +++ b/Docs/konflux-integration-guide.md @@ -1,12 +1,12 @@ -# ABEvalFlow Konflux Integration Guide +# Agentic Eval Flow Konflux Integration Guide -This guide explains how to integrate ABEvalFlow evaluation into any Konflux -application pipeline. ABEvalFlow provides generic evaluation tasks as Tekton +This guide explains how to integrate Agentic Eval Flow evaluation into any Konflux +application pipeline. Agentic Eval Flow provides generic evaluation tasks as Tekton Bundles that can evaluate A2A agents, MCP servers, and skills. ## Architecture -ABEvalFlow publishes **8 core tasks** as Tekton Bundles: +Agentic Eval Flow publishes **8 core tasks** as Tekton Bundles: ``` parse-snapshot → prepare → test → [red-team] → evaluate → analyze-scorecard → store → emit-result @@ -27,7 +27,7 @@ These tasks handle the entire evaluation lifecycle: | `emit-result` | Map scorecard to Konflux's `TEST_OUTPUT` format | Your application adds its own deployment and cleanup logic around these core -tasks. ABEvalFlow never deploys or manages your application. +tasks. Agentic Eval Flow never deploys or manages your application. ## Parameter Contract @@ -82,7 +82,7 @@ PIPELINE_REPO_REVISION: "main" For `harbor` in Konflux, the local mode runs with `environment.type: local` which does not perform the full scaffold/build/eval cycle. For full Harbor A/B testing -with container registry support, use the standalone ABEvalFlow pipeline on OpenShift. +with container registry support, use the standalone Agentic Eval Flow pipeline on OpenShift. ### Multi-component Applications @@ -113,14 +113,14 @@ The evaluation runs as a Pod on a separate workload cluster. Use this when: - You need the eval Pod co-located with the target for network access Required in this mode: -- `WORKLOAD_CLUSTER_URL` — API URL of the workload cluster -- `WORKLOAD_NAMESPACE` — Namespace to create the eval Pod in +- `WORKLOAD_CLUSTER_URL` -- API URL of the workload cluster +- `WORKLOAD_NAMESPACE` -- Namespace to create the eval Pod in - A Secret (named by `WORKLOAD_CREDENTIALS_SECRET`) with a `token` key containing a ServiceAccount token for the workload cluster ## Submissions -A **submission** is the evaluation definition package. It tells ABEvalFlow what +A **submission** is the evaluation definition package. It tells Agentic Eval Flow what to test and how to judge results. Structure depends on the eval engine: ### A2A Agent Submission @@ -201,7 +201,7 @@ Submissions can live in any Git repository. The pipeline accepts ### Pattern 1: Pre-deployed Target (simplest) If your agent or MCP server is already running (e.g., a long-lived service), -use ABEvalFlow's reference pipeline directly: +use Agentic Eval Flow's reference pipeline directly: ```yaml apiVersion: appstudio.redhat.com/v1beta2 @@ -214,7 +214,7 @@ metadata: spec: application: contexts: - - description: AI evaluation via ABEvalFlow + - description: AI evaluation via Agentic Eval Flow name: application resolverRef: resolver: git @@ -242,7 +242,7 @@ spec: ### Pattern 2: Pipeline-deployed Target If your target needs to be deployed for each evaluation run, create your own -pipeline that wraps ABEvalFlow's core tasks with deploy/cleanup steps. +pipeline that wraps Agentic Eval Flow's core tasks with deploy/cleanup steps. See the [full working example](https://github.com/ikrispin/abevalflow-konflux-example) for the Google Lightspeed Agent. @@ -251,7 +251,7 @@ Key steps: 1. Create a `deploy-.yaml` task that deploys your application and outputs the endpoint URL as a task result 2. Create a `cleanup-.yaml` task for the `finally:` block -3. Create a PipelineRun that chains: deploy → ABEvalFlow core tasks → cleanup +3. Create a PipelineRun that chains: deploy → Agentic Eval Flow core tasks → cleanup 4. Create a submission definition for your evaluation scenarios 5. Create an IntegrationTestScenario pointing to your pipeline @@ -337,7 +337,7 @@ make bundles 2. **Create a submission** defining your evaluation scenarios 3. **Provision secrets** in your Konflux tenant namespace 4. **Create an IntegrationTestScenario** in your tenant namespace -5. **Push a change** to your application — Konflux triggers the evaluation +5. **Push a change** to your application -- Konflux triggers the evaluation For a complete working example, see: [github.com/ikrispin/abevalflow-konflux-example](https://github.com/ikrispin/abevalflow-konflux-example) diff --git a/Docs/manual_trigger_guide.md b/Docs/manual_trigger_guide.md index c9ad9a8a..5c95354c 100644 --- a/Docs/manual_trigger_guide.md +++ b/Docs/manual_trigger_guide.md @@ -1,4 +1,4 @@ -# Manual Trigger Guide — Harbor, ASE & A2A +# Manual Trigger Guide -- Harbor, ASE & A2A Quick reference for manually triggering evaluations against the CI and monitoring pipelines. @@ -28,12 +28,12 @@ A2A monitoring runs are triggered automatically by three sources (plus manual Pi See [A2A Trigger Types](#a2a-trigger-types) below for full PipelineRun examples. -**Working examples — Monitoring pipeline:** +**Working examples -- Monitoring pipeline:** - Harbor: https://console-openshift-console.apps.cn-ai-lab.2vn8.p1.openshiftapps.com/k8s/ns/ab-eval-flow/tekton.dev~v1~PipelineRun/harbor-verify-t6spp/logs?taskName=analyze-and-check-degradation - ASE: https://console-openshift-console.apps.cn-ai-lab.2vn8.p1.openshiftapps.com/k8s/ns/ab-eval-flow/tekton.dev~v1~PipelineRun/ase-verify-sfmlw/logs?taskName=analyze-and-check-degradation - A2A: https://console-openshift-console.apps.cn-ai-lab.2vn8.p1.openshiftapps.com/k8s/ns/ab-eval-flow/tekton.dev~v1~PipelineRun/a2a-local-env-x8nbj/logs?taskName=analyze-and-check-degradation -**Working examples — CI pipeline:** +**Working examples -- CI pipeline:** - Harbor (all files): https://console-openshift-console.apps.cn-ai-lab.2vn8.p1.openshiftapps.com/k8s/ns/ab-eval-flow/tekton.dev~v1~PipelineRun/ci-harbor-with-instr-p92vj - Harbor (AI generation): https://console-openshift-console.apps.cn-ai-lab.2vn8.p1.openshiftapps.com/k8s/ns/ab-eval-flow/tekton.dev~v1~PipelineRun/ci-harbor-gen-instr-mn425 - ASE (all files): https://console-openshift-console.apps.cn-ai-lab.2vn8.p1.openshiftapps.com/k8s/ns/ab-eval-flow/tekton.dev~v1~PipelineRun/ci-ase-with-evals-7thhz @@ -55,7 +55,7 @@ Runs evaluations with degradation check and Slack notifications. No security sca ## Harbor Run (hello-world) -Uses `skill-submissions/hello-world` — a minimal Harbor task submission. +Uses `skill-submissions/hello-world` -- a minimal Harbor task submission. ```bash oc create -f - <<'YAML' @@ -101,7 +101,7 @@ YAML ## ASE Run (hello-world-full) -Uses `skill-submissions/hello-world-full` — a skill submission with `evals.json`. +Uses `skill-submissions/hello-world-full` -- a skill submission with `evals.json`. ```bash oc create -f - <<'YAML' @@ -141,7 +141,7 @@ YAML ## A2A Run (lightspeed-agent) -Uses `ABEvalFlow/a2a-agent-eval` — evaluates the deployed Lightspeed A2A agent. +Uses `Agentic Eval Flow/a2a-agent-eval` -- evaluates the deployed Lightspeed A2A agent. The agent must already be running at `http://lightspeed-agent.ab-eval-flow.svc:8000`. ```bash @@ -211,7 +211,7 @@ Fires when any file under `config/litellm/` (e.g., `config/litellm/configmap.yam - EventListener trigger: `litellm-config-push-trigger` - Uses the existing deployed agent at `http://lightspeed-agent.ab-eval-flow.svc:8000` -- **Requires:** GitHub webhook on the ABEvalFlow repo pointing to: +- **Requires:** GitHub webhook on the Agentic Eval Flow repo pointing to: `https://el-submission-listener-ab-eval-flow.apps.cn-ai-lab.2vn8.p1.openshiftapps.com` ### 3. 10-Day Scheduled @@ -264,7 +264,7 @@ spec: YAML ``` -Example for ephemeral deploy (Quay webhook equivalent — evaluates a specific image tag): +Example for ephemeral deploy (Quay webhook equivalent -- evaluates a specific image tag): ```bash oc create -f - <<'YAML' @@ -316,9 +316,9 @@ YAML Full evaluation run: prepare → test → evaluate → analyze → store. No degradation check or Slack. Security scan and quality review can be enabled/disabled per run. -## CI Harbor — all files present (no generation) +## CI Harbor -- all files present (no generation) -Uses `skill-submissions/hello-world` — has `instruction.md` and `tests/`. Generation is enabled +Uses `skill-submissions/hello-world` -- has `instruction.md` and `tests/`. Generation is enabled but skips because files already exist. Expects no `generated/` folder in MinIO. ```bash @@ -367,9 +367,9 @@ YAML --- -## CI Harbor — AI generation (no instruction.md) +## CI Harbor -- AI generation (no instruction.md) -Uses `skill-submissions/hello-world-no-instr` (PR #106) — only has `skills/SKILL.md` and +Uses `skill-submissions/hello-world-no-instr` (PR #106) -- only has `skills/SKILL.md` and `metadata.yaml` with `generation_mode: ai`. Pipeline generates `instruction.md`, `test_outputs.py`, and `llm_judge.py`. @@ -419,9 +419,9 @@ YAML --- -## CI ASE — all files present (no generation) +## CI ASE -- all files present (no generation) -Uses `skill-submissions/hello-world-full` — has `evals/evals.json`. Expects no `generated/` folder. +Uses `skill-submissions/hello-world-full` -- has `evals/evals.json`. Expects no `generated/` folder. ```bash oc create -f - <<'YAML' @@ -465,9 +465,9 @@ YAML --- -## CI ASE — AI generation (no evals.json) +## CI ASE -- AI generation (no evals.json) -Uses `skill-submissions/hello-world-minimal` (PR #104) — only has `SKILL.md`. Pipeline generates +Uses `skill-submissions/hello-world-minimal` (PR #104) -- only has `SKILL.md`. Pipeline generates `evals/evals.json` with `_generated_by: "ai"`. ```bash @@ -512,7 +512,7 @@ YAML --- -## CI MCPChecker — ExploitIQ +## CI MCPChecker -- ExploitIQ Uses `skill-submissions/exploitiq-mcp-eval` (branch `test/mcpchecker-exploitiq`). Requires `exploitiq-mcp-credentials` secret with a fresh `oc whoami -t` token from `ai-dev03`. @@ -572,7 +572,7 @@ YAML ## CI A2A -Uses `ABEvalFlow/a2a-agent-eval`. Security scan and quality review disabled (no test files). +Uses `Agentic Eval Flow/a2a-agent-eval`. Security scan and quality review disabled (no test files). ```bash oc create -f - <<'YAML' @@ -638,8 +638,8 @@ oc get pipelinerun -n ab-eval-flow --no-headers | grep "False" \ ## Slack Notifications Every completed monitoring run sends a Slack message to the team channel: -- ✅ `[ENGINE] Monitoring Pass` — score + baseline + ratio -- 🚨 `[ENGINE] Performance Degradation Detected` — score dropped below threshold +- ✅ `[ENGINE] Monitoring Pass` -- score + baseline + ratio +- 🚨 `[ENGINE] Performance Degradation Detected` -- score dropped below threshold The Run ID in the message is a clickable link to the OpenShift console. @@ -647,7 +647,7 @@ The Run ID in the message is a clickable link to the OpenShift console. ## Active Image -All eval steps (`harbor-eval`, `a2a-eval`) use `eval-base:local-env` — built from +All eval steps (`harbor-eval`, `a2a-eval`) use `eval-base:local-env` -- built from Harbor `feature/local-environment` branch with `claude-code` pre-installed. This is the canonical image; do not revert to `:latest`. @@ -661,7 +661,7 @@ This is the canonical image; do not revert to `:latest`. | `NonZeroAgentExitCodeError` in Harbor eval | Local registry `eval-base` out of sync with main registry | Re-run the skopeo sync (see `infrastructure_ops.md`) | | `Generated files: []` despite generation running | LLM call failed silently | Check stderr output in logs (now shown on failure) | | `generated/` folder missing despite AI generation | Files already existed (no AI header) or generation failed | Check `STEP-GENERATE-TESTS` logs for WARNING output | -| `debug/` folder missing in MinIO for Harbor runs | `results-dir` not passed to store task | Fixed in PR #32 — ensure `pipeline-repo-revision: main` | +| `debug/` folder missing in MinIO for Harbor runs | `results-dir` not passed to store task | Fixed in PR #32 -- ensure `pipeline-repo-revision: main` | | Degradation shows `0.00% → 0.00%` | `store` runs after `check-degradation`; monitor reads old DB runs | Fixed: current score passed from `report.json` via `--current-score` | | No Slack alert despite degradation | `\|\|` block caught exit code 1 from `monitor.py` | Fixed: only exit code 2 (error) is non-blocking now | | Slack Run ID shows `None` | `--run-id` not passed to `monitor.py` | Fixed: `--run-id "$(params.pipeline-run-id)"` now wired in task | diff --git a/Docs/mlflow_dashboard_proposal.md b/Docs/mlflow_dashboard_proposal.md index 0379c78e..289ed9b9 100644 --- a/Docs/mlflow_dashboard_proposal.md +++ b/Docs/mlflow_dashboard_proposal.md @@ -1,6 +1,6 @@ -# MLflow Dashboard Proposal — ABEvalFlow Observability +# MLflow Dashboard Proposal -- Agentic Eval Flow Observability -Phase C+D of APPENG-5370. Using MLflow's built-in dashboard and experiment tracking to visualize pipeline results, gate scores, certification levels, and LLM token usage — all in one place. +Phase C+D of APPENG-5370. Using MLflow's built-in dashboard and experiment tracking to visualize pipeline results, gate scores, certification levels, and LLM token usage -- all in one place. Each pipeline run becomes an MLflow run under an experiment (one experiment per submission). Metrics are logged from our existing PostgreSQL tables. @@ -72,10 +72,10 @@ Each submission (e.g. `hello-world`, `my-skill`) is an MLflow experiment. Every - See how uplift, pass rate, and gate scores changed between runs ### Metric charts (built-in) -- `treatment_pass_rate` over runs — is the skill improving? -- `gate_score_security` over runs — are security scores trending up? -- `total_tokens` over runs — is token usage growing? -- `uplift` over runs — is the skill's impact stable? +- `treatment_pass_rate` over runs -- is the skill improving? +- `gate_score_security` over runs -- are security scores trending up? +- `total_tokens` over runs -- is token usage growing? +- `uplift` over runs -- is the skill's impact stable? ### Experiment overview - Table of all runs with sortable columns (pass rate, uplift, certification, tokens) diff --git a/Docs/observability_tables.md b/Docs/observability_tables.md index 8a56a988..f8ee8738 100644 --- a/Docs/observability_tables.md +++ b/Docs/observability_tables.md @@ -28,7 +28,7 @@ These tables are populated automatically in the **store** step of the Tekton pip 2. `store_results.py` reads `_metrics_checkpoint.json` → writes to `observability_metrics` 3. The `_metrics_checkpoint.json` is created in the **test** phase finalize step from quality review token data -No configuration needed — the `enable-scorecard` pipeline flag defaults to `true`. +No configuration needed -- the `enable-scorecard` pipeline flag defaults to `true`. --- @@ -39,7 +39,7 @@ One row per pipeline run. Stores the unified gate verdict combining all evaluati | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `id` | UUID | No | Primary key, auto-generated | -| `pipeline_run_id` | VARCHAR(255) | No | Tekton PipelineRun name. Unique — one scorecard per run | +| `pipeline_run_id` | VARCHAR(255) | No | Tekton PipelineRun name. Unique -- one scorecard per run | | `submission_name` | VARCHAR(255) | No | Name of the evaluated skill/agent submission | | `eval_engine` | VARCHAR(50) | No | Evaluation engine used: harbor, ase, aeh, mcpchecker, a2a, both | | `recommendation` | VARCHAR(20) | No | Unified verdict: pass, warn, or fail | @@ -106,7 +106,7 @@ One row per certification level per pipeline run. Maximum 3 rows per scorecard ( **Indexes:** `scorecard_id`, `(level, passed)` -**Constraints:** Unique on `(scorecard_id, level)` — prevents duplicate certification rows for the same level. +**Constraints:** Unique on `(scorecard_id, level)` -- prevents duplicate certification rows for the same level. **Relationships:** Belongs to one `scorecard` (CASCADE delete on parent). @@ -114,7 +114,7 @@ One row per certification level per pipeline run. Maximum 3 rows per scorecard ( ## Table: `observability_metrics` -One row per pipeline run attempt with aggregated LLM token usage and phase timing. Enables cost tracking and performance monitoring. Retries create additional rows with incrementing `attempt_number`. No foreign key to scorecards — metrics may outlive scorecard lifecycle. +One row per pipeline run attempt with aggregated LLM token usage and phase timing. Enables cost tracking and performance monitoring. Retries create additional rows with incrementing `attempt_number`. No foreign key to scorecards -- metrics may outlive scorecard lifecycle. | Column | Type | Nullable | Description | |--------|------|----------|-------------| @@ -139,5 +139,5 @@ One row per pipeline run attempt with aggregated LLM token usage and phase timin **Indexes:** `submission_name`, `created_at`, `model_name` -**Constraints:** Unique on `(pipeline_run_id, attempt_number)` — prevents duplicate metrics for the same run attempt. +**Constraints:** Unique on `(pipeline_run_id, attempt_number)` -- prevents duplicate metrics for the same run attempt. diff --git a/Docs/persistence.md b/Docs/persistence.md new file mode 100644 index 00000000..c5f2b046 --- /dev/null +++ b/Docs/persistence.md @@ -0,0 +1,36 @@ +# Persistence + +## MinIO (Object Storage) + +Reports and artifacts are uploaded to MinIO under a timestamped prefix: + +``` +s3://ab-eval-reports/YYYYMMDD_hhmmss_{submission}_{run-id}/ +├── report.json # Main evaluation report +├── report.md # Human-readable report +├── scorecard.json # Unified scorecard +├── security_scans/ # Security scan results +│ └── security-scan.json +├── generated/ # AI-generated artifacts +│ ├── instruction.md +│ └── test_outputs.py +├── scaffolded/ # Scaffolded configs and review +│ └── _ai_review.json +└── trials/ # Per-trial artifacts (Harbor) + ├── trial_001/ + │ ├── agent/ + │ └── verifier/ + └── ... +``` + +## PostgreSQL (Results Database) + +Evaluation results are persisted for historical analysis and monitoring: + +- **Script:** `scripts/store_results.py` +- **Data stored:** + - Submission metadata + - Per-trial results (Harbor/ASE) + - Security scan findings + - Aggregate statistics + - Scorecard recommendation diff --git a/Docs/results_persistence_and_observability_plan.md b/Docs/results_persistence_and_observability_plan.md index df720e6f..aea75125 100644 --- a/Docs/results_persistence_and_observability_plan.md +++ b/Docs/results_persistence_and_observability_plan.md @@ -2,7 +2,7 @@ > **Jira:** APPENG-4985 (Results Persistence & Storage) > **Branch:** `APPENG-4985/results-persistence` -> **Depends on:** APPENG-4907 (`abevalflow/report.py` — `AnalysisResult` model) +> **Depends on:** APPENG-4907 (`abevalflow/report.py` -- `AnalysisResult` model) --- @@ -12,7 +12,7 @@ Persist every A/B evaluation run to PostgreSQL so results are queryable, compara --- -## Scope — What This Ticket Covers (ABEvalFlow side) +## Scope -- What This Ticket Covers (Agentic Eval Flow side) | Deliverable | Description | |---|---| @@ -21,12 +21,12 @@ Persist every A/B evaluation run to PostgreSQL so results are queryable, compara | `store_results.py` | CLI script: reads `report.json`, persists to PostgreSQL | | `query_results.py` | CLI script: historical queries (`list`, `latest`, `history`, `compare`) | | Observer protocol | Pluggable `ResultsObserver` interface for future backends | -| Tekton task | `store-results.yaml` — runs after `analyze-report` | +| Tekton task | `store-results.yaml` -- runs after `analyze-report` | | OpenShift manifests | PostgreSQL StatefulSet, Service, Secret template, PVC | | RBAC update | Pipeline SA gets read access to DB Secret | | Tests | Full coverage using SQLite in-memory (no PostgreSQL needed for CI) | -## Scope — What This Ticket Does NOT Cover +## Scope -- What This Ticket Does NOT Cover | Out of scope | Where it belongs | |---|---| @@ -120,18 +120,18 @@ One row per trial. Enables drill-down queries. --- -## 2. DB Engine — `abevalflow/db/engine.py` +## 2. DB Engine -- `abevalflow/db/engine.py` -- `get_engine(url: str | None = None) -> Engine` — creates SQLAlchemy engine from URL -- `init_db(engine: Engine)` — `Base.metadata.create_all(engine)`, retries up to 5 times with exponential backoff on `OperationalError` (via `tenacity`) for cold-start resilience; non-transient errors (auth, SSL) fail immediately -- `make_session(engine: Engine) -> sessionmaker[Session]` — returns a session factory bound to the given engine +- `get_engine(url: str | None = None) -> Engine` -- creates SQLAlchemy engine from URL +- `init_db(engine: Engine)` -- `Base.metadata.create_all(engine)`, retries up to 5 times with exponential backoff on `OperationalError` (via `tenacity`) for cold-start resilience; non-transient errors (auth, SSL) fail immediately +- `make_session(engine: Engine) -> sessionmaker[Session]` -- returns a session factory bound to the given engine - Connection URL from `DATABASE_URL` env var - Format: `postgresql+psycopg://user:pass@host:5432/abevalflow` - Falls back to SQLite for local dev / testing --- -## 3. Observer Protocol — `abevalflow/db/observer.py` +## 3. Observer Protocol -- `abevalflow/db/observer.py` ```python from typing import Protocol @@ -161,11 +161,11 @@ class ResultsObserver(Protocol): - `LANGFUSE_PUBLIC_KEY` set → load `LangfuseObserver` (future) - No env vars → no observers, PostgreSQL only -**Error isolation:** Each observer runs in a `try/except` — observer failures are logged as warnings, never fail the pipeline. +**Error isolation:** Each observer runs in a `try/except` -- observer failures are logged as warnings, never fail the pipeline. --- -## 4. Store Script — `scripts/store_results.py` +## 4. Store Script -- `scripts/store_results.py` ``` python scripts/store_results.py \ @@ -179,8 +179,8 @@ which emits a directory containing `report.json` and `report.md`. Logic: 1. Load and validate `{report-dir}/report.json` via `AnalysisResult.model_validate_json()` 2. Map `AnalysisResult` → `EvaluationRun` row (flatten provenance + summary) -3. Map each trial → `Trial` row (persist from `TrialResult.model_dump()` which includes the computed `passed` field — same rule as the Pydantic model) -4. Single transaction — all-or-nothing +3. Map each trial → `Trial` row (persist from `TrialResult.model_dump()` which includes the computed `passed` field -- same rule as the Pydantic model) +4. Single transaction -- all-or-nothing 5. **Idempotent:** if `pipeline_run_id` already exists, log warning and skip 6. After commit, invoke all registered `ResultsObserver` instances 7. Exit 0 on success, exit 1 on failure @@ -195,7 +195,7 @@ the host/db name only (masked connection string). --- -## 5. Query Script — `scripts/query_results.py` +## 5. Query Script -- `scripts/query_results.py` ``` python scripts/query_results.py list @@ -215,7 +215,7 @@ Output: formatted table to stdout (simple column alignment, no heavy dependency) --- -## 6. Tekton Task — `pipeline/tasks/store-results.yaml` +## 6. Tekton Task -- `pipeline/tasks/store-results.yaml` - Runs after `analyze-report` in the pipeline - Params: `report-dir`, `submission-name`, `pipeline-run-id`, `pipeline-repo-url`, `pipeline-repo-revision` @@ -228,13 +228,13 @@ Output: formatted table to stdout (simple column alignment, no heavy dependency) --- -## 7. OpenShift Manifests — `config/postgres/` +## 7. OpenShift Manifests -- `config/postgres/` | File | Content | |---|---| | `config/postgres/statefulset.yaml` | PostgreSQL 16 StatefulSet (single replica MVP) with inline `volumeClaimTemplates` (10Gi) | | `config/postgres/service.yaml` | ClusterIP Service (`ab-eval-db:5432`) | -| `config/postgres/secret.yaml` | Secret template (placeholder values — never real credentials) | +| `config/postgres/secret.yaml` | Secret template (placeholder values -- never real credentials) | Update `config/rbac.yaml`: grant pipeline SA read access to `ab-eval-db-credentials` Secret. @@ -262,8 +262,8 @@ Add to `pyproject.toml`: | `tests/test_store_results.py` | Store, idempotency, observer invocation, error handling | | `tests/test_query_results.py` | All subcommands against seeded data | -All unit tests use **SQLite in-memory / file-based** — no PostgreSQL required for CI. -An optional PostgreSQL integration test (deferred — add `@pytest.mark.skipif` +All unit tests use **SQLite in-memory / file-based** -- no PostgreSQL required for CI. +An optional PostgreSQL integration test (deferred -- add `@pytest.mark.skipif` when PostgreSQL is deployed) would validate dialect-specific behavior (JSONB, UUID). --- @@ -274,7 +274,7 @@ when PostgreSQL is deployed) would validate dialect-specific behavior (JSONB, UU - **Prefer taking from `main`** once APPENG-4907 merges - **Cherry-picked** onto this branch as commit `9567429` (from `APPENG-4907/analysis-reporting`) -- Both branches produce the identical file — git auto-merges cleanly when both land on main +- Both branches produce the identical file -- git auto-merges cleanly when both land on main - The store script imports `AnalysisResult` directly for type-safe validation at ingest time --- @@ -307,19 +307,19 @@ when PostgreSQL is deployed) would validate dialect-specific behavior (JSONB, UU --- -## Harbor Side — Observability Integration Points +## Harbor Side -- Observability Integration Points > This section documents what the Harbor fork needs for full observability. -> These changes are **not** part of APPENG-4985 — they belong in the Harbor fork repo. +> These changes are **not** part of APPENG-4985 -- they belong in the Harbor fork repo. ### Layer 1: Trial-Level LLM Tracing (Harbor Fork) Each Harbor trial pod runs an LLM agent. To trace those calls: -1. **OTel SDK in trial pods** — add `opentelemetry-sdk` + `opentelemetry-exporter-otlp` to the trial container's dependencies -2. **Configure via env vars** — pass `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME` through Harbor's `environment_kwargs` or task config -3. **Auto-instrumentation** — LiteLLM (if used) has OTel auto-instrumentation; otherwise manual spans around agent calls -4. **Backend routing** — the OTel collector routes traces to whichever backend is deployed: +1. **OTel SDK in trial pods** -- add `opentelemetry-sdk` + `opentelemetry-exporter-otlp` to the trial container's dependencies +2. **Configure via env vars** -- pass `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME` through Harbor's `environment_kwargs` or task config +3. **Auto-instrumentation** -- LiteLLM (if used) has OTel auto-instrumentation; otherwise manual spans around agent calls +4. **Backend routing** -- the OTel collector routes traces to whichever backend is deployed: - Langfuse (supports OTel ingestion) - MLflow (uses OTel internally for tracing) - Grafana Tempo / Jaeger (native OTel backends) @@ -328,20 +328,20 @@ Each Harbor trial pod runs an LLM agent. To trace those calls: Harbor already writes `result.json` per trial with `verifier_result.rewards.reward`, timing, and token usage. To surface aggregate metrics: -1. **Post-job callback** — after all trials complete, Harbor could call a webhook or write a summary JSON -2. **MLflow integration** — the existing `log_to_mlflow.py` pattern (currently a standalone script in the local clone) could be integrated into Harbor's CLI as `harbor log --backend mlflow` -3. **OTel metrics** — emit gauge/counter metrics for pass rate, mean reward, trial count using OTel Metrics API +1. **Post-job callback** -- after all trials complete, Harbor could call a webhook or write a summary JSON +2. **MLflow integration** -- the existing `log_to_mlflow.py` pattern (currently a standalone script in the local clone) could be integrated into Harbor's CLI as `harbor log --backend mlflow` +3. **OTel metrics** -- emit gauge/counter metrics for pass rate, mean reward, trial count using OTel Metrics API ### Layer 3: Connecting Both Layers The `pipeline_run_id` (Tekton run name) is the join key: -- ABEvalFlow stores it in `evaluation_runs.pipeline_run_id` (PostgreSQL) +- Agentic Eval Flow stores it in `evaluation_runs.pipeline_run_id` (PostgreSQL) - Harbor trial pods can set it as an OTel resource attribute (`pipeline.run.id`) - Any observability backend can correlate evaluation results with individual LLM traces using this key ### Recommended Sequence -1. **Now (APPENG-4985):** PostgreSQL + observer protocol in ABEvalFlow ← this ticket +1. **Now (APPENG-4985):** PostgreSQL + observer protocol in Agentic Eval Flow ← this ticket 2. **Next:** Pick an observability backend (MLflow or Langfuse) and implement one `ResultsObserver` adapter (~50 lines) 3. **Then:** Add OTel SDK to Harbor trial pods for LLM call tracing 4. **Finally:** Grafana dashboards over PostgreSQL for trend monitoring diff --git a/Docs/submission-formats.md b/Docs/submission-formats.md new file mode 100644 index 00000000..50a9125e --- /dev/null +++ b/Docs/submission-formats.md @@ -0,0 +1,133 @@ +# Submission Formats + +## Skill Submission (Harbor) + +For full agent evaluation with container isolation and A/B comparison: + +``` +my-skill/ +├── instruction.md # Task description (required, or generated from SKILL.md) +├── skills/ +│ └── SKILL.md # Skill definition (required) +├── tests/ +│ ├── test_outputs.py # Verification tests (required, or generated) +│ └── llm_judge.py # LLM-based judge (optional) +├── docs/ # Reference documentation (optional) +├── supportive/ # Mock MCPs, data files (optional, <50MB) +└── metadata.yaml # eval_engine: harbor (required) +``` + +## Skill Submission (ASE) + +For lightweight LLM-as-judge evaluation without containers: + +``` +my-skill/ +├── skills/ +│ └── SKILL.md # Skill definition (required) +├── evals/ +│ ├── evals.json # Evaluation prompts and assertions (optional, generated if missing) +│ └── files/ # Test data files (optional) +└── metadata.yaml # eval_engine: ase (required) +``` + +## MCP Server Submission + +For validating MCP server implementations: + +``` +my-mcp-server-eval/ +├── metadata.yaml # eval_engine: mcpchecker (required) +├── mcp-config.yaml # MCP server connection settings (required) +│ # - url: MCP server endpoint +│ # - auth: authentication config (if needed) +└── tasks/ + ├── task-1.yaml # Task definition with expected tool calls + └── task-2.yaml # Each task tests specific MCP functionality +``` + +MCPChecker validates that the MCP server correctly handles tool invocations and returns expected results. + +## Agent Submission (A2A Protocol) + +For evaluating agents that implement the A2A (Agent-to-Agent) protocol: + +``` +my-a2a-agent-eval/ +├── metadata.yaml # eval_engine: a2a (required) +├── agent-config.yaml # Agent endpoint and auth config (required) +│ # - endpoint: http://agent-service:8000 +│ # - auth: bearer token or API key +└── tasks/ + ├── instruction.md # Task description + ├── tests/ + │ └── test_outputs.py + └── task.toml # Task configuration +``` + +A2A evaluation connects to a deployed agent via the A2A protocol and runs evaluation tasks against it. + +## Agent Submission (Harbor) + +For evaluating general agents (non-A2A) with full container isolation: + +``` +my-agent-eval/ +├── metadata.yaml # eval_engine: harbor, persona: agent (required) +├── instruction.md # Task description (required) +├── tests/ +│ ├── test_outputs.py # Verification tests (required) +│ └── llm_judge.py # LLM-based judge (optional) +└── supportive/ # Environment files, data (optional) +``` + +Harbor creates treatment/control container variants and runs A/B comparison. + +## AEH Submission (Agent-Eval-Harness) + +For evaluating agents using the Agent-Eval-Harness framework with flexible LLM judges. +Trials run as Harbor jobs on OpenShift. + +**Single** (`aeh-mode=single`): + +``` +my-aeh-eval/ +├── metadata.yaml # eval_engine: aeh (required) +├── eval.yaml # AEH config (models, judges, thresholds, outputs) +├── skills/…/SKILL.md # Optional skill package +└── cases/ + └── case-001/ + └── input.yaml +``` + +**Pairwise** (`aeh-mode=pairwise`): + +``` +my-aeh-pairwise/ +├── metadata.yaml +├── eval-control.yaml # Baseline (often unskilled) +├── eval-treatment.yaml # Treatment + pairwise LLM judge +├── skills//SKILL.md +└── cases/ + └── case-001/ + └── input.yaml +``` + +Verified smoke samples in [skill-submissions](https://github.com/RHEcosystemAppEng/skill-submissions): +- Branch `eval/aeh-hello-world-single` -> `aeh-hello-world-single` +- Branch `eval/aeh-hello-world-pairwise` -> `aeh-hello-world-pairwise` + +Working trigger defaults: LiteLLM `claude-sonnet` via +`http://litellm.ab-eval-flow.svc.cluster.local:4000`, AEH image +`quay.io/ecosystem-appeng/agent-eval-harness:v1.0.3`. + +MinIO layout for AEH: `debug/harbor/` (raw Harbor jobs) + `debug/aeh/` +(`summary.yaml`, `report.html`, `cases/`). Pairwise HTML is regenerated with +`--baseline` so treatment `report.html` includes the pairwise section. + +See [Trigger Guide](trigger_guide.md) for full YAML examples, or run: + +```bash +./scripts/misc/trigger_test_runs.sh # all engines including AEH +./scripts/misc/trigger_test_runs.sh "$(git branch --show-current)" aeh # AEH only +``` diff --git a/Docs/trigger_guide.md b/Docs/trigger_guide.md index cfb6cb29..7f23f879 100644 --- a/Docs/trigger_guide.md +++ b/Docs/trigger_guide.md @@ -1,6 +1,6 @@ # How to Submit a Skill for A/B Evaluation -This guide explains how to submit a skill to the ABEvalFlow pipeline for +This guide explains how to submit a skill to the Agentic Eval Flow pipeline for automated evaluation. By the end, you'll know what files to prepare, how to submit them, and what happens next. @@ -8,7 +8,7 @@ submit them, and what happens next. ## What is this pipeline? -ABEvalFlow automatically tests whether an AI agent performs better **with** +Agentic Eval Flow automatically tests whether an AI agent performs better **with** your skill than **without** it. It does this by running the same task many times in two configurations: @@ -115,10 +115,10 @@ use the ASE format: submissions// ├── metadata.yaml # Required ├── skills/ -│ └── SKILL.md # Required — skill definition +│ └── SKILL.md # Required -- skill definition └── evals/ - ├── evals.json # Optional — generated if missing - └── files/ # Optional — test data files + ├── evals.json # Optional -- generated if missing + └── files/ # Optional -- test data files ``` Trigger with `eval-engine=ase`: @@ -144,11 +144,11 @@ MCP tools correctly and produce valid outputs. ``` submissions// -├── metadata.yaml # Required — eval_engine: mcpchecker -├── eval.yaml # Required — MCPChecker evaluation config -├── mcp-config.yaml # Required — MCP server connection settings +├── metadata.yaml # Required -- eval_engine: mcpchecker +├── eval.yaml # Required -- MCPChecker evaluation config +├── mcp-config.yaml # Required -- MCP server connection settings └── tasks/ - └── *.yaml # Required — at least one task definition + └── *.yaml # Required -- at least one task definition ``` **eval.yaml example:** @@ -215,12 +215,12 @@ from other agents using standardized A2A messaging. ``` submissions// -├── metadata.yaml # Required — eval_engine: a2a -├── agent_card.json # Required — A2A agent card +├── metadata.yaml # Required -- eval_engine: a2a +├── agent_card.json # Required -- A2A agent card ├── skills/ -│ └── SKILL.md # Required — skill definition +│ └── SKILL.md # Required -- skill definition └── evals/ - └── evals.json # Optional — evaluation scenarios (generated if missing) + └── evals.json # Optional -- evaluation scenarios (generated if missing) ``` **agent_card.json example:** @@ -279,14 +279,14 @@ OpenShift (OpenShiftEnvironment). ``` submissions// -├── metadata.yaml # Required — eval_engine: aeh -├── eval.yaml # Required — AEH evaluation config (judges, thresholds) -├── skills/ # Optional — nested or flat SKILL.md for treatment +├── metadata.yaml # Required -- eval_engine: aeh +├── eval.yaml # Required -- AEH evaluation config (judges, thresholds) +├── skills/ # Optional -- nested or flat SKILL.md for treatment │ └── …/SKILL.md └── cases/ └── case-001/ - ├── input.yaml # Required — task prompt / agent instruction - └── annotations.yaml # Optional — ground truth for judges + ├── input.yaml # Required -- task prompt / agent instruction + └── annotations.yaml # Optional -- ground truth for judges ``` **eval.yaml essentials** (LiteLLM model ids, not Anthropic native names): @@ -397,11 +397,11 @@ AEH-specific parameters: - `aeh-mode`: `single` (default) or `pairwise` - `aeh-control-config` / `aeh-treatment-config`: Pairwise config filenames (defaults: `eval-control.yaml` / `eval-treatment.yaml`) - `aeh-image`: Harbor trial image (use `quay.io/ecosystem-appeng/agent-eval-harness:v1.0.3` or newer) -- `aeh-runner`: Execution backend — currently `harbor` only +- `aeh-runner`: Execution backend -- currently `harbor` only **Note on execution backends:** - **harbor** (default): Containerized execution in OpenShift trial pods via AEH’s OpenShiftEnvironment. -- **vanilla**: Not yet implemented in ABEvalFlow. +- **vanilla**: Not yet implemented in Agentic Eval Flow. #### AEH Pairwise A/B Testing @@ -413,9 +413,9 @@ Pairwise runs control then treatment on the same cases, then `score.py pairwise` ``` submissions// -├── metadata.yaml # Required — eval_engine: aeh -├── eval-control.yaml # Required — control/baseline (often skill: "") -├── eval-treatment.yaml # Required — treatment (with skill package) +├── metadata.yaml # Required -- eval_engine: aeh +├── eval-control.yaml # Required -- control/baseline (often skill: "") +├── eval-treatment.yaml # Required -- treatment (with skill package) ├── skills//SKILL.md # Treatment skill (optional nested layout) └── cases/ └── case-001/ @@ -424,7 +424,7 @@ submissions// Both configs must share the same `skill:` namespace used for `$AGENT_EVAL_RUNS_DIR///`. Control may set `skill: ""` for an -unskilled baseline while treatment sets the real skill package name — the +unskilled baseline while treatment sets the real skill package name -- the pipeline still keys runs under the treatment skill name. **outputs:** is required when using a pairwise LLM judge (artifacts must be @@ -487,7 +487,7 @@ YAML | `wins_a` | Treatment preferred | | `wins_b` | Control preferred | | `ties` | No preference | -| `win_rate` | `wins_a / (wins_a + wins_b + ties)` — ties are non-wins | +| `win_rate` | `wins_a / (wins_a + wins_b + ties)` -- ties are non-wins | Recommendation: pass when treatment wins ≥50% of cases, **or** when the run is all-ties (no decisive losses). Scores still report the honest `win_rate` (e.g. @@ -499,7 +499,7 @@ all-ties (no decisive losses). Scores still report the honest `win_rate` (e.g. |------|----------| | `debug/aeh/treatment-*/summary.yaml` → `pairwise:` | Wins/ties + LLM reasoning (AEH native) | | `debug/aeh/treatment-*/report.html` | HTML regenerated with `--baseline` (includes pairwise) | -| `report.json` → `pairwise` | Aggregated ABEvalFlow report | +| `report.json` → `pairwise` | Aggregated Agentic Eval Flow report | | `debug/harbor/control|treatment//` | Raw Harbor trial trees | | `debug/aeh/control-*/` / `treatment-*/` | AEH mapped runs (`summary.yaml`, `run_result.json`, `cases/`) | @@ -516,7 +516,7 @@ instruction and tests automatically, set `generation_mode: ai` in submissions// ├── metadata.yaml # Must include: generation_mode: ai └── skills/ - └── SKILL.md # Required — the pipeline generates the rest + └── SKILL.md # Required -- the pipeline generates the rest ``` **Harbor mode:** The pipeline will use an LLM to generate `instruction.md` and @@ -593,14 +593,14 @@ gate_policy: ``` Gate modes: -- `disabled` — Gate is skipped entirely -- `warn` — Gate runs but failures don't block the pipeline -- `block` — Gate failures cause the scorecard to fail +- `disabled` -- Gate is skipped entirely +- `warn` -- Gate runs but failures don't block the pipeline +- `block` -- Gate failures cause the scorecard to fail Combination modes: -- `all_pass` — All blocking gates must pass (default) -- `any_pass` — At least one blocking gate must pass -- `weighted` — Weighted average of scores (≥0.7 pass, 0.5-0.7 warn, <0.5 fail) +- `all_pass` -- All blocking gates must pass (default) +- `any_pass` -- At least one blocking gate must pass +- `weighted` -- Weighted average of scores (≥0.7 pass, 0.5-0.7 warn, <0.5 fail) For Compass Facts integration (pushing evaluation results to Red Hat Compass): @@ -651,7 +651,7 @@ that returns a personalized greeting. ### skills/SKILL.md (required) The skill file that will be loaded into the agent during the **treatment** -runs. This is what you're evaluating — the guidance that should make the +runs. This is what you're evaluating -- the guidance that should make the agent perform better. Example: ```markdown @@ -749,18 +749,18 @@ Typical runtime: **5-30 minutes** depending on evaluation engine and task comple - **Tekton results:** task outputs available in PipelineRun status **MinIO (S3 object storage):** -- `report.json` / `report.md` — evaluation report with pass rates, uplift, p-values -- `scorecard.json` — unified verdict combining all evaluation gates (see below) -- `security-scan.json` / `security-scan.sarif` — security scan findings -- `generated/` — AI-generated files (instruction.md, test_outputs.py, evals.json) -- `debug/` — engine-specific trial / run trees +- `report.json` / `report.md` -- evaluation report with pass rates, uplift, p-values +- `scorecard.json` -- unified verdict combining all evaluation gates (see below) +- `security-scan.json` / `security-scan.sarif` -- security scan findings +- `generated/` -- AI-generated files (instruction.md, test_outputs.py, evals.json) +- `debug/` -- engine-specific trial / run trees - **Harbor / A2A:** `debug/` trial trees (`agent/`, `verifier/`, …) - **AEH:** `debug/harbor/{control,treatment}//` (raw Harbor jobs) and `debug/aeh//` (`summary.yaml`, `report.html`, `run_result.json`, `cases/`) **PostgreSQL database:** -- `analysis_results` table — evaluation summaries (pass rates, uplift, p-values) -- `security_scans` table — security scan results per pipeline run +- `analysis_results` table -- evaluation summaries (pass rates, uplift, p-values) +- `security_scans` table -- security scan results per pipeline run - Historical results queryable via `scripts/query_results.py` ### Understanding the Scorecard @@ -780,9 +780,9 @@ Each gate produces: - `findings`: list of issues found (for security/quality gates) The `scorecard.json` combines all gate results into a single recommendation: -- **pass** — All blocking gates passed -- **warn** — Warning gates failed but no blocking gates failed -- **fail** — One or more blocking gates failed +- **pass** -- All blocking gates passed +- **warn** -- Warning gates failed but no blocking gates failed +- **fail** -- One or more blocking gates failed Example scorecard output: ```json @@ -935,8 +935,8 @@ prompt comparison, custom). Set `experiment.type` in `metadata.yaml`. See `Docs/trigger_models_and_experiment_types.md` for details. **Q: Who do I contact for help?** -Reach out to the ABEvalFlow team or open an issue in the -[ABEvalFlow repository](https://github.com/RHEcosystemAppEng/agentic_eval_flow). +Reach out to the Agentic Eval Flow team or open an issue in the +[Agentic Eval Flow repository](https://github.com/RHEcosystemAppEng/agentic_eval_flow). --- diff --git a/Docs/trigger_models_and_experiment_types.md b/Docs/trigger_models_and_experiment_types.md index efd60f03..d4cf6b8a 100644 --- a/Docs/trigger_models_and_experiment_types.md +++ b/Docs/trigger_models_and_experiment_types.md @@ -17,7 +17,7 @@ flowchart TD AgentSub["submissions/compare-agents/\n(agent experiment)"] end - subgraph pipeline ["ABEvalFlow Pipeline"] + subgraph pipeline ["Agentic Eval Flow Pipeline"] Listener["EventListener\nfilters submissions/"] Validate["Validate"] ScaffoldAndRun["Scaffold, Build, Eval"] @@ -31,7 +31,7 @@ flowchart TD - A dedicated repo ([`RHEcosystemAppEng/skill-submissions`](https://github.com/RHEcosystemAppEng/skill-submissions)) exists solely for evaluation requests - Author pushes a folder under `submissions/` with `metadata.yaml` + whatever the experiment needs - EventListener watches this repo, fires on push -- The submission is ephemeral — it's an evaluation request, not permanent storage +- The submission is ephemeral -- it's an evaluation request, not permanent storage - Results go back to the author (report, PipelineRun status) **Pros:** @@ -58,7 +58,7 @@ flowchart TD PRCheck["GitHub check / label\ntriggers eval"] end - subgraph pipeline ["ABEvalFlow Pipeline"] + subgraph pipeline ["Agentic Eval Flow Pipeline"] Listener["EventListener or\nGH Action triggers pipeline"] Validate["Validate"] ScaffoldAndRun["Scaffold, Build, Eval"] @@ -91,10 +91,10 @@ flowchart TD | Experiment | PR content | Natural fit? | |---|---|---| -| **Skill** | New skill folder with SKILL.md + metadata.yaml | Yes — evaluating a contribution | -| **MCP** | metadata.yaml referencing a running MCP service | Somewhat — MCP isn't a "contribution" to the repo | -| **Agent compare** | metadata.yaml with two agent configs | No — no artifact being contributed | -| **Model compare** | metadata.yaml with two model configs | No — no artifact being contributed | +| **Skill** | New skill folder with SKILL.md + metadata.yaml | Yes -- evaluating a contribution | +| **MCP** | metadata.yaml referencing a running MCP service | Somewhat -- MCP isn't a "contribution" to the repo | +| **Agent compare** | metadata.yaml with two agent configs | No -- no artifact being contributed | +| **Model compare** | metadata.yaml with two model configs | No -- no artifact being contributed | ### Comparison @@ -104,24 +104,24 @@ flowchart TD | **Best for** | Any experiment type | Skills being contributed to a repo | | **Results delivery** | PipelineRun status, stored report | PR comment/check, gates merge | | **Lifecycle** | Fire-and-forget eval request | Tied to PR review cycle | -| **MCP/Agent/Model** | Natural fit | Awkward — no artifact being contributed | -| **Skill eval** | Works but disconnected from where skills live | Natural fit — eval gates the merge | +| **MCP/Agent/Model** | Natural fit | Awkward -- no artifact being contributed | +| **Skill eval** | Works but disconnected from where skills live | Natural fit -- eval gates the merge | | **Complexity** | Simpler (one trigger pattern) | More complex (PR events, commit status API, comment posting) | -| **EventListener changes** | Minimal — current design works | Needs PR event filtering, GitHub App/Actions integration | +| **EventListener changes** | Minimal -- current design works | Needs PR event filtering, GitHub App/Actions integration | ### Recommendation -**Option 1 is implemented** — [`skill-submissions`](https://github.com/RHEcosystemAppEng/skill-submissions) repo is live with webhook configured, matching the current EventListener design. +**Option 1 is implemented** -- [`skill-submissions`](https://github.com/RHEcosystemAppEng/skill-submissions) repo is live with webhook configured, matching the current EventListener design. -**Option 2 can be added later** for the specific case of "skill merged into a target repo needs eval" — this is a CI/CD integration (GitHub Action or webhook on PR) that calls the same pipeline but posts results back to the PR. +**Option 2 can be added later** for the specific case of "skill merged into a target repo needs eval" -- this is a CI/CD integration (GitHub Action or webhook on PR) that calls the same pipeline but posts results back to the PR. -The pipeline itself is the same either way — only the trigger and results-delivery differ. The A/B framework handles all experiment types regardless of how they're triggered. +The pipeline itself is the same either way -- only the trigger and results-delivery differ. The A/B framework handles all experiment types regardless of how they're triggered. --- ## Open Questions -1. **Pass/fail thresholds** — global defaults or configurable per submission? -2. **Notification mechanism** — GitHub checks, Slack, dashboard? -3. **Where do results live permanently?** — PVC, S3/MinIO, committed to a repo? -4. ~~**Submissions repo** — new repo or a `submissions/` folder in agentic-collections?~~ **Resolved:** standalone [`skill-submissions`](https://github.com/RHEcosystemAppEng/skill-submissions) repo. +1. **Pass/fail thresholds** -- global defaults or configurable per submission? +2. **Notification mechanism** -- GitHub checks, Slack, dashboard? +3. **Where do results live permanently?** -- PVC, S3/MinIO, committed to a repo? +4. ~~**Submissions repo** -- new repo or a `submissions/` folder in agentic-collections?~~ **Resolved:** standalone [`skill-submissions`](https://github.com/RHEcosystemAppEng/skill-submissions) repo. diff --git a/Docs/workstreams_roadmap.md b/Docs/workstreams_roadmap.md index c7f3135b..4222df6f 100644 --- a/Docs/workstreams_roadmap.md +++ b/Docs/workstreams_roadmap.md @@ -4,7 +4,7 @@ ## Overview -Four workstreams to complete the ABEvalFlow pipeline. WS1 is the critical path — it renames skilled/unskilled to treatment/control across the codebase and adds the A/B experiment framework. +Four workstreams to complete the Agentic Eval Flow pipeline. WS1 is the critical path -- it renames skilled/unskilled to treatment/control across the codebase and adds the A/B experiment framework. ```mermaid flowchart LR @@ -20,11 +20,11 @@ flowchart LR | Item | Status | |------|--------| -| PR #1 — Phase 1 validation (APPENG-4903) | Merged | -| PR #2 — Tekton triggers + validate task (APPENG-4903) | Merged | -| PR #3 — Phase 2 scaffolding (APPENG-4904) | Merged | -| PR #4 — Rename to ABEvalFlow | Merged | -| Branch `APPENG-4905/phase-3-build-push` | Stale — forked from `c98b547`, missing PRs #1-4. Abandoned. | +| PR #1 -- Phase 1 validation (APPENG-4903) | Merged | +| PR #2 -- Tekton triggers + validate task (APPENG-4903) | Merged | +| PR #3 -- Phase 2 scaffolding (APPENG-4904) | Merged | +| PR #4 -- Rename to Agentic Eval Flow | Merged | +| Branch `APPENG-4905/phase-3-build-push` | Stale -- forked from `c98b547`, missing PRs #1-4. Abandoned. | | Harbor OpenShift backend (`skills_eval_corrections`) | Feature-complete in fork, unit tested | --- @@ -56,18 +56,18 @@ See the detailed commit plan in [ab_testing_generalization_plan.md](./ab_testing **Branch:** `APPENG-4905/build-push-treatment-control` (to be created after WS1 merges) **Depends on:** WS1 merged -Recreate the build-push Tekton task from scratch on current `main` using treatment/control naming. The old `APPENG-4905/phase-3-build-push` branch is abandoned — it diverged from `c98b547` (before PRs #1-4) and would require a conflict-heavy rebase with no benefit. +Recreate the build-push Tekton task from scratch on current `main` using treatment/control naming. The old `APPENG-4905/phase-3-build-push` branch is abandoned -- it diverged from `c98b547` (before PRs #1-4) and would require a conflict-heavy rebase with no benefit. ### What to build -- `pipeline/tasks/build-push.yaml` — Buildah-based build and push for treatment/control images +- `pipeline/tasks/build-push.yaml` -- Buildah-based build and push for treatment/control images - Params: `treatment-task-dir`, `control-task-dir`, `skill-name`, `commit-sha`, `registry-url`, `registry-namespace` - Results: `treatment-image-ref`, `control-image-ref` (digest-based) - Steps: `build-push-treatment`, `build-push-control` (rootless Buildah, `--storage-driver=vfs`) - Image tags: `:treatment-`, `:control-` - Namespace: `ab-eval-flow` -- `config/rbac.yaml` — RoleBinding for `system:image-builder` in `ab-eval-flow` namespace -- Update `Docs/implementation_plan.md` — Phase 3 checkboxes +- `config/rbac.yaml` -- RoleBinding for `system:image-builder` in `ab-eval-flow` namespace +- Update `Docs/implementation_plan.md` -- Phase 3 checkboxes ### Reference @@ -99,7 +99,7 @@ Update [harbor_openshift_backend.md](./harbor_openshift_backend.md) to match the **Depends on:** WS2 merged (needs image ref handoff) -New `pipeline/tasks/harbor-eval.yaml` in ABEvalFlow: +New `pipeline/tasks/harbor-eval.yaml` in Agentic Eval Flow: - Params: `treatment-image-ref`, `control-image-ref`, `n-trials`, `namespace` - Runs `harbor run --env openshift --ek image_ref= --ek namespace=` - Collects results to workspace/PVC @@ -118,12 +118,12 @@ Update [trigger_models_and_experiment_types.md](./trigger_models_and_experiment_ ### Changes -1. **Option 1 stays primary** — standalone submission repo (to be created, e.g. `RHEcosystemAppEng/ab-eval-submissions`) -2. **Clarify "ephemeral"** — the git submission is persistent as a git artifact, but it's not the final destination; it's an evaluation request. For skills there is code to contribute, but for agent/model/MCP comparisons the output is a decision (env var change, configuration), not a code contribution -3. **Option 2 enhancement — hybrid approach** — GH Action triggered by a PR label from admins, calling the same pipeline. The eval platform just needs the gitops submission repo to run, no matter how it's created. Skill-admins control their own trigger policy -4. **Admin-gating** — not all developers should trigger evaluations; admin label/approval gates the pipeline. This is on the skill-owner side, separate from the eval platform -5. **Two-role separation** — skill-admin (controls trigger policy, labels PRs) vs eval-platform (runs the pipeline from the submission repo) -6. **Non-code experiments** — agent compare, model compare, MCP eval are env-var/config changes, not repo contributions — reinforces why Option 1 is the natural universal fit +1. **Option 1 stays primary** -- standalone submission repo (to be created, e.g. `RHEcosystemAppEng/ab-eval-submissions`) +2. **Clarify "ephemeral"** -- the git submission is persistent as a git artifact, but it's not the final destination; it's an evaluation request. For skills there is code to contribute, but for agent/model/MCP comparisons the output is a decision (env var change, configuration), not a code contribution +3. **Option 2 enhancement -- hybrid approach** -- GH Action triggered by a PR label from admins, calling the same pipeline. The eval platform just needs the gitops submission repo to run, no matter how it's created. Skill-admins control their own trigger policy +4. **Admin-gating** -- not all developers should trigger evaluations; admin label/approval gates the pipeline. This is on the skill-owner side, separate from the eval platform +5. **Two-role separation** -- skill-admin (controls trigger policy, labels PRs) vs eval-platform (runs the pipeline from the submission repo) +6. **Non-code experiments** -- agent compare, model compare, MCP eval are env-var/config changes, not repo contributions -- reinforces why Option 1 is the natural universal fit ### Scope @@ -133,11 +133,11 @@ Documentation update only. No code changes. The submission repo trigger (Option ## Execution Order -1. **WS1** — A/B generalization (branch exists, start coding) -2. **WS4** — trigger doc update (can be done during WS1 PR review) -3. **WS3A** — harbor doc update (can be done during WS1 PR review) -4. **WS2** — build-push with treatment/control naming (after WS1 merges) -5. **WS3B** — harbor-eval.yaml task (after WS2 merges) +1. **WS1** -- A/B generalization (branch exists, start coding) +2. **WS4** -- trigger doc update (can be done during WS1 PR review) +3. **WS3A** -- harbor doc update (can be done during WS1 PR review) +4. **WS2** -- build-push with treatment/control naming (after WS1 merges) +5. **WS3B** -- harbor-eval.yaml task (after WS2 merges) ## Jira Tickets diff --git a/README.md b/README.md index 197d0a5d..cbbf6baf 100644 --- a/README.md +++ b/README.md @@ -2,16 +2,14 @@ Automated Tekton-orchestrated pipeline on OpenShift for evaluating AI artifacts: -- **Skills** — Measures skill efficacy by comparing agent performance with and without skills (A/B "gap" testing) -- **MCP Servers** — Validates MCP server implementations via task-based verification -- **Agents** — Evaluates full agent behavior using Harbor (general agents) or A2A protocol (A2A-compliant agents) +- **Skills** -- Measures skill efficacy by comparing agent performance with and without skills (A/B "gap" testing) +- **MCP Servers** -- Validates MCP server implementations via task-based verification +- **Agents** -- Evaluates full agent behavior using Harbor (general agents) or A2A protocol (A2A-compliant agents) Produces statistical reports with pass rates, uplift metrics, significance tests, and a unified scorecard. ## Pipelines -Agentic Eval Flow provides two pipeline variants: - | Pipeline | Purpose | Key Differences | |----------|---------|-----------------| | **CI Pipeline** | Full evaluation for new submissions | Includes security scan, quality review, artifact generation | @@ -24,15 +22,15 @@ The pipeline executes in six main stages, with engine-specific steps within each ### 1. Prepare - Clone submission repository - Validate structure and `metadata.yaml` schema -- AI-assisted generation of missing test artifacts (optional): - - Harbor/A2A: generates `instruction.md` and `test_outputs.py` from `SKILL.md` - - ASE: generates `evals.json` from `SKILL.md` +- AI-assisted generation of missing test artifacts (optional) ### 2. Test (CI Pipeline only) -- **Quality Review** — AI-powered review of skill/test coherence (advisory) -- **Security Scan** — Cisco AI Defense scan for prompt injection, data exfiltration risks +- **Quality Review** -- AI-powered review of skill/test coherence (advisory) +- **Security Scan** -- [Cisco AI Defense](https://github.com/cisco-ai-defense/skill-scanner) scan for prompt injection, data exfiltration risks +- **Security & Quality Scan** -- [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval) deterministic scan (27 rule categories covering prompt injection, credential access, obfuscation, coercive overrides, stealth persistence, data exfiltration, description quality, broken references, and more) ### 3. Evaluate + Five evaluation engines, each suited for different artifact types: | Engine | Evaluates | Comparison Mode | Container Isolation | @@ -43,6 +41,8 @@ Five evaluation engines, each suited for different artifact types: | **MCPChecker** | MCP servers | Single-agent task verification | No | | **AEH** | Agents, skills | Judge-based evaluation | Yes (K8s pods) | +Engines are implemented in `abevalflow/engines/` using a registry pattern. + ### 4. Analyze - Compute pass rates, uplift (gap), statistical significance (p-value) - Generate `report.json` and `report.md` @@ -79,7 +79,7 @@ gate_policy: mode: warn ``` -See [Gate Policy Configuration](#gate-policy-configuration) for full options. +See [Gate Policy Configuration](Docs/gates-architecture.md#gate-policy-configuration) for full options. ## Repository Structure @@ -89,16 +89,7 @@ agentic_eval_flow/ ├── pipeline/ │ ├── pipeline.yaml # Main pipeline definition │ ├── triggers/ # EventListener, TriggerTemplate, TriggerBinding -│ └── tasks/ -│ ├── validate.yaml -│ ├── generate_tests.yaml -│ ├── test-quality-review.yaml -│ ├── security-scan.yaml -│ ├── scaffold.yaml -│ ├── build-push.yaml -│ ├── harbor-eval.yaml -│ ├── analyze-report.yaml -│ └── publish-store.yaml +│ └── tasks/ # Tekton task definitions (phases, components, post) ├── templates/ # Jinja2 templates (Dockerfiles, test.sh, task.toml) ├── scripts/ # Python scripts invoked by pipeline tasks ├── config/ # K8s manifests (RBAC, PostgreSQL, LiteLLM) @@ -109,559 +100,11 @@ agentic_eval_flow/ | Repository | Purpose | |---|---| -| [skill-submissions](https://github.com/RHEcosystemAppEng/skill-submissions) | Submission intake — users push skills, MCP evals, and agent evals here | +| [skill-submissions](https://github.com/RHEcosystemAppEng/skill-submissions) | Submission intake -- users push skills, MCP evals, and agent evals here | | [skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) | Harbor fork with OpenShift backend for Agentic Eval Flow | -| [All-Hands-AI/openhands-agent-monitor](https://github.com/All-Hands-AI/openhands-agent-monitor) | Harbor upstream — agent evaluation framework | +| [All-Hands-AI/openhands-agent-monitor](https://github.com/All-Hands-AI/openhands-agent-monitor) | Harbor upstream -- agent evaluation framework | | [cisco-ai-defense/skill-scanner](https://github.com/cisco-ai-defense/skill-scanner) | Security scanner for prompt injection and data exfiltration detection | - -## Evaluation Engines - -The pipeline supports five evaluation engines, each suited for different artifact types: - -| Engine | Artifact Type | Use Case | Comparison | Container Isolation | -|--------|---------------|----------|------------|---------------------| -| **Harbor** | Skills, Agents | Full evaluation with real tool execution | A/B (with vs without skill) | Yes | -| **ASE** | Skills only | Lightweight LLM-as-judge assertions | A/B (with vs without skill) | No | -| **A2A** | A2A Agents | A2A-protocol compliant agent evaluation | A/B (treatment vs control) | Yes | -| **MCPChecker** | MCP Servers | MCP server/tool verification | Single-agent task verification | No | -| **AEH** | Agents, Skills | Agent-Eval-Harness judge-based evaluation | Single or pairwise | Yes (K8s pods) | - -Engines are implemented in `abevalflow/engines/` using a registry pattern: - -``` -abevalflow/engines/ -├── __init__.py # Engine registry and factory -├── base.py # EvalEngine abstract base class -├── harbor.py # Harbor A/B evaluation -├── ase.py # ASE LLM-as-judge evaluation -├── a2a.py # A2A protocol evaluation -├── aeh.py # Agent-Eval-Harness evaluation -└── mcpchecker.py # MCPChecker task verification -``` - -## Gates Architecture - -Gates are evaluation checkpoints that produce standardized results. The unified scorecard aggregates all gate results to produce a final recommendation. - -### Gate Types - -| Category | Policy Key | Purpose | Implementation | -|----------|------------|---------|----------------| -| **evaluation** | `evaluation` | Results from the selected eval engine | Harbor, ASE, A2A, MCPChecker, or AEH | -| **security** | `security` | Security scanning results | Cisco AI Defense scanner | -| **quality** | `quality` | Quality review results | LLM-powered review | - -### Gate Modes - -Each gate operates in one of three modes: - -| Mode | Behavior | -|------|----------| -| `disabled` | Gate is skipped entirely | -| `warn` | Gate runs; failures produce warnings but don't block | -| `block` | Gate runs; failures cause the scorecard to fail | - -### GateResult Schema - -All gates produce a standardized `GateResult`: - -```python -class GateResult: - gate_type: GateType # engine, security, or quality - gate_name: str # Category name: "evaluation", "security", or "quality" - policy_key: str # Implementation: "harbor", "cisco", "llm-review", etc. - passed: bool # Whether the gate passed - score: float # Normalized score (0.0 to 1.0) - mode: GateMode # Mode that was applied (disabled/warn/block) - threshold: float | None # Threshold used for pass/fail - findings: list[Finding] # Issues discovered (security/quality gates) - details: dict # Implementation-specific data (e.g., {"engine": "harbor"}) - message: str # Human-readable summary -``` - -The `gate_name` is the category used in policy configuration, while `policy_key` identifies the specific implementation. - -### Existing Gates - -#### Evaluation Gate (`evaluation`) - -The primary gate that wraps the selected evaluation engine's results. - -- **Location:** `abevalflow/engines/*.py` (each engine produces evaluation gate results) -- **Input:** Engine-specific report from `reports/{submission}/` -- **Engines:** Harbor, ASE, A2A, MCPChecker (selected via `eval_engine` in metadata.yaml) -- **Pass criteria:** - - Harbor/ASE/A2A: `treatment_score - control_score >= threshold` (default threshold: 0.0) - - MCPChecker: All tasks pass verification -- **Score:** Mean reward or pass rate depending on engine - -#### Security Gate (`security`) - -Reads `security-scan.json` produced by the Cisco AI Defense scanner. - -- **Location:** `abevalflow/gates/security/cisco.py` -- **Input:** `reports/{submission}/security-scan.json` -- **Scanner:** Cisco AI Defense -- **Pass criteria:** - - `warn` mode: Always passes (findings are advisory) - - `block` mode: Fails if any HIGH or CRITICAL findings exist -- **Score:** Weighted average based on finding severities - -#### Quality Gate (`quality`) - -Reads `_ai_review.json` produced by the AI quality reviewer. - -- **Location:** `abevalflow/gates/quality/llm_review.py` -- **Input:** `{workspace}/_ai_review.json` -- **Reviewer:** LLM-powered quality review -- **Dimensions evaluated:** coherence, coverage, clarity, feasibility, robustness -- **Pass criteria:** - - `warn` mode: Passes unless recommendation is "fail" - - `block` mode: Passes only if `overall_score >= threshold` -- **Default threshold:** 0.6 - -## Scorecard - -The scorecard is the single source of truth for submission evaluation, aggregating all gate results with configurable policy. - -### Scorecard Schema - -```python -class Scorecard: - submission_name: str # Name of the evaluated submission - pipeline_run_id: str # Tekton PipelineRun ID - eval_engine: str # Primary evaluation engine used - gates: list[GateResult] # All gate results - policy: GatePolicy # Policy that was applied - recommendation: Recommendation # pass, warn, or fail - recommendation_reason: str # Human-readable explanation - gates_passed: int # Count of passed gates - gates_failed: int # Count of failed gates - blocking_gates_passed: int # Count of passed blocking gates - blocking_gates_failed: int # Count of failed blocking gates -``` - -### Combination Modes - -The scorecard supports three modes for combining gate results: - -| Mode | Logic | -|------|-------| -| `all_pass` | All blocking gates must pass; failing warn gates produce warnings | -| `any_pass` | At least one blocking gate must pass | -| `weighted` | Weighted average of gate scores determines outcome | - -### Output - -The scorecard is written to `reports/{submission}/scorecard.json` and includes: -- All gate results with scores and findings -- Final recommendation with reasoning -- Provenance metadata (commit SHA, branch, pipeline run ID) - -## Gate Policy Configuration - -Gate policies are configured in `metadata.yaml` under the `gate_policy` key: - -```yaml -# metadata.yaml -name: my-skill -eval_engine: harbor - -gate_policy: - default_mode: warn # Default mode for all gates - combination: all_pass # How to combine gate results - - gates: - # Security gate configuration - security: - mode: block # Fail the scorecard on security issues - threshold: 0.8 # Minimum score to pass - - # Quality gate configuration - quality: - mode: warn # Advisory only - threshold: 0.6 # Threshold for pass/fail - - # Engine gate configuration (uses eval_engine automatically) - evaluation: - mode: block - threshold: 0.0 # Any positive uplift passes -``` - -### GatePolicyItem Options - -| Field | Type | Default | Description | -|-------|------|---------|-------------| -| `mode` | `disabled`/`warn`/`block` | `warn` | Enforcement mode | -| `threshold` | `float` | Gate-specific | Score threshold for pass/fail | -| `weight` | `float` | `1.0` | Weight for weighted combination mode | - -## Compass Facts Integration - -The pipeline can push gate results to Red Hat Compass as Soundcheck facts for visibility in the developer portal. - -### Configuration - -Enable fact pushing in `metadata.yaml`: - -```yaml -gate_policy: - push_facts: - endpoint: https://compass.redhat.com/api/soundcheck/facts/ - entity_ref: component:default/my-component -``` - -### Fact Structure - -Each gate result is pushed as a separate fact. The fact reference includes both the category and implementation: - -```json -{ - "facts": [ - { - "factRef": "catalog:default/abevalflow_evaluation_harbor", - "entityRef": "component:default/my-component", - "data": { - "gate_name": "evaluation", - "passed": true, - "score": 0.85, - "mode": "block", - "message": "Harbor A/B: gap=0.15 >= threshold=0.0 -> PASS", - "evaluated_at": "2026-06-21T10:35:53Z" - } - } - ] -} -``` - -### Authentication - -The Compass API token is stored in a Kubernetes secret: - -```bash -oc create secret generic compass-facts-api --from-literal=token= -``` - -## Persistence - -### MinIO (Object Storage) - -Reports and artifacts are uploaded to MinIO under a timestamped prefix: - -``` -s3://ab-eval-reports/YYYYMMDD_hhmmss_{submission}_{run-id}/ -├── report.json # Main evaluation report -├── report.md # Human-readable report -├── scorecard.json # Unified scorecard -├── security_scans/ # Security scan results -│ └── security-scan.json -├── generated/ # AI-generated artifacts -│ ├── instruction.md -│ └── test_outputs.py -├── scaffolded/ # Scaffolded configs and review -│ └── _ai_review.json -└── trials/ # Per-trial artifacts (Harbor) - ├── trial_001/ - │ ├── agent/ - │ └── verifier/ - └── ... -``` - -### PostgreSQL (Results Database) - -Evaluation results are persisted for historical analysis and monitoring: - -- **Script:** `scripts/store_results.py` -- **Data stored:** - - Submission metadata - - Per-trial results (Harbor/ASE) - - Security scan findings - - Aggregate statistics - - Scorecard recommendation - -## Extensibility - -### Adding a New Engine - -1. Create a new file in `abevalflow/engines/`: - -```python -# abevalflow/engines/my_engine.py -from abevalflow.engines import register_engine -from abevalflow.engines.base import EvalEngine -from abevalflow.gates.base import GateResult, GateType - -@register_engine("my-engine") -class MyEngine(EvalEngine): - name = "my-engine" - - def read_result(self, reports_dir: Path) -> dict | None: - """Read engine results from reports directory.""" - result_path = reports_dir / "my-engine-report.json" - if not result_path.exists(): - return None - return json.loads(result_path.read_text()) - - def to_gate_result(self, raw_result: dict, policy: GatePolicy) -> GateResult: - """Convert engine result to standardized GateResult.""" - score = raw_result.get("score", 0.0) - threshold = policy.get_gate_policy(self.name).threshold or 0.0 - - return GateResult( - gate_type=GateType.ENGINE, - gate_name="evaluation", - policy_key=self.name, - passed=score >= threshold, - score=score, - mode=policy.get_gate_policy(self.name).mode, - message=f"MyEngine: score={score:.2f}", - ) -``` - -2. Import in `abevalflow/engines/__init__.py`: - -```python -from abevalflow.engines.my_engine import MyEngine -``` - -### Adding a New Security Gate - -1. Create a new file in `abevalflow/gates/security/`: - -```python -# abevalflow/gates/security/snyk.py -from abevalflow.gates.security import register_security_gate -from abevalflow.gates.security.base import SecurityGate -from abevalflow.gates.base import GateResult, GateType - -@register_security_gate("snyk") -class SnykGate(SecurityGate): - name = "snyk" - - def evaluate(self, reports_dir: Path, policy: GatePolicy) -> GateResult: - """Evaluate Snyk security scan results.""" - # Read snyk-report.json and produce GateResult - ... -``` - -2. Import in `abevalflow/gates/security/__init__.py`: - -```python -from abevalflow.gates.security.snyk import SnykGate -``` - -### Adding a New Quality Gate - -1. Create a new file in `abevalflow/gates/quality/`: - -```python -# abevalflow/gates/quality/custom_review.py -from abevalflow.gates.quality import register_quality_gate -from abevalflow.gates.quality.base import QualityGate -from abevalflow.gates.base import GateResult, GateType - -@register_quality_gate("custom-review") -class CustomReviewGate(QualityGate): - name = "custom-review" - - def evaluate(self, workspace_root: Path, policy: GatePolicy) -> GateResult: - """Evaluate custom quality review results.""" - # Read review artifacts and produce GateResult - ... -``` - -2. Import in `abevalflow/gates/quality/__init__.py`: - -```python -from abevalflow.gates.quality.custom_review import CustomReviewGate -``` - -### Adding a New Gate Category - -To add an entirely new gate category (e.g., "compliance", "performance"): - -1. **Add the GateType enum** in `abevalflow/gates/base.py`: - -```python -class GateType(str, Enum): - ENGINE = "engine" - SECURITY = "security" - QUALITY = "quality" - COMPLIANCE = "compliance" # New category -``` - -2. **Create the gate directory** at `abevalflow/gates/compliance/`: - -``` -abevalflow/gates/compliance/ -├── __init__.py # Registry and exports -├── base.py # ComplianceGate base class -└── my_checker.py # First implementation -``` - -3. **Create the base class** in `abevalflow/gates/compliance/base.py`: - -```python -from abc import abstractmethod -from abevalflow.gates.base import GateResult, GateType - -class ComplianceGate: - name: str - - @abstractmethod - def evaluate(self, reports_dir: Path, policy: GatePolicy) -> GateResult: - """Evaluate compliance and return standardized GateResult.""" - pass -``` - -4. **Update the scorecard aggregation** in `scripts/aggregate_scorecard.py`: - -```python -from abevalflow.gates.compliance import get_all_compliance_gates - -# In aggregate_scorecard(): -for compliance_gate in get_all_compliance_gates(): - if not policy.is_enabled(compliance_gate.name): - continue - gate_result = compliance_gate.evaluate(reports_dir, policy) - gates.append(gate_result) -``` - -5. **Add the category to policy schema** in `abevalflow/schemas.py` (documentation only, the schema is flexible) - -## Submission Formats - -### Skill Submission (Harbor) - -For full agent evaluation with container isolation and A/B comparison: - -``` -my-skill/ -├── instruction.md # Task description (required, or generated from SKILL.md) -├── skills/ -│ └── SKILL.md # Skill definition (required) -├── tests/ -│ ├── test_outputs.py # Verification tests (required, or generated) -│ └── llm_judge.py # LLM-based judge (optional) -├── docs/ # Reference documentation (optional) -├── supportive/ # Mock MCPs, data files (optional, <50MB) -└── metadata.yaml # eval_engine: harbor (required) -``` - -### Skill Submission (ASE) - -For lightweight LLM-as-judge evaluation without containers: - -``` -my-skill/ -├── skills/ -│ └── SKILL.md # Skill definition (required) -├── evals/ -│ ├── evals.json # Evaluation prompts and assertions (optional, generated if missing) -│ └── files/ # Test data files (optional) -└── metadata.yaml # eval_engine: ase (required) -``` - -### MCP Server Submission - -For validating MCP server implementations: - -``` -my-mcp-server-eval/ -├── metadata.yaml # eval_engine: mcpchecker (required) -├── mcp-config.yaml # MCP server connection settings (required) -│ # - url: MCP server endpoint -│ # - auth: authentication config (if needed) -└── tasks/ - ├── task-1.yaml # Task definition with expected tool calls - └── task-2.yaml # Each task tests specific MCP functionality -``` - -MCPChecker validates that the MCP server correctly handles tool invocations and returns expected results. - -### Agent Submission (A2A Protocol) - -For evaluating agents that implement the A2A (Agent-to-Agent) protocol: - -``` -my-a2a-agent-eval/ -├── metadata.yaml # eval_engine: a2a (required) -├── agent-config.yaml # Agent endpoint and auth config (required) -│ # - endpoint: http://agent-service:8000 -│ # - auth: bearer token or API key -└── tasks/ - ├── instruction.md # Task description - ├── tests/ - │ └── test_outputs.py - └── task.toml # Task configuration -``` - -A2A evaluation connects to a deployed agent via the A2A protocol and runs evaluation tasks against it. - -### Agent Submission (Harbor) - -For evaluating general agents (non-A2A) with full container isolation: - -``` -my-agent-eval/ -├── metadata.yaml # eval_engine: harbor, persona: agent (required) -├── instruction.md # Task description (required) -├── tests/ -│ ├── test_outputs.py # Verification tests (required) -│ └── llm_judge.py # LLM-based judge (optional) -└── supportive/ # Environment files, data (optional) -``` - -Harbor creates treatment/control container variants and runs A/B comparison. - -### AEH Submission (Agent-Eval-Harness) - -For evaluating agents using the Agent-Eval-Harness framework with flexible LLM judges. -Trials run as Harbor jobs on OpenShift. - -**Single** (`aeh-mode=single`): - -``` -my-aeh-eval/ -├── metadata.yaml # eval_engine: aeh (required) -├── eval.yaml # AEH config (models, judges, thresholds, outputs) -├── skills/…/SKILL.md # Optional skill package -└── cases/ - └── case-001/ - └── input.yaml -``` - -**Pairwise** (`aeh-mode=pairwise`): - -``` -my-aeh-pairwise/ -├── metadata.yaml -├── eval-control.yaml # Baseline (often unskilled) -├── eval-treatment.yaml # Treatment + pairwise LLM judge -├── skills//SKILL.md -└── cases/ - └── case-001/ - └── input.yaml -``` - -Verified smoke samples in [skill-submissions](https://github.com/RHEcosystemAppEng/skill-submissions): -- Branch `eval/aeh-hello-world-single` → `aeh-hello-world-single` -- Branch `eval/aeh-hello-world-pairwise` → `aeh-hello-world-pairwise` - -Working trigger defaults: LiteLLM `claude-sonnet` via -`http://litellm.ab-eval-flow.svc.cluster.local:4000`, AEH image -`quay.io/ecosystem-appeng/agent-eval-harness:v1.0.3`. - -MinIO layout for AEH: `debug/harbor/` (raw Harbor jobs) + `debug/aeh/` -(`summary.yaml`, `report.html`, `cases/`). Pairwise HTML is regenerated with -`--baseline` so treatment `report.html` includes the pairwise section. - -See [Trigger Guide](Docs/trigger_guide.md) for full YAML examples, or run: - -```bash -./scripts/misc/trigger_test_runs.sh # all engines including AEH -./scripts/misc/trigger_test_runs.sh "$(git branch --show-current)" aeh # AEH only -``` +| [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval) | Deterministic security and quality scanner for skill submissions (27 rule categories, 97 rules) | ## LLM Access @@ -683,7 +126,12 @@ The pipeline is LLM-agnostic. Three modes are supported: ## Documentation -- [Trigger Guide](Docs/trigger_guide.md) — How to submit skills, configure gate policies, and interpret scorecard results +- [Trigger Guide](Docs/trigger_guide.md) -- How to submit skills, configure gate policies, and interpret scorecard results +- [Gates Architecture](Docs/gates-architecture.md) -- Gate types, modes, GateResult schema, scorecard, and gate policy configuration +- [Submission Formats](Docs/submission-formats.md) -- Directory layouts for skill, agent, MCP, and AEH submissions +- [Extensibility](Docs/extensibility.md) -- How to add new engines, security gates, quality gates, and gate categories +- [Persistence](Docs/persistence.md) -- MinIO object storage layout and PostgreSQL results database +- [Compass Integration](Docs/compass_facts_integration.md) -- Pushing gate results to Red Hat Compass - [ADR: Skill Evaluation Pipeline](Docs/ADR_Skill_Evaluation_Pipeline_and_Harbor_Execution_Strategy.txt) ## License diff --git a/abevalflow/harbor_agents/__init__.py b/abevalflow/harbor_agents/__init__.py index fe8ef843..a5bb9ff1 100644 --- a/abevalflow/harbor_agents/__init__.py +++ b/abevalflow/harbor_agents/__init__.py @@ -1,4 +1,4 @@ -"""Harbor agent adapters for ABEvalFlow. +"""Harbor agent adapters for Agentic Eval Flow. This module provides custom Harbor agents that can be loaded via --agent-import-path for evaluation purposes. diff --git a/abevalflow/harbor_extensions/__init__.py b/abevalflow/harbor_extensions/__init__.py index 16b5f7eb..07fc238c 100644 --- a/abevalflow/harbor_extensions/__init__.py +++ b/abevalflow/harbor_extensions/__init__.py @@ -1,4 +1,4 @@ -"""Harbor environment extensions for ABEvalFlow OpenShift deployment. +"""Harbor environment extensions for Agentic Eval Flow OpenShift deployment. Includes OpenShiftEnvironment emptyDir mounts and AEH task enrichment (skills + annotations) applied before ``harbor run``. diff --git a/abevalflow/harbor_extensions/aeh_task_enrichment.py b/abevalflow/harbor_extensions/aeh_task_enrichment.py index ca801599..979ee98e 100644 --- a/abevalflow/harbor_extensions/aeh_task_enrichment.py +++ b/abevalflow/harbor_extensions/aeh_task_enrichment.py @@ -29,7 +29,7 @@ logger = logging.getLogger(__name__) _ANNOTATIONS_STAGE = """ -# ABEvalFlow: stage annotations for Harbor verifier judges. +# Agentic Eval Flow: stage annotations for Harbor verifier judges. # reward.py uses case_dir basename (e.g. "workspace") under dataset.path. if [ -f "@@WORKDIR@@/annotations.yaml" ]; then _aeh_case="$(basename "@@WORKDIR@@")" @@ -182,7 +182,7 @@ def _enrich_one_task( if judge_model: models["judge"] = judge_model if reward_inject is not None and not (isinstance(cfg.get("reward"), dict) and cfg["reward"]): - # Only ABEvalFlow-injected rewards get score_range [0, 1]. Do not + # Only Agentic Eval Flow-injected rewards get score_range [0, 1]. Do not # rewrite submission-authored weighted rewards that omit score_range # (AEH default [1, 5] may be intentional for Likert judges). cfg["reward"] = reward_inject @@ -278,7 +278,7 @@ def _dir_has_skill_md(skills_root: Path) -> bool: def _inject_annotations_stage(test_sh: Path) -> None: text = test_sh.read_text() - if "ABEvalFlow: stage annotations" in text: + if "Agentic Eval Flow: stage annotations" in text: return workdir = "/workspace" diff --git a/abevalflow/observability/__init__.py b/abevalflow/observability/__init__.py index 27077c2e..22567959 100644 --- a/abevalflow/observability/__init__.py +++ b/abevalflow/observability/__init__.py @@ -1,4 +1,4 @@ -"""Observability layer for ABEvalFlow pipeline metrics and tracing.""" +"""Observability layer for Agentic Eval Flow pipeline metrics and tracing.""" from abevalflow.observability.context import MetricsContext, TimingRecord, TokenUsage from abevalflow.observability.otel import get_tracer, is_otel_enabled diff --git a/abevalflow/observability/otel.py b/abevalflow/observability/otel.py index d17dce3b..05974548 100644 --- a/abevalflow/observability/otel.py +++ b/abevalflow/observability/otel.py @@ -1,4 +1,4 @@ -"""OpenTelemetry setup for ABEvalFlow pipeline tracing. +"""OpenTelemetry setup for Agentic Eval Flow pipeline tracing. Provides lazy-init OTEL instrumentation with three modes: diff --git a/config/konflux/secrets-template.yaml b/config/konflux/secrets-template.yaml index 981047dd..8089f72b 100644 --- a/config/konflux/secrets-template.yaml +++ b/config/konflux/secrets-template.yaml @@ -1,4 +1,4 @@ -# ABEvalFlow Konflux Secrets Template +# Agentic Eval Flow Konflux Secrets Template # # Apply these secrets to your Konflux tenant namespace. Not all secrets are # required -- it depends on which eval-mode and features you use. diff --git a/config/mlflow/deployment.yaml b/config/mlflow/deployment.yaml index 8dfe5d68..491ca86f 100644 --- a/config/mlflow/deployment.yaml +++ b/config/mlflow/deployment.yaml @@ -1,4 +1,4 @@ -# Lightweight dedicated MLflow tracking server for ABEvalFlow AEH runs. +# Lightweight dedicated MLflow tracking server for Agentic Eval Flow AEH runs. # Backend: sqlite on PVC. Artifacts: same PVC under /mlflow/artifacts. # Cluster DNS: http://abevalflow-mlflow.ab-eval-flow.svc.cluster.local:5000 apiVersion: apps/v1 diff --git a/config/mlflow/pvc.yaml b/config/mlflow/pvc.yaml index 6aecd0ec..0cd95121 100644 --- a/config/mlflow/pvc.yaml +++ b/config/mlflow/pvc.yaml @@ -1,4 +1,4 @@ -# Dedicated MLflow artifact + sqlite store for ABEvalFlow. +# Dedicated MLflow artifact + sqlite store for Agentic Eval Flow. # Apply into the target namespace (default matches shared cluster config): # oc apply -n ab-eval-flow -f config/mlflow/ apiVersion: v1 diff --git a/examples/mcpchecker-skill/metadata.yaml b/examples/mcpchecker-skill/metadata.yaml index 8f6d7349..dced5c52 100644 --- a/examples/mcpchecker-skill/metadata.yaml +++ b/examples/mcpchecker-skill/metadata.yaml @@ -7,7 +7,7 @@ name: example-mcpchecker description: Example MCPChecker evaluation for a sample MCP server version: "0.1.0" -author: ABEvalFlow Team +author: Agentic Eval Flow Team eval_engine: mcpchecker diff --git a/examples/sample_skill_ai/skills/SKILL.md b/examples/sample_skill_ai/skills/SKILL.md index 09c4793b..6b90a6ba 100644 --- a/examples/sample_skill_ai/skills/SKILL.md +++ b/examples/sample_skill_ai/skills/SKILL.md @@ -1,6 +1,6 @@ # Hello World Skill -A trivial file-creation skill used as a smoke test for the ABEvalFlow pipeline. +A trivial file-creation skill used as a smoke test for the Agentic Eval Flow pipeline. ## What the agent must do diff --git a/pipeline/cronjobs/monitoring-cronjob.yaml b/pipeline/cronjobs/monitoring-cronjob.yaml index 119de272..5c926d2b 100644 --- a/pipeline/cronjobs/monitoring-cronjob.yaml +++ b/pipeline/cronjobs/monitoring-cronjob.yaml @@ -41,7 +41,7 @@ spec: - | set -e - echo "=== ABEvalFlow Monitoring CronJob ===" + echo "=== Agentic Eval Flow Monitoring CronJob ===" echo "Time: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" # ── Helpers ────────────────────────────────────────────── @@ -166,7 +166,7 @@ spec: if [ "$HTTP_STATUS" != "200" ]; then echo " ERROR: Agent health check FAILED (endpoint=$HEALTH_ENDPOINT, status=$HTTP_STATUS)" - send_slack ":red_circle: *ABEvalFlow Monitoring* — ${SKILL} health check failed (HTTP $HTTP_STATUS). Skipping eval." + send_slack ":red_circle: *Agentic Eval Flow Monitoring* — ${SKILL} health check failed (HTTP $HTTP_STATUS). Skipping eval." continue fi echo " Agent health check PASSED (HTTP $HTTP_STATUS)" diff --git a/pipeline/integration/konflux-eval-pipelinerun.yaml b/pipeline/integration/konflux-eval-pipelinerun.yaml index 4302ad9c..016d3891 100644 --- a/pipeline/integration/konflux-eval-pipelinerun.yaml +++ b/pipeline/integration/konflux-eval-pipelinerun.yaml @@ -1,4 +1,4 @@ -# ABEvalFlow Generic Evaluation Pipeline for Konflux +# Agentic Eval Flow Generic Evaluation Pipeline for Konflux # # This is a REFERENCE pipeline that any Konflux application can use for # AI evaluation (agents, MCP servers, skills). It provides 7 core stages: @@ -128,7 +128,7 @@ spec: - name: PIPELINE_REPO_URL type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repo containing evaluation scripts + description: URL of the Agentic Eval Flow pipeline repo containing evaluation scripts - name: PIPELINE_REPO_REVISION type: string default: "main" diff --git a/pipeline/tasks/components/ase-eval.yaml b/pipeline/tasks/components/ase-eval.yaml index 9014eca1..c16197d7 100644 --- a/pipeline/tasks/components/ase-eval.yaml +++ b/pipeline/tasks/components/ase-eval.yaml @@ -45,7 +45,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/components/generate_tests.yaml b/pipeline/tasks/components/generate_tests.yaml index f91bbba6..7af0b451 100644 --- a/pipeline/tasks/components/generate_tests.yaml +++ b/pipeline/tasks/components/generate_tests.yaml @@ -21,7 +21,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/components/harbor-eval.yaml b/pipeline/tasks/components/harbor-eval.yaml index 22d6a00f..aafee1ba 100644 --- a/pipeline/tasks/components/harbor-eval.yaml +++ b/pipeline/tasks/components/harbor-eval.yaml @@ -50,7 +50,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/components/mcpchecker-eval.yaml b/pipeline/tasks/components/mcpchecker-eval.yaml index 56fd6644..137b4144 100644 --- a/pipeline/tasks/components/mcpchecker-eval.yaml +++ b/pipeline/tasks/components/mcpchecker-eval.yaml @@ -36,7 +36,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/components/scaffold.yaml b/pipeline/tasks/components/scaffold.yaml index aaf4d73b..e71abaa2 100644 --- a/pipeline/tasks/components/scaffold.yaml +++ b/pipeline/tasks/components/scaffold.yaml @@ -18,7 +18,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/components/test-quality-review.yaml b/pipeline/tasks/components/test-quality-review.yaml index 128cb3ca..de982d20 100644 --- a/pipeline/tasks/components/test-quality-review.yaml +++ b/pipeline/tasks/components/test-quality-review.yaml @@ -15,7 +15,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/components/validate.yaml b/pipeline/tasks/components/validate.yaml index 525e9375..0a8b6d1e 100644 --- a/pipeline/tasks/components/validate.yaml +++ b/pipeline/tasks/components/validate.yaml @@ -19,7 +19,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/konflux/analyze-scorecard.yaml b/pipeline/tasks/konflux/analyze-scorecard.yaml index 25bfbd93..111bbea1 100644 --- a/pipeline/tasks/konflux/analyze-scorecard.yaml +++ b/pipeline/tasks/konflux/analyze-scorecard.yaml @@ -45,7 +45,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/konflux/emit-result.yaml b/pipeline/tasks/konflux/emit-result.yaml index 89fc2ef1..996b2855 100644 --- a/pipeline/tasks/konflux/emit-result.yaml +++ b/pipeline/tasks/konflux/emit-result.yaml @@ -7,7 +7,7 @@ metadata: app.kubernetes.io/component: konflux spec: description: >- - Maps ABEvalFlow scorecard results to Konflux's standardized TEST_OUTPUT + Maps Agentic Eval Flow scorecard results to Konflux's standardized TEST_OUTPUT format. Reads the scorecard.json from the workspace and emits a JSON result with SUCCESS/WARNING/FAILURE status. params: @@ -50,7 +50,7 @@ spec: *) RESULT="ERROR" ;; esac - NOTE="ABEvalFlow: recommendation=${SCORECARD_REC}, certification=${CERT_LEVEL}, gates_passed=${GATES_PASSED}, gates_failed=${GATES_FAILED}" + NOTE="Agentic Eval Flow: recommendation=${SCORECARD_REC}, certification=${CERT_LEVEL}, gates_passed=${GATES_PASSED}, gates_failed=${GATES_FAILED}" elif [ -f "$REPORT" ]; then REPORT_REC=$(jq -r '.summary.recommendation // "fail"' "$REPORT") @@ -61,11 +61,11 @@ spec: *) RESULT="ERROR" ;; esac - NOTE="ABEvalFlow: recommendation=${REPORT_REC} (no scorecard)" + NOTE="Agentic Eval Flow: recommendation=${REPORT_REC} (no scorecard)" else RESULT="FAILURE" - NOTE="ABEvalFlow: no scorecard.json or report.json found" + NOTE="Agentic Eval Flow: no scorecard.json or report.json found" fi echo "Result: $RESULT" diff --git a/pipeline/tasks/konflux/evaluate.yaml b/pipeline/tasks/konflux/evaluate.yaml index 7aa82d5d..8dca1465 100644 --- a/pipeline/tasks/konflux/evaluate.yaml +++ b/pipeline/tasks/konflux/evaluate.yaml @@ -21,7 +21,7 @@ spec: mcpchecker + remote: supported (untested) ase + local: supported (no external endpoint needed) harbor + local: limited (no scaffold/build; uses local environment only) - harbor + remote: not supported (use standalone ABEvalFlow pipeline) + harbor + remote: not supported (use standalone Agentic Eval Flow pipeline) params: - name: eval-engine type: string diff --git a/pipeline/tasks/konflux/parse-snapshot.yaml b/pipeline/tasks/konflux/parse-snapshot.yaml index 573d9b49..d2538135 100644 --- a/pipeline/tasks/konflux/parse-snapshot.yaml +++ b/pipeline/tasks/konflux/parse-snapshot.yaml @@ -9,7 +9,7 @@ spec: description: >- Parses a Konflux Snapshot JSON to extract the component container image, git source URL, git revision, and component name. This bridges the Konflux - SNAPSHOT model with ABEvalFlow's parameter-based pipeline. + SNAPSHOT model with Agentic Eval Flow's parameter-based pipeline. params: - name: SNAPSHOT type: string diff --git a/pipeline/tasks/post/_deprecated_analyze-report.yaml b/pipeline/tasks/post/_deprecated_analyze-report.yaml index 13388ec2..50986a76 100644 --- a/pipeline/tasks/post/_deprecated_analyze-report.yaml +++ b/pipeline/tasks/post/_deprecated_analyze-report.yaml @@ -43,7 +43,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/pipeline/tasks/post/analyze-and-check-degradation.yaml b/pipeline/tasks/post/analyze-and-check-degradation.yaml index 71d2f556..12d8b186 100644 --- a/pipeline/tasks/post/analyze-and-check-degradation.yaml +++ b/pipeline/tasks/post/analyze-and-check-degradation.yaml @@ -43,7 +43,7 @@ spec: - name: pipeline-repo-url type: string default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git" - description: URL of the ABEvalFlow pipeline repository + description: URL of the Agentic Eval Flow pipeline repository - name: pipeline-repo-revision type: string default: "main" diff --git a/plan.md b/plan.md index 98e957d0..c2c4f351 100644 --- a/plan.md +++ b/plan.md @@ -173,7 +173,7 @@ Add this entry to `_ENVIRONMENT_REGISTRY` dict (after the OPENSHIFT entry): ## Step 4: Test locally ```bash -cd /Users/gziv/Dev/ABEvalFlow +cd "/Users/gziv/Dev/agentic_eval_flow" pip install -e /Users/gziv/Dev/skills_eval_corrections # Test with a simple task @@ -189,7 +189,7 @@ git commit -m "feat: add local environment type for direct subprocess execution" git push -u origin feature/local-environment ``` -## Step 6: Update ABEvalFlow evaluate.yaml +## Step 6: Update Agentic Eval Flow evaluate.yaml In the A2A Harbor config generation section, change environment type from `openshift` to `local`: ```python diff --git a/scripts/aggregate_aeh.py b/scripts/aggregate_aeh.py index c7ea3ecd..b9033e1b 100644 --- a/scripts/aggregate_aeh.py +++ b/scripts/aggregate_aeh.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 -"""Map AEH output to ABEvalFlow report format. +"""Map AEH output to Agentic Eval Flow report format. Reads agent-eval-harness output files (summary.yaml, run_result.json) and -produces a unified report.json compatible with ABEvalFlow's scorecard logic. +produces a unified report.json compatible with Agentic Eval Flow's scorecard logic. Supports both single-run and pairwise modes: - Single: One run directory @@ -148,7 +148,7 @@ def aggregate_single_run( threshold: Pass/fail threshold for mean_reward (matches GatePolicy default) Returns: - Dict in ABEvalFlow report format with full judge metadata + Dict in Agentic Eval Flow report format with full judge metadata """ summary_path = run_dir / "summary.yaml" @@ -254,7 +254,7 @@ def aggregate_pairwise_run( threshold: Pass/fail win-rate threshold (matches GatePolicy / engine) Returns: - Dict in ABEvalFlow report format with pairwise results + Dict in Agentic Eval Flow report format with pairwise results """ treatment_summary_path = treatment_dir / "summary.yaml" control_summary_path = control_dir / "summary.yaml" @@ -376,7 +376,7 @@ def aggregate_aeh_results( submission_name: str | None = None, threshold: float = DEFAULT_AEH_THRESHOLD, ) -> dict[str, Any]: - """Aggregate AEH results into ABEvalFlow report format. + """Aggregate AEH results into Agentic Eval Flow report format. Args: run_dir: Path to the AEH run output directory (treatment in pairwise mode) @@ -386,7 +386,7 @@ def aggregate_aeh_results( threshold: Pass/fail threshold aligned with AEHEngine / GatePolicy default Returns: - Dict in ABEvalFlow report format + Dict in Agentic Eval Flow report format """ if mode == "pairwise": if control_dir is None: @@ -428,7 +428,7 @@ def find_latest_run_dir(reports_dir: Path, submission_name: str) -> Path | None: def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser(description="Aggregate AEH results into ABEvalFlow report format") + parser = argparse.ArgumentParser(description="Aggregate AEH results into Agentic Eval Flow report format") parser.add_argument( "run_dir", type=Path, diff --git a/scripts/pyrit_crescendo/run_crescendo.py b/scripts/pyrit_crescendo/run_crescendo.py index 90d0be2f..24446050 100644 --- a/scripts/pyrit_crescendo/run_crescendo.py +++ b/scripts/pyrit_crescendo/run_crescendo.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Adaptive Crescendo multi-turn red-team runner. -Runs after Promptfoo in the ABEvalFlow red-team task. Generates attacker +Runs after Promptfoo in the Agentic Eval Flow red-team task. Generates attacker turns live based on the target's responses and grades objective success with an LLM-as-judge via LiteLLM. """ diff --git a/scripts/run_aeh.py b/scripts/run_aeh.py index db783c8e..aecb74dc 100644 --- a/scripts/run_aeh.py +++ b/scripts/run_aeh.py @@ -527,7 +527,7 @@ def _execute( return result.returncode def _prepare_enriched_tasks(self, config: Path, tasks_dir: Path) -> None: - """Generate Harbor tasks then apply ABEvalFlow skill/annotation fixes.""" + """Generate Harbor tasks then apply Agentic Eval Flow skill/annotation fixes.""" import shutil from agent_eval.config import EvalConfig diff --git a/submissions/aeh-hello-world/metadata.yaml b/submissions/aeh-hello-world/metadata.yaml index 78c4d520..846f7cf1 100644 --- a/submissions/aeh-hello-world/metadata.yaml +++ b/submissions/aeh-hello-world/metadata.yaml @@ -3,7 +3,7 @@ name: aeh-hello-world description: Sample AEH (Agent-Eval-Harness) submission for testing the pipeline integration persona: rh-developer version: "0.1.0" -author: ABEvalFlow Team +author: Agentic Eval Flow Team eval_engine: aeh tags: - sample diff --git a/submissions/aeh-pairwise-example/metadata.yaml b/submissions/aeh-pairwise-example/metadata.yaml index 3cdaa78c..10c6d231 100644 --- a/submissions/aeh-pairwise-example/metadata.yaml +++ b/submissions/aeh-pairwise-example/metadata.yaml @@ -3,7 +3,7 @@ name: aeh-pairwise-example description: Sample AEH pairwise A/B comparison submission (treatment vs control) persona: rh-developer version: "0.1.0" -author: ABEvalFlow Team +author: Agentic Eval Flow Team eval_engine: aeh tags: - sample diff --git a/tests/test_aeh_task_enrichment.py b/tests/test_aeh_task_enrichment.py index c8b52071..46626fb6 100644 --- a/tests/test_aeh_task_enrichment.py +++ b/tests/test_aeh_task_enrichment.py @@ -68,7 +68,7 @@ def test_enrich_copies_annotations_and_restores_dataset_path(tmp_path: Path): assert bundled["dataset"]["path"] == "cases" test_sh = (task / "tests" / "test.sh").read_text() - assert "ABEvalFlow: stage annotations" in test_sh + assert "Agentic Eval Flow: stage annotations" in test_sh assert "/tests/cases/" in test_sh