feat(eval): add measurement manifests for online scoring

[ascerra]

Summary

Add default measurement manifests for stock fullsend agents under eval/measurements/. These are the online-scoring policy that ships with the agents (“here is code, and here is what we measure on wild code runs”).

Companion engine PR: fullsend-ai/fullsend#6036 (ADR 0087, fullsend eval-measure, EM-001 trace_fitness).

Ownership (please read)

Concern	Repo
Scorer implementations (Go), parser, CLI, job wiring	fullsend#6036
Default manifests (this PR)	agents eval/measurements/<agent>.yaml
Org overrides / BYOA	Consumer FULLSEND_DIR

• Managed jobs resolve local ${FULLSEND_DIR}/eval/measurements/${AGENT}.yaml if present, else fetch agents@v0 from this path. Stock-agent users do not copy these files into every repo.
• Local files are for override, opt-out, or custom agents only.
• This PR is policy only (YAML). Executable scoring stays in fullsend because the measure CLI reads run-telemetry.jsonl produced by fullsend.
• Change guide: new Go scorer / (future) new declarative assert: → fullsend; new id / enable / disable / thresholds for a stock agent on an existing scorer → this repo; org-specific policy → consumer override.

Changes

• eval/measurements/{code,fix,prioritize,retro,review,triage}.yaml — each enables trace_fitness (em-001) @ version 1
• eval/measurements/README.md — ownership, resolution, when to PR where
• eval/README.md — short section + links to fullsend#6036 / ADR 0087
• Scribe intentionally omitted (no forge work-item identity for EM-001 today)

Test plan

• Manifest YAML shape matches fullsend registry loader (agent + measurements[].{id,scorer,version})
• Scribe excluded on purpose
• Cross-check with fullsend#6036 / ADR 0087 before merge
• After merge, ensure v0 (or the pin fullsend uses) includes these paths so wild runs stop skipping measure

Notes for reviewers

• Functional eval fixtures under eval/<agent>/ are unchanged and separate.
• Until this lands on the ref fullsend fetches (v0), measure steps skip fail-open — that is expected.

[qodo-code-review]

PR Summary by Qodo

Add per-agent measurement manifests for online scoring (trace_fitness em-001)

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Add default per-agent measurement manifests for managed-job online scoring.
• Enable trace_fitness (em-001 v1) for six stock agents; omit others (e.g., scribe).
• Document manifest ownership, resolution order, and separation from functional eval harness.

Diagram

graph TD
  J(["Managed agent job"]) --> CLI["fullsend eval-measure"]
  CLI --> D{"Local manifest?"}
  D -->|"yes"| L["FULLSEND_DIR manifest"]
  D -->|"no"| R["agents@v0 manifest"]
  CLI --> T[("run-telemetry.jsonl")] --> S["trace_fitness scorer\n(em-001 v1)"] --> O[("Measurement results")]

  subgraph Legend
    direction LR
    _proc["Process/CLI"] ~~~ _dec{"Decision"} ~~~ _data[("Data")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Bake defaults into fullsend (no agents@v0 fetch)

• ➕ Single release artifact controls both scorer code and default policy
• ➕ No reliance on agents@v0 pin being updated to enable scoring
• ➖ Harder for agents repo to own/iterate on policy independently
• ➖ Forces fullsend release for simple enable/disable/threshold policy changes

2. Single manifest with per-agent sections

• ➕ Fewer files to manage/version together
• ➕ Easier to diff global policy changes across agents
• ➖ Makes per-agent overrides less discoverable and more conflict-prone
• ➖ Resolution logic becomes more complex (merge/selection vs direct fetch)

3. Remote registry service for measurement policy

• ➕ Policy updates without Git pins; central governance and rollout controls
• ➕ Can support targeting/experiments by org/repo
• ➖ Adds new infrastructure dependency and failure modes
• ➖ Overkill for initial ship; higher operational complexity

Recommendation: Keep the current approach: per-agent YAML manifests in this repo fetched via agents@v0 with explicit FULLSEND_DIR override. It cleanly separates engine/scorer implementation (fullsend) from policy defaults (agents), keeps overrides simple, and matches the ownership model described in the docs.

Files changed (8) +106 / -0

Documentation (2) +70 / -0

README.mdDocument measurement manifests as online-scoring policy +13/-0

Document measurement manifests as online-scoring policy

• Adds a new section explaining that 'eval/measurements/' defines default online-scoring policy for managed jobs. Clarifies separation from functional eval harness scenarios and links to the companion fullsend PR and ADR.

eval/README.md

README.mdAdd measurements README covering ownership and resolution +57/-0

Add measurements README covering ownership and resolution

• Introduces documentation for what measurement manifests are, why they live with agents, and how managed jobs resolve local overrides vs agents@v0 defaults. Defines responsibility boundaries between fullsend (scorer code) and agents (policy) and lists initial agent coverage for em-001.

eval/measurements/README.md

Other (6) +36 / -0

code.yamlAdd default manifest for code agent (em-001 v1) +6/-0

Add default manifest for code agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'code' agent.

eval/measurements/code.yaml

fix.yamlAdd default manifest for fix agent (em-001 v1) +6/-0

Add default manifest for fix agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'fix' agent.

eval/measurements/fix.yaml

prioritize.yamlAdd default manifest for prioritize agent (em-001 v1) +6/-0

Add default manifest for prioritize agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'prioritize' agent.

eval/measurements/prioritize.yaml

retro.yamlAdd default manifest for retro agent (em-001 v1) +6/-0

Add default manifest for retro agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'retro' agent.

eval/measurements/retro.yaml

review.yamlAdd default manifest for review agent (em-001 v1) +6/-0

Add default manifest for review agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'review' agent.

eval/measurements/review.yaml

triage.yamlAdd default manifest for triage agent (em-001 v1) +6/-0

Add default manifest for triage agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'triage' agent.

eval/measurements/triage.yaml

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 11:46 AM UTC · Ended 11:51 AM UTC

Commit: de3a60a · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (1)

1. ID casing inconsistency ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The measurement manifests set the trace_fitness measurement id to em-001, but the measurement
README refers to the same measurement as EM-001, creating ambiguity about the canonical identifier
to use in configs/docs/tooling.

Code

eval/measurements/code.yaml[R3-5]

+  - id: em-001
+    scorer: trace_fitness
+    version: 1

Relevance

●●● Strong

Trivial consistency fix; team has accepted casing-alignment changes in nearby review history.

PR-#592

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The repo’s new measurement README explicitly calls the scorer “EM-001” while each new manifest
defines the measurement id as “em-001”, so the PR introduces two different canonical spellings for
the same measurement identifier.

eval/measurements/README.md[10-11]
eval/measurements/code.yaml[1-5]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The measurement ID is referenced with different casing in different newly-added sources: manifests use `em-001`, while documentation calls it `EM-001`. This makes it unclear which identifier is canonical.

## Issue Context
All per-agent manifests added in this PR use `id: em-001`, but the README in the same directory describes the scorer as `EM-001`.

## Fix Focus Areas
- eval/measurements/README.md[10-11]
- eval/measurements/code.yaml[1-5]
- eval/measurements/fix.yaml[1-5]
- eval/measurements/prioritize.yaml[1-5]
- eval/measurements/retro.yaml[1-5]
- eval/measurements/review.yaml[1-5]
- eval/measurements/triage.yaml[1-5]

## Proposed fix
Pick one canonical representation (either `EM-001` everywhere or `em-001` everywhere) and update both:
1) the documentation string in `eval/measurements/README.md`, and
2) all manifest `id:` fields (if needed) to match.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Missing linked issue authorization 📜 Skill insight § Compliance

Description

This PR introduces a new measurement-manifest feature and adds 20+ lines of new configuration/docs,
but the provided PR description does not link an authorizing issue. This violates the requirement to
have explicit authorization for non-trivial work.

Code

eval/measurements/README.md[R1-4]

+# Measurement manifests
+
+Per-agent YAML that selects which **eval measurement** scorers run after a
+managed agent job (`fullsend eval-measure`). This is **not** the functional

Relevance

● Weak

Similar “add explicit authorization/linked issue” governance suggestions were rejected previously.

PR-#476
PR-#631

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Rule 1538390 requires an explicit linked issue for non-trivial changes. The diff adds a new
eval/measurements/ feature surface (manifests + documentation), but the provided PR description
contains no linked issue reference (e.g., Fixes #...).

eval/measurements/README.md[1-4]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR introduces a non-trivial new feature/config surface (measurement manifests) but has no linked issue authorizing the work in the PR description.

## Issue Context
Compliance requires non-trivial changes (20+ changed lines or structural changes) to include an explicit linked issue (e.g., `Fixes #123` / `Refs #123`, or a full issue URL) that authorizes the change.

## Fix Focus Areas
- eval/measurements/README.md[1-4]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

Tip of the day

💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous review results

Review updated until commit 228c7b5 ⚖️ Balanced

Results up to commit de3a60a ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (1)

1. ID casing inconsistency ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The measurement manifests set the trace_fitness measurement id to em-001, but the measurement
README refers to the same measurement as EM-001, creating ambiguity about the canonical identifier
to use in configs/docs/tooling.

Code

eval/measurements/code.yaml[R3-5]

+  - id: em-001
+    scorer: trace_fitness
+    version: 1

Relevance

●●● Strong

Trivial consistency fix; team has accepted casing-alignment changes in nearby review history.

PR-#592

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The repo’s new measurement README explicitly calls the scorer “EM-001” while each new manifest
defines the measurement id as “em-001”, so the PR introduces two different canonical spellings for
the same measurement identifier.

eval/measurements/README.md[10-11]
eval/measurements/code.yaml[1-5]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The measurement ID is referenced with different casing in different newly-added sources: manifests use `em-001`, while documentation calls it `EM-001`. This makes it unclear which identifier is canonical.

## Issue Context
All per-agent manifests added in this PR use `id: em-001`, but the README in the same directory describes the scorer as `EM-001`.

## Fix Focus Areas
- eval/measurements/README.md[10-11]
- eval/measurements/code.yaml[1-5]
- eval/measurements/fix.yaml[1-5]
- eval/measurements/prioritize.yaml[1-5]
- eval/measurements/retro.yaml[1-5]
- eval/measurements/review.yaml[1-5]
- eval/measurements/triage.yaml[1-5]

## Proposed fix
Pick one canonical representation (either `EM-001` everywhere or `em-001` everywhere) and update both:
1) the documentation string in `eval/measurements/README.md`, and
2) all manifest `id:` fields (if needed) to match.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Missing linked issue authorization 📜 Skill insight § Compliance

Description

This PR introduces a new measurement-manifest feature and adds 20+ lines of new configuration/docs,
but the provided PR description does not link an authorizing issue. This violates the requirement to
have explicit authorization for non-trivial work.

Code

eval/measurements/README.md[R1-4]

+# Measurement manifests
+
+Per-agent YAML that selects which **eval measurement** scorers run after a
+managed agent job (`fullsend eval-measure`). This is **not** the functional

Relevance

● Weak

Similar “add explicit authorization/linked issue” governance suggestions were rejected previously.

PR-#476
PR-#631

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Rule 1538390 requires an explicit linked issue for non-trivial changes. The diff adds a new
eval/measurements/ feature surface (manifests + documentation), but the provided PR description
contains no linked issue reference (e.g., Fixes #...).

eval/measurements/README.md[1-4]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR introduces a non-trivial new feature/config surface (measurement manifests) but has no linked issue authorizing the work in the PR description.

## Issue Context
Compliance requires non-trivial changes (20+ changed lines or structural changes) to include an explicit linked issue (e.g., `Fixes #123` / `Refs #123`, or a full issue URL) that authorizes the change.

## Fix Focus Areas
- eval/measurements/README.md[1-4]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 11:52 AM UTC · Ended 11:54 AM UTC

Commit: fe9703a · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:55 AM UTC · Completed 12:08 PM UTC

Commit: c1dbd68 · View workflow run →

[fullsend-ai-review]

Review

Findings

Low

• [yaml-document-marker-consistency] eval/measurements/code.yaml:1 — The six new measurement manifests use the YAML document start marker (---) on line 1, while all existing YAML files under eval/ (e.g. eval/code/eval.yaml, eval/triage/cases/001-bug-url-encoding/input.yaml) omit it. This introduces a minor style inconsistency within the eval/ tree. Since these files live in their own subdirectory and serve a distinct purpose, this is cosmetic rather than harmful, but dropping the --- would align with the established convention.

• [incomplete documentation] LOCAL.md:133 — LOCAL.md describes the eval/ directory as containing only 'functional test scenarios that run agents against ephemeral GitHub repos and score the results.' With this PR, eval/ now also contains eval/measurements/ — default online-scoring measurement manifests consumed by fullsend eval-measure. The description is not wrong but is now incomplete; a reader following this guide would not learn that eval/ also houses measurement manifests.
  Remediation: Add a sentence after line 135 mentioning the measurement manifests, e.g.: 'The eval/measurements/ subdirectory holds default online-scoring manifests (see eval/measurements/README.md).'

Previous run

Review

Findings

Low

• [documentation-link-consistency] eval/README.md:151 — Reference to 'fullsend ADR 0087' uses plain text while adjacent references in the same sentence use markdown hyperlinks (e.g., [fullsend#6036](...)). The eval/measurements/README.md already hyperlinks the same ADR with a direct URL, so the link target is known and available.

---

Labels: PR adds YAML policy files and documentation for eval measurement manifests

Previous run (2)

Review

Findings

Low

• [scope-architectural-inconsistency] eval/measurements/prioritize.yaml — Measurement manifest exists for the 'prioritize' agent, but no functional eval directory exists at eval/prioritize/. The measurements README explains these are separate systems, but an explicit note that measurement coverage is independent of functional eval coverage would prevent future confusion.

• [scope-architectural-inconsistency] eval/measurements/retro.yaml — Same as above for the 'retro' agent — no functional eval directory at eval/retro/. Consider a one-line clarification in the measurements README.

• [scope-missing-dependency] eval/measurements/README.md:14 — References 'fullsend ADR 0087' as the authoritative specification without a direct URL. The ADR lives in the external fullsend-ai/fullsend repository (a standard cross-repo reference pattern), but adding a direct link when the ADR is published would improve discoverability.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 1:28 PM UTC · Ended 1:33 PM UTC

Commit: 74b8959 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 1:34 PM UTC · Completed 1:48 PM UTC

Commit: 93d3db9 · View workflow run →

[waynesun09]

MEDIUM: No CI/pre-commit validation for eval/measurements/*.yaml — bad scorer/agent values silently no-op forever

(Referencing .pre-commit-config.yaml line 49, which isn't part of this PR's diff, so noting it here in the review body instead of inline.)

Confirmed on head 93d3db9f: the repo has a pre-commit hook lint-agent-docs (./hack/lint-agent-docs) scoped via files: ^(harness/|docs/|hack/lint-agent-docs) that validates every harness/*.yaml has a valid doc: field, and a separate eval/lint-cases.sh that validates eval/<agent>/cases and eval.yaml judges — but neither covers the new eval/measurements/*.yaml files added in this PR, and no .github/workflows/*.yml step does either. Only the generic check-yaml pre-commit hook (syntax-only) applies.

On the consumer side, in the companion PR fullsend-ai/fullsend#6036, internal/evalmeasure/registry.go's ScoreTrace has:

switch m.Scorer {
case ScorerFitness: ...
default: // Unknown scorers are skipped (forward-compatible)
}

i.e. a typo'd scorer: value (e.g. trace-fitness with a hyphen) or a misspelled agent: value that never matches a trace's agent name would pass this repo's CI cleanly, merge, and then silently produce zero measurements for that agent in production with no error/log/CI failure.

The six current manifests all correctly use the literal trace_fitness matching const ScorerFitness = "trace_fitness" in fitness.go, so nothing is broken today, but there's no guardrail against this on the next manifest edit or new-agent addition.

Suggestion: Add a small lint script (sibling to hack/lint-agent-docs, or extend eval/lint-cases.sh) that validates eval/measurements/*.yaml: agent: matches an existing agents/<name>.md, measurements[].id is unique per file, and .scorer/.version are present/well-formed (and, once the fullsend-side registry is stable, checked against a known-scorer allow-list). Wire it into .pre-commit-config.yaml (files: ^eval/measurements/) so a typo fails CI instead of silently no-op'ing at runtime.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Ended 11:50 AM UTC

Commit: 228c7b5 · View workflow run →

[fullsend-ai-review]

🤖 Review · ❌ Terminated · Started 11:51 AM UTC · Ended 12:04 PM UTC

Commit: 228c7b5 · View workflow run →

[qodo-code-review]

Code review by qodo was updated up to the latest commit 228c7b5

[fullsend-ai-review]

[low] yaml-document-marker-consistency

The six new measurement manifests use the YAML document start marker (---) on line 1, while all existing YAML files under eval/ (e.g. eval/code/eval.yaml, eval/triage/cases/001-bug-url-encoding/input.yaml) omit it. This introduces a minor style inconsistency within the eval/ tree. Since these files live in their own subdirectory and serve a distinct purpose, this is cosmetic rather than harmful, but dropping the --- would align with the established convention.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:51 AM UTC · Completed 12:04 PM UTC

Commit: 228c7b5 · View workflow run →

[waynesun09]

Automated review sweep — 3 findings (0 duplicates skipped after checking existing inline comments/reviews on this PR).

[waynesun09]

MEDIUM: review.yaml enables em-001 even though its work_item sub-check can never pass for review traces

Verified against the companion PR's actual code (fullsend-ai/fullsend#6036, internal/evalmeasure/fitness.go @ 615dd691c2): ScoreFitnessNamed requires 8/8 sub-checks to pass for label="pass". One sub-check, workItemOK(run), requires fullsend.work_item_id to be set and not the literal "unknown". The code's own comment states: '"unknown" is the CLI sentinel when no ISSUE_*/GITHUB_ISSUE_URL is set (common for review, which wires PR_NUMBER / GITHUB_PR_URL instead).' This is confirmed by the companion PR's own test fixture testdata/review-unknown-workitem.jsonl (fullsend.work_item_id="unknown") and score_test.go's TestScoreFitness_ReviewUnknownWorkItemFails, which asserts Explanation contains work_item=fail and missing: work_item for an agent:review trace. Since this agents-repo PR enables em-001 for review, review's fitness score is structurally capped at 7/8=0.875 and will always report label=fail by design — nothing in this PR or the companion PR's manifest/README documents that cap specifically for review, and it is presented as uniform across all six enabled agents.

Suggestion: Add a note next to review.yaml (or in the README's per-agent table) that review's em-001 label will always read 'fail' until the scorer gains a PR-based identity check or review starts populating fullsend.work_item_id, so downstream trend consumers don't mistake the permanent fail for a regression.

[waynesun09]

MEDIUM: EM-001/em-001 casing inconsistency reintroduced after prior fix

A prior review thread on this PR (resolved) flagged the manifests using 'em-001' while the README used 'EM-001', and commit e8b75c0 ('fix(eval): normalize ID casing and add YAML document markers') fixed that original occurrence, correctly using lowercase in the 'First ship' section and the id: field. However, a later commit, 4d5728a ('docs(eval): clarify measurement manifest ownership'), rewrote/expanded the README and introduced a brand-new sentence at this line — 'Platform checks like EM-001 (trace_fitness) still get enabled here for each stock agent' — using uppercase again, while lines 47/54 in the same file (added by the same rewrite) correctly use lowercase 'em-001' matching the manifests' literal id: fields. This is a new, unresolved instance of the exact ambiguity the earlier fix addressed, at a different location than the already-resolved thread, so it was never caught by the earlier fix.

Suggestion: Change 'Platform checks like EM-001' to 'Platform checks like em-001' so the file consistently uses the manifest's literal id: em-001 casing everywhere.

[waynesun09]

MEDIUM: ADR 0087 hyperlink 404s (target doesn't exist on fullsend main)

Both this line and eval/README.md:151 render a live markdown hyperlink to https://github.com/fullsend-ai/fullsend/blob/main/docs/ADRs/0087-eval-measurements-online-trace-scoring.md. Verified with curl (https://raw.githubusercontent.com/fullsend-ai/fullsend/main/docs/ADRs/0087-eval-measurements-online-trace-scoring.md -> HTTP 404) and via gh pr view 6036 --repo fullsend-ai/fullsend (state OPEN, mergedAt null) that ADR 0087 only exists on the still-unmerged companion PR fullsend-ai/fullsend#6036. This is distinct from the two already-posted review comments on this PR (one asked to add a direct URL 'when the ADR is published', the other flagged plain-text-vs-hyperlink inconsistency) — commit 228c7b5 ('docs(eval): hyperlink ADR 0087 in eval README', current head) added the real hyperlink before the ADR was published, so the link now renders and 404s for anyone who clicks it prior to #6036 merging.

Suggestion: Hold merging this PR until fullsend-ai/fullsend#6036 lands, or temporarily point both links at the PR itself (https://github.com/fullsend-ai/fullsend/pull/6036) and swap to the permanent docs/ADRs/0087 URL once #6036 merges.