Skip to content

feat(eval): findings diagnostic chain with dual entry points - #301

Merged
MeloMei merged 3 commits into
mainfrom
feat/eval-skill-findings
Aug 2, 2026
Merged

feat(eval): findings diagnostic chain with dual entry points#301
MeloMei merged 3 commits into
mainfrom
feat/eval-skill-findings

Conversation

@MeloMei

@MeloMei MeloMei commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Eval 阶段三:把描述性数字升级为带证据的结论(finding),并让结论在 Skills 页被撞见。

做了什么

  • 新增纯函数规则求值器(src/core/skill-eval-findings.ts),四条摩擦规则:已装未用 / 工具反复失败 / 触发轮次成本偏高 / 触发轮次频繁出错——每条规则标注严重度(low/medium,不设 high)、证据强度、样本量,样本不足即沉默
  • 新增 listSkillToolOutcomes 查询:按工具名聚合 linked-turn 触发 turn 内的 kind='tool' span,只计 status='failed'(aborted 是用户中断不算失败),session_id 白名单从一开始就带 codex
  • Eval 页实况卡片上方新增 Findings 段:每条 finding 可展开查看 observation + repairDirection + evidence,零 finding 时明示"不等于没问题"
  • Skills 页技能列表新增 finding 数量徽标,点击跳转 Eval 页对应 Skill(双向入口),Eval 关闭时不渲染任何徽标
  • 措辞纪律进测试:observation 不得出现因果词,repairDirection 只能是"Consider checking..."

边界

  • V1 零改动,不新增表不新增迁移
  • 不做因果断言、不设 high 严重度、findings 不落库(查询期计算)
  • Qoder 会话在全部摩擦规则上沉默(无 trace_spans/token),显示 unobserved

测试

  • 29 个阈值表驱动测试覆盖四条规则的边界(恰好命中 / 差一个样本 / 差一个百分点)
  • V2 全量 292 文件 / 2526 passed,V1 97 文件 / 1235 passed,release-note:check 通过

Copilot AI review requested due to automatic review settings August 2, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds “findings” as a higher-level diagnostic layer in the Eval experience: it evaluates a small set of table-driven friction rules against observed skill usage/performance data, exposes them via IPC, and surfaces them both in the Eval page and as a badge entry point from the Skills page.

Changes:

  • Introduces a pure-function findings evaluator (4 rules) plus wording-discipline tests to keep observations non-causal and repair directions in a constrained format.
  • Adds IPC/service/store plumbing to compute per-skill findings and aggregated finding counts for Skills page badges.
  • Updates the renderer UI/CSS to display findings in Eval and show clickable findings-count badges in the Skills list that deep-link into Eval.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
apps/main-2.0/src/shared/ipc/skills.ts Adds IPC endpoints for findings + finding-counts.
apps/main-2.0/src/renderer/src/styles/skills-page.css Styles the Skills-page findings badge link.
apps/main-2.0/src/renderer/src/styles/eval.css Adds styling for the new Findings card UI.
apps/main-2.0/src/renderer/src/features/skills/skills-page.tsx Threads findings-count data + navigation callback into the Skills list.
apps/main-2.0/src/renderer/src/features/skills/skill-library-list.tsx Renders findings badge per skill with click-to-navigate behavior.
apps/main-2.0/src/renderer/src/features/eval/eval-page.tsx Fetches and renders findings per selected skill; consumes preselected-skill navigation.
apps/main-2.0/src/renderer/src/App.tsx Fetches global finding counts when Eval is enabled; wires Skills→Eval deep link.
apps/main-2.0/src/preload/skills.ts Exposes findings IPC methods to the renderer API.
apps/main-2.0/src/main/services/skill-service.ts Implements getSkillFindings + getSkillFindingCounts and calls the evaluator.
apps/main-2.0/src/main/services/skill-service.test.ts Updates service harness mocks for the new store method.
apps/main-2.0/src/main/ipc/skills.ts Registers IPC handlers for findings and finding counts.
apps/main-2.0/src/core/skill-eval-findings.ts Adds table-driven findings evaluator + wording assertion helper.
apps/main-2.0/src/core/skill-eval-findings.test.ts Adds boundary + wording-discipline tests for the 4 rules.
apps/main-2.0/src/core/session-store.ts Adds session-store wrapper for listing tool outcomes.
apps/main-2.0/src/core/postgres/skill-repository.ts Adds SkillToolOutcome model + query to aggregate tool-call outcomes per skill.
.release-notes/eval-skill-findings.md Adds user-facing release note for Findings + dual entry points.
Suppressed comments (2)

apps/main-2.0/src/main/services/skill-service.ts:820

  • getSkillFindingCounts uses linkedTriggers > 0 to decide whether a skill is "exercised" vs "never-used", which can incorrectly label triggered skills (with no linked triggers) as "never-used" and incorrectly count the installed-never-exercised finding. It also currently can’t represent "unobserved" for agents like trae, which can make the badge counts disagree with the Eval overview semantics.
      const installed = installedNames.has(name);
      const overviewLike = overviewItem
        ? {
            observation: (overviewItem.linkedTriggers > 0 ? "exercised" : "never-used") as "exercised" | "never-used" | "unobserved",
            installed,

apps/main-2.0/src/main/services/skill-service.ts:812

  • getSkillFindingCounts runs getSkillPerformanceSignals + listSkillToolOutcomes inside a for ... of loop, resulting in 2×N round-trips (and serialized execution) for N skills. Since this is used to render a badge list, it can become noticeably slow with many installed/triggered skills.
    for (const name of allNames) {
      const [signals, toolOutcomes] = await Promise.all([
        store.getSkillPerformanceSignals(name),
        store.listSkillToolOutcomes(name),
      ]);

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apps/main-2.0/src/renderer/src/features/eval/eval-page.tsx Outdated
Comment thread apps/main-2.0/src/core/skill-eval-findings.ts Outdated
Comment thread apps/main-2.0/src/main/services/skill-service.ts Outdated
Comment thread apps/main-2.0/src/core/postgres/skill-repository.ts
MeloMei added 2 commits August 2, 2026 11:39
# Conflicts:
#	apps/main-2.0/src/main/services/skill-service.test.ts
@MeloMei
MeloMei merged commit 9d62561 into main Aug 2, 2026
4 checks passed
@MeloMei
MeloMei deleted the feat/eval-skill-findings branch August 2, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants