feat(eval): findings diagnostic chain with dual entry points - #301
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds “findings” as a higher-level diagnostic layer in the Eval experience: it evaluates a small set of table-driven friction rules against observed skill usage/performance data, exposes them via IPC, and surfaces them both in the Eval page and as a badge entry point from the Skills page.
Changes:
- Introduces a pure-function findings evaluator (4 rules) plus wording-discipline tests to keep observations non-causal and repair directions in a constrained format.
- Adds IPC/service/store plumbing to compute per-skill findings and aggregated finding counts for Skills page badges.
- Updates the renderer UI/CSS to display findings in Eval and show clickable findings-count badges in the Skills list that deep-link into Eval.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| apps/main-2.0/src/shared/ipc/skills.ts | Adds IPC endpoints for findings + finding-counts. |
| apps/main-2.0/src/renderer/src/styles/skills-page.css | Styles the Skills-page findings badge link. |
| apps/main-2.0/src/renderer/src/styles/eval.css | Adds styling for the new Findings card UI. |
| apps/main-2.0/src/renderer/src/features/skills/skills-page.tsx | Threads findings-count data + navigation callback into the Skills list. |
| apps/main-2.0/src/renderer/src/features/skills/skill-library-list.tsx | Renders findings badge per skill with click-to-navigate behavior. |
| apps/main-2.0/src/renderer/src/features/eval/eval-page.tsx | Fetches and renders findings per selected skill; consumes preselected-skill navigation. |
| apps/main-2.0/src/renderer/src/App.tsx | Fetches global finding counts when Eval is enabled; wires Skills→Eval deep link. |
| apps/main-2.0/src/preload/skills.ts | Exposes findings IPC methods to the renderer API. |
| apps/main-2.0/src/main/services/skill-service.ts | Implements getSkillFindings + getSkillFindingCounts and calls the evaluator. |
| apps/main-2.0/src/main/services/skill-service.test.ts | Updates service harness mocks for the new store method. |
| apps/main-2.0/src/main/ipc/skills.ts | Registers IPC handlers for findings and finding counts. |
| apps/main-2.0/src/core/skill-eval-findings.ts | Adds table-driven findings evaluator + wording assertion helper. |
| apps/main-2.0/src/core/skill-eval-findings.test.ts | Adds boundary + wording-discipline tests for the 4 rules. |
| apps/main-2.0/src/core/session-store.ts | Adds session-store wrapper for listing tool outcomes. |
| apps/main-2.0/src/core/postgres/skill-repository.ts | Adds SkillToolOutcome model + query to aggregate tool-call outcomes per skill. |
| .release-notes/eval-skill-findings.md | Adds user-facing release note for Findings + dual entry points. |
Suppressed comments (2)
apps/main-2.0/src/main/services/skill-service.ts:820
getSkillFindingCountsuseslinkedTriggers > 0to decide whether a skill is "exercised" vs "never-used", which can incorrectly label triggered skills (with no linked triggers) as "never-used" and incorrectly count the installed-never-exercised finding. It also currently can’t represent "unobserved" for agents liketrae, which can make the badge counts disagree with the Eval overview semantics.
const installed = installedNames.has(name);
const overviewLike = overviewItem
? {
observation: (overviewItem.linkedTriggers > 0 ? "exercised" : "never-used") as "exercised" | "never-used" | "unobserved",
installed,
apps/main-2.0/src/main/services/skill-service.ts:812
getSkillFindingCountsrunsgetSkillPerformanceSignals+listSkillToolOutcomesinside afor ... ofloop, resulting in 2×N round-trips (and serialized execution) for N skills. Since this is used to render a badge list, it can become noticeably slow with many installed/triggered skills.
for (const name of allNames) {
const [signals, toolOutcomes] = await Promise.all([
store.getSkillPerformanceSignals(name),
store.listSkillToolOutcomes(name),
]);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…servation logic, ORDER BY, aria-label)
# Conflicts: # apps/main-2.0/src/main/services/skill-service.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eval 阶段三:把描述性数字升级为带证据的结论(finding),并让结论在 Skills 页被撞见。
做了什么
边界
测试