From 5dfbb8d72c5a5a4f4650fce63cdb60aaab610b08 Mon Sep 17 00:00:00 2001 From: Byron Wall Date: Thu, 12 Nov 2026 21:54:14 -0500 Subject: [PATCH] docs: retire metric-system-overhaul All seven milestones merged on 2026-10-16 (#73-#79). The pilot README now links the plan at a pinned revision. Move the durable facts into the lasting docs, add the closure record to docs/initiative-history.json, and remove the planning folder. Co-Authored-By: Claude Opus 5.5 --- docs/adding-a-metric.md | 13 + docs/initiative-history.json | 22 + .../implementation-plan.md | 106 ---- .../initiative-map.json | 558 ------------------ .../metric-system-overhaul/intent-brief.md | 64 -- .../metric-system-overhaul/shape-brief.md | 80 --- docs/research/metric-pilot-2026-10/README.md | 2 +- 7 files changed, 36 insertions(+), 809 deletions(-) delete mode 100644 docs/intent/metric-system-overhaul/implementation-plan.md delete mode 100644 docs/intent/metric-system-overhaul/initiative-map.json delete mode 100644 docs/intent/metric-system-overhaul/intent-brief.md delete mode 100644 docs/intent/metric-system-overhaul/shape-brief.md diff --git a/docs/adding-a-metric.md b/docs/adding-a-metric.md index 8f012105..f796168a 100644 --- a/docs/adding-a-metric.md +++ b/docs/adding-a-metric.md @@ -30,3 +30,16 @@ For a metric that flows end to end (analyzer → API → treemap, Hot Spots, exp - Confirm the metric appears in the **Color** and **Rank by** pickers (Rank by only when it isn't neutral), ranks Hot Spots with its unit and N/A group, and shows in the tooltip and code viewer for the languages that measure it. - Avoid `// @ts-ignore` / `// @ts-expect-error`; instead, update types so the pipeline is type-safe from API response through to the UI. + +## What the catalog keeps, and why + +Check a new metric against these decisions before you add it. + +- Each metric must give its own actionable signal. A metric that mostly repeats another one is cut, not kept as a variant. +- Removed on purpose: the Low Comments penalty (replaced by `comment_noise`), hardcoded and duplicated TSX string counts, and the class, type and import risk weights. Removed metrics get no compatibility layer. +- TODO counts are neutral detail, never risk. +- React and Solid components have their own component-scoped metrics, such as hooks, effects, JSX depth and props. +- Findings come from the project's own `tsc` and `eslint` when present. Otherwise srcly runs `ruff` and its own TypeScript rules. Each finding names its tool. +- Named 2x2 presets use absolute cut-offs, not cut-offs relative to the repo. +- Below the cut line: clones, coverage, CRAP, temporal coupling and trends. +- Tune defaults with evidence. The October 2026 pilot adjudicated 729 top-10 cases on four repositories, then set the defaults (for example, `comment_noise` flags 1.0 comment line per code line or more). Its method, results and open follow-ups are in `docs/research/metric-pilot-2026-10/README.md`. Rerun a pilot with `server/scripts/evaluate_metric_portfolio.py`. diff --git a/docs/initiative-history.json b/docs/initiative-history.json index 8b6b0918..c1052fe8 100644 --- a/docs/initiative-history.json +++ b/docs/initiative-history.json @@ -84,5 +84,27 @@ "follow_ups": [ "Longer-horizon ideas in claim C8 (change islands, PR splittability, ripple frontier) were never planned and are not tracked." ] + }, + { + "title": "Metric system overhaul", + "slug": "metric-system-overhaul", + "outcome": "completed", + "retired_at": "2026-11-12", + "summary": "The metric system overhaul landed as seven stacked PRs: trust fixes, catalog cuts with comment noise and component metrics, one server-owned catalog with a single-metric list, Git activity, 2x2 presets, zero-config findings, and a four-repo pilot that tuned the defaults.", + "reason": "All seven milestones merged on 2026-10-16 (#73-#79); the map already recorded completion.", + "documentation": [ + "docs/adding-a-metric.md", + "docs/CODE_MAP.md", + "docs/research/metric-pilot-2026-10/README.md" + ], + "revision": "0bfb05ba", + "follow_ups": [ + "Scope LOC (and commit count) to code languages, so Markdown, HTML and CSS stop ranking beside code.", + "Make parameter_count's file and folder value the max arity with a witness instead of a sum.", + "Fix the LOC description or count: TS and Python LOC include comment lines; then revisit comment_ratio.", + "Weight findings by severity, or set stylistic rules (eslint curly) apart, so one project rule can't fill the Findings preset.", + "Detect components that return .map(...) of JSX (explorEDA DslCode.tsx).", + "Run the next pilot on a TS repo without tsc or eslint and a non-author repo, with a second reviewer." + ] } ] diff --git a/docs/intent/metric-system-overhaul/implementation-plan.md b/docs/intent/metric-system-overhaul/implementation-plan.md deleted file mode 100644 index 0a1d37cd..00000000 --- a/docs/intent/metric-system-overhaul/implementation-plan.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: "Metric system overhaul — implementation plan" -slug: "metric-system-overhaul" -phase: plan -status: current -last_updated: "2026-10-15" ---- - -# Metric system overhaul — implementation plan - -## Plan at a glance - -There are seven stacked PRs, each a Conventional Commit and each squash-mergeable on its own. Trust comes first: a new signal on top of an N/A-as-zero pipeline would just mislead faster. The catalog then moves to one server-owned definition before the list, scatter and findings build on it. Without that, every new metric would need three hand edits (`reporting.py`, `metricsStore.tsx`, `compare/diff.py`). Git activity reuses the Git plumbing that compare mode already uses. Findings arrive last among the features because they add the only new runtime dependency (`ruff`) and external processes. The pilot closes the stack and tunes thresholds and defaults with evidence from local repos. - -Pushback against the research: it planned to keep the old catalog during evaluation and gate each initiative on Byron's review. Here, cuts land early (PR 2) and the agent adjudicates the pilot. That's cheaper, and any cut is one revert away. - -## Implementation strategy - -- **First proof:** a fixture scan whose zero, N/A and witness values match across API, report and explorer. -- **Primary seams:** `analysis_types.py` → `analysis.py` (measurement), a new catalog module (definitions), `activity.py` and `findings/` (external processes behind one function each). -- **Fast local loop:** `cd server && uv run pytest tests/test_complexity_semantics.py tests/test_reporting.py -q`, plus `cd client && pnpm test`. -- **Local dependencies:** fixture directories under `server/tests/fixtures/`, plus temp Git repositories that tests build with `git init`. -- **Live confirmation:** scan srcly and recipes-modern in the browser (API on 8000, Vite on 5173). -- **Rollout and rollback:** one revertable PR per step. Activity and findings degrade to N/A when the tool is missing, slow, or fails. - -## Milestone 1: Zero and N/A stay distinct, and maxima name their function (PR `fix(metrics): …`) - -- **Change — availability and scope kind** - - Add the executable kind and per-metric availability to `FunctionMetrics`/`FileMetrics` and to `Metrics` in `models.py`. Unsupported, skipped and failed become `null` with a reason, not `0`. - - `convert_function` keeps the executable kind. Folder `complexity` carries the witness path and line. -- **Change — ranking math** - - Fix `reporting.py:rank_nodes` so zero values count toward the denominator. Align the cohort in `Explorer.tsx` (files or functions, never mixed). -- **Verification:** a fixture with nested TS/Python functions, a measured zero, and a skipped large file. Monotonicity test: raising one input never lowers a score. - -### Desired end state - -- The report, explorer detail and API show N/A for unsupported metrics. -- Folder max complexity opens the right function. - -## Milestone 2: The catalog is honest and smaller (PR `feat(metrics)!: …`) - -- **Change — cuts:** delete `comment_density` as "Low Comments", `tsx_hardcoded_string_volume`, `tsx_duplicated_string_count`, and the risk weights on class, type, export and import counts in `PROFILE_WEIGHTS`. Demote `todo_count` to neutral detail. Fix README complexity wording. -- **Change — comment noise:** add `comment_noise` with three parts: comment ratio > 0.5 in a scope of at least 10 code lines, a docblock longer than 1.5× its body and at least 8 lines, and runs of at least 3 commented lines that parse as code. These are absolute thresholds; the pilot may tune them. -- **Change — React components:** detect components (a capitalized function or arrow returning JSX). Record per component: LOC, hooks, effects (`useEffect`/`createEffect`), local JSX depth, declared props, inline handlers, and JSX conditionals. The file value is the max, and it names the component. -- **Verification:** React and Solid component fixtures, plus an over-commented fixture and a commented-out code fixture. - -### Desired end state - -- No metric scores "fewer comments" as risk. -- React fixtures rank the god component first. - -## Milestone 3: One catalog drives the list, the report and compare (PR `feat(metrics): …`) - -- **Change:** create `server/app/services/metric_catalog.py` for id, label, unit, group, direction, scope, languages and description. Serve it from `GET /api/metrics/catalog`. `reporting.METRIC_DEFINITIONS` and `compare/diff.COMPARE_METRICS` derive from it. The client's `HOTSPOT_METRICS` becomes a fetched store. Update the `AGENTS.md` add-a-metric rule and `docs/CODE_MAP.md`. -- **Change — the single-metric list:** the Hot Spots list ranks by one metric. Each row shows the value with its unit, the witness function, and a cohort label. N/A rows sit in a collapsed group. -- **Verification:** a parity test where the same fixture gives the same top 10 from `srcly report` and from the client ranking function. A compare test confirms comment noise counts as "worse". - -### Desired end state - -- Adding a metric needs one catalog entry plus its analyzer. - -## Milestone 4: Activity explains "why now" (PR `feat(activity): …`) - -- **Change:** add `services/activity.py`, which reuses the runner in `compare/git_refs.py`. One `git log --numstat --no-merges --since=` pass gives `commit_count` and `churn_lines` per file. The default window is 90 days. Shallow clones, dirty trees and missing `.git` come back as N/A with a reason. The cache keys on HEAD plus the window. -- **Verification:** a temp repo with a rename, a merge, and a bulk commit. Counts match `git log`. - -### Desired end state - -- Activity shows in the catalog and the list on any Git repo with no setup. - -## Milestone 5: 2x2 presets (PR `feat(scatter): …`) - -- **Change:** add a scatter view beside the treemap with four presets: Activity, Blast radius (fan-in from the deps graph), Component weight, and Comment noise. Each preset has named quadrants, draggable cut-offs on absolute defaults, and a file/function toggle. Clicking a point opens its source. The report gets `--quadrant `. -- **Verification:** a browser check on srcly and recipes-modern, plus a unit test of quadrant assignment. - -### Desired end state - -- Each preset puts known hot files in the expected quadrant, and a point click opens its source. - -## Milestone 6: Findings just work (PR `feat(findings): …`) - -- **Change:** add a `services/findings/` runner. It uses project `node_modules/.bin/tsc --noEmit` and eslint when they and their config exist. Otherwise it falls back to `ruff check --output-format json`, added via `uv add ruff`, and to built-in tree-sitter rules (explicit `any`, unexplained `@ts-ignore`, non-null assertion runs). The runner is a separate job with timeouts. The result is a per-file `finding_count` plus a source-linked list labelled with the tool that ran. -- **Verification:** fixtures with and without project tools, and a timeout injection. - -### Desired end state - -- Any repo shows findings with no setup, labelled by the tool that ran; a tool failure shows N/A, never zero. - -## Milestone 7: The pilot tunes defaults (PR `feat(metrics): tune defaults from pilot`) - -- **Change:** add `server/scripts/evaluate_metric_portfolio.py`. Per language and scope, it reports missing rates, Spearman correlation and top-10/25 overlap. The agent then reviews the top 10 per lens and preset on srcly, recipes-modern, explorEDA and context-recorder. A row counts as useful only when a cited source line justifies action. Results update the thresholds, quadrant cut-offs and the default metric. The record goes in `docs/research/`. - -### Desired end state - -- Every default metric has at least two useful cases in two repos, or it is demoted. - -## Open decisions and spikes - -- **Component detection on Solid.** Decision: whether the capitalized-JSX heuristic is enough. Evidence: the srcly client and recipes-modern. Fallback: label it React-only. -- **Bundled TS rules.** Decision: whether tree-sitter rules give useful findings without tsc. Evidence: pilot hit rate. Fallback: show only project-tool findings for TS. - -## Below the cut line - -- Clones, coverage, CRAP, mutation, temporal coupling, and trends over snapshots. -- People or ownership metrics, Maintainability Index, and Halstead estimates. -- Persistent triage states. diff --git a/docs/intent/metric-system-overhaul/initiative-map.json b/docs/intent/metric-system-overhaul/initiative-map.json deleted file mode 100644 index e0ee8da4..00000000 --- a/docs/intent/metric-system-overhaul/initiative-map.json +++ /dev/null @@ -1,558 +0,0 @@ -{ - "schema_version": "0.5.0", - "initiative": { - "title": "Metric system overhaul", - "slug": "metric-system-overhaul", - "scale": "program", - "current_phase": "execution", - "last_updated": "2026-10-16", - "summary": "Turn the codex/metrics-research findings (metric trust, portfolio, Git activity, imported evidence) into a stack of PRs landing now.", - "status": "completed", - "next_action": "None: all seven milestones landed as stacked PRs #73-#79. Follow-ups are in docs/research/metric-pilot-2026-10/README.md." - }, - "artifacts": { - "intent_brief": { - "path": "intent-brief.md", - "status": "current", - "updated_at": "2026-10-15" - }, - "brainstorm": { - "path": "brainstorm.md", - "status": "not-requested", - "updated_at": null - }, - "shape_brief": { - "path": "shape-brief.md", - "status": "current", - "updated_at": "2026-10-15" - }, - "implementation_plan": { - "path": "implementation-plan.md", - "status": "current", - "updated_at": "2026-10-15" - } - }, - "sources": [ - { - "id": "research-branch", - "type": "research", - "location": "origin/codex/metrics-research", - "anchor": "docs/research/metrics-2026-10-15 and docs/intent/{metric-trust,metric-portfolio,git-activity,imported-quality-evidence}", - "authority": "research" - }, - { - "id": "main-compare", - "type": "repository", - "location": "main", - "anchor": "server/app/services/compare (PR compare mode added after research baseline 83dcbf4)", - "authority": "repository" - }, - { - "id": "qa-round-1", - "type": "question-round", - "location": "conversation", - "anchor": "1", - "authority": "user" - }, - { - "id": "qa-round-2", - "type": "question-round", - "location": "conversation", - "anchor": "2", - "authority": "user" - }, - { - "id": "local-corpus", - "type": "repository", - "location": "~/Projects", - "anchor": "Local git repos usable as pilot corpus: srcly, recipes-modern (React), explorEDA (TSX+Python, 1273 commits), product-grid-mgmt, shadcn-ui, humanlayer, context-recorder (Python)", - "authority": "observation" - }, - { - "id": "qa-round-3", - "type": "question-round", - "location": "conversation", - "anchor": "3", - "authority": "user" - } - ], - "intent": { - "claims": [ - { - "id": "C1", - "statement": "Explorer, headless reports, and PR compare are equally important consumers; one shared metric definition should drive all three.", - "kind": "goal", - "sources": [ - "qa-round-1" - ], - "confidence": "high", - "core": true - }, - { - "id": "C2", - "statement": "Work happens now as a stack of PRs: fix issues first, then improve the system.", - "kind": "decision", - "sources": [ - "qa-round-1" - ], - "confidence": "high" - }, - { - "id": "C3", - "statement": "All four research initiatives are in scope now, not gated behind each other.", - "kind": "decision", - "sources": [ - "qa-round-1" - ], - "confidence": "high" - }, - { - "id": "C4", - "statement": "Cut boldly: remove misleading risk directions and noisy metrics now, with no compatibility layer.", - "kind": "decision", - "sources": [ - "qa-round-1" - ], - "confidence": "high" - }, - { - "id": "C5", - "statement": "The agent runs the portfolio pilot itself on local repos and adjudicates cases; Byron is minimally available. Use consider-contrast for consequential choices.", - "kind": "decision", - "sources": [ - "qa-round-2" - ], - "confidence": "high" - }, - { - "id": "C6", - "statement": "Must have an excellent single-metric hotspot list. Add a two-axis scatter with a named 2x2 quadrant model.", - "kind": "goal", - "sources": [ - "qa-round-2" - ], - "confidence": "high", - "core": true - }, - { - "id": "C7", - "statement": "Keep React/TSX-specific metrics that help target React and TSX refactors; choose the set by consider-contrast instead of deleting them all.", - "kind": "goal", - "sources": [ - "qa-round-2" - ], - "confidence": "high" - }, - { - "id": "C8", - "statement": "Excessive comments matter now (AI-era over-commenting); the low-comments penalty is the wrong direction. TODOs are low priority.", - "kind": "goal", - "sources": [ - "qa-round-2" - ], - "confidence": "high" - }, - { - "id": "C9", - "statement": "Srcly must just work: run lint/type tools itself with no bespoke config, rather than import user-supplied artifacts. Lint/type findings are in scope.", - "kind": "constraint", - "sources": [ - "qa-round-2" - ], - "confidence": "high" - }, - { - "id": "C10", - "statement": "Ship several named 2x2 scatter presets (e.g. activity, blast radius, React component weight, comment noise) rather than one fixed pair.", - "kind": "decision", - "sources": [ - "qa-round-3" - ], - "confidence": "high" - }, - { - "id": "C11", - "statement": "Excessive comments = high comment ratio, docblocks much longer than the body, and commented-out code, judged against absolute thresholds, never the repo norm.", - "kind": "decision", - "sources": [ - "qa-round-3" - ], - "confidence": "high" - }, - { - "id": "C12", - "statement": "Findings use project tsc/eslint when present, otherwise Srcly bundled defaults (ruff for Python, built-in tree-sitter rules for TS), labelled by which ran.", - "kind": "decision", - "sources": [ - "qa-round-3" - ], - "confidence": "high" - }, - { - "id": "C13", - "statement": "Stack order confirmed: trust fixes, catalog cuts + comment flip, single-metric list + shared definitions, Git activity on by default, 2x2 presets, findings, local-repo pilot. Clones and coverage below the cut line.", - "kind": "decision", - "sources": [ - "qa-round-3" - ], - "confidence": "high" - } - ], - "themes": [], - "conflicts": [], - "human_questions": [] - }, - "brainstorm": { - "status": "not-requested", - "requested_count": 7, - "options": [], - "selected_option": null - }, - "shape": { - "status": "current", - "appetite": "Seven stacked PRs landing now", - "problem": "Metrics conflate zero with N/A, rank inconsistently across surfaces, score some fields in the wrong direction, and lack activity/comment-noise/component/finding signals.", - "recommended_shape": "Trust fixes, one server-owned catalog, bold cuts plus comment-noise and component metrics, single-metric list, Git activity, 2x2 presets, zero-config findings, agent-run pilot.", - "success_moves": [ - "Trust before signals so new data is not read as zero", - "Single server catalog removes three hand-mirrored tables", - "Reuse compare git_refs runner for activity", - "External tools behind one runner with timeout -> N/A", - "Absolute comment thresholds tuned by pilot", - "Agent-run hybrid pilot replaces human-gated review" - ], - "first_proof": { - "question": "Do zero and N/A stay distinct and maxima name the right function across API, report and explorer?", - "artifact": "Fixture scan with nested functions, measured zero, skipped file, over-commented function", - "decision_rule": "Proceed to catalog cuts only if all three surfaces agree" - }, - "rabbit_holes": [ - "Cross-language scoring framework", - "Bundling a TS compiler without node", - "Statistical significance claims from four repos" - ], - "open_decisions": [] - }, - "implementation": { - "status": "current", - "repository_snapshot": { - "root": null, - "revision": null, - "worktree_status": null, - "subsystems": [], - "test_commands": [], - "deployment_surfaces": [] - }, - "dependencies": [ - { - "id": "git", - "name": "Local git CLI", - "material": true, - "application_seam": "services/activity.py via compare/git_refs runner", - "failure_modes": [ - "no .git", - "shallow clone", - "slow history" - ], - "proof_rungs": [ - "temp repo fixtures", - "real repo scan" - ], - "first_proof_role": { - "mode": "absent", - "rationale": "M1 needs no git" - }, - "local_proof": "pytest temp repos", - "live_proof": "scan srcly and explorEDA", - "rollout": "N/A when unavailable" - }, - { - "id": "lint-tools", - "name": "ruff, project tsc/eslint", - "material": true, - "application_seam": "services/findings runner (one function per tool)", - "failure_modes": [ - "missing tool", - "no config", - "timeout", - "crash" - ], - "proof_rungs": [ - "fixtures with fake tool outputs", - "real tool on fixtures", - "real repo scan" - ], - "first_proof_role": { - "mode": "absent", - "rationale": "Findings arrive in M6" - }, - "local_proof": "pytest with injected tool output and timeouts", - "live_proof": "scan recipes-modern and context-recorder", - "rollout": "Separate job; N/A on failure" - } - ], - "fast_local_loop": { - "command": "cd server && uv run pytest -q; cd client && pnpm test" - }, - "milestones": [ - { - "id": "m1", - "title": "Zero vs N/A and witnesses", - "outcome": "Unsupported metrics show N/A; maxima open their function; consistent ranking cohort", - "dependencies_introduced": [], - "proof": "Fixture scan parity across API/report/explorer; monotonicity test", - "valid_intermediate_state": "Existing catalog with correct availability", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "https://github.com/byronwall/srcly/pull/73 (fix(metrics): zero vs N/A, witnesses, ranking denominator)" - ] - }, - { - "id": "m2", - "title": "Catalog cuts, comment noise, React components", - "outcome": "No inverted comment penalty; comment_noise and component metrics present", - "dependencies_introduced": [], - "proof": "React/Solid and comment fixtures", - "valid_intermediate_state": "Smaller catalog, all surfaces working", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "https://github.com/byronwall/srcly/pull/74 (catalog cuts, comment_noise, React/Solid component metrics)" - ] - }, - { - "id": "m3", - "title": "One catalog and single-metric list", - "outcome": "Server catalog drives client, report, compare; excellent single-metric list", - "dependencies_introduced": [], - "proof": "Report vs client top-10 parity test", - "valid_intermediate_state": "Same metrics, one source of truth", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "https://github.com/byronwall/srcly/pull/75 (server-owned metric catalog, single-metric Hot Spots list, report parity test)" - ] - }, - { - "id": "m4", - "title": "Git activity", - "outcome": "Commit count and churn per file on any git repo", - "dependencies_introduced": [ - "git" - ], - "proof": "Temp repo with rename/merge/bulk commit matches git log", - "valid_intermediate_state": "Activity N/A without git", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "https://github.com/byronwall/srcly/pull/76 (Git activity: commit_count, churn_lines, N/A reasons)" - ] - }, - { - "id": "m5", - "title": "2x2 presets", - "outcome": "Four named scatter presets with source drilldown", - "dependencies_introduced": [], - "proof": "Quadrant unit test and browser check", - "valid_intermediate_state": "Scatter optional beside treemap", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "https://github.com/byronwall/srcly/pull/77 (2x2 scatter presets with absolute cut-offs)" - ] - }, - { - "id": "m6", - "title": "Zero-config findings", - "outcome": "Project tools or bundled ruff/tree-sitter rules produce labelled findings", - "dependencies_introduced": [ - "ruff", - "project tsc/eslint" - ], - "proof": "Fixtures with/without tools, timeout injection", - "valid_intermediate_state": "Findings N/A on failure", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "https://github.com/byronwall/srcly/pull/78 (zero-config findings: tsc/eslint/ruff/srcly TS rules)" - ] - }, - { - "id": "m7", - "title": "Pilot tunes defaults", - "outcome": "Defaults, thresholds and cut-offs backed by four-repo review", - "dependencies_introduced": [], - "proof": "Evaluation script output plus adjudicated cases in docs/research", - "valid_intermediate_state": "Tuned defaults", - "rollback": "Revert PR", - "status": "completed", - "evidence": [ - "Branch claude/metrics-pr7-pilot-952fa2: server/scripts/evaluate_metric_portfolio.py + tests; 729 adjudicated cases on srcly, recipes-modern, explorEDA, context-recorder; decision record docs/research/metric-pilot-2026-10/README.md; comment-noise ratio 0.5→1.0, comment-noise 2x2 cut-off 0.15→0.5, profile weights retuned", - "https://github.com/byronwall/srcly/pull/79 (feat(metrics): tune defaults from a four-repo pilot)" - ] - } - ], - "cut_line": [ - "Clones, coverage, CRAP, mutation", - "Temporal coupling, trends", - "People metrics, MI, Halstead", - "Persistent triage" - ], - "open_decisions": [ - "Solid component detection adequacy", - "Bundled TS rules usefulness" - ] - }, - "history": [ - { - "date": "2026-10-15", - "event": "created", - "note": "Umbrella over codex/metrics-research initiatives; round 1 recorded" - }, - { - "date": "2026-10-15", - "event": "shaped-and-planned", - "note": "Three question rounds; intent, shape, plan current" - }, - { - "date": "2026-10-16", - "event": "completed", - "note": "M7 pilot tuned the defaults; all milestones landed as stacked PRs #73-#79" - } - ], - "closure": { - "summary": "The metric system overhaul landed as seven stacked PRs: trust fixes, catalog cuts with comment noise and component metrics, one server-owned catalog with a single-metric list, Git activity, 2x2 presets, zero-config findings, and a four-repo pilot that tuned the defaults.", - "evidence": [ - "M7 pilot: 729 cases adjudicated on srcly, recipes-modern, explorEDA and context-recorder (docs/research/metric-pilot-2026-10/README.md, cases.tsv).", - "cd server && uv run pytest: 344 passed on the M7 branch.", - "cd client && pnpm test: 298 passed; pnpm build succeeded on the M7 branch.", - "PRs #73-#78 each recorded their own tests and screenshots." - ], - "reconciliation": [ - { - "claim_id": "C1", - "disposition": "delivered", - "evidence": "One server catalog (GET /api/metrics/catalog) feeds explorer, report and compare; parity test (PR #75)." - }, - { - "claim_id": "C2", - "disposition": "delivered", - "evidence": "Seven stacked PRs, #73 (trust fixes) through #79 (the pilot)." - }, - { - "claim_id": "C3", - "disposition": "delivered", - "evidence": "Trust (#73), portfolio (#74, #75, M7), Git activity (#76), quality evidence as zero-config findings (#78)." - }, - { - "claim_id": "C4", - "disposition": "delivered", - "evidence": "Low Comments, hardcoded/duplicated text and raw TSX counters cut with no compatibility layer (#74); M7 demoted further metrics by evidence." - }, - { - "claim_id": "C5", - "disposition": "delivered", - "evidence": "The agent ran the pilot on four local repos and adjudicated 729 cases (docs/research/metric-pilot-2026-10)." - }, - { - "claim_id": "C6", - "disposition": "delivered", - "evidence": "Single-metric Hot Spots list (#75) and the 2x2 scatter view with named quadrants (#77)." - }, - { - "claim_id": "C7", - "disposition": "delivered", - "evidence": "Component-scoped React/Solid metrics (#74); M7 kept LOC, hooks, effects, JSX depth and props as defaults, demoted inline handlers and JSX conditionals." - }, - { - "claim_id": "C8", - "disposition": "delivered", - "evidence": "comment_noise replaces the low-comments penalty (#74); TODOs are neutral detail." - }, - { - "claim_id": "C9", - "disposition": "delivered", - "evidence": "Findings run tsc/eslint/ruff/srcly TS rules with no config (#78)." - }, - { - "claim_id": "C10", - "disposition": "delivered", - "evidence": "Activity, Blast radius, Component weight, Comment noise and Findings presets (#77, #78); M7 tuned the comment-noise cut-off." - }, - { - "claim_id": "C11", - "disposition": "changed", - "evidence": "All three rules shipped with absolute thresholds (#74); M7 raised the narrated-code ratio from 0.5 to 1.0 comment lines per code line on pilot evidence, still absolute." - }, - { - "claim_id": "C12", - "disposition": "delivered", - "evidence": "Project tsc/eslint when present, otherwise ruff and srcly's TS rules, labelled by tool (#78)." - }, - { - "claim_id": "C13", - "disposition": "delivered", - "evidence": "Landed in the confirmed order #73 → #79." - } - ], - "follow_ups": [ - "Scope LOC (and commit count) to code languages, so Markdown, HTML and CSS stop ranking beside code.", - "Make parameter_count's file and folder value the max arity with a witness instead of a sum.", - "Fix the LOC description or count: TS and Python LOC include comment lines; then revisit comment_ratio.", - "Weight findings by severity, or set stylistic rules (eslint curly) apart, so one project rule can't fill the Findings preset.", - "Detect components that return .map(...) of JSX (explorEDA DslCode.tsx).", - "Run the next pilot on a TS repo without tsc or eslint and a non-author repo, with a second reviewer." - ] - }, - "question_rounds": [ - { - "round": 1, - "focus": "goals and stakes", - "questions": [ - "main consumer", - "appetite", - "trust vs new signal order", - "pruning" - ], - "answers": [ - "All equally", - "Stacked PRs now: fix then improve", - "Do all initiatives now", - "Cut boldly now" - ] - }, - { - "round": 2, - "focus": "behavior and limits", - "questions": [ - "pilot", - "default ranking", - "deletions", - "evidence sources" - ], - "answers": [ - "Agent runs pilot on local repos via consider-contrast", - "Great single-metric list plus named 2x2 scatter", - "Keep useful React/TSX set; excessive comments important; TODO less", - "Srcly runs tools itself; lint/type findings" - ] - }, - { - "round": 3, - "focus": "boundaries and confirmation", - "questions": [ - "2x2 axes", - "excessive comments", - "no tools present", - "stack order" - ], - "answers": [ - "Several named presets", - "Absolute thresholds: ratio, large docblocks, commented-out code", - "Bundle defaults", - "Yes, that order" - ] - } - ] -} diff --git a/docs/intent/metric-system-overhaul/intent-brief.md b/docs/intent/metric-system-overhaul/intent-brief.md deleted file mode 100644 index bbbf0386..00000000 --- a/docs/intent/metric-system-overhaul/intent-brief.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: "Metric system overhaul" -slug: "metric-system-overhaul" -phase: intent -status: current -last_updated: "2026-10-15" ---- - -# Metric system overhaul - -## My read - -Srcly's metrics should point at code that is worth refactoring. Today some of them mislead. Zero and "not measured" look the same. Reports and the explorer rank different populations with different math. Some fields score the wrong direction, such as treating low comment density as risky. The research on `codex/metrics-research` found these problems and proposed four gated initiatives: metric trust, portfolio evaluation, Git activity, and imported evidence. - -Byron wants all of it now, as a stack of PRs that first fixes and then improves. The research gated each step on his review. That changes here: the agent runs the evaluation on local repositories and makes the calls, and Byron stays mostly out of the loop. The result should "just work", with no per-repo configuration. - -## What matters most - -- One metric definition drives the explorer, `srcly report`, and PR compare equally. -- An excellent single-metric hot-spot list, where every value leads to its source. -- Named 2x2 scatter presets that show how code ranks on two axes at once. -- Signals that matter now: excessive comments, useful React/TSX refactor targets, Git activity, and lint/type findings. - -## The experience or behavior you appear to want - -Open a repo with no setup. Pick a metric and get a ranked list you can trust, where each row opens the responsible function. Switch to a 2x2 preset such as "Activity" and see which complex files are also churning. Agents get the same ranking from `srcly report`. PR compare uses the same better/worse directions. - -## Boundaries - -### Must be true - -- A measured zero and N/A differ everywhere. -- Excessive-comment checks use absolute thresholds, never the repo's own norm. -- Lint/type findings use the project's tools when present. Otherwise Srcly uses bundled defaults and labels which ones ran. -- Git activity turns on automatically when `.git` exists. - -### Must be avoided - -- Compatibility layers for removed or renamed fields. -- Bespoke configuration as a precondition for useful results. -- Combined scores that hide their inputs. - -## What seems settled - -- Cut boldly. Misleading directions and noisy fields go away. -- Keep a curated React/TSX set that targets component refactors; the agent picks it. -- TODOs are low priority. -- Stack order: trust fixes → catalog cuts and the comment flip → single-metric list with shared definitions → Git activity → 2x2 presets → findings → local-repo pilot. -- Clones and coverage come later. - -## Possibilities, not decisions - -- Quadrant names and cut-off values. -- Exact comment thresholds. The pilot tunes them. - -## Current reality that matters - -- PR compare (`server/app/services/compare/`) landed after the research baseline. It has its own direction table that inherits the report's risk directions, so changes to the catalog affect it. -- `metricsStore.tsx` already records `scope` and `languages`, which covers part of the research's TS-support fix. -- Python dependencies have no lint tool yet. `ruff` would be new. - -## Next step after confirmation - -Shape and plan (done: see [shape](shape-brief.md) and [plan](implementation-plan.md)). diff --git a/docs/intent/metric-system-overhaul/shape-brief.md b/docs/intent/metric-system-overhaul/shape-brief.md deleted file mode 100644 index b0079e3a..00000000 --- a/docs/intent/metric-system-overhaul/shape-brief.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: "Metric system overhaul — shape brief" -slug: "metric-system-overhaul" -phase: shape -status: current -last_updated: "2026-10-15" ---- - -# Metric system overhaul — what we are adding - -**Outcome:** Trustworthy metrics, one shared catalog, and new signals (activity, comment noise, React components, findings) that point at code worth refactoring. - -**Primary flow:** Scan with no setup → pick one metric or a 2x2 preset → open the responsible source → the same story appears in `srcly report` and PR compare. - -## Feature scope - -```text -Metric system -├── PLANNED ADDITIONS -│ ├── Trust -│ │ ├── Measured zero vs N/A (unsupported, skipped, failed) end to end -│ │ ├── Real functions separate from display scopes; max carries its function -│ │ └── Fixed ranking denominator; report and explorer rank the same cohort -│ ├── One catalog -│ │ ├── Server-owned definitions: label, group, direction, scope, languages -│ │ └── Served to the client and reused by PR compare -│ ├── Catalog cuts and new signals -│ │ ├── Delete: Low Comments, hardcoded/duplicated text, raw class/type/export penalties -│ │ ├── Comment noise: ratio, oversized docblocks, commented-out code -│ │ └── React components: size, hooks, effects, JSX depth, props, inline handlers -│ ├── Single-metric hot-spot list -│ │ └── Value, unit, witness function, cohort label, N/A rows apart -│ ├── Git activity (on when .git exists) -│ │ └── Commits in window per file; churn shown separately -│ ├── 2x2 scatter presets -│ │ └── Activity · Blast radius · Component weight · Comment noise -│ └── Findings -│ └── Project tsc/eslint if present, otherwise bundled ruff + tree-sitter rules -├── EVALUATE BEFORE COMMITTING -│ └── Local-repo pilot tunes defaults, thresholds and quadrant cut-offs -└── LATER POSSIBILITIES - └── Clones, coverage, CRAP, temporal coupling, trends -``` - -## Behavior - -| Situation | Expected result | -| --- | --- | -| Metric doesn't apply to a file | N/A, excluded from ranking and color | -| No `.git` or shallow clone | Activity is N/A with the reason; source metrics are unaffected | -| No project tsc/eslint | Bundled rules run, labelled "srcly defaults" | -| Function is 60% comments | Flagged by comment noise whatever the repo norm | -| Folder shows max complexity | Clicking it opens the responsible function | - -## Examples - -Proposed 2x2 presets, with names to be tuned by the pilot: - -| Preset | X × Y | High/high | High X only | High Y only | -| --- | --- | --- | --- | --- | -| Activity | complexity × commits | Hot mess | Sleeping giant | Busy, fine | -| Blast radius | complexity × fan-in | Load-bearing tangle | Isolated mess | Hub | -| Component weight | component LOC × hooks | God component | Long markup | Hook soup | -| Comment noise | comment ratio × complexity | Documented complexity | Narrated trivia | Bare complexity | - -## Decisions and boundaries - -**Appetite:** Seven stacked PRs, landing now. Each one leaves Srcly working. - -**Key decision — React set (consider-contrast):** I scored three options on refactor targeting, soundness across React and Solid, and cost. Keeping every TSX field scored 4/3/9. The research's demote-all scored 3/8/8. A component-scoped set scored 9/7/5. Chosen: **component-scoped**. Detect components (functions returning JSX). Measure size, hooks, effects (`useEffect` or Solid `createEffect`), local JSX depth, declared props and inline handlers per component. Delete hardcoded/duplicated text. Redefine render branching as JSX conditionals per component. - -**Key decision — pilot (consider-contrast):** Options were a full statistical study (rigor 9, speed 2), case review only (rigor 4, speed 8), or a hybrid (rigor 7, speed 7). Chosen: **hybrid**. A script reports redundancy, missing rates and top-k overlap. The agent then reviews the top 10 per lens on four repos against a written rubric. A row counts only when a cited source line justifies action. The pilot runs last, after activity and findings exist, so it evaluates them too. - -**Key decision — one catalog:** The server serves the catalog at runtime. That avoids codegen, which the research rejected, and avoids hand-mirrored tables. - -**Boundary:** No compatibility layer. No per-repo config is required. External tools run with timeouts, and a failure shows as N/A. Each PR reverts as one commit. - -**First proof:** **Try:** Scan a TS/Python fixture with nested functions, a measured zero, a skipped file, and an over-commented function. **Observe:** The API, report and explorer show the same values, N/A rows, and witness functions. **Decide:** Continue to the catalog cuts only when zero and N/A stay distinct through all three surfaces. - -See the [intent](intent-brief.md) and the [implementation plan](implementation-plan.md). diff --git a/docs/research/metric-pilot-2026-10/README.md b/docs/research/metric-pilot-2026-10/README.md index 4b332645..3ec1dea7 100644 --- a/docs/research/metric-pilot-2026-10/README.md +++ b/docs/research/metric-pilot-2026-10/README.md @@ -7,7 +7,7 @@ status: decided # Metric pilot, October 2026 -Milestone 7 of the [metric system overhaul](../../intent/metric-system-overhaul/implementation-plan.md). It is the hybrid pilot chosen in the [shape brief](../../intent/metric-system-overhaul/shape-brief.md). A script measured redundancy, missing rates and top-k overlap. Then every case in the top 10 of each Hot Spots lens and each 2x2 preset was read in source and given a verdict. The defaults were tuned from those verdicts. +Milestone 7 of the [metric system overhaul](https://github.com/byronwall/srcly/blob/0bfb05ba/docs/intent/metric-system-overhaul/implementation-plan.md). It is the hybrid pilot chosen in the [shape brief](https://github.com/byronwall/srcly/blob/0bfb05ba/docs/intent/metric-system-overhaul/shape-brief.md). A script measured redundancy, missing rates and top-k overlap. Then every case in the top 10 of each Hot Spots lens and each 2x2 preset was read in source and given a verdict. The defaults were tuned from those verdicts. Byron was away and asked the agent to make these calls. One reviewer (Claude) adjudicated every case. The review was not blind: the reviewer knew which metric surfaced each case.